1============================= 2User Guide for AMDGPU Backend 3============================= 4 5.. contents:: 6 :local: 7 8.. toctree:: 9 :hidden: 10 11 AMDGPU/AMDGPUAsmGFX7 12 AMDGPU/AMDGPUAsmGFX8 13 AMDGPU/AMDGPUAsmGFX9 14 AMDGPU/AMDGPUAsmGFX900 15 AMDGPU/AMDGPUAsmGFX904 16 AMDGPU/AMDGPUAsmGFX906 17 AMDGPU/AMDGPUAsmGFX908 18 AMDGPU/AMDGPUAsmGFX90a 19 AMDGPU/AMDGPUAsmGFX10 20 AMDGPU/AMDGPUAsmGFX1011 21 AMDGPU/AMDGPUAsmGFX1013 22 AMDGPU/AMDGPUAsmGFX1030 23 AMDGPUModifierSyntax 24 AMDGPUOperandSyntax 25 AMDGPUInstructionSyntax 26 AMDGPUInstructionNotation 27 AMDGPUDwarfExtensionsForHeterogeneousDebugging 28 AMDGPUDwarfExtensionAllowLocationDescriptionOnTheDwarfExpressionStack/AMDGPUDwarfExtensionAllowLocationDescriptionOnTheDwarfExpressionStack 29 30Introduction 31============ 32 33The AMDGPU backend provides ISA code generation for AMD GPUs, starting with the 34R600 family up until the current GCN families. It lives in the 35``llvm/lib/Target/AMDGPU`` directory. 36 37LLVM 38==== 39 40.. _amdgpu-target-triples: 41 42Target Triples 43-------------- 44 45Use the Clang option ``-target <Architecture>-<Vendor>-<OS>-<Environment>`` 46to specify the target triple: 47 48 .. table:: AMDGPU Architectures 49 :name: amdgpu-architecture-table 50 51 ============ ============================================================== 52 Architecture Description 53 ============ ============================================================== 54 ``r600`` AMD GPUs HD2XXX-HD6XXX for graphics and compute shaders. 55 ``amdgcn`` AMD GPUs GCN GFX6 onwards for graphics and compute shaders. 56 ============ ============================================================== 57 58 .. table:: AMDGPU Vendors 59 :name: amdgpu-vendor-table 60 61 ============ ============================================================== 62 Vendor Description 63 ============ ============================================================== 64 ``amd`` Can be used for all AMD GPU usage. 65 ``mesa3d`` Can be used if the OS is ``mesa3d``. 66 ============ ============================================================== 67 68 .. table:: AMDGPU Operating Systems 69 :name: amdgpu-os 70 71 ============== ============================================================ 72 OS Description 73 ============== ============================================================ 74 *<empty>* Defaults to the *unknown* OS. 75 ``amdhsa`` Compute kernels executed on HSA [HSA]_ compatible runtimes 76 such as: 77 78 - AMD's ROCm™ runtime [AMD-ROCm]_ using the *rocm-amdhsa* 79 loader on Linux. See *AMD ROCm Platform Release Notes* 80 [AMD-ROCm-Release-Notes]_ for supported hardware and 81 software. 82 - AMD's PAL runtime using the *pal-amdhsa* loader on 83 Windows. 84 85 ``amdpal`` Graphic shaders and compute kernels executed on AMD's PAL 86 runtime using the *pal-amdpal* loader on Windows and Linux 87 Pro. 88 ``mesa3d`` Graphic shaders and compute kernels executed on AMD's Mesa 89 3D runtime using the *mesa-mesa3d* loader on Linux. 90 ============== ============================================================ 91 92 .. table:: AMDGPU Environments 93 :name: amdgpu-environment-table 94 95 ============ ============================================================== 96 Environment Description 97 ============ ============================================================== 98 *<empty>* Default. 99 ============ ============================================================== 100 101.. _amdgpu-processors: 102 103Processors 104---------- 105 106Use the Clang options ``-mcpu=<target-id>`` or ``--offload-arch=<target-id>`` to 107specify the AMDGPU processor together with optional target features. See 108:ref:`amdgpu-target-id` and :ref:`amdgpu-target-features` for AMD GPU target 109specific information. 110 111Every processor supports every OS ABI (see :ref:`amdgpu-os`) with the following exceptions: 112 113* ``amdhsa`` is not supported in ``r600`` architecture (see :ref:`amdgpu-architecture-table`). 114 115 116 .. table:: AMDGPU Processors 117 :name: amdgpu-processor-table 118 119 =========== =============== ============ ===== ================= =============== =============== ====================== 120 Processor Alternative Target dGPU/ Target Target OS Support Example 121 Processor Triple APU Features Properties *(see* Products 122 Architecture Supported `amdgpu-os`_ 123 *and 124 corresponding 125 runtime release 126 notes for 127 current 128 information and 129 level of 130 support)* 131 =========== =============== ============ ===== ================= =============== =============== ====================== 132 **Radeon HD 2000/3000 Series (R600)** [AMD-RADEON-HD-2000-3000]_ 133 ----------------------------------------------------------------------------------------------------------------------- 134 ``r600`` ``r600`` dGPU - Does not 135 support 136 generic 137 address 138 space 139 ``r630`` ``r600`` dGPU - Does not 140 support 141 generic 142 address 143 space 144 ``rs880`` ``r600`` dGPU - Does not 145 support 146 generic 147 address 148 space 149 ``rv670`` ``r600`` dGPU - Does not 150 support 151 generic 152 address 153 space 154 **Radeon HD 4000 Series (R700)** [AMD-RADEON-HD-4000]_ 155 ----------------------------------------------------------------------------------------------------------------------- 156 ``rv710`` ``r600`` dGPU - Does not 157 support 158 generic 159 address 160 space 161 ``rv730`` ``r600`` dGPU - Does not 162 support 163 generic 164 address 165 space 166 ``rv770`` ``r600`` dGPU - Does not 167 support 168 generic 169 address 170 space 171 **Radeon HD 5000 Series (Evergreen)** [AMD-RADEON-HD-5000]_ 172 ----------------------------------------------------------------------------------------------------------------------- 173 ``cedar`` ``r600`` dGPU - Does not 174 support 175 generic 176 address 177 space 178 ``cypress`` ``r600`` dGPU - Does not 179 support 180 generic 181 address 182 space 183 ``juniper`` ``r600`` dGPU - Does not 184 support 185 generic 186 address 187 space 188 ``redwood`` ``r600`` dGPU - Does not 189 support 190 generic 191 address 192 space 193 ``sumo`` ``r600`` dGPU - Does not 194 support 195 generic 196 address 197 space 198 **Radeon HD 6000 Series (Northern Islands)** [AMD-RADEON-HD-6000]_ 199 ----------------------------------------------------------------------------------------------------------------------- 200 ``barts`` ``r600`` dGPU - Does not 201 support 202 generic 203 address 204 space 205 ``caicos`` ``r600`` dGPU - Does not 206 support 207 generic 208 address 209 space 210 ``cayman`` ``r600`` dGPU - Does not 211 support 212 generic 213 address 214 space 215 ``turks`` ``r600`` dGPU - Does not 216 support 217 generic 218 address 219 space 220 **GCN GFX6 (Southern Islands (SI))** [AMD-GCN-GFX6]_ 221 ----------------------------------------------------------------------------------------------------------------------- 222 ``gfx600`` - ``tahiti`` ``amdgcn`` dGPU - Does not - *pal-amdpal* 223 support 224 generic 225 address 226 space 227 ``gfx601`` - ``pitcairn`` ``amdgcn`` dGPU - Does not - *pal-amdpal* 228 - ``verde`` support 229 generic 230 address 231 space 232 ``gfx602`` - ``hainan`` ``amdgcn`` dGPU - Does not - *pal-amdpal* 233 - ``oland`` support 234 generic 235 address 236 space 237 **GCN GFX7 (Sea Islands (CI))** [AMD-GCN-GFX7]_ 238 ----------------------------------------------------------------------------------------------------------------------- 239 ``gfx700`` - ``kaveri`` ``amdgcn`` APU - Offset - *rocm-amdhsa* - A6-7000 240 flat - *pal-amdhsa* - A6 Pro-7050B 241 scratch - *pal-amdpal* - A8-7100 242 - A8 Pro-7150B 243 - A10-7300 244 - A10 Pro-7350B 245 - FX-7500 246 - A8-7200P 247 - A10-7400P 248 - FX-7600P 249 ``gfx701`` - ``hawaii`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - FirePro W8100 250 flat - *pal-amdhsa* - FirePro W9100 251 scratch - *pal-amdpal* - FirePro S9150 252 - FirePro S9170 253 ``gfx702`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - Radeon R9 290 254 flat - *pal-amdhsa* - Radeon R9 290x 255 scratch - *pal-amdpal* - Radeon R390 256 - Radeon R390x 257 ``gfx703`` - ``kabini`` ``amdgcn`` APU - Offset - *pal-amdhsa* - E1-2100 258 - ``mullins`` flat - *pal-amdpal* - E1-2200 259 scratch - E1-2500 260 - E2-3000 261 - E2-3800 262 - A4-5000 263 - A4-5100 264 - A6-5200 265 - A4 Pro-3340B 266 ``gfx704`` - ``bonaire`` ``amdgcn`` dGPU - Offset - *pal-amdhsa* - Radeon HD 7790 267 flat - *pal-amdpal* - Radeon HD 8770 268 scratch - R7 260 269 - R7 260X 270 ``gfx705`` ``amdgcn`` APU - Offset - *pal-amdhsa* *TBA* 271 flat - *pal-amdpal* 272 scratch .. TODO:: 273 274 Add product 275 names. 276 277 **GCN GFX8 (Volcanic Islands (VI))** [AMD-GCN-GFX8]_ 278 ----------------------------------------------------------------------------------------------------------------------- 279 ``gfx801`` - ``carrizo`` ``amdgcn`` APU - xnack - Offset - *rocm-amdhsa* - A6-8500P 280 flat - *pal-amdhsa* - Pro A6-8500B 281 scratch - *pal-amdpal* - A8-8600P 282 - Pro A8-8600B 283 - FX-8800P 284 - Pro A12-8800B 285 - A10-8700P 286 - Pro A10-8700B 287 - A10-8780P 288 - A10-9600P 289 - A10-9630P 290 - A12-9700P 291 - A12-9730P 292 - FX-9800P 293 - FX-9830P 294 - E2-9010 295 - A6-9210 296 - A9-9410 297 ``gfx802`` - ``iceland`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - Radeon R9 285 298 - ``tonga`` flat - *pal-amdhsa* - Radeon R9 380 299 scratch - *pal-amdpal* - Radeon R9 385 300 ``gfx803`` - ``fiji`` ``amdgcn`` dGPU - *rocm-amdhsa* - Radeon R9 Nano 301 - *pal-amdhsa* - Radeon R9 Fury 302 - *pal-amdpal* - Radeon R9 FuryX 303 - Radeon Pro Duo 304 - FirePro S9300x2 305 - Radeon Instinct MI8 306 \ - ``polaris10`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - Radeon RX 470 307 flat - *pal-amdhsa* - Radeon RX 480 308 scratch - *pal-amdpal* - Radeon Instinct MI6 309 \ - ``polaris11`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - Radeon RX 460 310 flat - *pal-amdhsa* 311 scratch - *pal-amdpal* 312 ``gfx805`` - ``tongapro`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - FirePro S7150 313 flat - *pal-amdhsa* - FirePro S7100 314 scratch - *pal-amdpal* - FirePro W7100 315 - Mobile FirePro 316 M7170 317 ``gfx810`` - ``stoney`` ``amdgcn`` APU - xnack - Offset - *rocm-amdhsa* *TBA* 318 flat - *pal-amdhsa* 319 scratch - *pal-amdpal* .. TODO:: 320 321 Add product 322 names. 323 324 **GCN GFX9 (Vega)** [AMD-GCN-GFX900-GFX904-VEGA]_ [AMD-GCN-GFX906-VEGA7NM]_ [AMD-GCN-GFX908-CDNA1]_ [AMD-GCN-GFX90A-CDNA2]_ 325 ----------------------------------------------------------------------------------------------------------------------- 326 ``gfx900`` ``amdgcn`` dGPU - xnack - Absolute - *rocm-amdhsa* - Radeon Vega 327 flat - *pal-amdhsa* Frontier Edition 328 scratch - *pal-amdpal* - Radeon RX Vega 56 329 - Radeon RX Vega 64 330 - Radeon RX Vega 64 331 Liquid 332 - Radeon Instinct MI25 333 ``gfx902`` ``amdgcn`` APU - xnack - Absolute - *rocm-amdhsa* - Ryzen 3 2200G 334 flat - *pal-amdhsa* - Ryzen 5 2400G 335 scratch - *pal-amdpal* 336 ``gfx904`` ``amdgcn`` dGPU - xnack - *rocm-amdhsa* *TBA* 337 - *pal-amdhsa* 338 - *pal-amdpal* .. TODO:: 339 340 Add product 341 names. 342 343 ``gfx906`` ``amdgcn`` dGPU - sramecc - Absolute - *rocm-amdhsa* - Radeon Instinct MI50 344 - xnack flat - *pal-amdhsa* - Radeon Instinct MI60 345 scratch - *pal-amdpal* - Radeon VII 346 - Radeon Pro VII 347 ``gfx908`` ``amdgcn`` dGPU - sramecc - *rocm-amdhsa* - AMD Instinct MI100 Accelerator 348 - xnack - Absolute 349 flat 350 scratch 351 ``gfx909`` ``amdgcn`` APU - xnack - Absolute - *pal-amdpal* *TBA* 352 flat 353 scratch .. TODO:: 354 355 Add product 356 names. 357 358 ``gfx90a`` ``amdgcn`` dGPU - sramecc - Absolute - *rocm-amdhsa* *TBA* 359 - tgsplit flat 360 - xnack scratch .. TODO:: 361 - Packed 362 work-item Add product 363 IDs names. 364 365 ``gfx90c`` ``amdgcn`` APU - xnack - Absolute - *pal-amdpal* - Ryzen 7 4700G 366 flat - Ryzen 7 4700GE 367 scratch - Ryzen 5 4600G 368 - Ryzen 5 4600GE 369 - Ryzen 3 4300G 370 - Ryzen 3 4300GE 371 - Ryzen Pro 4000G 372 - Ryzen 7 Pro 4700G 373 - Ryzen 7 Pro 4750GE 374 - Ryzen 5 Pro 4650G 375 - Ryzen 5 Pro 4650GE 376 - Ryzen 3 Pro 4350G 377 - Ryzen 3 Pro 4350GE 378 379 ``gfx940`` ``amdgcn`` dGPU - sramecc - Architected *TBA* 380 - tgsplit flat 381 - xnack scratch .. TODO:: 382 - Packed 383 work-item Add product 384 IDs names. 385 386 **GCN GFX10.1 (RDNA 1)** [AMD-GCN-GFX10-RDNA1]_ 387 ----------------------------------------------------------------------------------------------------------------------- 388 ``gfx1010`` ``amdgcn`` dGPU - cumode - Absolute - *rocm-amdhsa* - Radeon RX 5700 389 - wavefrontsize64 flat - *pal-amdhsa* - Radeon RX 5700 XT 390 - xnack scratch - *pal-amdpal* - Radeon Pro 5600 XT 391 - Radeon Pro 5600M 392 ``gfx1011`` ``amdgcn`` dGPU - cumode - *rocm-amdhsa* - Radeon Pro V520 393 - wavefrontsize64 - Absolute - *pal-amdhsa* 394 - xnack flat - *pal-amdpal* 395 scratch 396 ``gfx1012`` ``amdgcn`` dGPU - cumode - Absolute - *rocm-amdhsa* - Radeon RX 5500 397 - wavefrontsize64 flat - *pal-amdhsa* - Radeon RX 5500 XT 398 - xnack scratch - *pal-amdpal* 399 ``gfx1013`` ``amdgcn`` APU - cumode - Absolute - *rocm-amdhsa* *TBA* 400 - wavefrontsize64 flat - *pal-amdhsa* 401 - xnack scratch - *pal-amdpal* .. TODO:: 402 403 Add product 404 names. 405 406 **GCN GFX10.3 (RDNA 2)** [AMD-GCN-GFX10-RDNA2]_ 407 ----------------------------------------------------------------------------------------------------------------------- 408 ``gfx1030`` ``amdgcn`` dGPU - cumode - Absolute - *rocm-amdhsa* - Radeon RX 6800 409 - wavefrontsize64 flat - *pal-amdhsa* - Radeon RX 6800 XT 410 scratch - *pal-amdpal* - Radeon RX 6900 XT 411 ``gfx1031`` ``amdgcn`` dGPU - cumode - Absolute - *rocm-amdhsa* - Radeon RX 6700 XT 412 - wavefrontsize64 flat - *pal-amdhsa* 413 scratch - *pal-amdpal* 414 ``gfx1032`` ``amdgcn`` dGPU - cumode - Absolute - *rocm-amdhsa* *TBA* 415 - wavefrontsize64 flat - *pal-amdhsa* 416 scratch - *pal-amdpal* .. TODO:: 417 418 Add product 419 names. 420 421 ``gfx1033`` ``amdgcn`` APU - cumode - Absolute - *pal-amdpal* *TBA* 422 - wavefrontsize64 flat 423 scratch .. TODO:: 424 425 Add product 426 names. 427 ``gfx1034`` ``amdgcn`` dGPU - cumode - Absolute - *pal-amdpal* *TBA* 428 - wavefrontsize64 flat 429 scratch .. TODO:: 430 431 Add product 432 names. 433 434 ``gfx1035`` ``amdgcn`` APU - cumode - Absolute - *pal-amdpal* *TBA* 435 - wavefrontsize64 flat 436 scratch .. TODO:: 437 Add product 438 names. 439 440 ``gfx1036`` ``amdgcn`` APU - cumode - Absolute - *pal-amdpal* *TBA* 441 - wavefrontsize64 flat 442 scratch .. TODO:: 443 444 Add product 445 names. 446 447 =========== =============== ============ ===== ================= =============== =============== ====================== 448 449.. _amdgpu-target-features: 450 451Target Features 452--------------- 453 454Target features control how code is generated to support certain 455processor specific features. Not all target features are supported by 456all processors. The runtime must ensure that the features supported by 457the device used to execute the code match the features enabled when 458generating the code. A mismatch of features may result in incorrect 459execution, or a reduction in performance. 460 461The target features supported by each processor is listed in 462:ref:`amdgpu-processor-table`. 463 464Target features are controlled by exactly one of the following Clang 465options: 466 467``-mcpu=<target-id>`` or ``--offload-arch=<target-id>`` 468 469 The ``-mcpu`` and ``--offload-arch`` can specify the target feature as 470 optional components of the target ID. If omitted, the target feature has the 471 ``any`` value. See :ref:`amdgpu-target-id`. 472 473``-m[no-]<target-feature>`` 474 475 Target features not specified by the target ID are specified using a 476 separate option. These target features can have an ``on`` or ``off`` 477 value. ``on`` is specified by omitting the ``no-`` prefix, and 478 ``off`` is specified by including the ``no-`` prefix. The default 479 if not specified is ``off``. 480 481For example: 482 483``-mcpu=gfx908:xnack+`` 484 Enable the ``xnack`` feature. 485``-mcpu=gfx908:xnack-`` 486 Disable the ``xnack`` feature. 487``-mcumode`` 488 Enable the ``cumode`` feature. 489``-mno-cumode`` 490 Disable the ``cumode`` feature. 491 492 .. table:: AMDGPU Target Features 493 :name: amdgpu-target-features-table 494 495 =============== ============================ ================================================== 496 Target Feature Clang Option to Control Description 497 Name 498 =============== ============================ ================================================== 499 cumode - ``-m[no-]cumode`` Control the wavefront execution mode used 500 when generating code for kernels. When disabled 501 native WGP wavefront execution mode is used, 502 when enabled CU wavefront execution mode is used 503 (see :ref:`amdgpu-amdhsa-memory-model`). 504 505 sramecc - ``-mcpu`` If specified, generate code that can only be 506 - ``--offload-arch`` loaded and executed in a process that has a 507 matching setting for SRAMECC. 508 509 If not specified for code object V2 to V3, generate 510 code that can be loaded and executed in a process 511 with SRAMECC enabled. 512 513 If not specified for code object V4 or above, generate 514 code that can be loaded and executed in a process 515 with either setting of SRAMECC. 516 517 tgsplit ``-m[no-]tgsplit`` Enable/disable generating code that assumes 518 work-groups are launched in threadgroup split mode. 519 When enabled the waves of a work-group may be 520 launched in different CUs. 521 522 wavefrontsize64 - ``-m[no-]wavefrontsize64`` Control the wavefront size used when 523 generating code for kernels. When disabled 524 native wavefront size 32 is used, when enabled 525 wavefront size 64 is used. 526 527 xnack - ``-mcpu`` If specified, generate code that can only be 528 - ``--offload-arch`` loaded and executed in a process that has a 529 matching setting for XNACK replay. 530 531 If not specified for code object V2 to V3, generate 532 code that can be loaded and executed in a process 533 with XNACK replay enabled. 534 535 If not specified for code object V4 or above, generate 536 code that can be loaded and executed in a process 537 with either setting of XNACK replay. 538 539 XNACK replay can be used for demand paging and 540 page migration. If enabled in the device, then if 541 a page fault occurs the code may execute 542 incorrectly unless generated with XNACK replay 543 enabled, or generated for code object V4 or above without 544 specifying XNACK replay. Executing code that was 545 generated with XNACK replay enabled, or generated 546 for code object V4 or above without specifying XNACK replay, 547 on a device that does not have XNACK replay 548 enabled will execute correctly but may be less 549 performant than code generated for XNACK replay 550 disabled. 551 =============== ============================ ================================================== 552 553.. _amdgpu-target-id: 554 555Target ID 556--------- 557 558AMDGPU supports target IDs. See `Clang Offload Bundler 559<https://clang.llvm.org/docs/ClangOffloadBundler.html>`_ for a general 560description. The AMDGPU target specific information is: 561 562**processor** 563 Is an AMDGPU processor or alternative processor name specified in 564 :ref:`amdgpu-processor-table`. The non-canonical form target ID allows both 565 the primary processor and alternative processor names. The canonical form 566 target ID only allow the primary processor name. 567 568**target-feature** 569 Is a target feature name specified in :ref:`amdgpu-target-features-table` that 570 is supported by the processor. The target features supported by each processor 571 is specified in :ref:`amdgpu-processor-table`. Those that can be specified in 572 a target ID are marked as being controlled by ``-mcpu`` and 573 ``--offload-arch``. Each target feature must appear at most once in a target 574 ID. The non-canonical form target ID allows the target features to be 575 specified in any order. The canonical form target ID requires the target 576 features to be specified in alphabetic order. 577 578.. _amdgpu-target-id-v2-v3: 579 580Code Object V2 to V3 Target ID 581~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 582 583The target ID syntax for code object V2 to V3 is the same as defined in `Clang 584Offload Bundler <https://clang.llvm.org/docs/ClangOffloadBundler.html>`_ except 585when used in the :ref:`amdgpu-assembler-directive-amdgcn-target` assembler 586directive and the bundle entry ID. In those cases it has the following BNF 587syntax: 588 589.. code:: 590 591 <target-id> ::== <processor> ( "+" <target-feature> )* 592 593Where a target feature is omitted if *Off* and present if *On* or *Any*. 594 595.. note:: 596 597 The code object V2 to V3 cannot represent *Any* and treats it the same as 598 *On*. 599 600.. _amdgpu-embedding-bundled-objects: 601 602Embedding Bundled Code Objects 603------------------------------ 604 605AMDGPU supports the HIP and OpenMP languages that perform code object embedding 606as described in `Clang Offload Bundler 607<https://clang.llvm.org/docs/ClangOffloadBundler.html>`_. 608 609.. note:: 610 611 The target ID syntax used for code object V2 to V3 for a bundle entry ID 612 differs from that used elsewhere. See :ref:`amdgpu-target-id-v2-v3`. 613 614.. _amdgpu-address-spaces: 615 616Address Spaces 617-------------- 618 619The AMDGPU architecture supports a number of memory address spaces. The address 620space names use the OpenCL standard names, with some additions. 621 622The AMDGPU address spaces correspond to target architecture specific LLVM 623address space numbers used in LLVM IR. 624 625The AMDGPU address spaces are described in 626:ref:`amdgpu-address-spaces-table`. Only 64-bit process address spaces are 627supported for the ``amdgcn`` target. 628 629 .. table:: AMDGPU Address Spaces 630 :name: amdgpu-address-spaces-table 631 632 ================================= =============== =========== ================ ======= ============================ 633 .. 64-Bit Process Address Space 634 --------------------------------- --------------- ----------- ---------------- ------------------------------------ 635 Address Space Name LLVM IR Address HSA Segment Hardware Address NULL Value 636 Space Number Name Name Size 637 ================================= =============== =========== ================ ======= ============================ 638 Generic 0 flat flat 64 0x0000000000000000 639 Global 1 global global 64 0x0000000000000000 640 Region 2 N/A GDS 32 *not implemented for AMDHSA* 641 Local 3 group LDS 32 0xFFFFFFFF 642 Constant 4 constant *same as global* 64 0x0000000000000000 643 Private 5 private scratch 32 0xFFFFFFFF 644 Constant 32-bit 6 *TODO* 0x00000000 645 Buffer Fat Pointer (experimental) 7 *TODO* 646 ================================= =============== =========== ================ ======= ============================ 647 648**Generic** 649 The generic address space is supported unless the *Target Properties* column 650 of :ref:`amdgpu-processor-table` specifies *Does not support generic address 651 space*. 652 653 The generic address space uses the hardware flat address support for two fixed 654 ranges of virtual addresses (the private and local apertures), that are 655 outside the range of addressable global memory, to map from a flat address to 656 a private or local address. This uses FLAT instructions that can take a flat 657 address and access global, private (scratch), and group (LDS) memory depending 658 on if the address is within one of the aperture ranges. 659 660 Flat access to scratch requires hardware aperture setup and setup in the 661 kernel prologue (see :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`). Flat 662 access to LDS requires hardware aperture setup and M0 (GFX7-GFX8) register 663 setup (see :ref:`amdgpu-amdhsa-kernel-prolog-m0`). 664 665 To convert between a private or group address space address (termed a segment 666 address) and a flat address the base address of the corresponding aperture 667 can be used. For GFX7-GFX8 these are available in the 668 :ref:`amdgpu-amdhsa-hsa-aql-queue` the address of which can be obtained with 669 Queue Ptr SGPR (see :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). For 670 GFX9-GFX10 the aperture base addresses are directly available as inline 671 constant registers ``SRC_SHARED_BASE/LIMIT`` and ``SRC_PRIVATE_BASE/LIMIT``. 672 In 64-bit address mode the aperture sizes are 2^32 bytes and the base is 673 aligned to 2^32 which makes it easier to convert from flat to segment or 674 segment to flat. 675 676 A global address space address has the same value when used as a flat address 677 so no conversion is needed. 678 679**Global and Constant** 680 The global and constant address spaces both use global virtual addresses, 681 which are the same virtual address space used by the CPU. However, some 682 virtual addresses may only be accessible to the CPU, some only accessible 683 by the GPU, and some by both. 684 685 Using the constant address space indicates that the data will not change 686 during the execution of the kernel. This allows scalar read instructions to 687 be used. As the constant address space could only be modified on the host 688 side, a generic pointer loaded from the constant address space is safe to be 689 assumed as a global pointer since only the device global memory is visible 690 and managed on the host side. The vector and scalar L1 caches are invalidated 691 of volatile data before each kernel dispatch execution to allow constant 692 memory to change values between kernel dispatches. 693 694**Region** 695 The region address space uses the hardware Global Data Store (GDS). All 696 wavefronts executing on the same device will access the same memory for any 697 given region address. However, the same region address accessed by wavefronts 698 executing on different devices will access different memory. It is higher 699 performance than global memory. It is allocated by the runtime. The data 700 store (DS) instructions can be used to access it. 701 702**Local** 703 The local address space uses the hardware Local Data Store (LDS) which is 704 automatically allocated when the hardware creates the wavefronts of a 705 work-group, and freed when all the wavefronts of a work-group have 706 terminated. All wavefronts belonging to the same work-group will access the 707 same memory for any given local address. However, the same local address 708 accessed by wavefronts belonging to different work-groups will access 709 different memory. It is higher performance than global memory. The data store 710 (DS) instructions can be used to access it. 711 712**Private** 713 The private address space uses the hardware scratch memory support which 714 automatically allocates memory when it creates a wavefront and frees it when 715 a wavefronts terminates. The memory accessed by a lane of a wavefront for any 716 given private address will be different to the memory accessed by another lane 717 of the same or different wavefront for the same private address. 718 719 If a kernel dispatch uses scratch, then the hardware allocates memory from a 720 pool of backing memory allocated by the runtime for each wavefront. The lanes 721 of the wavefront access this using dword (4 byte) interleaving. The mapping 722 used from private address to backing memory address is: 723 724 ``wavefront-scratch-base + 725 ((private-address / 4) * wavefront-size * 4) + 726 (wavefront-lane-id * 4) + (private-address % 4)`` 727 728 If each lane of a wavefront accesses the same private address, the 729 interleaving results in adjacent dwords being accessed and hence requires 730 fewer cache lines to be fetched. 731 732 There are different ways that the wavefront scratch base address is 733 determined by a wavefront (see 734 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 735 736 Scratch memory can be accessed in an interleaved manner using buffer 737 instructions with the scratch buffer descriptor and per wavefront scratch 738 offset, by the scratch instructions, or by flat instructions. Multi-dword 739 access is not supported except by flat and scratch instructions in 740 GFX9-GFX10. 741 742**Constant 32-bit** 743 *TODO* 744 745**Buffer Fat Pointer** 746 The buffer fat pointer is an experimental address space that is currently 747 unsupported in the backend. It exposes a non-integral pointer that is in 748 the future intended to support the modelling of 128-bit buffer descriptors 749 plus a 32-bit offset into the buffer (in total encapsulating a 160-bit 750 *pointer*), allowing normal LLVM load/store/atomic operations to be used to 751 model the buffer descriptors used heavily in graphics workloads targeting 752 the backend. 753 754.. _amdgpu-memory-scopes: 755 756Memory Scopes 757------------- 758 759This section provides LLVM memory synchronization scopes supported by the AMDGPU 760backend memory model when the target triple OS is ``amdhsa`` (see 761:ref:`amdgpu-amdhsa-memory-model` and :ref:`amdgpu-target-triples`). 762 763The memory model supported is based on the HSA memory model [HSA]_ which is 764based in turn on HRF-indirect with scope inclusion [HRF]_. The happens-before 765relation is transitive over the synchronizes-with relation independent of scope 766and synchronizes-with allows the memory scope instances to be inclusive (see 767table :ref:`amdgpu-amdhsa-llvm-sync-scopes-table`). 768 769This is different to the OpenCL [OpenCL]_ memory model which does not have scope 770inclusion and requires the memory scopes to exactly match. However, this 771is conservatively correct for OpenCL. 772 773 .. table:: AMDHSA LLVM Sync Scopes 774 :name: amdgpu-amdhsa-llvm-sync-scopes-table 775 776 ======================= =================================================== 777 LLVM Sync Scope Description 778 ======================= =================================================== 779 *none* The default: ``system``. 780 781 Synchronizes with, and participates in modification 782 and seq_cst total orderings with, other operations 783 (except image operations) for all address spaces 784 (except private, or generic that accesses private) 785 provided the other operation's sync scope is: 786 787 - ``system``. 788 - ``agent`` and executed by a thread on the same 789 agent. 790 - ``workgroup`` and executed by a thread in the 791 same work-group. 792 - ``wavefront`` and executed by a thread in the 793 same wavefront. 794 795 ``agent`` Synchronizes with, and participates in modification 796 and seq_cst total orderings with, other operations 797 (except image operations) for all address spaces 798 (except private, or generic that accesses private) 799 provided the other operation's sync scope is: 800 801 - ``system`` or ``agent`` and executed by a thread 802 on the same agent. 803 - ``workgroup`` and executed by a thread in the 804 same work-group. 805 - ``wavefront`` and executed by a thread in the 806 same wavefront. 807 808 ``workgroup`` Synchronizes with, and participates in modification 809 and seq_cst total orderings with, other operations 810 (except image operations) for all address spaces 811 (except private, or generic that accesses private) 812 provided the other operation's sync scope is: 813 814 - ``system``, ``agent`` or ``workgroup`` and 815 executed by a thread in the same work-group. 816 - ``wavefront`` and executed by a thread in the 817 same wavefront. 818 819 ``wavefront`` Synchronizes with, and participates in modification 820 and seq_cst total orderings with, other operations 821 (except image operations) for all address spaces 822 (except private, or generic that accesses private) 823 provided the other operation's sync scope is: 824 825 - ``system``, ``agent``, ``workgroup`` or 826 ``wavefront`` and executed by a thread in the 827 same wavefront. 828 829 ``singlethread`` Only synchronizes with and participates in 830 modification and seq_cst total orderings with, 831 other operations (except image operations) running 832 in the same thread for all address spaces (for 833 example, in signal handlers). 834 835 ``one-as`` Same as ``system`` but only synchronizes with other 836 operations within the same address space. 837 838 ``agent-one-as`` Same as ``agent`` but only synchronizes with other 839 operations within the same address space. 840 841 ``workgroup-one-as`` Same as ``workgroup`` but only synchronizes with 842 other operations within the same address space. 843 844 ``wavefront-one-as`` Same as ``wavefront`` but only synchronizes with 845 other operations within the same address space. 846 847 ``singlethread-one-as`` Same as ``singlethread`` but only synchronizes with 848 other operations within the same address space. 849 ======================= =================================================== 850 851LLVM IR Intrinsics 852------------------ 853 854The AMDGPU backend implements the following LLVM IR intrinsics. 855 856*This section is WIP.* 857 858.. TODO:: 859 860 List AMDGPU intrinsics. 861 862LLVM IR Attributes 863------------------ 864 865The AMDGPU backend supports the following LLVM IR attributes. 866 867 .. table:: AMDGPU LLVM IR Attributes 868 :name: amdgpu-llvm-ir-attributes-table 869 870 ======================================= ========================================================== 871 LLVM Attribute Description 872 ======================================= ========================================================== 873 "amdgpu-flat-work-group-size"="min,max" Specify the minimum and maximum flat work group sizes that 874 will be specified when the kernel is dispatched. Generated 875 by the ``amdgpu_flat_work_group_size`` CLANG attribute [CLANG-ATTR]_. 876 The implied default value is 1,1024. 877 878 "amdgpu-implicitarg-num-bytes"="n" Number of kernel argument bytes to add to the kernel 879 argument block size for the implicit arguments. This 880 varies by OS and language (for OpenCL see 881 :ref:`opencl-kernel-implicit-arguments-appended-for-amdhsa-os-table`). 882 "amdgpu-num-sgpr"="n" Specifies the number of SGPRs to use. Generated by 883 the ``amdgpu_num_sgpr`` CLANG attribute [CLANG-ATTR]_. 884 "amdgpu-num-vgpr"="n" Specifies the number of VGPRs to use. Generated by the 885 ``amdgpu_num_vgpr`` CLANG attribute [CLANG-ATTR]_. 886 "amdgpu-waves-per-eu"="m,n" Specify the minimum and maximum number of waves per 887 execution unit. Generated by the ``amdgpu_waves_per_eu`` 888 CLANG attribute [CLANG-ATTR]_. This is an optimization hint, 889 and the backend may not be able to satisfy the request. If 890 the specified range is incompatible with the function's 891 "amdgpu-flat-work-group-size" value, the implied occupancy 892 bounds by the workgroup size takes precedence. 893 894 "amdgpu-ieee" true/false. Specify whether the function expects the IEEE field of the 895 mode register to be set on entry. Overrides the default for 896 the calling convention. 897 "amdgpu-dx10-clamp" true/false. Specify whether the function expects the DX10_CLAMP field of 898 the mode register to be set on entry. Overrides the default 899 for the calling convention. 900 901 "amdgpu-no-workitem-id-x" Indicates the function does not depend on the value of the 902 llvm.amdgcn.workitem.id.x intrinsic. If a function is marked with this 903 attribute, or reached through a call site marked with this attribute, 904 the value returned by the intrinsic is undefined. The backend can 905 generally infer this during code generation, so typically there is no 906 benefit to frontends marking functions with this. 907 908 "amdgpu-no-workitem-id-y" The same as amdgpu-no-workitem-id-x, except for the 909 llvm.amdgcn.workitem.id.y intrinsic. 910 911 "amdgpu-no-workitem-id-z" The same as amdgpu-no-workitem-id-x, except for the 912 llvm.amdgcn.workitem.id.z intrinsic. 913 914 "amdgpu-no-workgroup-id-x" The same as amdgpu-no-workitem-id-x, except for the 915 llvm.amdgcn.workgroup.id.x intrinsic. 916 917 "amdgpu-no-workgroup-id-y" The same as amdgpu-no-workitem-id-x, except for the 918 llvm.amdgcn.workgroup.id.y intrinsic. 919 920 "amdgpu-no-workgroup-id-z" The same as amdgpu-no-workitem-id-x, except for the 921 llvm.amdgcn.workgroup.id.z intrinsic. 922 923 "amdgpu-no-dispatch-ptr" The same as amdgpu-no-workitem-id-x, except for the 924 llvm.amdgcn.dispatch.ptr intrinsic. 925 926 "amdgpu-no-implicitarg-ptr" The same as amdgpu-no-workitem-id-x, except for the 927 llvm.amdgcn.implicitarg.ptr intrinsic. 928 929 "amdgpu-no-dispatch-id" The same as amdgpu-no-workitem-id-x, except for the 930 llvm.amdgcn.dispatch.id intrinsic. 931 932 "amdgpu-no-queue-ptr" Similar to amdgpu-no-workitem-id-x, except for the 933 llvm.amdgcn.queue.ptr intrinsic. Note that unlike the other ABI hint 934 attributes, the queue pointer may be required in situations where the 935 intrinsic call does not directly appear in the program. Some subtargets 936 require the queue pointer for to handle some addrspacecasts, as well 937 as the llvm.amdgcn.is.shared, llvm.amdgcn.is.private, llvm.trap, and 938 llvm.debug intrinsics. 939 940 "amdgpu-no-hostcall-ptr" Similar to amdgpu-no-implicitarg-ptr, except specific to the implicit 941 kernel argument that holds the pointer to the hostcall buffer. If this 942 attribute is absent, then the amdgpu-no-implicitarg-ptr is also removed. 943 944 "amdgpu-no-heap-ptr" Similar to amdgpu-no-implicitarg-ptr, except specific to the implicit 945 kernel argument that holds the pointer to an initialized memory buffer 946 that conforms to the requirements of the malloc/free device library V1 947 version implementation. If this attribute is absent, then the 948 amdgpu-no-implicitarg-ptr is also removed. 949 950 ======================================= ========================================================== 951 952.. _amdgpu-elf-code-object: 953 954ELF Code Object 955=============== 956 957The AMDGPU backend generates a standard ELF [ELF]_ relocatable code object that 958can be linked by ``lld`` to produce a standard ELF shared code object which can 959be loaded and executed on an AMDGPU target. 960 961.. _amdgpu-elf-header: 962 963Header 964------ 965 966The AMDGPU backend uses the following ELF header: 967 968 .. table:: AMDGPU ELF Header 969 :name: amdgpu-elf-header-table 970 971 ========================== =============================== 972 Field Value 973 ========================== =============================== 974 ``e_ident[EI_CLASS]`` ``ELFCLASS64`` 975 ``e_ident[EI_DATA]`` ``ELFDATA2LSB`` 976 ``e_ident[EI_OSABI]`` - ``ELFOSABI_NONE`` 977 - ``ELFOSABI_AMDGPU_HSA`` 978 - ``ELFOSABI_AMDGPU_PAL`` 979 - ``ELFOSABI_AMDGPU_MESA3D`` 980 ``e_ident[EI_ABIVERSION]`` - ``ELFABIVERSION_AMDGPU_HSA_V2`` 981 - ``ELFABIVERSION_AMDGPU_HSA_V3`` 982 - ``ELFABIVERSION_AMDGPU_HSA_V4`` 983 - ``ELFABIVERSION_AMDGPU_HSA_V5`` 984 - ``ELFABIVERSION_AMDGPU_PAL`` 985 - ``ELFABIVERSION_AMDGPU_MESA3D`` 986 ``e_type`` - ``ET_REL`` 987 - ``ET_DYN`` 988 ``e_machine`` ``EM_AMDGPU`` 989 ``e_entry`` 0 990 ``e_flags`` See :ref:`amdgpu-elf-header-e_flags-v2-table`, 991 :ref:`amdgpu-elf-header-e_flags-table-v3`, 992 and :ref:`amdgpu-elf-header-e_flags-table-v4-onwards` 993 ========================== =============================== 994 995.. 996 997 .. table:: AMDGPU ELF Header Enumeration Values 998 :name: amdgpu-elf-header-enumeration-values-table 999 1000 =============================== ===== 1001 Name Value 1002 =============================== ===== 1003 ``EM_AMDGPU`` 224 1004 ``ELFOSABI_NONE`` 0 1005 ``ELFOSABI_AMDGPU_HSA`` 64 1006 ``ELFOSABI_AMDGPU_PAL`` 65 1007 ``ELFOSABI_AMDGPU_MESA3D`` 66 1008 ``ELFABIVERSION_AMDGPU_HSA_V2`` 0 1009 ``ELFABIVERSION_AMDGPU_HSA_V3`` 1 1010 ``ELFABIVERSION_AMDGPU_HSA_V4`` 2 1011 ``ELFABIVERSION_AMDGPU_HSA_V5`` 3 1012 ``ELFABIVERSION_AMDGPU_PAL`` 0 1013 ``ELFABIVERSION_AMDGPU_MESA3D`` 0 1014 =============================== ===== 1015 1016``e_ident[EI_CLASS]`` 1017 The ELF class is: 1018 1019 * ``ELFCLASS32`` for ``r600`` architecture. 1020 1021 * ``ELFCLASS64`` for ``amdgcn`` architecture which only supports 64-bit 1022 process address space applications. 1023 1024``e_ident[EI_DATA]`` 1025 All AMDGPU targets use ``ELFDATA2LSB`` for little-endian byte ordering. 1026 1027``e_ident[EI_OSABI]`` 1028 One of the following AMDGPU target architecture specific OS ABIs 1029 (see :ref:`amdgpu-os`): 1030 1031 * ``ELFOSABI_NONE`` for *unknown* OS. 1032 1033 * ``ELFOSABI_AMDGPU_HSA`` for ``amdhsa`` OS. 1034 1035 * ``ELFOSABI_AMDGPU_PAL`` for ``amdpal`` OS. 1036 1037 * ``ELFOSABI_AMDGPU_MESA3D`` for ``mesa3D`` OS. 1038 1039``e_ident[EI_ABIVERSION]`` 1040 The ABI version of the AMDGPU target architecture specific OS ABI to which the code 1041 object conforms: 1042 1043 * ``ELFABIVERSION_AMDGPU_HSA_V2`` is used to specify the version of AMD HSA 1044 runtime ABI for code object V2. Specify using the Clang option 1045 ``-mcode-object-version=2``. 1046 1047 * ``ELFABIVERSION_AMDGPU_HSA_V3`` is used to specify the version of AMD HSA 1048 runtime ABI for code object V3. Specify using the Clang option 1049 ``-mcode-object-version=3``. 1050 1051 * ``ELFABIVERSION_AMDGPU_HSA_V4`` is used to specify the version of AMD HSA 1052 runtime ABI for code object V4. Specify using the Clang option 1053 ``-mcode-object-version=4``. This is the default code object 1054 version if not specified. 1055 1056 * ``ELFABIVERSION_AMDGPU_HSA_V5`` is used to specify the version of AMD HSA 1057 runtime ABI for code object V5. Specify using the Clang option 1058 ``-mcode-object-version=5``. 1059 1060 * ``ELFABIVERSION_AMDGPU_PAL`` is used to specify the version of AMD PAL 1061 runtime ABI. 1062 1063 * ``ELFABIVERSION_AMDGPU_MESA3D`` is used to specify the version of AMD MESA 1064 3D runtime ABI. 1065 1066``e_type`` 1067 Can be one of the following values: 1068 1069 1070 ``ET_REL`` 1071 The type produced by the AMDGPU backend compiler as it is relocatable code 1072 object. 1073 1074 ``ET_DYN`` 1075 The type produced by the linker as it is a shared code object. 1076 1077 The AMD HSA runtime loader requires a ``ET_DYN`` code object. 1078 1079``e_machine`` 1080 The value ``EM_AMDGPU`` is used for the machine for all processors supported 1081 by the ``r600`` and ``amdgcn`` architectures (see 1082 :ref:`amdgpu-processor-table`). The specific processor is specified in the 1083 ``NT_AMD_HSA_ISA_VERSION`` note record for code object V2 (see 1084 :ref:`amdgpu-note-records-v2`) and in the ``EF_AMDGPU_MACH`` bit field of the 1085 ``e_flags`` for code object V3 and above (see 1086 :ref:`amdgpu-elf-header-e_flags-table-v3` and 1087 :ref:`amdgpu-elf-header-e_flags-table-v4-onwards`). 1088 1089``e_entry`` 1090 The entry point is 0 as the entry points for individual kernels must be 1091 selected in order to invoke them through AQL packets. 1092 1093``e_flags`` 1094 The AMDGPU backend uses the following ELF header flags: 1095 1096 .. table:: AMDGPU ELF Header ``e_flags`` for Code Object V2 1097 :name: amdgpu-elf-header-e_flags-v2-table 1098 1099 ===================================== ===== ============================= 1100 Name Value Description 1101 ===================================== ===== ============================= 1102 ``EF_AMDGPU_FEATURE_XNACK_V2`` 0x01 Indicates if the ``xnack`` 1103 target feature is 1104 enabled for all code 1105 contained in the code object. 1106 If the processor 1107 does not support the 1108 ``xnack`` target 1109 feature then must 1110 be 0. 1111 See 1112 :ref:`amdgpu-target-features`. 1113 ``EF_AMDGPU_FEATURE_TRAP_HANDLER_V2`` 0x02 Indicates if the trap 1114 handler is enabled for all 1115 code contained in the code 1116 object. If the processor 1117 does not support a trap 1118 handler then must be 0. 1119 See 1120 :ref:`amdgpu-target-features`. 1121 ===================================== ===== ============================= 1122 1123 .. table:: AMDGPU ELF Header ``e_flags`` for Code Object V3 1124 :name: amdgpu-elf-header-e_flags-table-v3 1125 1126 ================================= ===== ============================= 1127 Name Value Description 1128 ================================= ===== ============================= 1129 ``EF_AMDGPU_MACH`` 0x0ff AMDGPU processor selection 1130 mask for 1131 ``EF_AMDGPU_MACH_xxx`` values 1132 defined in 1133 :ref:`amdgpu-ef-amdgpu-mach-table`. 1134 ``EF_AMDGPU_FEATURE_XNACK_V3`` 0x100 Indicates if the ``xnack`` 1135 target feature is 1136 enabled for all code 1137 contained in the code object. 1138 If the processor 1139 does not support the 1140 ``xnack`` target 1141 feature then must 1142 be 0. 1143 See 1144 :ref:`amdgpu-target-features`. 1145 ``EF_AMDGPU_FEATURE_SRAMECC_V3`` 0x200 Indicates if the ``sramecc`` 1146 target feature is 1147 enabled for all code 1148 contained in the code object. 1149 If the processor 1150 does not support the 1151 ``sramecc`` target 1152 feature then must 1153 be 0. 1154 See 1155 :ref:`amdgpu-target-features`. 1156 ================================= ===== ============================= 1157 1158 .. table:: AMDGPU ELF Header ``e_flags`` for Code Object V4 and After 1159 :name: amdgpu-elf-header-e_flags-table-v4-onwards 1160 1161 ============================================ ===== =================================== 1162 Name Value Description 1163 ============================================ ===== =================================== 1164 ``EF_AMDGPU_MACH`` 0x0ff AMDGPU processor selection 1165 mask for 1166 ``EF_AMDGPU_MACH_xxx`` values 1167 defined in 1168 :ref:`amdgpu-ef-amdgpu-mach-table`. 1169 ``EF_AMDGPU_FEATURE_XNACK_V4`` 0x300 XNACK selection mask for 1170 ``EF_AMDGPU_FEATURE_XNACK_*_V4`` 1171 values. 1172 ``EF_AMDGPU_FEATURE_XNACK_UNSUPPORTED_V4`` 0x000 XNACK unsuppored. 1173 ``EF_AMDGPU_FEATURE_XNACK_ANY_V4`` 0x100 XNACK can have any value. 1174 ``EF_AMDGPU_FEATURE_XNACK_OFF_V4`` 0x200 XNACK disabled. 1175 ``EF_AMDGPU_FEATURE_XNACK_ON_V4`` 0x300 XNACK enabled. 1176 ``EF_AMDGPU_FEATURE_SRAMECC_V4`` 0xc00 SRAMECC selection mask for 1177 ``EF_AMDGPU_FEATURE_SRAMECC_*_V4`` 1178 values. 1179 ``EF_AMDGPU_FEATURE_SRAMECC_UNSUPPORTED_V4`` 0x000 SRAMECC unsuppored. 1180 ``EF_AMDGPU_FEATURE_SRAMECC_ANY_V4`` 0x400 SRAMECC can have any value. 1181 ``EF_AMDGPU_FEATURE_SRAMECC_OFF_V4`` 0x800 SRAMECC disabled, 1182 ``EF_AMDGPU_FEATURE_SRAMECC_ON_V4`` 0xc00 SRAMECC enabled. 1183 ============================================ ===== =================================== 1184 1185 .. table:: AMDGPU ``EF_AMDGPU_MACH`` Values 1186 :name: amdgpu-ef-amdgpu-mach-table 1187 1188 ==================================== ========== ============================= 1189 Name Value Description (see 1190 :ref:`amdgpu-processor-table`) 1191 ==================================== ========== ============================= 1192 ``EF_AMDGPU_MACH_NONE`` 0x000 *not specified* 1193 ``EF_AMDGPU_MACH_R600_R600`` 0x001 ``r600`` 1194 ``EF_AMDGPU_MACH_R600_R630`` 0x002 ``r630`` 1195 ``EF_AMDGPU_MACH_R600_RS880`` 0x003 ``rs880`` 1196 ``EF_AMDGPU_MACH_R600_RV670`` 0x004 ``rv670`` 1197 ``EF_AMDGPU_MACH_R600_RV710`` 0x005 ``rv710`` 1198 ``EF_AMDGPU_MACH_R600_RV730`` 0x006 ``rv730`` 1199 ``EF_AMDGPU_MACH_R600_RV770`` 0x007 ``rv770`` 1200 ``EF_AMDGPU_MACH_R600_CEDAR`` 0x008 ``cedar`` 1201 ``EF_AMDGPU_MACH_R600_CYPRESS`` 0x009 ``cypress`` 1202 ``EF_AMDGPU_MACH_R600_JUNIPER`` 0x00a ``juniper`` 1203 ``EF_AMDGPU_MACH_R600_REDWOOD`` 0x00b ``redwood`` 1204 ``EF_AMDGPU_MACH_R600_SUMO`` 0x00c ``sumo`` 1205 ``EF_AMDGPU_MACH_R600_BARTS`` 0x00d ``barts`` 1206 ``EF_AMDGPU_MACH_R600_CAICOS`` 0x00e ``caicos`` 1207 ``EF_AMDGPU_MACH_R600_CAYMAN`` 0x00f ``cayman`` 1208 ``EF_AMDGPU_MACH_R600_TURKS`` 0x010 ``turks`` 1209 *reserved* 0x011 - Reserved for ``r600`` 1210 0x01f architecture processors. 1211 ``EF_AMDGPU_MACH_AMDGCN_GFX600`` 0x020 ``gfx600`` 1212 ``EF_AMDGPU_MACH_AMDGCN_GFX601`` 0x021 ``gfx601`` 1213 ``EF_AMDGPU_MACH_AMDGCN_GFX700`` 0x022 ``gfx700`` 1214 ``EF_AMDGPU_MACH_AMDGCN_GFX701`` 0x023 ``gfx701`` 1215 ``EF_AMDGPU_MACH_AMDGCN_GFX702`` 0x024 ``gfx702`` 1216 ``EF_AMDGPU_MACH_AMDGCN_GFX703`` 0x025 ``gfx703`` 1217 ``EF_AMDGPU_MACH_AMDGCN_GFX704`` 0x026 ``gfx704`` 1218 *reserved* 0x027 Reserved. 1219 ``EF_AMDGPU_MACH_AMDGCN_GFX801`` 0x028 ``gfx801`` 1220 ``EF_AMDGPU_MACH_AMDGCN_GFX802`` 0x029 ``gfx802`` 1221 ``EF_AMDGPU_MACH_AMDGCN_GFX803`` 0x02a ``gfx803`` 1222 ``EF_AMDGPU_MACH_AMDGCN_GFX810`` 0x02b ``gfx810`` 1223 ``EF_AMDGPU_MACH_AMDGCN_GFX900`` 0x02c ``gfx900`` 1224 ``EF_AMDGPU_MACH_AMDGCN_GFX902`` 0x02d ``gfx902`` 1225 ``EF_AMDGPU_MACH_AMDGCN_GFX904`` 0x02e ``gfx904`` 1226 ``EF_AMDGPU_MACH_AMDGCN_GFX906`` 0x02f ``gfx906`` 1227 ``EF_AMDGPU_MACH_AMDGCN_GFX908`` 0x030 ``gfx908`` 1228 ``EF_AMDGPU_MACH_AMDGCN_GFX909`` 0x031 ``gfx909`` 1229 ``EF_AMDGPU_MACH_AMDGCN_GFX90C`` 0x032 ``gfx90c`` 1230 ``EF_AMDGPU_MACH_AMDGCN_GFX1010`` 0x033 ``gfx1010`` 1231 ``EF_AMDGPU_MACH_AMDGCN_GFX1011`` 0x034 ``gfx1011`` 1232 ``EF_AMDGPU_MACH_AMDGCN_GFX1012`` 0x035 ``gfx1012`` 1233 ``EF_AMDGPU_MACH_AMDGCN_GFX1030`` 0x036 ``gfx1030`` 1234 ``EF_AMDGPU_MACH_AMDGCN_GFX1031`` 0x037 ``gfx1031`` 1235 ``EF_AMDGPU_MACH_AMDGCN_GFX1032`` 0x038 ``gfx1032`` 1236 ``EF_AMDGPU_MACH_AMDGCN_GFX1033`` 0x039 ``gfx1033`` 1237 ``EF_AMDGPU_MACH_AMDGCN_GFX602`` 0x03a ``gfx602`` 1238 ``EF_AMDGPU_MACH_AMDGCN_GFX705`` 0x03b ``gfx705`` 1239 ``EF_AMDGPU_MACH_AMDGCN_GFX805`` 0x03c ``gfx805`` 1240 ``EF_AMDGPU_MACH_AMDGCN_GFX1035`` 0x03d ``gfx1035`` 1241 ``EF_AMDGPU_MACH_AMDGCN_GFX1034`` 0x03e ``gfx1034`` 1242 ``EF_AMDGPU_MACH_AMDGCN_GFX90A`` 0x03f ``gfx90a`` 1243 ``EF_AMDGPU_MACH_AMDGCN_GFX940`` 0x040 ``gfx940`` 1244 *reserved* 0x041 Reserved. 1245 ``EF_AMDGPU_MACH_AMDGCN_GFX1013`` 0x042 ``gfx1013`` 1246 *reserved* 0x043 Reserved. 1247 *reserved* 0x044 Reserved. 1248 ``EF_AMDGPU_MACH_AMDGCN_GFX1036`` 0x045 ``gfx1036`` 1249 ==================================== ========== ============================= 1250 1251Sections 1252-------- 1253 1254An AMDGPU target ELF code object has the standard ELF sections which include: 1255 1256 .. table:: AMDGPU ELF Sections 1257 :name: amdgpu-elf-sections-table 1258 1259 ================== ================ ================================= 1260 Name Type Attributes 1261 ================== ================ ================================= 1262 ``.bss`` ``SHT_NOBITS`` ``SHF_ALLOC`` + ``SHF_WRITE`` 1263 ``.data`` ``SHT_PROGBITS`` ``SHF_ALLOC`` + ``SHF_WRITE`` 1264 ``.debug_``\ *\** ``SHT_PROGBITS`` *none* 1265 ``.dynamic`` ``SHT_DYNAMIC`` ``SHF_ALLOC`` 1266 ``.dynstr`` ``SHT_PROGBITS`` ``SHF_ALLOC`` 1267 ``.dynsym`` ``SHT_PROGBITS`` ``SHF_ALLOC`` 1268 ``.got`` ``SHT_PROGBITS`` ``SHF_ALLOC`` + ``SHF_WRITE`` 1269 ``.hash`` ``SHT_HASH`` ``SHF_ALLOC`` 1270 ``.note`` ``SHT_NOTE`` *none* 1271 ``.rela``\ *name* ``SHT_RELA`` *none* 1272 ``.rela.dyn`` ``SHT_RELA`` *none* 1273 ``.rodata`` ``SHT_PROGBITS`` ``SHF_ALLOC`` 1274 ``.shstrtab`` ``SHT_STRTAB`` *none* 1275 ``.strtab`` ``SHT_STRTAB`` *none* 1276 ``.symtab`` ``SHT_SYMTAB`` *none* 1277 ``.text`` ``SHT_PROGBITS`` ``SHF_ALLOC`` + ``SHF_EXECINSTR`` 1278 ================== ================ ================================= 1279 1280These sections have their standard meanings (see [ELF]_) and are only generated 1281if needed. 1282 1283``.debug``\ *\** 1284 The standard DWARF sections. See :ref:`amdgpu-dwarf-debug-information` for 1285 information on the DWARF produced by the AMDGPU backend. 1286 1287``.dynamic``, ``.dynstr``, ``.dynsym``, ``.hash`` 1288 The standard sections used by a dynamic loader. 1289 1290``.note`` 1291 See :ref:`amdgpu-note-records` for the note records supported by the AMDGPU 1292 backend. 1293 1294``.rela``\ *name*, ``.rela.dyn`` 1295 For relocatable code objects, *name* is the name of the section that the 1296 relocation records apply. For example, ``.rela.text`` is the section name for 1297 relocation records associated with the ``.text`` section. 1298 1299 For linked shared code objects, ``.rela.dyn`` contains all the relocation 1300 records from each of the relocatable code object's ``.rela``\ *name* sections. 1301 1302 See :ref:`amdgpu-relocation-records` for the relocation records supported by 1303 the AMDGPU backend. 1304 1305``.text`` 1306 The executable machine code for the kernels and functions they call. Generated 1307 as position independent code. See :ref:`amdgpu-code-conventions` for 1308 information on conventions used in the isa generation. 1309 1310.. _amdgpu-note-records: 1311 1312Note Records 1313------------ 1314 1315The AMDGPU backend code object contains ELF note records in the ``.note`` 1316section. The set of generated notes and their semantics depend on the code 1317object version; see :ref:`amdgpu-note-records-v2` and 1318:ref:`amdgpu-note-records-v3-onwards`. 1319 1320As required by ``ELFCLASS32`` and ``ELFCLASS64``, minimal zero-byte padding 1321must be generated after the ``name`` field to ensure the ``desc`` field is 4 1322byte aligned. In addition, minimal zero-byte padding must be generated to 1323ensure the ``desc`` field size is a multiple of 4 bytes. The ``sh_addralign`` 1324field of the ``.note`` section must be at least 4 to indicate at least 8 byte 1325alignment. 1326 1327.. _amdgpu-note-records-v2: 1328 1329Code Object V2 Note Records 1330~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1331 1332.. warning:: 1333 Code object V2 is not the default code object version emitted by 1334 this version of LLVM. 1335 1336The AMDGPU backend code object uses the following ELF note record in the 1337``.note`` section when compiling for code object V2. 1338 1339The note record vendor field is "AMD". 1340 1341Additional note records may be present, but any which are not documented here 1342are deprecated and should not be used. 1343 1344 .. table:: AMDGPU Code Object V2 ELF Note Records 1345 :name: amdgpu-elf-note-records-v2-table 1346 1347 ===== ===================================== ====================================== 1348 Name Type Description 1349 ===== ===================================== ====================================== 1350 "AMD" ``NT_AMD_HSA_CODE_OBJECT_VERSION`` Code object version. 1351 "AMD" ``NT_AMD_HSA_HSAIL`` HSAIL properties generated by the HSAIL 1352 Finalizer and not the LLVM compiler. 1353 "AMD" ``NT_AMD_HSA_ISA_VERSION`` Target ISA version. 1354 "AMD" ``NT_AMD_HSA_METADATA`` Metadata null terminated string in 1355 YAML [YAML]_ textual format. 1356 "AMD" ``NT_AMD_HSA_ISA_NAME`` Target ISA name. 1357 ===== ===================================== ====================================== 1358 1359.. 1360 1361 .. table:: AMDGPU Code Object V2 ELF Note Record Enumeration Values 1362 :name: amdgpu-elf-note-record-enumeration-values-v2-table 1363 1364 ===================================== ===== 1365 Name Value 1366 ===================================== ===== 1367 ``NT_AMD_HSA_CODE_OBJECT_VERSION`` 1 1368 ``NT_AMD_HSA_HSAIL`` 2 1369 ``NT_AMD_HSA_ISA_VERSION`` 3 1370 *reserved* 4-9 1371 ``NT_AMD_HSA_METADATA`` 10 1372 ``NT_AMD_HSA_ISA_NAME`` 11 1373 ===================================== ===== 1374 1375``NT_AMD_HSA_CODE_OBJECT_VERSION`` 1376 Specifies the code object version number. The description field has the 1377 following layout: 1378 1379 .. code:: c 1380 1381 struct amdgpu_hsa_note_code_object_version_s { 1382 uint32_t major_version; 1383 uint32_t minor_version; 1384 }; 1385 1386 The ``major_version`` has a value less than or equal to 2. 1387 1388``NT_AMD_HSA_HSAIL`` 1389 Specifies the HSAIL properties used by the HSAIL Finalizer. The description 1390 field has the following layout: 1391 1392 .. code:: c 1393 1394 struct amdgpu_hsa_note_hsail_s { 1395 uint32_t hsail_major_version; 1396 uint32_t hsail_minor_version; 1397 uint8_t profile; 1398 uint8_t machine_model; 1399 uint8_t default_float_round; 1400 }; 1401 1402``NT_AMD_HSA_ISA_VERSION`` 1403 Specifies the target ISA version. The description field has the following layout: 1404 1405 .. code:: c 1406 1407 struct amdgpu_hsa_note_isa_s { 1408 uint16_t vendor_name_size; 1409 uint16_t architecture_name_size; 1410 uint32_t major; 1411 uint32_t minor; 1412 uint32_t stepping; 1413 char vendor_and_architecture_name[1]; 1414 }; 1415 1416 ``vendor_name_size`` and ``architecture_name_size`` are the length of the 1417 vendor and architecture names respectively, including the NUL character. 1418 1419 ``vendor_and_architecture_name`` contains the NUL terminates string for the 1420 vendor, immediately followed by the NUL terminated string for the 1421 architecture. 1422 1423 This note record is used by the HSA runtime loader. 1424 1425 Code object V2 only supports a limited number of processors and has fixed 1426 settings for target features. See 1427 :ref:`amdgpu-elf-note-record-supported_processors-v2-table` for a list of 1428 processors and the corresponding target ID. In the table the note record ISA 1429 name is a concatenation of the vendor name, architecture name, major, minor, 1430 and stepping separated by a ":". 1431 1432 The target ID column shows the processor name and fixed target features used 1433 by the LLVM compiler. The LLVM compiler does not generate a 1434 ``NT_AMD_HSA_HSAIL`` note record. 1435 1436 A code object generated by the Finalizer also uses code object V2 and always 1437 generates a ``NT_AMD_HSA_HSAIL`` note record. The processor name and 1438 ``sramecc`` target feature is as shown in 1439 :ref:`amdgpu-elf-note-record-supported_processors-v2-table` but the ``xnack`` 1440 target feature is specified by the ``EF_AMDGPU_FEATURE_XNACK_V2`` ``e_flags`` 1441 bit. 1442 1443``NT_AMD_HSA_ISA_NAME`` 1444 Specifies the target ISA name as a non-NUL terminated string. 1445 1446 This note record is not used by the HSA runtime loader. 1447 1448 See the ``NT_AMD_HSA_ISA_VERSION`` note record description of the code object 1449 V2's limited support of processors and fixed settings for target features. 1450 1451 See :ref:`amdgpu-elf-note-record-supported_processors-v2-table` for a mapping 1452 from the string to the corresponding target ID. If the ``xnack`` target 1453 feature is supported and enabled, the string produced by the LLVM compiler 1454 will may have a ``+xnack`` appended. The Finlizer did not do the appending and 1455 instead used the ``EF_AMDGPU_FEATURE_XNACK_V2`` ``e_flags`` bit. 1456 1457``NT_AMD_HSA_METADATA`` 1458 Specifies extensible metadata associated with the code objects executed on HSA 1459 [HSA]_ compatible runtimes (see :ref:`amdgpu-os`). It is required when the 1460 target triple OS is ``amdhsa`` (see :ref:`amdgpu-target-triples`). See 1461 :ref:`amdgpu-amdhsa-code-object-metadata-v2` for the syntax of the code object 1462 metadata string. 1463 1464 .. table:: AMDGPU Code Object V2 Supported Processors and Fixed Target Feature Settings 1465 :name: amdgpu-elf-note-record-supported_processors-v2-table 1466 1467 ===================== ========================== 1468 Note Record ISA Name Target ID 1469 ===================== ========================== 1470 ``AMD:AMDGPU:6:0:0`` ``gfx600`` 1471 ``AMD:AMDGPU:6:0:1`` ``gfx601`` 1472 ``AMD:AMDGPU:6:0:2`` ``gfx602`` 1473 ``AMD:AMDGPU:7:0:0`` ``gfx700`` 1474 ``AMD:AMDGPU:7:0:1`` ``gfx701`` 1475 ``AMD:AMDGPU:7:0:2`` ``gfx702`` 1476 ``AMD:AMDGPU:7:0:3`` ``gfx703`` 1477 ``AMD:AMDGPU:7:0:4`` ``gfx704`` 1478 ``AMD:AMDGPU:7:0:5`` ``gfx705`` 1479 ``AMD:AMDGPU:8:0:0`` ``gfx802`` 1480 ``AMD:AMDGPU:8:0:1`` ``gfx801:xnack+`` 1481 ``AMD:AMDGPU:8:0:2`` ``gfx802`` 1482 ``AMD:AMDGPU:8:0:3`` ``gfx803`` 1483 ``AMD:AMDGPU:8:0:4`` ``gfx803`` 1484 ``AMD:AMDGPU:8:0:5`` ``gfx805`` 1485 ``AMD:AMDGPU:8:1:0`` ``gfx810:xnack+`` 1486 ``AMD:AMDGPU:9:0:0`` ``gfx900:xnack-`` 1487 ``AMD:AMDGPU:9:0:1`` ``gfx900:xnack+`` 1488 ``AMD:AMDGPU:9:0:2`` ``gfx902:xnack-`` 1489 ``AMD:AMDGPU:9:0:3`` ``gfx902:xnack+`` 1490 ``AMD:AMDGPU:9:0:4`` ``gfx904:xnack-`` 1491 ``AMD:AMDGPU:9:0:5`` ``gfx904:xnack+`` 1492 ``AMD:AMDGPU:9:0:6`` ``gfx906:sramecc-:xnack-`` 1493 ``AMD:AMDGPU:9:0:7`` ``gfx906:sramecc-:xnack+`` 1494 ``AMD:AMDGPU:9:0:12`` ``gfx90c:xnack-`` 1495 ===================== ========================== 1496 1497.. _amdgpu-note-records-v3-onwards: 1498 1499Code Object V3 and Above Note Records 1500~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1501 1502The AMDGPU backend code object uses the following ELF note record in the 1503``.note`` section when compiling for code object V3 and above. 1504 1505The note record vendor field is "AMDGPU". 1506 1507Additional note records may be present, but any which are not documented here 1508are deprecated and should not be used. 1509 1510 .. table:: AMDGPU Code Object V3 and Above ELF Note Records 1511 :name: amdgpu-elf-note-records-table-v3-onwards 1512 1513 ======== ============================== ====================================== 1514 Name Type Description 1515 ======== ============================== ====================================== 1516 "AMDGPU" ``NT_AMDGPU_METADATA`` Metadata in Message Pack [MsgPack]_ 1517 binary format. 1518 ======== ============================== ====================================== 1519 1520.. 1521 1522 .. table:: AMDGPU Code Object V3 and Above ELF Note Record Enumeration Values 1523 :name: amdgpu-elf-note-record-enumeration-values-table-v3-onwards 1524 1525 ============================== ===== 1526 Name Value 1527 ============================== ===== 1528 *reserved* 0-31 1529 ``NT_AMDGPU_METADATA`` 32 1530 ============================== ===== 1531 1532``NT_AMDGPU_METADATA`` 1533 Specifies extensible metadata associated with an AMDGPU code object. It is 1534 encoded as a map in the Message Pack [MsgPack]_ binary data format. See 1535 :ref:`amdgpu-amdhsa-code-object-metadata-v3`, 1536 :ref:`amdgpu-amdhsa-code-object-metadata-v4` and 1537 :ref:`amdgpu-amdhsa-code-object-metadata-v5` for the map keys defined for the 1538 ``amdhsa`` OS. 1539 1540.. _amdgpu-symbols: 1541 1542Symbols 1543------- 1544 1545Symbols include the following: 1546 1547 .. table:: AMDGPU ELF Symbols 1548 :name: amdgpu-elf-symbols-table 1549 1550 ===================== ================== ================ ================== 1551 Name Type Section Description 1552 ===================== ================== ================ ================== 1553 *link-name* ``STT_OBJECT`` - ``.data`` Global variable 1554 - ``.rodata`` 1555 - ``.bss`` 1556 *link-name*\ ``.kd`` ``STT_OBJECT`` - ``.rodata`` Kernel descriptor 1557 *link-name* ``STT_FUNC`` - ``.text`` Kernel entry point 1558 *link-name* ``STT_OBJECT`` - SHN_AMDGPU_LDS Global variable in LDS 1559 ===================== ================== ================ ================== 1560 1561Global variable 1562 Global variables both used and defined by the compilation unit. 1563 1564 If the symbol is defined in the compilation unit then it is allocated in the 1565 appropriate section according to if it has initialized data or is readonly. 1566 1567 If the symbol is external then its section is ``STN_UNDEF`` and the loader 1568 will resolve relocations using the definition provided by another code object 1569 or explicitly defined by the runtime. 1570 1571 If the symbol resides in local/group memory (LDS) then its section is the 1572 special processor specific section name ``SHN_AMDGPU_LDS``, and the 1573 ``st_value`` field describes alignment requirements as it does for common 1574 symbols. 1575 1576 .. TODO:: 1577 1578 Add description of linked shared object symbols. Seems undefined symbols 1579 are marked as STT_NOTYPE. 1580 1581Kernel descriptor 1582 Every HSA kernel has an associated kernel descriptor. It is the address of the 1583 kernel descriptor that is used in the AQL dispatch packet used to invoke the 1584 kernel, not the kernel entry point. The layout of the HSA kernel descriptor is 1585 defined in :ref:`amdgpu-amdhsa-kernel-descriptor`. 1586 1587Kernel entry point 1588 Every HSA kernel also has a symbol for its machine code entry point. 1589 1590.. _amdgpu-relocation-records: 1591 1592Relocation Records 1593------------------ 1594 1595AMDGPU backend generates ``Elf64_Rela`` relocation records. Supported 1596relocatable fields are: 1597 1598``word32`` 1599 This specifies a 32-bit field occupying 4 bytes with arbitrary byte 1600 alignment. These values use the same byte order as other word values in the 1601 AMDGPU architecture. 1602 1603``word64`` 1604 This specifies a 64-bit field occupying 8 bytes with arbitrary byte 1605 alignment. These values use the same byte order as other word values in the 1606 AMDGPU architecture. 1607 1608Following notations are used for specifying relocation calculations: 1609 1610**A** 1611 Represents the addend used to compute the value of the relocatable field. 1612 1613**G** 1614 Represents the offset into the global offset table at which the relocation 1615 entry's symbol will reside during execution. 1616 1617**GOT** 1618 Represents the address of the global offset table. 1619 1620**P** 1621 Represents the place (section offset for ``et_rel`` or address for ``et_dyn``) 1622 of the storage unit being relocated (computed using ``r_offset``). 1623 1624**S** 1625 Represents the value of the symbol whose index resides in the relocation 1626 entry. Relocations not using this must specify a symbol index of 1627 ``STN_UNDEF``. 1628 1629**B** 1630 Represents the base address of a loaded executable or shared object which is 1631 the difference between the ELF address and the actual load address. 1632 Relocations using this are only valid in executable or shared objects. 1633 1634The following relocation types are supported: 1635 1636 .. table:: AMDGPU ELF Relocation Records 1637 :name: amdgpu-elf-relocation-records-table 1638 1639 ========================== ======= ===== ========== ============================== 1640 Relocation Type Kind Value Field Calculation 1641 ========================== ======= ===== ========== ============================== 1642 ``R_AMDGPU_NONE`` 0 *none* *none* 1643 ``R_AMDGPU_ABS32_LO`` Static, 1 ``word32`` (S + A) & 0xFFFFFFFF 1644 Dynamic 1645 ``R_AMDGPU_ABS32_HI`` Static, 2 ``word32`` (S + A) >> 32 1646 Dynamic 1647 ``R_AMDGPU_ABS64`` Static, 3 ``word64`` S + A 1648 Dynamic 1649 ``R_AMDGPU_REL32`` Static 4 ``word32`` S + A - P 1650 ``R_AMDGPU_REL64`` Static 5 ``word64`` S + A - P 1651 ``R_AMDGPU_ABS32`` Static, 6 ``word32`` S + A 1652 Dynamic 1653 ``R_AMDGPU_GOTPCREL`` Static 7 ``word32`` G + GOT + A - P 1654 ``R_AMDGPU_GOTPCREL32_LO`` Static 8 ``word32`` (G + GOT + A - P) & 0xFFFFFFFF 1655 ``R_AMDGPU_GOTPCREL32_HI`` Static 9 ``word32`` (G + GOT + A - P) >> 32 1656 ``R_AMDGPU_REL32_LO`` Static 10 ``word32`` (S + A - P) & 0xFFFFFFFF 1657 ``R_AMDGPU_REL32_HI`` Static 11 ``word32`` (S + A - P) >> 32 1658 *reserved* 12 1659 ``R_AMDGPU_RELATIVE64`` Dynamic 13 ``word64`` B + A 1660 ``R_AMDGPU_REL16`` Static 14 ``word16`` ((S + A - P) - 4) / 4 1661 ========================== ======= ===== ========== ============================== 1662 1663``R_AMDGPU_ABS32_LO`` and ``R_AMDGPU_ABS32_HI`` are only supported by 1664the ``mesa3d`` OS, which does not support ``R_AMDGPU_ABS64``. 1665 1666There is no current OS loader support for 32-bit programs and so 1667``R_AMDGPU_ABS32`` is not used. 1668 1669.. _amdgpu-loaded-code-object-path-uniform-resource-identifier: 1670 1671Loaded Code Object Path Uniform Resource Identifier (URI) 1672--------------------------------------------------------- 1673 1674The AMD GPU code object loader represents the path of the ELF shared object from 1675which the code object was loaded as a textual Uniform Resource Identifier (URI). 1676Note that the code object is the in memory loaded relocated form of the ELF 1677shared object. Multiple code objects may be loaded at different memory 1678addresses in the same process from the same ELF shared object. 1679 1680The loaded code object path URI syntax is defined by the following BNF syntax: 1681 1682.. code:: 1683 1684 code_object_uri ::== file_uri | memory_uri 1685 file_uri ::== "file://" file_path [ range_specifier ] 1686 memory_uri ::== "memory://" process_id range_specifier 1687 range_specifier ::== [ "#" | "?" ] "offset=" number "&" "size=" number 1688 file_path ::== URI_ENCODED_OS_FILE_PATH 1689 process_id ::== DECIMAL_NUMBER 1690 number ::== HEX_NUMBER | DECIMAL_NUMBER | OCTAL_NUMBER 1691 1692**number** 1693 Is a C integral literal where hexadecimal values are prefixed by "0x" or "0X", 1694 and octal values by "0". 1695 1696**file_path** 1697 Is the file's path specified as a URI encoded UTF-8 string. In URI encoding, 1698 every character that is not in the regular expression ``[a-zA-Z0-9/_.~-]`` is 1699 encoded as two uppercase hexadecimal digits proceeded by "%". Directories in 1700 the path are separated by "/". 1701 1702**offset** 1703 Is a 0-based byte offset to the start of the code object. For a file URI, it 1704 is from the start of the file specified by the ``file_path``, and if omitted 1705 defaults to 0. For a memory URI, it is the memory address and is required. 1706 1707**size** 1708 Is the number of bytes in the code object. For a file URI, if omitted it 1709 defaults to the size of the file. It is required for a memory URI. 1710 1711**process_id** 1712 Is the identity of the process owning the memory. For Linux it is the C 1713 unsigned integral decimal literal for the process ID (PID). 1714 1715For example: 1716 1717.. code:: 1718 1719 file:///dir1/dir2/file1 1720 file:///dir3/dir4/file2#offset=0x2000&size=3000 1721 memory://1234#offset=0x20000&size=3000 1722 1723.. _amdgpu-dwarf-debug-information: 1724 1725DWARF Debug Information 1726======================= 1727 1728.. warning:: 1729 1730 This section describes **provisional support** for AMDGPU DWARF [DWARF]_ that 1731 is not currently fully implemented and is subject to change. 1732 1733AMDGPU generates DWARF [DWARF]_ debugging information ELF sections (see 1734:ref:`amdgpu-elf-code-object`) which contain information that maps the code 1735object executable code and data to the source language constructs. It can be 1736used by tools such as debuggers and profilers. It uses features defined in 1737:doc:`AMDGPUDwarfExtensionsForHeterogeneousDebugging` that are made available in 1738DWARF Version 4 and DWARF Version 5 as an LLVM vendor extension. 1739 1740This section defines the AMDGPU target architecture specific DWARF mappings. 1741 1742.. _amdgpu-dwarf-register-identifier: 1743 1744Register Identifier 1745------------------- 1746 1747This section defines the AMDGPU target architecture register numbers used in 1748DWARF operation expressions (see DWARF Version 5 section 2.5 and 1749:ref:`amdgpu-dwarf-operation-expressions`) and Call Frame Information 1750instructions (see DWARF Version 5 section 6.4 and 1751:ref:`amdgpu-dwarf-call-frame-information`). 1752 1753A single code object can contain code for kernels that have different wavefront 1754sizes. The vector registers and some scalar registers are based on the wavefront 1755size. AMDGPU defines distinct DWARF registers for each wavefront size. This 1756simplifies the consumer of the DWARF so that each register has a fixed size, 1757rather than being dynamic according to the wavefront size mode. Similarly, 1758distinct DWARF registers are defined for those registers that vary in size 1759according to the process address size. This allows a consumer to treat a 1760specific AMDGPU processor as a single architecture regardless of how it is 1761configured at run time. The compiler explicitly specifies the DWARF registers 1762that match the mode in which the code it is generating will be executed. 1763 1764DWARF registers are encoded as numbers, which are mapped to architecture 1765registers. The mapping for AMDGPU is defined in 1766:ref:`amdgpu-dwarf-register-mapping-table`. All AMDGPU targets use the same 1767mapping. 1768 1769.. table:: AMDGPU DWARF Register Mapping 1770 :name: amdgpu-dwarf-register-mapping-table 1771 1772 ============== ================= ======== ================================== 1773 DWARF Register AMDGPU Register Bit Size Description 1774 ============== ================= ======== ================================== 1775 0 PC_32 32 Program Counter (PC) when 1776 executing in a 32-bit process 1777 address space. Used in the CFI to 1778 describe the PC of the calling 1779 frame. 1780 1 EXEC_MASK_32 32 Execution Mask Register when 1781 executing in wavefront 32 mode. 1782 2-15 *Reserved* *Reserved for highly accessed 1783 registers using DWARF shortcut.* 1784 16 PC_64 64 Program Counter (PC) when 1785 executing in a 64-bit process 1786 address space. Used in the CFI to 1787 describe the PC of the calling 1788 frame. 1789 17 EXEC_MASK_64 64 Execution Mask Register when 1790 executing in wavefront 64 mode. 1791 18-31 *Reserved* *Reserved for highly accessed 1792 registers using DWARF shortcut.* 1793 32-95 SGPR0-SGPR63 32 Scalar General Purpose 1794 Registers. 1795 96-127 *Reserved* *Reserved for frequently accessed 1796 registers using DWARF 1-byte ULEB.* 1797 128 STATUS 32 Status Register. 1798 129-511 *Reserved* *Reserved for future Scalar 1799 Architectural Registers.* 1800 512 VCC_32 32 Vector Condition Code Register 1801 when executing in wavefront 32 1802 mode. 1803 513-767 *Reserved* *Reserved for future Vector 1804 Architectural Registers when 1805 executing in wavefront 32 mode.* 1806 768 VCC_64 64 Vector Condition Code Register 1807 when executing in wavefront 64 1808 mode. 1809 769-1023 *Reserved* *Reserved for future Vector 1810 Architectural Registers when 1811 executing in wavefront 64 mode.* 1812 1024-1087 *Reserved* *Reserved for padding.* 1813 1088-1129 SGPR64-SGPR105 32 Scalar General Purpose Registers. 1814 1130-1535 *Reserved* *Reserved for future Scalar 1815 General Purpose Registers.* 1816 1536-1791 VGPR0-VGPR255 32*32 Vector General Purpose Registers 1817 when executing in wavefront 32 1818 mode. 1819 1792-2047 *Reserved* *Reserved for future Vector 1820 General Purpose Registers when 1821 executing in wavefront 32 mode.* 1822 2048-2303 AGPR0-AGPR255 32*32 Vector Accumulation Registers 1823 when executing in wavefront 32 1824 mode. 1825 2304-2559 *Reserved* *Reserved for future Vector 1826 Accumulation Registers when 1827 executing in wavefront 32 mode.* 1828 2560-2815 VGPR0-VGPR255 64*32 Vector General Purpose Registers 1829 when executing in wavefront 64 1830 mode. 1831 2816-3071 *Reserved* *Reserved for future Vector 1832 General Purpose Registers when 1833 executing in wavefront 64 mode.* 1834 3072-3327 AGPR0-AGPR255 64*32 Vector Accumulation Registers 1835 when executing in wavefront 64 1836 mode. 1837 3328-3583 *Reserved* *Reserved for future Vector 1838 Accumulation Registers when 1839 executing in wavefront 64 mode.* 1840 ============== ================= ======== ================================== 1841 1842The vector registers are represented as the full size for the wavefront. They 1843are organized as consecutive dwords (32-bits), one per lane, with the dword at 1844the least significant bit position corresponding to lane 0 and so forth. DWARF 1845location expressions involving the ``DW_OP_LLVM_offset`` and 1846``DW_OP_LLVM_push_lane`` operations are used to select the part of the vector 1847register corresponding to the lane that is executing the current thread of 1848execution in languages that are implemented using a SIMD or SIMT execution 1849model. 1850 1851If the wavefront size is 32 lanes then the wavefront 32 mode register 1852definitions are used. If the wavefront size is 64 lanes then the wavefront 64 1853mode register definitions are used. Some AMDGPU targets support executing in 1854both wavefront 32 and wavefront 64 mode. The register definitions corresponding 1855to the wavefront mode of the generated code will be used. 1856 1857If code is generated to execute in a 32-bit process address space, then the 185832-bit process address space register definitions are used. If code is generated 1859to execute in a 64-bit process address space, then the 64-bit process address 1860space register definitions are used. The ``amdgcn`` target only supports the 186164-bit process address space. 1862 1863.. _amdgpu-dwarf-address-class-identifier: 1864 1865Address Class Identifier 1866------------------------ 1867 1868The DWARF address class represents the source language memory space. See DWARF 1869Version 5 section 2.12 which is updated by the *DWARF Extensions For 1870Heterogeneous Debugging* section :ref:`amdgpu-dwarf-segment_addresses`. 1871 1872The DWARF address class mapping used for AMDGPU is defined in 1873:ref:`amdgpu-dwarf-address-class-mapping-table`. 1874 1875.. table:: AMDGPU DWARF Address Class Mapping 1876 :name: amdgpu-dwarf-address-class-mapping-table 1877 1878 ========================= ====== ================= 1879 DWARF AMDGPU 1880 -------------------------------- ----------------- 1881 Address Class Name Value Address Space 1882 ========================= ====== ================= 1883 ``DW_ADDR_none`` 0x0000 Generic (Flat) 1884 ``DW_ADDR_LLVM_global`` 0x0001 Global 1885 ``DW_ADDR_LLVM_constant`` 0x0002 Global 1886 ``DW_ADDR_LLVM_group`` 0x0003 Local (group/LDS) 1887 ``DW_ADDR_LLVM_private`` 0x0004 Private (Scratch) 1888 ``DW_ADDR_AMDGPU_region`` 0x8000 Region (GDS) 1889 ========================= ====== ================= 1890 1891The DWARF address class values defined in the *DWARF Extensions For 1892Heterogeneous Debugging* section :ref:`amdgpu-dwarf-segment_addresses` are used. 1893 1894In addition, ``DW_ADDR_AMDGPU_region`` is encoded as a vendor extension. This is 1895available for use for the AMD extension for access to the hardware GDS memory 1896which is scratchpad memory allocated per device. 1897 1898For AMDGPU if no ``DW_AT_address_class`` attribute is present, then the default 1899address class of ``DW_ADDR_none`` is used. 1900 1901See :ref:`amdgpu-dwarf-address-space-identifier` for information on the AMDGPU 1902mapping of DWARF address classes to DWARF address spaces, including address size 1903and NULL value. 1904 1905.. _amdgpu-dwarf-address-space-identifier: 1906 1907Address Space Identifier 1908------------------------ 1909 1910DWARF address spaces correspond to target architecture specific linear 1911addressable memory areas. See DWARF Version 5 section 2.12 and *DWARF Extensions 1912For Heterogeneous Debugging* section :ref:`amdgpu-dwarf-segment_addresses`. 1913 1914The DWARF address space mapping used for AMDGPU is defined in 1915:ref:`amdgpu-dwarf-address-space-mapping-table`. 1916 1917.. table:: AMDGPU DWARF Address Space Mapping 1918 :name: amdgpu-dwarf-address-space-mapping-table 1919 1920 ======================================= ===== ======= ======== ================= ======================= 1921 DWARF AMDGPU Notes 1922 --------------------------------------- ----- ---------------- ----------------- ----------------------- 1923 Address Space Name Value Address Bit Size Address Space 1924 --------------------------------------- ----- ------- -------- ----------------- ----------------------- 1925 .. 64-bit 32-bit 1926 process process 1927 address address 1928 space space 1929 ======================================= ===== ======= ======== ================= ======================= 1930 ``DW_ASPACE_none`` 0x00 64 32 Global *default address space* 1931 ``DW_ASPACE_AMDGPU_generic`` 0x01 64 32 Generic (Flat) 1932 ``DW_ASPACE_AMDGPU_region`` 0x02 32 32 Region (GDS) 1933 ``DW_ASPACE_AMDGPU_local`` 0x03 32 32 Local (group/LDS) 1934 *Reserved* 0x04 1935 ``DW_ASPACE_AMDGPU_private_lane`` 0x05 32 32 Private (Scratch) *focused lane* 1936 ``DW_ASPACE_AMDGPU_private_wave`` 0x06 32 32 Private (Scratch) *unswizzled wavefront* 1937 ======================================= ===== ======= ======== ================= ======================= 1938 1939See :ref:`amdgpu-address-spaces` for information on the AMDGPU address spaces 1940including address size and NULL value. 1941 1942The ``DW_ASPACE_none`` address space is the default target architecture address 1943space used in DWARF operations that do not specify an address space. It 1944therefore has to map to the global address space so that the ``DW_OP_addr*`` and 1945related operations can refer to addresses in the program code. 1946 1947The ``DW_ASPACE_AMDGPU_generic`` address space allows location expressions to 1948specify the flat address space. If the address corresponds to an address in the 1949local address space, then it corresponds to the wavefront that is executing the 1950focused thread of execution. If the address corresponds to an address in the 1951private address space, then it corresponds to the lane that is executing the 1952focused thread of execution for languages that are implemented using a SIMD or 1953SIMT execution model. 1954 1955.. note:: 1956 1957 CUDA-like languages such as HIP that do not have address spaces in the 1958 language type system, but do allow variables to be allocated in different 1959 address spaces, need to explicitly specify the ``DW_ASPACE_AMDGPU_generic`` 1960 address space in the DWARF expression operations as the default address space 1961 is the global address space. 1962 1963The ``DW_ASPACE_AMDGPU_local`` address space allows location expressions to 1964specify the local address space corresponding to the wavefront that is executing 1965the focused thread of execution. 1966 1967The ``DW_ASPACE_AMDGPU_private_lane`` address space allows location expressions 1968to specify the private address space corresponding to the lane that is executing 1969the focused thread of execution for languages that are implemented using a SIMD 1970or SIMT execution model. 1971 1972The ``DW_ASPACE_AMDGPU_private_wave`` address space allows location expressions 1973to specify the unswizzled private address space corresponding to the wavefront 1974that is executing the focused thread of execution. The wavefront view of private 1975memory is the per wavefront unswizzled backing memory layout defined in 1976:ref:`amdgpu-address-spaces`, such that address 0 corresponds to the first 1977location for the backing memory of the wavefront (namely the address is not 1978offset by ``wavefront-scratch-base``). The following formula can be used to 1979convert from a ``DW_ASPACE_AMDGPU_private_lane`` address to a 1980``DW_ASPACE_AMDGPU_private_wave`` address: 1981 1982:: 1983 1984 private-address-wavefront = 1985 ((private-address-lane / 4) * wavefront-size * 4) + 1986 (wavefront-lane-id * 4) + (private-address-lane % 4) 1987 1988If the ``DW_ASPACE_AMDGPU_private_lane`` address is dword aligned, and the start 1989of the dwords for each lane starting with lane 0 is required, then this 1990simplifies to: 1991 1992:: 1993 1994 private-address-wavefront = 1995 private-address-lane * wavefront-size 1996 1997A compiler can use the ``DW_ASPACE_AMDGPU_private_wave`` address space to read a 1998complete spilled vector register back into a complete vector register in the 1999CFI. The frame pointer can be a private lane address which is dword aligned, 2000which can be shifted to multiply by the wavefront size, and then used to form a 2001private wavefront address that gives a location for a contiguous set of dwords, 2002one per lane, where the vector register dwords are spilled. The compiler knows 2003the wavefront size since it generates the code. Note that the type of the 2004address may have to be converted as the size of a 2005``DW_ASPACE_AMDGPU_private_lane`` address may be smaller than the size of a 2006``DW_ASPACE_AMDGPU_private_wave`` address. 2007 2008.. _amdgpu-dwarf-lane-identifier: 2009 2010Lane identifier 2011--------------- 2012 2013DWARF lane identifies specify a target architecture lane position for hardware 2014that executes in a SIMD or SIMT manner, and on which a source language maps its 2015threads of execution onto those lanes. The DWARF lane identifier is pushed by 2016the ``DW_OP_LLVM_push_lane`` DWARF expression operation. See DWARF Version 5 2017section 2.5 which is updated by *DWARF Extensions For Heterogeneous Debugging* 2018section :ref:`amdgpu-dwarf-operation-expressions`. 2019 2020For AMDGPU, the lane identifier corresponds to the hardware lane ID of a 2021wavefront. It is numbered from 0 to the wavefront size minus 1. 2022 2023Operation Expressions 2024--------------------- 2025 2026DWARF expressions are used to compute program values and the locations of 2027program objects. See DWARF Version 5 section 2.5 and 2028:ref:`amdgpu-dwarf-operation-expressions`. 2029 2030DWARF location descriptions describe how to access storage which includes memory 2031and registers. When accessing storage on AMDGPU, bytes are ordered with least 2032significant bytes first, and bits are ordered within bytes with least 2033significant bits first. 2034 2035For AMDGPU CFI expressions, ``DW_OP_LLVM_select_bit_piece`` is used to describe 2036unwinding vector registers that are spilled under the execution mask to memory: 2037the zero-single location description is the vector register, and the one-single 2038location description is the spilled memory location description. The 2039``DW_OP_LLVM_form_aspace_address`` is used to specify the address space of the 2040memory location description. 2041 2042In AMDGPU expressions, ``DW_OP_LLVM_select_bit_piece`` is used by the 2043``DW_AT_LLVM_lane_pc`` attribute expression where divergent control flow is 2044controlled by the execution mask. An undefined location description together 2045with ``DW_OP_LLVM_extend`` is used to indicate the lane was not active on entry 2046to the subprogram. See :ref:`amdgpu-dwarf-dw-at-llvm-lane-pc` for an example. 2047 2048Debugger Information Entry Attributes 2049------------------------------------- 2050 2051This section describes how certain debugger information entry attributes are 2052used by AMDGPU. See the sections in DWARF Version 5 section 3.3.5 and 3.1.1 2053which are updated by *DWARF Extensions For Heterogeneous Debugging* section 2054:ref:`amdgpu-dwarf-low-level-information` and 2055:ref:`amdgpu-dwarf-full-and-partial-compilation-unit-entries`. 2056 2057.. _amdgpu-dwarf-dw-at-llvm-lane-pc: 2058 2059``DW_AT_LLVM_lane_pc`` 2060~~~~~~~~~~~~~~~~~~~~~~ 2061 2062For AMDGPU, the ``DW_AT_LLVM_lane_pc`` attribute is used to specify the program 2063location of the separate lanes of a SIMT thread. 2064 2065If the lane is an active lane then this will be the same as the current program 2066location. 2067 2068If the lane is inactive, but was active on entry to the subprogram, then this is 2069the program location in the subprogram at which execution of the lane is 2070conceptual positioned. 2071 2072If the lane was not active on entry to the subprogram, then this will be the 2073undefined location. A client debugger can check if the lane is part of a valid 2074work-group by checking that the lane is in the range of the associated 2075work-group within the grid, accounting for partial work-groups. If it is not, 2076then the debugger can omit any information for the lane. Otherwise, the debugger 2077may repeatedly unwind the stack and inspect the ``DW_AT_LLVM_lane_pc`` of the 2078calling subprogram until it finds a non-undefined location. Conceptually the 2079lane only has the call frames that it has a non-undefined 2080``DW_AT_LLVM_lane_pc``. 2081 2082The following example illustrates how the AMDGPU backend can generate a DWARF 2083location list expression for the nested ``IF/THEN/ELSE`` structures of the 2084following subprogram pseudo code for a target with 64 lanes per wavefront. 2085 2086.. code:: 2087 :number-lines: 2088 2089 SUBPROGRAM X 2090 BEGIN 2091 a; 2092 IF (c1) THEN 2093 b; 2094 IF (c2) THEN 2095 c; 2096 ELSE 2097 d; 2098 ENDIF 2099 e; 2100 ELSE 2101 f; 2102 ENDIF 2103 g; 2104 END 2105 2106The AMDGPU backend may generate the following pseudo LLVM MIR to manipulate the 2107execution mask (``EXEC``) to linearize the control flow. The condition is 2108evaluated to make a mask of the lanes for which the condition evaluates to true. 2109First the ``THEN`` region is executed by setting the ``EXEC`` mask to the 2110logical ``AND`` of the current ``EXEC`` mask with the condition mask. Then the 2111``ELSE`` region is executed by negating the ``EXEC`` mask and logical ``AND`` of 2112the saved ``EXEC`` mask at the start of the region. After the ``IF/THEN/ELSE`` 2113region the ``EXEC`` mask is restored to the value it had at the beginning of the 2114region. This is shown below. Other approaches are possible, but the basic 2115concept is the same. 2116 2117.. code:: 2118 :number-lines: 2119 2120 $lex_start: 2121 a; 2122 %1 = EXEC 2123 %2 = c1 2124 $lex_1_start: 2125 EXEC = %1 & %2 2126 $if_1_then: 2127 b; 2128 %3 = EXEC 2129 %4 = c2 2130 $lex_1_1_start: 2131 EXEC = %3 & %4 2132 $lex_1_1_then: 2133 c; 2134 EXEC = ~EXEC & %3 2135 $lex_1_1_else: 2136 d; 2137 EXEC = %3 2138 $lex_1_1_end: 2139 e; 2140 EXEC = ~EXEC & %1 2141 $lex_1_else: 2142 f; 2143 EXEC = %1 2144 $lex_1_end: 2145 g; 2146 $lex_end: 2147 2148To create the DWARF location list expression that defines the location 2149description of a vector of lane program locations, the LLVM MIR ``DBG_VALUE`` 2150pseudo instruction can be used to annotate the linearized control flow. This can 2151be done by defining an artificial variable for the lane PC. The DWARF location 2152list expression created for it is used as the value of the 2153``DW_AT_LLVM_lane_pc`` attribute on the subprogram's debugger information entry. 2154 2155A DWARF procedure is defined for each well nested structured control flow region 2156which provides the conceptual lane program location for a lane if it is not 2157active (namely it is divergent). The DWARF operation expression for each region 2158conceptually inherits the value of the immediately enclosing region and modifies 2159it according to the semantics of the region. 2160 2161For an ``IF/THEN/ELSE`` region the divergent program location is at the start of 2162the region for the ``THEN`` region since it is executed first. For the ``ELSE`` 2163region the divergent program location is at the end of the ``IF/THEN/ELSE`` 2164region since the ``THEN`` region has completed. 2165 2166The lane PC artificial variable is assigned at each region transition. It uses 2167the immediately enclosing region's DWARF procedure to compute the program 2168location for each lane assuming they are divergent, and then modifies the result 2169by inserting the current program location for each lane that the ``EXEC`` mask 2170indicates is active. 2171 2172By having separate DWARF procedures for each region, they can be reused to 2173define the value for any nested region. This reduces the total size of the DWARF 2174operation expressions. 2175 2176The following provides an example using pseudo LLVM MIR. 2177 2178.. code:: 2179 :number-lines: 2180 2181 $lex_start: 2182 DEFINE_DWARF %__uint_64 = DW_TAG_base_type[ 2183 DW_AT_name = "__uint64"; 2184 DW_AT_byte_size = 8; 2185 DW_AT_encoding = DW_ATE_unsigned; 2186 ]; 2187 DEFINE_DWARF %__active_lane_pc = DW_TAG_dwarf_procedure[ 2188 DW_AT_name = "__active_lane_pc"; 2189 DW_AT_location = [ 2190 DW_OP_regx PC; 2191 DW_OP_LLVM_extend 64, 64; 2192 DW_OP_regval_type EXEC, %uint_64; 2193 DW_OP_LLVM_select_bit_piece 64, 64; 2194 ]; 2195 ]; 2196 DEFINE_DWARF %__divergent_lane_pc = DW_TAG_dwarf_procedure[ 2197 DW_AT_name = "__divergent_lane_pc"; 2198 DW_AT_location = [ 2199 DW_OP_LLVM_undefined; 2200 DW_OP_LLVM_extend 64, 64; 2201 ]; 2202 ]; 2203 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 2204 DW_OP_call_ref %__divergent_lane_pc; 2205 DW_OP_call_ref %__active_lane_pc; 2206 ]; 2207 a; 2208 %1 = EXEC; 2209 DBG_VALUE %1, $noreg, %__lex_1_save_exec; 2210 %2 = c1; 2211 $lex_1_start: 2212 EXEC = %1 & %2; 2213 $lex_1_then: 2214 DEFINE_DWARF %__divergent_lane_pc_1_then = DW_TAG_dwarf_procedure[ 2215 DW_AT_name = "__divergent_lane_pc_1_then"; 2216 DW_AT_location = DIExpression[ 2217 DW_OP_call_ref %__divergent_lane_pc; 2218 DW_OP_addrx &lex_1_start; 2219 DW_OP_stack_value; 2220 DW_OP_LLVM_extend 64, 64; 2221 DW_OP_call_ref %__lex_1_save_exec; 2222 DW_OP_deref_type 64, %__uint_64; 2223 DW_OP_LLVM_select_bit_piece 64, 64; 2224 ]; 2225 ]; 2226 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 2227 DW_OP_call_ref %__divergent_lane_pc_1_then; 2228 DW_OP_call_ref %__active_lane_pc; 2229 ]; 2230 b; 2231 %3 = EXEC; 2232 DBG_VALUE %3, %__lex_1_1_save_exec; 2233 %4 = c2; 2234 $lex_1_1_start: 2235 EXEC = %3 & %4; 2236 $lex_1_1_then: 2237 DEFINE_DWARF %__divergent_lane_pc_1_1_then = DW_TAG_dwarf_procedure[ 2238 DW_AT_name = "__divergent_lane_pc_1_1_then"; 2239 DW_AT_location = DIExpression[ 2240 DW_OP_call_ref %__divergent_lane_pc_1_then; 2241 DW_OP_addrx &lex_1_1_start; 2242 DW_OP_stack_value; 2243 DW_OP_LLVM_extend 64, 64; 2244 DW_OP_call_ref %__lex_1_1_save_exec; 2245 DW_OP_deref_type 64, %__uint_64; 2246 DW_OP_LLVM_select_bit_piece 64, 64; 2247 ]; 2248 ]; 2249 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 2250 DW_OP_call_ref %__divergent_lane_pc_1_1_then; 2251 DW_OP_call_ref %__active_lane_pc; 2252 ]; 2253 c; 2254 EXEC = ~EXEC & %3; 2255 $lex_1_1_else: 2256 DEFINE_DWARF %__divergent_lane_pc_1_1_else = DW_TAG_dwarf_procedure[ 2257 DW_AT_name = "__divergent_lane_pc_1_1_else"; 2258 DW_AT_location = DIExpression[ 2259 DW_OP_call_ref %__divergent_lane_pc_1_then; 2260 DW_OP_addrx &lex_1_1_end; 2261 DW_OP_stack_value; 2262 DW_OP_LLVM_extend 64, 64; 2263 DW_OP_call_ref %__lex_1_1_save_exec; 2264 DW_OP_deref_type 64, %__uint_64; 2265 DW_OP_LLVM_select_bit_piece 64, 64; 2266 ]; 2267 ]; 2268 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 2269 DW_OP_call_ref %__divergent_lane_pc_1_1_else; 2270 DW_OP_call_ref %__active_lane_pc; 2271 ]; 2272 d; 2273 EXEC = %3; 2274 $lex_1_1_end: 2275 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 2276 DW_OP_call_ref %__divergent_lane_pc; 2277 DW_OP_call_ref %__active_lane_pc; 2278 ]; 2279 e; 2280 EXEC = ~EXEC & %1; 2281 $lex_1_else: 2282 DEFINE_DWARF %__divergent_lane_pc_1_else = DW_TAG_dwarf_procedure[ 2283 DW_AT_name = "__divergent_lane_pc_1_else"; 2284 DW_AT_location = DIExpression[ 2285 DW_OP_call_ref %__divergent_lane_pc; 2286 DW_OP_addrx &lex_1_end; 2287 DW_OP_stack_value; 2288 DW_OP_LLVM_extend 64, 64; 2289 DW_OP_call_ref %__lex_1_save_exec; 2290 DW_OP_deref_type 64, %__uint_64; 2291 DW_OP_LLVM_select_bit_piece 64, 64; 2292 ]; 2293 ]; 2294 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 2295 DW_OP_call_ref %__divergent_lane_pc_1_else; 2296 DW_OP_call_ref %__active_lane_pc; 2297 ]; 2298 f; 2299 EXEC = %1; 2300 $lex_1_end: 2301 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc DIExpression[ 2302 DW_OP_call_ref %__divergent_lane_pc; 2303 DW_OP_call_ref %__active_lane_pc; 2304 ]; 2305 g; 2306 $lex_end: 2307 2308The DWARF procedure ``%__active_lane_pc`` is used to update the lane pc elements 2309that are active, with the current program location. 2310 2311Artificial variables %__lex_1_save_exec and %__lex_1_1_save_exec are created for 2312the execution masks saved on entry to a region. Using the ``DBG_VALUE`` pseudo 2313instruction, location list entries will be created that describe where the 2314artificial variables are allocated at any given program location. The compiler 2315may allocate them to registers or spill them to memory. 2316 2317The DWARF procedures for each region use the values of the saved execution mask 2318artificial variables to only update the lanes that are active on entry to the 2319region. All other lanes retain the value of the enclosing region where they were 2320last active. If they were not active on entry to the subprogram, then will have 2321the undefined location description. 2322 2323Other structured control flow regions can be handled similarly. For example, 2324loops would set the divergent program location for the region at the end of the 2325loop. Any lanes active will be in the loop, and any lanes not active must have 2326exited the loop. 2327 2328An ``IF/THEN/ELSEIF/ELSEIF/...`` region can be treated as a nest of 2329``IF/THEN/ELSE`` regions. 2330 2331The DWARF procedures can use the active lane artificial variable described in 2332:ref:`amdgpu-dwarf-amdgpu-dw-at-llvm-active-lane` rather than the actual 2333``EXEC`` mask in order to support whole or quad wavefront mode. 2334 2335.. _amdgpu-dwarf-amdgpu-dw-at-llvm-active-lane: 2336 2337``DW_AT_LLVM_active_lane`` 2338~~~~~~~~~~~~~~~~~~~~~~~~~~ 2339 2340The ``DW_AT_LLVM_active_lane`` attribute on a subprogram debugger information 2341entry is used to specify the lanes that are conceptually active for a SIMT 2342thread. 2343 2344The execution mask may be modified to implement whole or quad wavefront mode 2345operations. For example, all lanes may need to temporarily be made active to 2346execute a whole wavefront operation. Such regions would save the ``EXEC`` mask, 2347update it to enable the necessary lanes, perform the operations, and then 2348restore the ``EXEC`` mask from the saved value. While executing the whole 2349wavefront region, the conceptual execution mask is the saved value, not the 2350``EXEC`` value. 2351 2352This is handled by defining an artificial variable for the active lane mask. The 2353active lane mask artificial variable would be the actual ``EXEC`` mask for 2354normal regions, and the saved execution mask for regions where the mask is 2355temporarily updated. The location list expression created for this artificial 2356variable is used to define the value of the ``DW_AT_LLVM_active_lane`` 2357attribute. 2358 2359``DW_AT_LLVM_augmentation`` 2360~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2361 2362For AMDGPU, the ``DW_AT_LLVM_augmentation`` attribute of a compilation unit 2363debugger information entry has the following value for the augmentation string: 2364 2365:: 2366 2367 [amdgpu:v0.0] 2368 2369The "vX.Y" specifies the major X and minor Y version number of the AMDGPU 2370extensions used in the DWARF of the compilation unit. The version number 2371conforms to [SEMVER]_. 2372 2373Call Frame Information 2374---------------------- 2375 2376DWARF Call Frame Information (CFI) describes how a consumer can virtually 2377*unwind* call frames in a running process or core dump. See DWARF Version 5 2378section 6.4 and :ref:`amdgpu-dwarf-call-frame-information`. 2379 2380For AMDGPU, the Common Information Entry (CIE) fields have the following values: 2381 23821. ``augmentation`` string contains the following null-terminated UTF-8 string: 2383 2384 :: 2385 2386 [amd:v0.0] 2387 2388 The ``vX.Y`` specifies the major X and minor Y version number of the AMDGPU 2389 extensions used in this CIE or to the FDEs that use it. The version number 2390 conforms to [SEMVER]_. 2391 23922. ``address_size`` for the ``Global`` address space is defined in 2393 :ref:`amdgpu-dwarf-address-space-identifier`. 2394 23953. ``segment_selector_size`` is 0 as AMDGPU does not use a segment selector. 2396 23974. ``code_alignment_factor`` is 4 bytes. 2398 2399 .. TODO:: 2400 2401 Add to :ref:`amdgpu-processor-table` table. 2402 24035. ``data_alignment_factor`` is 4 bytes. 2404 2405 .. TODO:: 2406 2407 Add to :ref:`amdgpu-processor-table` table. 2408 24096. ``return_address_register`` is ``PC_32`` for 32-bit processes and ``PC_64`` 2410 for 64-bit processes defined in :ref:`amdgpu-dwarf-register-identifier`. 2411 24127. ``initial_instructions`` Since a subprogram X with fewer registers can be 2413 called from subprogram Y that has more allocated, X will not change any of 2414 the extra registers as it cannot access them. Therefore, the default rule 2415 for all columns is ``same value``. 2416 2417For AMDGPU the register number follows the numbering defined in 2418:ref:`amdgpu-dwarf-register-identifier`. 2419 2420For AMDGPU the instructions are variable size. A consumer can subtract 1 from 2421the return address to get the address of a byte within the call site 2422instructions. See DWARF Version 5 section 6.4.4. 2423 2424Accelerated Access 2425------------------ 2426 2427See DWARF Version 5 section 6.1. 2428 2429Lookup By Name Section Header 2430~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2431 2432See DWARF Version 5 section 6.1.1.4.1 and :ref:`amdgpu-dwarf-lookup-by-name`. 2433 2434For AMDGPU the lookup by name section header table: 2435 2436``augmentation_string_size`` (uword) 2437 2438 Set to the length of the ``augmentation_string`` value which is always a 2439 multiple of 4. 2440 2441``augmentation_string`` (sequence of UTF-8 characters) 2442 2443 Contains the following UTF-8 string null padded to a multiple of 4 bytes: 2444 2445 :: 2446 2447 [amdgpu:v0.0] 2448 2449 The "vX.Y" specifies the major X and minor Y version number of the AMDGPU 2450 extensions used in the DWARF of this index. The version number conforms to 2451 [SEMVER]_. 2452 2453 .. note:: 2454 2455 This is different to the DWARF Version 5 definition that requires the first 2456 4 characters to be the vendor ID. But this is consistent with the other 2457 augmentation strings and does allow multiple vendor contributions. However, 2458 backwards compatibility may be more desirable. 2459 2460Lookup By Address Section Header 2461~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2462 2463See DWARF Version 5 section 6.1.2. 2464 2465For AMDGPU the lookup by address section header table: 2466 2467``address_size`` (ubyte) 2468 2469 Match the address size for the ``Global`` address space defined in 2470 :ref:`amdgpu-dwarf-address-space-identifier`. 2471 2472``segment_selector_size`` (ubyte) 2473 2474 AMDGPU does not use a segment selector so this is 0. The entries in the 2475 ``.debug_aranges`` do not have a segment selector. 2476 2477Line Number Information 2478----------------------- 2479 2480See DWARF Version 5 section 6.2 and :ref:`amdgpu-dwarf-line-number-information`. 2481 2482AMDGPU does not use the ``isa`` state machine registers and always sets it to 0. 2483The instruction set must be obtained from the ELF file header ``e_flags`` field 2484in the ``EF_AMDGPU_MACH`` bit position (see :ref:`ELF Header 2485<amdgpu-elf-header>`). See DWARF Version 5 section 6.2.2. 2486 2487.. TODO:: 2488 2489 Should the ``isa`` state machine register be used to indicate if the code is 2490 in wavefront32 or wavefront64 mode? Or used to specify the architecture ISA? 2491 2492For AMDGPU the line number program header fields have the following values (see 2493DWARF Version 5 section 6.2.4): 2494 2495``address_size`` (ubyte) 2496 Matches the address size for the ``Global`` address space defined in 2497 :ref:`amdgpu-dwarf-address-space-identifier`. 2498 2499``segment_selector_size`` (ubyte) 2500 AMDGPU does not use a segment selector so this is 0. 2501 2502``minimum_instruction_length`` (ubyte) 2503 For GFX9-GFX10 this is 4. 2504 2505``maximum_operations_per_instruction`` (ubyte) 2506 For GFX9-GFX10 this is 1. 2507 2508Source text for online-compiled programs (for example, those compiled by the 2509OpenCL language runtime) may be embedded into the DWARF Version 5 line table. 2510See DWARF Version 5 section 6.2.4.1 which is updated by *DWARF Extensions For 2511Heterogeneous Debugging* section :ref:`DW_LNCT_LLVM_source 2512<amdgpu-dwarf-line-number-information-dw-lnct-llvm-source>`. 2513 2514The Clang option used to control source embedding in AMDGPU is defined in 2515:ref:`amdgpu-clang-debug-options-table`. 2516 2517 .. table:: AMDGPU Clang Debug Options 2518 :name: amdgpu-clang-debug-options-table 2519 2520 ==================== ================================================== 2521 Debug Flag Description 2522 ==================== ================================================== 2523 -g[no-]embed-source Enable/disable embedding source text in DWARF 2524 debug sections. Useful for environments where 2525 source cannot be written to disk, such as 2526 when performing online compilation. 2527 ==================== ================================================== 2528 2529For example: 2530 2531``-gembed-source`` 2532 Enable the embedded source. 2533 2534``-gno-embed-source`` 2535 Disable the embedded source. 2536 253732-Bit and 64-Bit DWARF Formats 2538------------------------------- 2539 2540See DWARF Version 5 section 7.4 and 2541:ref:`amdgpu-dwarf-32-bit-and-64-bit-dwarf-formats`. 2542 2543For AMDGPU: 2544 2545* For the ``amdgcn`` target architecture only the 64-bit process address space 2546 is supported. 2547 2548* The producer can generate either 32-bit or 64-bit DWARF format. LLVM generates 2549 the 32-bit DWARF format. 2550 2551Unit Headers 2552------------ 2553 2554For AMDGPU the following values apply for each of the unit headers described in 2555DWARF Version 5 sections 7.5.1.1, 7.5.1.2, and 7.5.1.3: 2556 2557``address_size`` (ubyte) 2558 Matches the address size for the ``Global`` address space defined in 2559 :ref:`amdgpu-dwarf-address-space-identifier`. 2560 2561.. _amdgpu-code-conventions: 2562 2563Code Conventions 2564================ 2565 2566This section provides code conventions used for each supported target triple OS 2567(see :ref:`amdgpu-target-triples`). 2568 2569AMDHSA 2570------ 2571 2572This section provides code conventions used when the target triple OS is 2573``amdhsa`` (see :ref:`amdgpu-target-triples`). 2574 2575.. _amdgpu-amdhsa-code-object-metadata: 2576 2577Code Object Metadata 2578~~~~~~~~~~~~~~~~~~~~ 2579 2580The code object metadata specifies extensible metadata associated with the code 2581objects executed on HSA [HSA]_ compatible runtimes (see :ref:`amdgpu-os`). The 2582encoding and semantics of this metadata depends on the code object version; see 2583:ref:`amdgpu-amdhsa-code-object-metadata-v2`, 2584:ref:`amdgpu-amdhsa-code-object-metadata-v3`, 2585:ref:`amdgpu-amdhsa-code-object-metadata-v4` and 2586:ref:`amdgpu-amdhsa-code-object-metadata-v5`. 2587 2588Code object metadata is specified in a note record (see 2589:ref:`amdgpu-note-records`) and is required when the target triple OS is 2590``amdhsa`` (see :ref:`amdgpu-target-triples`). It must contain the minimum 2591information necessary to support the HSA compatible runtime kernel queries. For 2592example, the segment sizes needed in a dispatch packet. In addition, a 2593high-level language runtime may require other information to be included. For 2594example, the AMD OpenCL runtime records kernel argument information. 2595 2596.. _amdgpu-amdhsa-code-object-metadata-v2: 2597 2598Code Object V2 Metadata 2599+++++++++++++++++++++++ 2600 2601.. warning:: 2602 Code object V2 is not the default code object version emitted by this version 2603 of LLVM. 2604 2605Code object V2 metadata is specified by the ``NT_AMD_HSA_METADATA`` note record 2606(see :ref:`amdgpu-note-records-v2`). 2607 2608The metadata is specified as a YAML formatted string (see [YAML]_ and 2609:doc:`YamlIO`). 2610 2611.. TODO:: 2612 2613 Is the string null terminated? It probably should not if YAML allows it to 2614 contain null characters, otherwise it should be. 2615 2616The metadata is represented as a single YAML document comprised of the mapping 2617defined in table :ref:`amdgpu-amdhsa-code-object-metadata-map-v2-table` and 2618referenced tables. 2619 2620For boolean values, the string values of ``false`` and ``true`` are used for 2621false and true respectively. 2622 2623Additional information can be added to the mappings. To avoid conflicts, any 2624non-AMD key names should be prefixed by "*vendor-name*.". 2625 2626 .. table:: AMDHSA Code Object V2 Metadata Map 2627 :name: amdgpu-amdhsa-code-object-metadata-map-v2-table 2628 2629 ========== ============== ========= ======================================= 2630 String Key Value Type Required? Description 2631 ========== ============== ========= ======================================= 2632 "Version" sequence of Required - The first integer is the major 2633 2 integers version. Currently 1. 2634 - The second integer is the minor 2635 version. Currently 0. 2636 "Printf" sequence of Each string is encoded information 2637 strings about a printf function call. The 2638 encoded information is organized as 2639 fields separated by colon (':'): 2640 2641 ``ID:N:S[0]:S[1]:...:S[N-1]:FormatString`` 2642 2643 where: 2644 2645 ``ID`` 2646 A 32-bit integer as a unique id for 2647 each printf function call 2648 2649 ``N`` 2650 A 32-bit integer equal to the number 2651 of arguments of printf function call 2652 minus 1 2653 2654 ``S[i]`` (where i = 0, 1, ... , N-1) 2655 32-bit integers for the size in bytes 2656 of the i-th FormatString argument of 2657 the printf function call 2658 2659 FormatString 2660 The format string passed to the 2661 printf function call. 2662 "Kernels" sequence of Required Sequence of the mappings for each 2663 mapping kernel in the code object. See 2664 :ref:`amdgpu-amdhsa-code-object-kernel-metadata-map-v2-table` 2665 for the definition of the mapping. 2666 ========== ============== ========= ======================================= 2667 2668.. 2669 2670 .. table:: AMDHSA Code Object V2 Kernel Metadata Map 2671 :name: amdgpu-amdhsa-code-object-kernel-metadata-map-v2-table 2672 2673 ================= ============== ========= ================================ 2674 String Key Value Type Required? Description 2675 ================= ============== ========= ================================ 2676 "Name" string Required Source name of the kernel. 2677 "SymbolName" string Required Name of the kernel 2678 descriptor ELF symbol. 2679 "Language" string Source language of the kernel. 2680 Values include: 2681 2682 - "OpenCL C" 2683 - "OpenCL C++" 2684 - "HCC" 2685 - "OpenMP" 2686 2687 "LanguageVersion" sequence of - The first integer is the major 2688 2 integers version. 2689 - The second integer is the 2690 minor version. 2691 "Attrs" mapping Mapping of kernel attributes. 2692 See 2693 :ref:`amdgpu-amdhsa-code-object-kernel-attribute-metadata-map-v2-table` 2694 for the mapping definition. 2695 "Args" sequence of Sequence of mappings of the 2696 mapping kernel arguments. See 2697 :ref:`amdgpu-amdhsa-code-object-kernel-argument-metadata-map-v2-table` 2698 for the definition of the mapping. 2699 "CodeProps" mapping Mapping of properties related to 2700 the kernel code. See 2701 :ref:`amdgpu-amdhsa-code-object-kernel-code-properties-metadata-map-v2-table` 2702 for the mapping definition. 2703 ================= ============== ========= ================================ 2704 2705.. 2706 2707 .. table:: AMDHSA Code Object V2 Kernel Attribute Metadata Map 2708 :name: amdgpu-amdhsa-code-object-kernel-attribute-metadata-map-v2-table 2709 2710 =================== ============== ========= ============================== 2711 String Key Value Type Required? Description 2712 =================== ============== ========= ============================== 2713 "ReqdWorkGroupSize" sequence of If not 0, 0, 0 then all values 2714 3 integers must be >=1 and the dispatch 2715 work-group size X, Y, Z must 2716 correspond to the specified 2717 values. Defaults to 0, 0, 0. 2718 2719 Corresponds to the OpenCL 2720 ``reqd_work_group_size`` 2721 attribute. 2722 "WorkGroupSizeHint" sequence of The dispatch work-group size 2723 3 integers X, Y, Z is likely to be the 2724 specified values. 2725 2726 Corresponds to the OpenCL 2727 ``work_group_size_hint`` 2728 attribute. 2729 "VecTypeHint" string The name of a scalar or vector 2730 type. 2731 2732 Corresponds to the OpenCL 2733 ``vec_type_hint`` attribute. 2734 2735 "RuntimeHandle" string The external symbol name 2736 associated with a kernel. 2737 OpenCL runtime allocates a 2738 global buffer for the symbol 2739 and saves the kernel's address 2740 to it, which is used for 2741 device side enqueueing. Only 2742 available for device side 2743 enqueued kernels. 2744 =================== ============== ========= ============================== 2745 2746.. 2747 2748 .. table:: AMDHSA Code Object V2 Kernel Argument Metadata Map 2749 :name: amdgpu-amdhsa-code-object-kernel-argument-metadata-map-v2-table 2750 2751 ================= ============== ========= ================================ 2752 String Key Value Type Required? Description 2753 ================= ============== ========= ================================ 2754 "Name" string Kernel argument name. 2755 "TypeName" string Kernel argument type name. 2756 "Size" integer Required Kernel argument size in bytes. 2757 "Align" integer Required Kernel argument alignment in 2758 bytes. Must be a power of two. 2759 "ValueKind" string Required Kernel argument kind that 2760 specifies how to set up the 2761 corresponding argument. 2762 Values include: 2763 2764 "ByValue" 2765 The argument is copied 2766 directly into the kernarg. 2767 2768 "GlobalBuffer" 2769 A global address space pointer 2770 to the buffer data is passed 2771 in the kernarg. 2772 2773 "DynamicSharedPointer" 2774 A group address space pointer 2775 to dynamically allocated LDS 2776 is passed in the kernarg. 2777 2778 "Sampler" 2779 A global address space 2780 pointer to a S# is passed in 2781 the kernarg. 2782 2783 "Image" 2784 A global address space 2785 pointer to a T# is passed in 2786 the kernarg. 2787 2788 "Pipe" 2789 A global address space pointer 2790 to an OpenCL pipe is passed in 2791 the kernarg. 2792 2793 "Queue" 2794 A global address space pointer 2795 to an OpenCL device enqueue 2796 queue is passed in the 2797 kernarg. 2798 2799 "HiddenGlobalOffsetX" 2800 The OpenCL grid dispatch 2801 global offset for the X 2802 dimension is passed in the 2803 kernarg. 2804 2805 "HiddenGlobalOffsetY" 2806 The OpenCL grid dispatch 2807 global offset for the Y 2808 dimension is passed in the 2809 kernarg. 2810 2811 "HiddenGlobalOffsetZ" 2812 The OpenCL grid dispatch 2813 global offset for the Z 2814 dimension is passed in the 2815 kernarg. 2816 2817 "HiddenNone" 2818 An argument that is not used 2819 by the kernel. Space needs to 2820 be left for it, but it does 2821 not need to be set up. 2822 2823 "HiddenPrintfBuffer" 2824 A global address space pointer 2825 to the runtime printf buffer 2826 is passed in kernarg. 2827 2828 "HiddenHostcallBuffer" 2829 A global address space pointer 2830 to the runtime hostcall buffer 2831 is passed in kernarg. 2832 2833 "HiddenDefaultQueue" 2834 A global address space pointer 2835 to the OpenCL device enqueue 2836 queue that should be used by 2837 the kernel by default is 2838 passed in the kernarg. 2839 2840 "HiddenCompletionAction" 2841 A global address space pointer 2842 to help link enqueued kernels into 2843 the ancestor tree for determining 2844 when the parent kernel has finished. 2845 2846 "HiddenMultiGridSyncArg" 2847 A global address space pointer for 2848 multi-grid synchronization is 2849 passed in the kernarg. 2850 2851 "ValueType" string Unused and deprecated. This should no longer 2852 be emitted, but is accepted for compatibility. 2853 2854 2855 "PointeeAlign" integer Alignment in bytes of pointee 2856 type for pointer type kernel 2857 argument. Must be a power 2858 of 2. Only present if 2859 "ValueKind" is 2860 "DynamicSharedPointer". 2861 "AddrSpaceQual" string Kernel argument address space 2862 qualifier. Only present if 2863 "ValueKind" is "GlobalBuffer" or 2864 "DynamicSharedPointer". Values 2865 are: 2866 2867 - "Private" 2868 - "Global" 2869 - "Constant" 2870 - "Local" 2871 - "Generic" 2872 - "Region" 2873 2874 .. TODO:: 2875 2876 Is GlobalBuffer only Global 2877 or Constant? Is 2878 DynamicSharedPointer always 2879 Local? Can HCC allow Generic? 2880 How can Private or Region 2881 ever happen? 2882 2883 "AccQual" string Kernel argument access 2884 qualifier. Only present if 2885 "ValueKind" is "Image" or 2886 "Pipe". Values 2887 are: 2888 2889 - "ReadOnly" 2890 - "WriteOnly" 2891 - "ReadWrite" 2892 2893 .. TODO:: 2894 2895 Does this apply to 2896 GlobalBuffer? 2897 2898 "ActualAccQual" string The actual memory accesses 2899 performed by the kernel on the 2900 kernel argument. Only present if 2901 "ValueKind" is "GlobalBuffer", 2902 "Image", or "Pipe". This may be 2903 more restrictive than indicated 2904 by "AccQual" to reflect what the 2905 kernel actual does. If not 2906 present then the runtime must 2907 assume what is implied by 2908 "AccQual" and "IsConst". Values 2909 are: 2910 2911 - "ReadOnly" 2912 - "WriteOnly" 2913 - "ReadWrite" 2914 2915 "IsConst" boolean Indicates if the kernel argument 2916 is const qualified. Only present 2917 if "ValueKind" is 2918 "GlobalBuffer". 2919 2920 "IsRestrict" boolean Indicates if the kernel argument 2921 is restrict qualified. Only 2922 present if "ValueKind" is 2923 "GlobalBuffer". 2924 2925 "IsVolatile" boolean Indicates if the kernel argument 2926 is volatile qualified. Only 2927 present if "ValueKind" is 2928 "GlobalBuffer". 2929 2930 "IsPipe" boolean Indicates if the kernel argument 2931 is pipe qualified. Only present 2932 if "ValueKind" is "Pipe". 2933 2934 .. TODO:: 2935 2936 Can GlobalBuffer be pipe 2937 qualified? 2938 2939 ================= ============== ========= ================================ 2940 2941.. 2942 2943 .. table:: AMDHSA Code Object V2 Kernel Code Properties Metadata Map 2944 :name: amdgpu-amdhsa-code-object-kernel-code-properties-metadata-map-v2-table 2945 2946 ============================ ============== ========= ===================== 2947 String Key Value Type Required? Description 2948 ============================ ============== ========= ===================== 2949 "KernargSegmentSize" integer Required The size in bytes of 2950 the kernarg segment 2951 that holds the values 2952 of the arguments to 2953 the kernel. 2954 "GroupSegmentFixedSize" integer Required The amount of group 2955 segment memory 2956 required by a 2957 work-group in 2958 bytes. This does not 2959 include any 2960 dynamically allocated 2961 group segment memory 2962 that may be added 2963 when the kernel is 2964 dispatched. 2965 "PrivateSegmentFixedSize" integer Required The amount of fixed 2966 private address space 2967 memory required for a 2968 work-item in 2969 bytes. If the kernel 2970 uses a dynamic call 2971 stack then additional 2972 space must be added 2973 to this value for the 2974 call stack. 2975 "KernargSegmentAlign" integer Required The maximum byte 2976 alignment of 2977 arguments in the 2978 kernarg segment. Must 2979 be a power of 2. 2980 "WavefrontSize" integer Required Wavefront size. Must 2981 be a power of 2. 2982 "NumSGPRs" integer Required Number of scalar 2983 registers used by a 2984 wavefront for 2985 GFX6-GFX10. This 2986 includes the special 2987 SGPRs for VCC, Flat 2988 Scratch (GFX7-GFX10) 2989 and XNACK (for 2990 GFX8-GFX10). It does 2991 not include the 16 2992 SGPR added if a trap 2993 handler is 2994 enabled. It is not 2995 rounded up to the 2996 allocation 2997 granularity. 2998 "NumVGPRs" integer Required Number of vector 2999 registers used by 3000 each work-item for 3001 GFX6-GFX10 3002 "MaxFlatWorkGroupSize" integer Required Maximum flat 3003 work-group size 3004 supported by the 3005 kernel in work-items. 3006 Must be >=1 and 3007 consistent with 3008 ReqdWorkGroupSize if 3009 not 0, 0, 0. 3010 "NumSpilledSGPRs" integer Number of stores from 3011 a scalar register to 3012 a register allocator 3013 created spill 3014 location. 3015 "NumSpilledVGPRs" integer Number of stores from 3016 a vector register to 3017 a register allocator 3018 created spill 3019 location. 3020 ============================ ============== ========= ===================== 3021 3022.. _amdgpu-amdhsa-code-object-metadata-v3: 3023 3024Code Object V3 Metadata 3025+++++++++++++++++++++++ 3026 3027.. warning:: 3028 Code object V3 is not the default code object version emitted by this version 3029 of LLVM. 3030 3031Code object V3 and above metadata is specified by the ``NT_AMDGPU_METADATA`` note 3032record (see :ref:`amdgpu-note-records-v3-onwards`). 3033 3034The metadata is represented as Message Pack formatted binary data (see 3035[MsgPack]_). The top level is a Message Pack map that includes the 3036keys defined in table 3037:ref:`amdgpu-amdhsa-code-object-metadata-map-table-v3` and referenced 3038tables. 3039 3040Additional information can be added to the maps. To avoid conflicts, 3041any key names should be prefixed by "*vendor-name*." where 3042``vendor-name`` can be the name of the vendor and specific vendor 3043tool that generates the information. The prefix is abbreviated to 3044simply "." when it appears within a map that has been added by the 3045same *vendor-name*. 3046 3047 .. table:: AMDHSA Code Object V3 Metadata Map 3048 :name: amdgpu-amdhsa-code-object-metadata-map-table-v3 3049 3050 ================= ============== ========= ======================================= 3051 String Key Value Type Required? Description 3052 ================= ============== ========= ======================================= 3053 "amdhsa.version" sequence of Required - The first integer is the major 3054 2 integers version. Currently 1. 3055 - The second integer is the minor 3056 version. Currently 0. 3057 "amdhsa.printf" sequence of Each string is encoded information 3058 strings about a printf function call. The 3059 encoded information is organized as 3060 fields separated by colon (':'): 3061 3062 ``ID:N:S[0]:S[1]:...:S[N-1]:FormatString`` 3063 3064 where: 3065 3066 ``ID`` 3067 A 32-bit integer as a unique id for 3068 each printf function call 3069 3070 ``N`` 3071 A 32-bit integer equal to the number 3072 of arguments of printf function call 3073 minus 1 3074 3075 ``S[i]`` (where i = 0, 1, ... , N-1) 3076 32-bit integers for the size in bytes 3077 of the i-th FormatString argument of 3078 the printf function call 3079 3080 FormatString 3081 The format string passed to the 3082 printf function call. 3083 "amdhsa.kernels" sequence of Required Sequence of the maps for each 3084 map kernel in the code object. See 3085 :ref:`amdgpu-amdhsa-code-object-kernel-metadata-map-table-v3` 3086 for the definition of the keys included 3087 in that map. 3088 ================= ============== ========= ======================================= 3089 3090.. 3091 3092 .. table:: AMDHSA Code Object V3 Kernel Metadata Map 3093 :name: amdgpu-amdhsa-code-object-kernel-metadata-map-table-v3 3094 3095 =================================== ============== ========= ================================ 3096 String Key Value Type Required? Description 3097 =================================== ============== ========= ================================ 3098 ".name" string Required Source name of the kernel. 3099 ".symbol" string Required Name of the kernel 3100 descriptor ELF symbol. 3101 ".language" string Source language of the kernel. 3102 Values include: 3103 3104 - "OpenCL C" 3105 - "OpenCL C++" 3106 - "HCC" 3107 - "HIP" 3108 - "OpenMP" 3109 - "Assembler" 3110 3111 ".language_version" sequence of - The first integer is the major 3112 2 integers version. 3113 - The second integer is the 3114 minor version. 3115 ".args" sequence of Sequence of maps of the 3116 map kernel arguments. See 3117 :ref:`amdgpu-amdhsa-code-object-kernel-argument-metadata-map-table-v3` 3118 for the definition of the keys 3119 included in that map. 3120 ".reqd_workgroup_size" sequence of If not 0, 0, 0 then all values 3121 3 integers must be >=1 and the dispatch 3122 work-group size X, Y, Z must 3123 correspond to the specified 3124 values. Defaults to 0, 0, 0. 3125 3126 Corresponds to the OpenCL 3127 ``reqd_work_group_size`` 3128 attribute. 3129 ".workgroup_size_hint" sequence of The dispatch work-group size 3130 3 integers X, Y, Z is likely to be the 3131 specified values. 3132 3133 Corresponds to the OpenCL 3134 ``work_group_size_hint`` 3135 attribute. 3136 ".vec_type_hint" string The name of a scalar or vector 3137 type. 3138 3139 Corresponds to the OpenCL 3140 ``vec_type_hint`` attribute. 3141 3142 ".device_enqueue_symbol" string The external symbol name 3143 associated with a kernel. 3144 OpenCL runtime allocates a 3145 global buffer for the symbol 3146 and saves the kernel's address 3147 to it, which is used for 3148 device side enqueueing. Only 3149 available for device side 3150 enqueued kernels. 3151 ".kernarg_segment_size" integer Required The size in bytes of 3152 the kernarg segment 3153 that holds the values 3154 of the arguments to 3155 the kernel. 3156 ".group_segment_fixed_size" integer Required The amount of group 3157 segment memory 3158 required by a 3159 work-group in 3160 bytes. This does not 3161 include any 3162 dynamically allocated 3163 group segment memory 3164 that may be added 3165 when the kernel is 3166 dispatched. 3167 ".private_segment_fixed_size" integer Required The amount of fixed 3168 private address space 3169 memory required for a 3170 work-item in 3171 bytes. If the kernel 3172 uses a dynamic call 3173 stack then additional 3174 space must be added 3175 to this value for the 3176 call stack. 3177 ".kernarg_segment_align" integer Required The maximum byte 3178 alignment of 3179 arguments in the 3180 kernarg segment. Must 3181 be a power of 2. 3182 ".wavefront_size" integer Required Wavefront size. Must 3183 be a power of 2. 3184 ".sgpr_count" integer Required Number of scalar 3185 registers required by a 3186 wavefront for 3187 GFX6-GFX9. A register 3188 is required if it is 3189 used explicitly, or 3190 if a higher numbered 3191 register is used 3192 explicitly. This 3193 includes the special 3194 SGPRs for VCC, Flat 3195 Scratch (GFX7-GFX9) 3196 and XNACK (for 3197 GFX8-GFX9). It does 3198 not include the 16 3199 SGPR added if a trap 3200 handler is 3201 enabled. It is not 3202 rounded up to the 3203 allocation 3204 granularity. 3205 ".vgpr_count" integer Required Number of vector 3206 registers required by 3207 each work-item for 3208 GFX6-GFX9. A register 3209 is required if it is 3210 used explicitly, or 3211 if a higher numbered 3212 register is used 3213 explicitly. 3214 ".agpr_count" integer Required Number of accumulator 3215 registers required by 3216 each work-item for 3217 GFX90A, GFX908. 3218 ".max_flat_workgroup_size" integer Required Maximum flat 3219 work-group size 3220 supported by the 3221 kernel in work-items. 3222 Must be >=1 and 3223 consistent with 3224 ReqdWorkGroupSize if 3225 not 0, 0, 0. 3226 ".sgpr_spill_count" integer Number of stores from 3227 a scalar register to 3228 a register allocator 3229 created spill 3230 location. 3231 ".vgpr_spill_count" integer Number of stores from 3232 a vector register to 3233 a register allocator 3234 created spill 3235 location. 3236 ".kind" string The kind of the kernel 3237 with the following 3238 values: 3239 3240 "normal" 3241 Regular kernels. 3242 3243 "init" 3244 These kernels must be 3245 invoked after loading 3246 the containing code 3247 object and must 3248 complete before any 3249 normal and fini 3250 kernels in the same 3251 code object are 3252 invoked. 3253 3254 "fini" 3255 These kernels must be 3256 invoked before 3257 unloading the 3258 containing code object 3259 and after all init and 3260 normal kernels in the 3261 same code object have 3262 been invoked and 3263 completed. 3264 3265 If omitted, "normal" is 3266 assumed. 3267 =================================== ============== ========= ================================ 3268 3269.. 3270 3271 .. table:: AMDHSA Code Object V3 Kernel Argument Metadata Map 3272 :name: amdgpu-amdhsa-code-object-kernel-argument-metadata-map-table-v3 3273 3274 ====================== ============== ========= ================================ 3275 String Key Value Type Required? Description 3276 ====================== ============== ========= ================================ 3277 ".name" string Kernel argument name. 3278 ".type_name" string Kernel argument type name. 3279 ".size" integer Required Kernel argument size in bytes. 3280 ".offset" integer Required Kernel argument offset in 3281 bytes. The offset must be a 3282 multiple of the alignment 3283 required by the argument. 3284 ".value_kind" string Required Kernel argument kind that 3285 specifies how to set up the 3286 corresponding argument. 3287 Values include: 3288 3289 "by_value" 3290 The argument is copied 3291 directly into the kernarg. 3292 3293 "global_buffer" 3294 A global address space pointer 3295 to the buffer data is passed 3296 in the kernarg. 3297 3298 "dynamic_shared_pointer" 3299 A group address space pointer 3300 to dynamically allocated LDS 3301 is passed in the kernarg. 3302 3303 "sampler" 3304 A global address space 3305 pointer to a S# is passed in 3306 the kernarg. 3307 3308 "image" 3309 A global address space 3310 pointer to a T# is passed in 3311 the kernarg. 3312 3313 "pipe" 3314 A global address space pointer 3315 to an OpenCL pipe is passed in 3316 the kernarg. 3317 3318 "queue" 3319 A global address space pointer 3320 to an OpenCL device enqueue 3321 queue is passed in the 3322 kernarg. 3323 3324 "hidden_global_offset_x" 3325 The OpenCL grid dispatch 3326 global offset for the X 3327 dimension is passed in the 3328 kernarg. 3329 3330 "hidden_global_offset_y" 3331 The OpenCL grid dispatch 3332 global offset for the Y 3333 dimension is passed in the 3334 kernarg. 3335 3336 "hidden_global_offset_z" 3337 The OpenCL grid dispatch 3338 global offset for the Z 3339 dimension is passed in the 3340 kernarg. 3341 3342 "hidden_none" 3343 An argument that is not used 3344 by the kernel. Space needs to 3345 be left for it, but it does 3346 not need to be set up. 3347 3348 "hidden_printf_buffer" 3349 A global address space pointer 3350 to the runtime printf buffer 3351 is passed in kernarg. 3352 3353 "hidden_hostcall_buffer" 3354 A global address space pointer 3355 to the runtime hostcall buffer 3356 is passed in kernarg. 3357 3358 "hidden_default_queue" 3359 A global address space pointer 3360 to the OpenCL device enqueue 3361 queue that should be used by 3362 the kernel by default is 3363 passed in the kernarg. 3364 3365 "hidden_completion_action" 3366 A global address space pointer 3367 to help link enqueued kernels into 3368 the ancestor tree for determining 3369 when the parent kernel has finished. 3370 3371 "hidden_multigrid_sync_arg" 3372 A global address space pointer for 3373 multi-grid synchronization is 3374 passed in the kernarg. 3375 3376 ".value_type" string Unused and deprecated. This should no longer 3377 be emitted, but is accepted for compatibility. 3378 3379 ".pointee_align" integer Alignment in bytes of pointee 3380 type for pointer type kernel 3381 argument. Must be a power 3382 of 2. Only present if 3383 ".value_kind" is 3384 "dynamic_shared_pointer". 3385 ".address_space" string Kernel argument address space 3386 qualifier. Only present if 3387 ".value_kind" is "global_buffer" or 3388 "dynamic_shared_pointer". Values 3389 are: 3390 3391 - "private" 3392 - "global" 3393 - "constant" 3394 - "local" 3395 - "generic" 3396 - "region" 3397 3398 .. TODO:: 3399 3400 Is "global_buffer" only "global" 3401 or "constant"? Is 3402 "dynamic_shared_pointer" always 3403 "local"? Can HCC allow "generic"? 3404 How can "private" or "region" 3405 ever happen? 3406 3407 ".access" string Kernel argument access 3408 qualifier. Only present if 3409 ".value_kind" is "image" or 3410 "pipe". Values 3411 are: 3412 3413 - "read_only" 3414 - "write_only" 3415 - "read_write" 3416 3417 .. TODO:: 3418 3419 Does this apply to 3420 "global_buffer"? 3421 3422 ".actual_access" string The actual memory accesses 3423 performed by the kernel on the 3424 kernel argument. Only present if 3425 ".value_kind" is "global_buffer", 3426 "image", or "pipe". This may be 3427 more restrictive than indicated 3428 by ".access" to reflect what the 3429 kernel actual does. If not 3430 present then the runtime must 3431 assume what is implied by 3432 ".access" and ".is_const" . Values 3433 are: 3434 3435 - "read_only" 3436 - "write_only" 3437 - "read_write" 3438 3439 ".is_const" boolean Indicates if the kernel argument 3440 is const qualified. Only present 3441 if ".value_kind" is 3442 "global_buffer". 3443 3444 ".is_restrict" boolean Indicates if the kernel argument 3445 is restrict qualified. Only 3446 present if ".value_kind" is 3447 "global_buffer". 3448 3449 ".is_volatile" boolean Indicates if the kernel argument 3450 is volatile qualified. Only 3451 present if ".value_kind" is 3452 "global_buffer". 3453 3454 ".is_pipe" boolean Indicates if the kernel argument 3455 is pipe qualified. Only present 3456 if ".value_kind" is "pipe". 3457 3458 .. TODO:: 3459 3460 Can "global_buffer" be pipe 3461 qualified? 3462 3463 ====================== ============== ========= ================================ 3464 3465.. _amdgpu-amdhsa-code-object-metadata-v4: 3466 3467Code Object V4 Metadata 3468+++++++++++++++++++++++ 3469 3470Code object V4 metadata is the same as 3471:ref:`amdgpu-amdhsa-code-object-metadata-v3` with the changes and additions 3472defined in table :ref:`amdgpu-amdhsa-code-object-metadata-map-table-v4`. 3473 3474 .. table:: AMDHSA Code Object V4 Metadata Map Changes 3475 :name: amdgpu-amdhsa-code-object-metadata-map-table-v4 3476 3477 ================= ============== ========= ======================================= 3478 String Key Value Type Required? Description 3479 ================= ============== ========= ======================================= 3480 "amdhsa.version" sequence of Required - The first integer is the major 3481 2 integers version. Currently 1. 3482 - The second integer is the minor 3483 version. Currently 1. 3484 "amdhsa.target" string Required The target name of the code using the syntax: 3485 3486 .. code:: 3487 3488 <target-triple> [ "-" <target-id> ] 3489 3490 A canonical target ID must be 3491 used. See :ref:`amdgpu-target-triples` 3492 and :ref:`amdgpu-target-id`. 3493 ================= ============== ========= ======================================= 3494 3495.. _amdgpu-amdhsa-code-object-metadata-v5: 3496 3497Code Object V5 Metadata 3498+++++++++++++++++++++++ 3499 3500.. warning:: 3501 Code object V5 is not the default code object version emitted by this version 3502 of LLVM. 3503 3504 3505Code object V5 metadata is the same as 3506:ref:`amdgpu-amdhsa-code-object-metadata-v4` with the changes defined in table 3507:ref:`amdgpu-amdhsa-code-object-metadata-map-table-v5` and table 3508:ref:`amdgpu-amdhsa-code-object-kernel-argument-metadata-map-table-v5`. 3509 3510 .. table:: AMDHSA Code Object V5 Metadata Map Changes 3511 :name: amdgpu-amdhsa-code-object-metadata-map-table-v5 3512 3513 ================= ============== ========= ======================================= 3514 String Key Value Type Required? Description 3515 ================= ============== ========= ======================================= 3516 "amdhsa.version" sequence of Required - The first integer is the major 3517 2 integers version. Currently 1. 3518 - The second integer is the minor 3519 version. Currently 2. 3520 ================= ============== ========= ======================================= 3521 3522.. 3523 3524 .. table:: AMDHSA Code Object V5 Kernel Argument Metadata Map Additions and Changes 3525 :name: amdgpu-amdhsa-code-object-kernel-argument-metadata-map-table-v5 3526 3527 ====================== ============== ========= ================================ 3528 String Key Value Type Required? Description 3529 ====================== ============== ========= ================================ 3530 ".value_kind" string Required Kernel argument kind that 3531 specifies how to set up the 3532 corresponding argument. 3533 Values include: 3534 the same as code object V3 metadata 3535 (see :ref:`amdgpu-amdhsa-code-object-kernel-argument-metadata-map-table-v3`) 3536 with the following additions: 3537 3538 "hidden_block_count_x" 3539 The grid dispatch work-group count for the X dimension 3540 is passed in the kernarg. Some languages, such as OpenCL, 3541 support a last work-group in each dimension being partial. 3542 This count only includes the non-partial work-group count. 3543 This is not the same as the value in the AQL dispatch packet, 3544 which has the grid size in work-items. 3545 3546 "hidden_block_count_y" 3547 The grid dispatch work-group count for the Y dimension 3548 is passed in the kernarg. Some languages, such as OpenCL, 3549 support a last work-group in each dimension being partial. 3550 This count only includes the non-partial work-group count. 3551 This is not the same as the value in the AQL dispatch packet, 3552 which has the grid size in work-items. If the grid dimensionality 3553 is 1, then must be 1. 3554 3555 "hidden_block_count_z" 3556 The grid dispatch work-group count for the Z dimension 3557 is passed in the kernarg. Some languages, such as OpenCL, 3558 support a last work-group in each dimension being partial. 3559 This count only includes the non-partial work-group count. 3560 This is not the same as the value in the AQL dispatch packet, 3561 which has the grid size in work-items. If the grid dimensionality 3562 is 1 or 2, then must be 1. 3563 3564 "hidden_group_size_x" 3565 The grid dispatch work-group size for the X dimension is 3566 passed in the kernarg. This size only applies to the 3567 non-partial work-groups. This is the same value as the AQL 3568 dispatch packet work-group size. 3569 3570 "hidden_group_size_y" 3571 The grid dispatch work-group size for the Y dimension is 3572 passed in the kernarg. This size only applies to the 3573 non-partial work-groups. This is the same value as the AQL 3574 dispatch packet work-group size. If the grid dimensionality 3575 is 1, then must be 1. 3576 3577 "hidden_group_size_z" 3578 The grid dispatch work-group size for the Z dimension is 3579 passed in the kernarg. This size only applies to the 3580 non-partial work-groups. This is the same value as the AQL 3581 dispatch packet work-group size. If the grid dimensionality 3582 is 1 or 2, then must be 1. 3583 3584 "hidden_remainder_x" 3585 The grid dispatch work group size of the the partial work group 3586 of the X dimension, if it exists. Must be zero if a partial 3587 work group does not exist in the X dimension. 3588 3589 "hidden_remainder_y" 3590 The grid dispatch work group size of the the partial work group 3591 of the Y dimension, if it exists. Must be zero if a partial 3592 work group does not exist in the Y dimension. 3593 3594 "hidden_remainder_z" 3595 The grid dispatch work group size of the the partial work group 3596 of the Z dimension, if it exists. Must be zero if a partial 3597 work group does not exist in the Z dimension. 3598 3599 "hidden_grid_dims" 3600 The grid dispatch dimensionality. This is the same value 3601 as the AQL dispatch packet dimensionality. Must be a value 3602 between 1 and 3. 3603 3604 "hidden_heap_v1" 3605 A global address space pointer to an initialized memory 3606 buffer that conforms to the requirements of the malloc/free 3607 device library V1 version implementation. 3608 3609 "hidden_private_base" 3610 The high 32 bits of the flat addressing private aperture base. 3611 Only used by GFX8 to allow conversion between private segment 3612 and flat addresses. See :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`. 3613 3614 "hidden_shared_base" 3615 The high 32 bits of the flat addressing shared aperture base. 3616 Only used by GFX8 to allow conversion between shared segment 3617 and flat addresses. See :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`. 3618 3619 "hidden_queue_ptr" 3620 A global memory address space pointer to the ROCm runtime 3621 ``struct amd_queue_t`` structure for the HSA queue of the 3622 associated dispatch AQL packet. It is only required for pre-GFX9 3623 devices for the trap handler ABI (see :ref:`amdgpu-amdhsa-trap-handler-abi`). 3624 3625 ====================== ============== ========= ================================ 3626 3627.. 3628 3629Kernel Dispatch 3630~~~~~~~~~~~~~~~ 3631 3632The HSA architected queuing language (AQL) defines a user space memory interface 3633that can be used to control the dispatch of kernels, in an agent independent 3634way. An agent can have zero or more AQL queues created for it using an HSA 3635compatible runtime (see :ref:`amdgpu-os`), in which AQL packets (all of which 3636are 64 bytes) can be placed. See the *HSA Platform System Architecture 3637Specification* [HSA]_ for the AQL queue mechanics and packet layouts. 3638 3639The packet processor of a kernel agent is responsible for detecting and 3640dispatching HSA kernels from the AQL queues associated with it. For AMD GPUs the 3641packet processor is implemented by the hardware command processor (CP), 3642asynchronous dispatch controller (ADC) and shader processor input controller 3643(SPI). 3644 3645An HSA compatible runtime can be used to allocate an AQL queue object. It uses 3646the kernel mode driver to initialize and register the AQL queue with CP. 3647 3648To dispatch a kernel the following actions are performed. This can occur in the 3649CPU host program, or from an HSA kernel executing on a GPU. 3650 36511. A pointer to an AQL queue for the kernel agent on which the kernel is to be 3652 executed is obtained. 36532. A pointer to the kernel descriptor (see 3654 :ref:`amdgpu-amdhsa-kernel-descriptor`) of the kernel to execute is obtained. 3655 It must be for a kernel that is contained in a code object that was loaded 3656 by an HSA compatible runtime on the kernel agent with which the AQL queue is 3657 associated. 36583. Space is allocated for the kernel arguments using the HSA compatible runtime 3659 allocator for a memory region with the kernarg property for the kernel agent 3660 that will execute the kernel. It must be at least 16-byte aligned. 36614. Kernel argument values are assigned to the kernel argument memory 3662 allocation. The layout is defined in the *HSA Programmer's Language 3663 Reference* [HSA]_. For AMDGPU the kernel execution directly accesses the 3664 kernel argument memory in the same way constant memory is accessed. (Note 3665 that the HSA specification allows an implementation to copy the kernel 3666 argument contents to another location that is accessed by the kernel.) 36675. An AQL kernel dispatch packet is created on the AQL queue. The HSA compatible 3668 runtime api uses 64-bit atomic operations to reserve space in the AQL queue 3669 for the packet. The packet must be set up, and the final write must use an 3670 atomic store release to set the packet kind to ensure the packet contents are 3671 visible to the kernel agent. AQL defines a doorbell signal mechanism to 3672 notify the kernel agent that the AQL queue has been updated. These rules, and 3673 the layout of the AQL queue and kernel dispatch packet is defined in the *HSA 3674 System Architecture Specification* [HSA]_. 36756. A kernel dispatch packet includes information about the actual dispatch, 3676 such as grid and work-group size, together with information from the code 3677 object about the kernel, such as segment sizes. The HSA compatible runtime 3678 queries on the kernel symbol can be used to obtain the code object values 3679 which are recorded in the :ref:`amdgpu-amdhsa-code-object-metadata`. 36807. CP executes micro-code and is responsible for detecting and setting up the 3681 GPU to execute the wavefronts of a kernel dispatch. 36828. CP ensures that when the a wavefront starts executing the kernel machine 3683 code, the scalar general purpose registers (SGPR) and vector general purpose 3684 registers (VGPR) are set up as required by the machine code. The required 3685 setup is defined in the :ref:`amdgpu-amdhsa-kernel-descriptor`. The initial 3686 register state is defined in 3687 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`. 36889. The prolog of the kernel machine code (see 3689 :ref:`amdgpu-amdhsa-kernel-prolog`) sets up the machine state as necessary 3690 before continuing executing the machine code that corresponds to the kernel. 369110. When the kernel dispatch has completed execution, CP signals the completion 3692 signal specified in the kernel dispatch packet if not 0. 3693 3694.. _amdgpu-amdhsa-memory-spaces: 3695 3696Memory Spaces 3697~~~~~~~~~~~~~ 3698 3699The memory space properties are: 3700 3701 .. table:: AMDHSA Memory Spaces 3702 :name: amdgpu-amdhsa-memory-spaces-table 3703 3704 ================= =========== ======== ======= ================== 3705 Memory Space Name HSA Segment Hardware Address NULL Value 3706 Name Name Size 3707 ================= =========== ======== ======= ================== 3708 Private private scratch 32 0x00000000 3709 Local group LDS 32 0xFFFFFFFF 3710 Global global global 64 0x0000000000000000 3711 Constant constant *same as 64 0x0000000000000000 3712 global* 3713 Generic flat flat 64 0x0000000000000000 3714 Region N/A GDS 32 *not implemented 3715 for AMDHSA* 3716 ================= =========== ======== ======= ================== 3717 3718The global and constant memory spaces both use global virtual addresses, which 3719are the same virtual address space used by the CPU. However, some virtual 3720addresses may only be accessible to the CPU, some only accessible by the GPU, 3721and some by both. 3722 3723Using the constant memory space indicates that the data will not change during 3724the execution of the kernel. This allows scalar read instructions to be 3725used. The vector and scalar L1 caches are invalidated of volatile data before 3726each kernel dispatch execution to allow constant memory to change values between 3727kernel dispatches. 3728 3729The local memory space uses the hardware Local Data Store (LDS) which is 3730automatically allocated when the hardware creates work-groups of wavefronts, and 3731freed when all the wavefronts of a work-group have terminated. The data store 3732(DS) instructions can be used to access it. 3733 3734The private memory space uses the hardware scratch memory support. If the kernel 3735uses scratch, then the hardware allocates memory that is accessed using 3736wavefront lane dword (4 byte) interleaving. The mapping used from private 3737address to physical address is: 3738 3739 ``wavefront-scratch-base + 3740 (private-address * wavefront-size * 4) + 3741 (wavefront-lane-id * 4)`` 3742 3743There are different ways that the wavefront scratch base address is determined 3744by a wavefront (see :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). This 3745memory can be accessed in an interleaved manner using buffer instruction with 3746the scratch buffer descriptor and per wavefront scratch offset, by the scratch 3747instructions, or by flat instructions. If each lane of a wavefront accesses the 3748same private address, the interleaving results in adjacent dwords being accessed 3749and hence requires fewer cache lines to be fetched. Multi-dword access is not 3750supported except by flat and scratch instructions in GFX9-GFX10. 3751 3752The generic address space uses the hardware flat address support available in 3753GFX7-GFX10. This uses two fixed ranges of virtual addresses (the private and 3754local apertures), that are outside the range of addressible global memory, to 3755map from a flat address to a private or local address. 3756 3757FLAT instructions can take a flat address and access global, private (scratch) 3758and group (LDS) memory depending on if the address is within one of the 3759aperture ranges. Flat access to scratch requires hardware aperture setup and 3760setup in the kernel prologue (see 3761:ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`). Flat access to LDS requires 3762hardware aperture setup and M0 (GFX7-GFX8) register setup (see 3763:ref:`amdgpu-amdhsa-kernel-prolog-m0`). 3764 3765To convert between a segment address and a flat address the base address of the 3766apertures address can be used. For GFX7-GFX8 these are available in the 3767:ref:`amdgpu-amdhsa-hsa-aql-queue` the address of which can be obtained with 3768Queue Ptr SGPR (see :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). For 3769GFX9-GFX10 the aperture base addresses are directly available as inline constant 3770registers ``SRC_SHARED_BASE/LIMIT`` and ``SRC_PRIVATE_BASE/LIMIT``. In 64 bit 3771address mode the aperture sizes are 2^32 bytes and the base is aligned to 2^32 3772which makes it easier to convert from flat to segment or segment to flat. 3773 3774Image and Samplers 3775~~~~~~~~~~~~~~~~~~ 3776 3777Image and sample handles created by an HSA compatible runtime (see 3778:ref:`amdgpu-os`) are 64-bit addresses of a hardware 32-byte V# and 48 byte S# 3779object respectively. In order to support the HSA ``query_sampler`` operations 3780two extra dwords are used to store the HSA BRIG enumeration values for the 3781queries that are not trivially deducible from the S# representation. 3782 3783HSA Signals 3784~~~~~~~~~~~ 3785 3786HSA signal handles created by an HSA compatible runtime (see :ref:`amdgpu-os`) 3787are 64-bit addresses of a structure allocated in memory accessible from both the 3788CPU and GPU. The structure is defined by the runtime and subject to change 3789between releases. For example, see [AMD-ROCm-github]_. 3790 3791.. _amdgpu-amdhsa-hsa-aql-queue: 3792 3793HSA AQL Queue 3794~~~~~~~~~~~~~ 3795 3796The HSA AQL queue structure is defined by an HSA compatible runtime (see 3797:ref:`amdgpu-os`) and subject to change between releases. For example, see 3798[AMD-ROCm-github]_. For some processors it contains fields needed to implement 3799certain language features such as the flat address aperture bases. It also 3800contains fields used by CP such as managing the allocation of scratch memory. 3801 3802.. _amdgpu-amdhsa-kernel-descriptor: 3803 3804Kernel Descriptor 3805~~~~~~~~~~~~~~~~~ 3806 3807A kernel descriptor consists of the information needed by CP to initiate the 3808execution of a kernel, including the entry point address of the machine code 3809that implements the kernel. 3810 3811Code Object V3 Kernel Descriptor 3812++++++++++++++++++++++++++++++++ 3813 3814CP microcode requires the Kernel descriptor to be allocated on 64-byte 3815alignment. 3816 3817The fields used by CP for code objects before V3 also match those specified in 3818:ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 3819 3820 .. table:: Code Object V3 Kernel Descriptor 3821 :name: amdgpu-amdhsa-kernel-descriptor-v3-table 3822 3823 ======= ======= =============================== ============================ 3824 Bits Size Field Name Description 3825 ======= ======= =============================== ============================ 3826 31:0 4 bytes GROUP_SEGMENT_FIXED_SIZE The amount of fixed local 3827 address space memory 3828 required for a work-group 3829 in bytes. This does not 3830 include any dynamically 3831 allocated local address 3832 space memory that may be 3833 added when the kernel is 3834 dispatched. 3835 63:32 4 bytes PRIVATE_SEGMENT_FIXED_SIZE The amount of fixed 3836 private address space 3837 memory required for a 3838 work-item in bytes. 3839 Additional space may need to 3840 be added to this value if 3841 the call stack has 3842 non-inlined function calls. 3843 95:64 4 bytes KERNARG_SIZE The size of the kernarg 3844 memory pointed to by the 3845 AQL dispatch packet. The 3846 kernarg memory is used to 3847 pass arguments to the 3848 kernel. 3849 3850 * If the kernarg pointer in 3851 the dispatch packet is NULL 3852 then there are no kernel 3853 arguments. 3854 * If the kernarg pointer in 3855 the dispatch packet is 3856 not NULL and this value 3857 is 0 then the kernarg 3858 memory size is 3859 unspecified. 3860 * If the kernarg pointer in 3861 the dispatch packet is 3862 not NULL and this value 3863 is not 0 then the value 3864 specifies the kernarg 3865 memory size in bytes. It 3866 is recommended to provide 3867 a value as it may be used 3868 by CP to optimize making 3869 the kernarg memory 3870 visible to the kernel 3871 code. 3872 3873 127:96 4 bytes Reserved, must be 0. 3874 191:128 8 bytes KERNEL_CODE_ENTRY_BYTE_OFFSET Byte offset (possibly 3875 negative) from base 3876 address of kernel 3877 descriptor to kernel's 3878 entry point instruction 3879 which must be 256 byte 3880 aligned. 3881 351:272 20 Reserved, must be 0. 3882 bytes 3883 383:352 4 bytes COMPUTE_PGM_RSRC3 GFX6-GFX9 3884 Reserved, must be 0. 3885 GFX90A, GFX940 3886 Compute Shader (CS) 3887 program settings used by 3888 CP to set up 3889 ``COMPUTE_PGM_RSRC3`` 3890 configuration 3891 register. See 3892 :ref:`amdgpu-amdhsa-compute_pgm_rsrc3-gfx90a-table`. 3893 GFX10 3894 Compute Shader (CS) 3895 program settings used by 3896 CP to set up 3897 ``COMPUTE_PGM_RSRC3`` 3898 configuration 3899 register. See 3900 :ref:`amdgpu-amdhsa-compute_pgm_rsrc3-gfx10-table`. 3901 415:384 4 bytes COMPUTE_PGM_RSRC1 Compute Shader (CS) 3902 program settings used by 3903 CP to set up 3904 ``COMPUTE_PGM_RSRC1`` 3905 configuration 3906 register. See 3907 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 3908 447:416 4 bytes COMPUTE_PGM_RSRC2 Compute Shader (CS) 3909 program settings used by 3910 CP to set up 3911 ``COMPUTE_PGM_RSRC2`` 3912 configuration 3913 register. See 3914 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 3915 458:448 7 bits *See separate bits below.* Enable the setup of the 3916 SGPR user data registers 3917 (see 3918 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 3919 3920 The total number of SGPR 3921 user data registers 3922 requested must not exceed 3923 16 and match value in 3924 ``compute_pgm_rsrc2.user_sgpr.user_sgpr_count``. 3925 Any requests beyond 16 3926 will be ignored. 3927 >448 1 bit ENABLE_SGPR_PRIVATE_SEGMENT If the *Target Properties* 3928 _BUFFER column of 3929 :ref:`amdgpu-processor-table` 3930 specifies *Architected flat 3931 scratch* then not supported 3932 and must be 0, 3933 >449 1 bit ENABLE_SGPR_DISPATCH_PTR 3934 >450 1 bit ENABLE_SGPR_QUEUE_PTR 3935 >451 1 bit ENABLE_SGPR_KERNARG_SEGMENT_PTR 3936 >452 1 bit ENABLE_SGPR_DISPATCH_ID 3937 >453 1 bit ENABLE_SGPR_FLAT_SCRATCH_INIT If the *Target Properties* 3938 column of 3939 :ref:`amdgpu-processor-table` 3940 specifies *Architected flat 3941 scratch* then not supported 3942 and must be 0, 3943 >454 1 bit ENABLE_SGPR_PRIVATE_SEGMENT 3944 _SIZE 3945 457:455 3 bits Reserved, must be 0. 3946 458 1 bit ENABLE_WAVEFRONT_SIZE32 GFX6-GFX9 3947 Reserved, must be 0. 3948 GFX10 3949 - If 0 execute in 3950 wavefront size 64 mode. 3951 - If 1 execute in 3952 native wavefront size 3953 32 mode. 3954 463:459 1 bit Reserved, must be 0. 3955 464 1 bit RESERVED_464 Deprecated, must be 0. 3956 467:465 3 bits Reserved, must be 0. 3957 468 1 bit RESERVED_468 Deprecated, must be 0. 3958 469:471 3 bits Reserved, must be 0. 3959 511:472 5 bytes Reserved, must be 0. 3960 512 **Total size 64 bytes.** 3961 ======= ==================================================================== 3962 3963.. 3964 3965 .. table:: compute_pgm_rsrc1 for GFX6-GFX10 3966 :name: amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table 3967 3968 ======= ======= =============================== =========================================================================== 3969 Bits Size Field Name Description 3970 ======= ======= =============================== =========================================================================== 3971 5:0 6 bits GRANULATED_WORKITEM_VGPR_COUNT Number of vector register 3972 blocks used by each work-item; 3973 granularity is device 3974 specific: 3975 3976 GFX6-GFX9 3977 - vgprs_used 0..256 3978 - max(0, ceil(vgprs_used / 4) - 1) 3979 GFX90A, GFX940 3980 - vgprs_used 0..512 3981 - vgprs_used = align(arch_vgprs, 4) 3982 + acc_vgprs 3983 - max(0, ceil(vgprs_used / 8) - 1) 3984 GFX10 (wavefront size 64) 3985 - max_vgpr 1..256 3986 - max(0, ceil(vgprs_used / 4) - 1) 3987 GFX10 (wavefront size 32) 3988 - max_vgpr 1..256 3989 - max(0, ceil(vgprs_used / 8) - 1) 3990 3991 Where vgprs_used is defined 3992 as the highest VGPR number 3993 explicitly referenced plus 3994 one. 3995 3996 Used by CP to set up 3997 ``COMPUTE_PGM_RSRC1.VGPRS``. 3998 3999 The 4000 :ref:`amdgpu-assembler` 4001 calculates this 4002 automatically for the 4003 selected processor from 4004 values provided to the 4005 `.amdhsa_kernel` directive 4006 by the 4007 `.amdhsa_next_free_vgpr` 4008 nested directive (see 4009 :ref:`amdhsa-kernel-directives-table`). 4010 9:6 4 bits GRANULATED_WAVEFRONT_SGPR_COUNT Number of scalar register 4011 blocks used by a wavefront; 4012 granularity is device 4013 specific: 4014 4015 GFX6-GFX8 4016 - sgprs_used 0..112 4017 - max(0, ceil(sgprs_used / 8) - 1) 4018 GFX9 4019 - sgprs_used 0..112 4020 - 2 * max(0, ceil(sgprs_used / 16) - 1) 4021 GFX10 4022 Reserved, must be 0. 4023 (128 SGPRs always 4024 allocated.) 4025 4026 Where sgprs_used is 4027 defined as the highest 4028 SGPR number explicitly 4029 referenced plus one, plus 4030 a target specific number 4031 of additional special 4032 SGPRs for VCC, 4033 FLAT_SCRATCH (GFX7+) and 4034 XNACK_MASK (GFX8+), and 4035 any additional 4036 target specific 4037 limitations. It does not 4038 include the 16 SGPRs added 4039 if a trap handler is 4040 enabled. 4041 4042 The target specific 4043 limitations and special 4044 SGPR layout are defined in 4045 the hardware 4046 documentation, which can 4047 be found in the 4048 :ref:`amdgpu-processors` 4049 table. 4050 4051 Used by CP to set up 4052 ``COMPUTE_PGM_RSRC1.SGPRS``. 4053 4054 The 4055 :ref:`amdgpu-assembler` 4056 calculates this 4057 automatically for the 4058 selected processor from 4059 values provided to the 4060 `.amdhsa_kernel` directive 4061 by the 4062 `.amdhsa_next_free_sgpr` 4063 and `.amdhsa_reserve_*` 4064 nested directives (see 4065 :ref:`amdhsa-kernel-directives-table`). 4066 11:10 2 bits PRIORITY Must be 0. 4067 4068 Start executing wavefront 4069 at the specified priority. 4070 4071 CP is responsible for 4072 filling in 4073 ``COMPUTE_PGM_RSRC1.PRIORITY``. 4074 13:12 2 bits FLOAT_ROUND_MODE_32 Wavefront starts execution 4075 with specified rounding 4076 mode for single (32 4077 bit) floating point 4078 precision floating point 4079 operations. 4080 4081 Floating point rounding 4082 mode values are defined in 4083 :ref:`amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table`. 4084 4085 Used by CP to set up 4086 ``COMPUTE_PGM_RSRC1.FLOAT_MODE``. 4087 15:14 2 bits FLOAT_ROUND_MODE_16_64 Wavefront starts execution 4088 with specified rounding 4089 denorm mode for half/double (16 4090 and 64-bit) floating point 4091 precision floating point 4092 operations. 4093 4094 Floating point rounding 4095 mode values are defined in 4096 :ref:`amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table`. 4097 4098 Used by CP to set up 4099 ``COMPUTE_PGM_RSRC1.FLOAT_MODE``. 4100 17:16 2 bits FLOAT_DENORM_MODE_32 Wavefront starts execution 4101 with specified denorm mode 4102 for single (32 4103 bit) floating point 4104 precision floating point 4105 operations. 4106 4107 Floating point denorm mode 4108 values are defined in 4109 :ref:`amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table`. 4110 4111 Used by CP to set up 4112 ``COMPUTE_PGM_RSRC1.FLOAT_MODE``. 4113 19:18 2 bits FLOAT_DENORM_MODE_16_64 Wavefront starts execution 4114 with specified denorm mode 4115 for half/double (16 4116 and 64-bit) floating point 4117 precision floating point 4118 operations. 4119 4120 Floating point denorm mode 4121 values are defined in 4122 :ref:`amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table`. 4123 4124 Used by CP to set up 4125 ``COMPUTE_PGM_RSRC1.FLOAT_MODE``. 4126 20 1 bit PRIV Must be 0. 4127 4128 Start executing wavefront 4129 in privilege trap handler 4130 mode. 4131 4132 CP is responsible for 4133 filling in 4134 ``COMPUTE_PGM_RSRC1.PRIV``. 4135 21 1 bit ENABLE_DX10_CLAMP Wavefront starts execution 4136 with DX10 clamp mode 4137 enabled. Used by the vector 4138 ALU to force DX10 style 4139 treatment of NaN's (when 4140 set, clamp NaN to zero, 4141 otherwise pass NaN 4142 through). 4143 4144 Used by CP to set up 4145 ``COMPUTE_PGM_RSRC1.DX10_CLAMP``. 4146 22 1 bit DEBUG_MODE Must be 0. 4147 4148 Start executing wavefront 4149 in single step mode. 4150 4151 CP is responsible for 4152 filling in 4153 ``COMPUTE_PGM_RSRC1.DEBUG_MODE``. 4154 23 1 bit ENABLE_IEEE_MODE Wavefront starts execution 4155 with IEEE mode 4156 enabled. Floating point 4157 opcodes that support 4158 exception flag gathering 4159 will quiet and propagate 4160 signaling-NaN inputs per 4161 IEEE 754-2008. Min_dx10 and 4162 max_dx10 become IEEE 4163 754-2008 compliant due to 4164 signaling-NaN propagation 4165 and quieting. 4166 4167 Used by CP to set up 4168 ``COMPUTE_PGM_RSRC1.IEEE_MODE``. 4169 24 1 bit BULKY Must be 0. 4170 4171 Only one work-group allowed 4172 to execute on a compute 4173 unit. 4174 4175 CP is responsible for 4176 filling in 4177 ``COMPUTE_PGM_RSRC1.BULKY``. 4178 25 1 bit CDBG_USER Must be 0. 4179 4180 Flag that can be used to 4181 control debugging code. 4182 4183 CP is responsible for 4184 filling in 4185 ``COMPUTE_PGM_RSRC1.CDBG_USER``. 4186 26 1 bit FP16_OVFL GFX6-GFX8 4187 Reserved, must be 0. 4188 GFX9-GFX10 4189 Wavefront starts execution 4190 with specified fp16 overflow 4191 mode. 4192 4193 - If 0, fp16 overflow generates 4194 +/-INF values. 4195 - If 1, fp16 overflow that is the 4196 result of an +/-INF input value 4197 or divide by 0 produces a +/-INF, 4198 otherwise clamps computed 4199 overflow to +/-MAX_FP16 as 4200 appropriate. 4201 4202 Used by CP to set up 4203 ``COMPUTE_PGM_RSRC1.FP16_OVFL``. 4204 28:27 2 bits Reserved, must be 0. 4205 29 1 bit WGP_MODE GFX6-GFX9 4206 Reserved, must be 0. 4207 GFX10 4208 - If 0 execute work-groups in 4209 CU wavefront execution mode. 4210 - If 1 execute work-groups on 4211 in WGP wavefront execution mode. 4212 4213 See :ref:`amdgpu-amdhsa-memory-model`. 4214 4215 Used by CP to set up 4216 ``COMPUTE_PGM_RSRC1.WGP_MODE``. 4217 30 1 bit MEM_ORDERED GFX6-GFX9 4218 Reserved, must be 0. 4219 GFX10 4220 Controls the behavior of the 4221 s_waitcnt's vmcnt and vscnt 4222 counters. 4223 4224 - If 0 vmcnt reports completion 4225 of load and atomic with return 4226 out of order with sample 4227 instructions, and the vscnt 4228 reports the completion of 4229 store and atomic without 4230 return in order. 4231 - If 1 vmcnt reports completion 4232 of load, atomic with return 4233 and sample instructions in 4234 order, and the vscnt reports 4235 the completion of store and 4236 atomic without return in order. 4237 4238 Used by CP to set up 4239 ``COMPUTE_PGM_RSRC1.MEM_ORDERED``. 4240 31 1 bit FWD_PROGRESS GFX6-GFX9 4241 Reserved, must be 0. 4242 GFX10 4243 - If 0 execute SIMD wavefronts 4244 using oldest first policy. 4245 - If 1 execute SIMD wavefronts to 4246 ensure wavefronts will make some 4247 forward progress. 4248 4249 Used by CP to set up 4250 ``COMPUTE_PGM_RSRC1.FWD_PROGRESS``. 4251 32 **Total size 4 bytes** 4252 ======= =================================================================================================================== 4253 4254.. 4255 4256 .. table:: compute_pgm_rsrc2 for GFX6-GFX10 4257 :name: amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table 4258 4259 ======= ======= =============================== =========================================================================== 4260 Bits Size Field Name Description 4261 ======= ======= =============================== =========================================================================== 4262 0 1 bit ENABLE_PRIVATE_SEGMENT * Enable the setup of the 4263 private segment. 4264 * If the *Target Properties* 4265 column of 4266 :ref:`amdgpu-processor-table` 4267 does not specify 4268 *Architected flat 4269 scratch* then enable the 4270 setup of the SGPR 4271 wavefront scratch offset 4272 system register (see 4273 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4274 * If the *Target Properties* 4275 column of 4276 :ref:`amdgpu-processor-table` 4277 specifies *Architected 4278 flat scratch* then enable 4279 the setup of the 4280 FLAT_SCRATCH register 4281 pair (see 4282 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4283 4284 Used by CP to set up 4285 ``COMPUTE_PGM_RSRC2.SCRATCH_EN``. 4286 5:1 5 bits USER_SGPR_COUNT The total number of SGPR 4287 user data 4288 registers requested. This 4289 number must be greater than 4290 or equal to the number of user 4291 data registers enabled. 4292 4293 Used by CP to set up 4294 ``COMPUTE_PGM_RSRC2.USER_SGPR``. 4295 6 1 bit ENABLE_TRAP_HANDLER Must be 0. 4296 4297 This bit represents 4298 ``COMPUTE_PGM_RSRC2.TRAP_PRESENT``, 4299 which is set by the CP if 4300 the runtime has installed a 4301 trap handler. 4302 7 1 bit ENABLE_SGPR_WORKGROUP_ID_X Enable the setup of the 4303 system SGPR register for 4304 the work-group id in the X 4305 dimension (see 4306 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4307 4308 Used by CP to set up 4309 ``COMPUTE_PGM_RSRC2.TGID_X_EN``. 4310 8 1 bit ENABLE_SGPR_WORKGROUP_ID_Y Enable the setup of the 4311 system SGPR register for 4312 the work-group id in the Y 4313 dimension (see 4314 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4315 4316 Used by CP to set up 4317 ``COMPUTE_PGM_RSRC2.TGID_Y_EN``. 4318 9 1 bit ENABLE_SGPR_WORKGROUP_ID_Z Enable the setup of the 4319 system SGPR register for 4320 the work-group id in the Z 4321 dimension (see 4322 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4323 4324 Used by CP to set up 4325 ``COMPUTE_PGM_RSRC2.TGID_Z_EN``. 4326 10 1 bit ENABLE_SGPR_WORKGROUP_INFO Enable the setup of the 4327 system SGPR register for 4328 work-group information (see 4329 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4330 4331 Used by CP to set up 4332 ``COMPUTE_PGM_RSRC2.TGID_SIZE_EN``. 4333 12:11 2 bits ENABLE_VGPR_WORKITEM_ID Enable the setup of the 4334 VGPR system registers used 4335 for the work-item ID. 4336 :ref:`amdgpu-amdhsa-system-vgpr-work-item-id-enumeration-values-table` 4337 defines the values. 4338 4339 Used by CP to set up 4340 ``COMPUTE_PGM_RSRC2.TIDIG_CMP_CNT``. 4341 13 1 bit ENABLE_EXCEPTION_ADDRESS_WATCH Must be 0. 4342 4343 Wavefront starts execution 4344 with address watch 4345 exceptions enabled which 4346 are generated when L1 has 4347 witnessed a thread access 4348 an *address of 4349 interest*. 4350 4351 CP is responsible for 4352 filling in the address 4353 watch bit in 4354 ``COMPUTE_PGM_RSRC2.EXCP_EN_MSB`` 4355 according to what the 4356 runtime requests. 4357 14 1 bit ENABLE_EXCEPTION_MEMORY Must be 0. 4358 4359 Wavefront starts execution 4360 with memory violation 4361 exceptions exceptions 4362 enabled which are generated 4363 when a memory violation has 4364 occurred for this wavefront from 4365 L1 or LDS 4366 (write-to-read-only-memory, 4367 mis-aligned atomic, LDS 4368 address out of range, 4369 illegal address, etc.). 4370 4371 CP sets the memory 4372 violation bit in 4373 ``COMPUTE_PGM_RSRC2.EXCP_EN_MSB`` 4374 according to what the 4375 runtime requests. 4376 23:15 9 bits GRANULATED_LDS_SIZE Must be 0. 4377 4378 CP uses the rounded value 4379 from the dispatch packet, 4380 not this value, as the 4381 dispatch may contain 4382 dynamically allocated group 4383 segment memory. CP writes 4384 directly to 4385 ``COMPUTE_PGM_RSRC2.LDS_SIZE``. 4386 4387 Amount of group segment 4388 (LDS) to allocate for each 4389 work-group. Granularity is 4390 device specific: 4391 4392 GFX6 4393 roundup(lds-size / (64 * 4)) 4394 GFX7-GFX10 4395 roundup(lds-size / (128 * 4)) 4396 4397 24 1 bit ENABLE_EXCEPTION_IEEE_754_FP Wavefront starts execution 4398 _INVALID_OPERATION with specified exceptions 4399 enabled. 4400 4401 Used by CP to set up 4402 ``COMPUTE_PGM_RSRC2.EXCP_EN`` 4403 (set from bits 0..6). 4404 4405 IEEE 754 FP Invalid 4406 Operation 4407 25 1 bit ENABLE_EXCEPTION_FP_DENORMAL FP Denormal one or more 4408 _SOURCE input operands is a 4409 denormal number 4410 26 1 bit ENABLE_EXCEPTION_IEEE_754_FP IEEE 754 FP Division by 4411 _DIVISION_BY_ZERO Zero 4412 27 1 bit ENABLE_EXCEPTION_IEEE_754_FP IEEE 754 FP FP Overflow 4413 _OVERFLOW 4414 28 1 bit ENABLE_EXCEPTION_IEEE_754_FP IEEE 754 FP Underflow 4415 _UNDERFLOW 4416 29 1 bit ENABLE_EXCEPTION_IEEE_754_FP IEEE 754 FP Inexact 4417 _INEXACT 4418 30 1 bit ENABLE_EXCEPTION_INT_DIVIDE_BY Integer Division by Zero 4419 _ZERO (rcp_iflag_f32 instruction 4420 only) 4421 31 1 bit Reserved, must be 0. 4422 32 **Total size 4 bytes.** 4423 ======= =================================================================================================================== 4424 4425.. 4426 4427 .. table:: compute_pgm_rsrc3 for GFX90A, GFX940 4428 :name: amdgpu-amdhsa-compute_pgm_rsrc3-gfx90a-table 4429 4430 ======= ======= =============================== =========================================================================== 4431 Bits Size Field Name Description 4432 ======= ======= =============================== =========================================================================== 4433 5:0 6 bits ACCUM_OFFSET Offset of a first AccVGPR in the unified register file. Granularity 4. 4434 Value 0-63. 0 - accum-offset = 4, 1 - accum-offset = 8, ..., 4435 63 - accum-offset = 256. 4436 6:15 10 Reserved, must be 0. 4437 bits 4438 16 1 bit TG_SPLIT - If 0 the waves of a work-group are 4439 launched in the same CU. 4440 - If 1 the waves of a work-group can be 4441 launched in different CUs. The waves 4442 cannot use S_BARRIER or LDS. 4443 17:31 15 Reserved, must be 0. 4444 bits 4445 32 **Total size 4 bytes.** 4446 ======= =================================================================================================================== 4447 4448.. 4449 4450 .. table:: compute_pgm_rsrc3 for GFX10 4451 :name: amdgpu-amdhsa-compute_pgm_rsrc3-gfx10-table 4452 4453 ======= ======= =============================== =========================================================================== 4454 Bits Size Field Name Description 4455 ======= ======= =============================== =========================================================================== 4456 3:0 4 bits SHARED_VGPR_COUNT Number of shared VGPR blocks when executing in subvector mode. For 4457 wavefront size 64 the value is 0-15, representing 0-120 VGPRs (granularity 4458 of 8), such that (compute_pgm_rsrc1.vgprs +1)*4 + shared_vgpr_count*8 does 4459 not exceed 256. For wavefront size 32 shared_vgpr_count must be 0. 4460 31:4 28 Reserved, must be 0. 4461 bits 4462 32 **Total size 4 bytes.** 4463 ======= =================================================================================================================== 4464 4465.. 4466 4467 .. table:: Floating Point Rounding Mode Enumeration Values 4468 :name: amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table 4469 4470 ====================================== ===== ============================== 4471 Enumeration Name Value Description 4472 ====================================== ===== ============================== 4473 FLOAT_ROUND_MODE_NEAR_EVEN 0 Round Ties To Even 4474 FLOAT_ROUND_MODE_PLUS_INFINITY 1 Round Toward +infinity 4475 FLOAT_ROUND_MODE_MINUS_INFINITY 2 Round Toward -infinity 4476 FLOAT_ROUND_MODE_ZERO 3 Round Toward 0 4477 ====================================== ===== ============================== 4478 4479.. 4480 4481 .. table:: Floating Point Denorm Mode Enumeration Values 4482 :name: amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table 4483 4484 ====================================== ===== ============================== 4485 Enumeration Name Value Description 4486 ====================================== ===== ============================== 4487 FLOAT_DENORM_MODE_FLUSH_SRC_DST 0 Flush Source and Destination 4488 Denorms 4489 FLOAT_DENORM_MODE_FLUSH_DST 1 Flush Output Denorms 4490 FLOAT_DENORM_MODE_FLUSH_SRC 2 Flush Source Denorms 4491 FLOAT_DENORM_MODE_FLUSH_NONE 3 No Flush 4492 ====================================== ===== ============================== 4493 4494.. 4495 4496 .. table:: System VGPR Work-Item ID Enumeration Values 4497 :name: amdgpu-amdhsa-system-vgpr-work-item-id-enumeration-values-table 4498 4499 ======================================== ===== ============================ 4500 Enumeration Name Value Description 4501 ======================================== ===== ============================ 4502 SYSTEM_VGPR_WORKITEM_ID_X 0 Set work-item X dimension 4503 ID. 4504 SYSTEM_VGPR_WORKITEM_ID_X_Y 1 Set work-item X and Y 4505 dimensions ID. 4506 SYSTEM_VGPR_WORKITEM_ID_X_Y_Z 2 Set work-item X, Y and Z 4507 dimensions ID. 4508 SYSTEM_VGPR_WORKITEM_ID_UNDEFINED 3 Undefined. 4509 ======================================== ===== ============================ 4510 4511.. _amdgpu-amdhsa-initial-kernel-execution-state: 4512 4513Initial Kernel Execution State 4514~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 4515 4516This section defines the register state that will be set up by the packet 4517processor prior to the start of execution of every wavefront. This is limited by 4518the constraints of the hardware controllers of CP/ADC/SPI. 4519 4520The order of the SGPR registers is defined, but the compiler can specify which 4521ones are actually setup in the kernel descriptor using the ``enable_sgpr_*`` bit 4522fields (see :ref:`amdgpu-amdhsa-kernel-descriptor`). The register numbers used 4523for enabled registers are dense starting at SGPR0: the first enabled register is 4524SGPR0, the next enabled register is SGPR1 etc.; disabled registers do not have 4525an SGPR number. 4526 4527The initial SGPRs comprise up to 16 User SRGPs that are set by CP and apply to 4528all wavefronts of the grid. It is possible to specify more than 16 User SGPRs 4529using the ``enable_sgpr_*`` bit fields, in which case only the first 16 are 4530actually initialized. These are then immediately followed by the System SGPRs 4531that are set up by ADC/SPI and can have different values for each wavefront of 4532the grid dispatch. 4533 4534SGPR register initial state is defined in 4535:ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 4536 4537 .. table:: SGPR Register Set Up Order 4538 :name: amdgpu-amdhsa-sgpr-register-set-up-order-table 4539 4540 ========== ========================== ====== ============================== 4541 SGPR Order Name Number Description 4542 (kernel descriptor enable of 4543 field) SGPRs 4544 ========== ========================== ====== ============================== 4545 First Private Segment Buffer 4 See 4546 (enable_sgpr_private :ref:`amdgpu-amdhsa-kernel-prolog-private-segment-buffer`. 4547 _segment_buffer) 4548 then Dispatch Ptr 2 64-bit address of AQL dispatch 4549 (enable_sgpr_dispatch_ptr) packet for kernel dispatch 4550 actually executing. 4551 then Queue Ptr 2 64-bit address of amd_queue_t 4552 (enable_sgpr_queue_ptr) object for AQL queue on which 4553 the dispatch packet was 4554 queued. 4555 then Kernarg Segment Ptr 2 64-bit address of Kernarg 4556 (enable_sgpr_kernarg segment. This is directly 4557 _segment_ptr) copied from the 4558 kernarg_address in the kernel 4559 dispatch packet. 4560 4561 Having CP load it once avoids 4562 loading it at the beginning of 4563 every wavefront. 4564 then Dispatch Id 2 64-bit Dispatch ID of the 4565 (enable_sgpr_dispatch_id) dispatch packet being 4566 executed. 4567 then Flat Scratch Init 2 See 4568 (enable_sgpr_flat_scratch :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`. 4569 _init) 4570 then Private Segment Size 1 The 32-bit byte size of a 4571 (enable_sgpr_private single work-item's memory 4572 _segment_size) allocation. This is the 4573 value from the kernel 4574 dispatch packet Private 4575 Segment Byte Size rounded up 4576 by CP to a multiple of 4577 DWORD. 4578 4579 Having CP load it once avoids 4580 loading it at the beginning of 4581 every wavefront. 4582 4583 This is not used for 4584 GFX7-GFX8 since it is the same 4585 value as the second SGPR of 4586 Flat Scratch Init. However, it 4587 may be needed for GFX9-GFX10 which 4588 changes the meaning of the 4589 Flat Scratch Init value. 4590 then Work-Group Id X 1 32-bit work-group id in X 4591 (enable_sgpr_workgroup_id dimension of grid for 4592 _X) wavefront. 4593 then Work-Group Id Y 1 32-bit work-group id in Y 4594 (enable_sgpr_workgroup_id dimension of grid for 4595 _Y) wavefront. 4596 then Work-Group Id Z 1 32-bit work-group id in Z 4597 (enable_sgpr_workgroup_id dimension of grid for 4598 _Z) wavefront. 4599 then Work-Group Info 1 {first_wavefront, 14'b0000, 4600 (enable_sgpr_workgroup ordered_append_term[10:0], 4601 _info) threadgroup_size_in_wavefronts[5:0]} 4602 then Scratch Wavefront Offset 1 See 4603 (enable_sgpr_private :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`. 4604 _segment_wavefront_offset) and 4605 :ref:`amdgpu-amdhsa-kernel-prolog-private-segment-buffer`. 4606 ========== ========================== ====== ============================== 4607 4608The order of the VGPR registers is defined, but the compiler can specify which 4609ones are actually setup in the kernel descriptor using the ``enable_vgpr*`` bit 4610fields (see :ref:`amdgpu-amdhsa-kernel-descriptor`). The register numbers used 4611for enabled registers are dense starting at VGPR0: the first enabled register is 4612VGPR0, the next enabled register is VGPR1 etc.; disabled registers do not have a 4613VGPR number. 4614 4615There are different methods used for the VGPR initial state: 4616 4617* Unless the *Target Properties* column of :ref:`amdgpu-processor-table` 4618 specifies otherwise, a separate VGPR register is used per work-item ID. The 4619 VGPR register initial state for this method is defined in 4620 :ref:`amdgpu-amdhsa-vgpr-register-set-up-order-for-unpacked-work-item-id-method-table`. 4621* If *Target Properties* column of :ref:`amdgpu-processor-table` 4622 specifies *Packed work-item IDs*, the initial value of VGPR0 register is used 4623 for all work-item IDs. The register layout for this method is defined in 4624 :ref:`amdgpu-amdhsa-register-layout-for-packed-work-item-id-method-table`. 4625 4626 .. table:: VGPR Register Set Up Order for Unpacked Work-Item ID Method 4627 :name: amdgpu-amdhsa-vgpr-register-set-up-order-for-unpacked-work-item-id-method-table 4628 4629 ========== ========================== ====== ============================== 4630 VGPR Order Name Number Description 4631 (kernel descriptor enable of 4632 field) VGPRs 4633 ========== ========================== ====== ============================== 4634 First Work-Item Id X 1 32-bit work-item id in X 4635 (Always initialized) dimension of work-group for 4636 wavefront lane. 4637 then Work-Item Id Y 1 32-bit work-item id in Y 4638 (enable_vgpr_workitem_id dimension of work-group for 4639 > 0) wavefront lane. 4640 then Work-Item Id Z 1 32-bit work-item id in Z 4641 (enable_vgpr_workitem_id dimension of work-group for 4642 > 1) wavefront lane. 4643 ========== ========================== ====== ============================== 4644 4645.. 4646 4647 .. table:: Register Layout for Packed Work-Item ID Method 4648 :name: amdgpu-amdhsa-register-layout-for-packed-work-item-id-method-table 4649 4650 ======= ======= ================ ========================================= 4651 Bits Size Field Name Description 4652 ======= ======= ================ ========================================= 4653 0:9 10 bits Work-Item Id X Work-item id in X 4654 dimension of work-group for 4655 wavefront lane. 4656 4657 Always initialized. 4658 4659 10:19 10 bits Work-Item Id Y Work-item id in Y 4660 dimension of work-group for 4661 wavefront lane. 4662 4663 Initialized if enable_vgpr_workitem_id > 4664 0, otherwise set to 0. 4665 20:29 10 bits Work-Item Id Z Work-item id in Z 4666 dimension of work-group for 4667 wavefront lane. 4668 4669 Initialized if enable_vgpr_workitem_id > 4670 1, otherwise set to 0. 4671 30:31 2 bits Reserved, set to 0. 4672 ======= ======= ================ ========================================= 4673 4674The setting of registers is done by GPU CP/ADC/SPI hardware as follows: 4675 46761. SGPRs before the Work-Group Ids are set by CP using the 16 User Data 4677 registers. 46782. Work-group Id registers X, Y, Z are set by ADC which supports any 4679 combination including none. 46803. Scratch Wavefront Offset is set by SPI in a per wavefront basis which is why 4681 its value cannot be included with the flat scratch init value which is per 4682 queue (see :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`). 46834. The VGPRs are set by SPI which only supports specifying either (X), (X, Y) 4684 or (X, Y, Z). 46855. Flat Scratch register pair initialization is described in 4686 :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`. 4687 4688The global segment can be accessed either using buffer instructions (GFX6 which 4689has V# 64-bit address support), flat instructions (GFX7-GFX10), or global 4690instructions (GFX9-GFX10). 4691 4692If buffer operations are used, then the compiler can generate a V# with the 4693following properties: 4694 4695* base address of 0 4696* no swizzle 4697* ATC: 1 if IOMMU present (such as APU) 4698* ptr64: 1 4699* MTYPE set to support memory coherence that matches the runtime (such as CC for 4700 APU and NC for dGPU). 4701 4702.. _amdgpu-amdhsa-kernel-prolog: 4703 4704Kernel Prolog 4705~~~~~~~~~~~~~ 4706 4707The compiler performs initialization in the kernel prologue depending on the 4708target and information about things like stack usage in the kernel and called 4709functions. Some of this initialization requires the compiler to request certain 4710User and System SGPRs be present in the 4711:ref:`amdgpu-amdhsa-initial-kernel-execution-state` via the 4712:ref:`amdgpu-amdhsa-kernel-descriptor`. 4713 4714.. _amdgpu-amdhsa-kernel-prolog-cfi: 4715 4716CFI 4717+++ 4718 47191. The CFI return address is undefined. 4720 47212. The CFI CFA is defined using an expression which evaluates to a location 4722 description that comprises one memory location description for the 4723 ``DW_ASPACE_AMDGPU_private_lane`` address space address ``0``. 4724 4725.. _amdgpu-amdhsa-kernel-prolog-m0: 4726 4727M0 4728++ 4729 4730GFX6-GFX8 4731 The M0 register must be initialized with a value at least the total LDS size 4732 if the kernel may access LDS via DS or flat operations. Total LDS size is 4733 available in dispatch packet. For M0, it is also possible to use maximum 4734 possible value of LDS for given target (0x7FFF for GFX6 and 0xFFFF for 4735 GFX7-GFX8). 4736GFX9-GFX10 4737 The M0 register is not used for range checking LDS accesses and so does not 4738 need to be initialized in the prolog. 4739 4740.. _amdgpu-amdhsa-kernel-prolog-stack-pointer: 4741 4742Stack Pointer 4743+++++++++++++ 4744 4745If the kernel has function calls it must set up the ABI stack pointer described 4746in :ref:`amdgpu-amdhsa-function-call-convention-non-kernel-functions` by setting 4747SGPR32 to the unswizzled scratch offset of the address past the last local 4748allocation. 4749 4750.. _amdgpu-amdhsa-kernel-prolog-frame-pointer: 4751 4752Frame Pointer 4753+++++++++++++ 4754 4755If the kernel needs a frame pointer for the reasons defined in 4756``SIFrameLowering`` then SGPR33 is used and is always set to ``0`` in the 4757kernel prolog. If a frame pointer is not required then all uses of the frame 4758pointer are replaced with immediate ``0`` offsets. 4759 4760.. _amdgpu-amdhsa-kernel-prolog-flat-scratch: 4761 4762Flat Scratch 4763++++++++++++ 4764 4765There are different methods used for initializing flat scratch: 4766 4767* If the *Target Properties* column of :ref:`amdgpu-processor-table` 4768 specifies *Does not support generic address space*: 4769 4770 Flat scratch is not supported and there is no flat scratch register pair. 4771 4772* If the *Target Properties* column of :ref:`amdgpu-processor-table` 4773 specifies *Offset flat scratch*: 4774 4775 If the kernel or any function it calls may use flat operations to access 4776 scratch memory, the prolog code must set up the FLAT_SCRATCH register pair 4777 (FLAT_SCRATCH_LO/FLAT_SCRATCH_HI). Initialization uses Flat Scratch Init and 4778 Scratch Wavefront Offset SGPR registers (see 4779 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`): 4780 4781 1. The low word of Flat Scratch Init is the 32-bit byte offset from 4782 ``SH_HIDDEN_PRIVATE_BASE_VIMID`` to the base of scratch backing memory 4783 being managed by SPI for the queue executing the kernel dispatch. This is 4784 the same value used in the Scratch Segment Buffer V# base address. 4785 4786 CP obtains this from the runtime. (The Scratch Segment Buffer base address 4787 is ``SH_HIDDEN_PRIVATE_BASE_VIMID`` plus this offset.) 4788 4789 The prolog must add the value of Scratch Wavefront Offset to get the 4790 wavefront's byte scratch backing memory offset from 4791 ``SH_HIDDEN_PRIVATE_BASE_VIMID``. 4792 4793 The Scratch Wavefront Offset must also be used as an offset with Private 4794 segment address when using the Scratch Segment Buffer. 4795 4796 Since FLAT_SCRATCH_LO is in units of 256 bytes, the offset must be right 4797 shifted by 8 before moving into FLAT_SCRATCH_HI. 4798 4799 FLAT_SCRATCH_HI corresponds to SGPRn-4 on GFX7, and SGPRn-6 on GFX8 (where 4800 SGPRn is the highest numbered SGPR allocated to the wavefront). 4801 FLAT_SCRATCH_HI is multiplied by 256 (as it is in units of 256 bytes) and 4802 added to ``SH_HIDDEN_PRIVATE_BASE_VIMID`` to calculate the per wavefront 4803 FLAT SCRATCH BASE in flat memory instructions that access the scratch 4804 aperture. 4805 2. The second word of Flat Scratch Init is 32-bit byte size of a single 4806 work-items scratch memory usage. 4807 4808 CP obtains this from the runtime, and it is always a multiple of DWORD. CP 4809 checks that the value in the kernel dispatch packet Private Segment Byte 4810 Size is not larger and requests the runtime to increase the queue's scratch 4811 size if necessary. 4812 4813 CP directly loads from the kernel dispatch packet Private Segment Byte Size 4814 field and rounds up to a multiple of DWORD. Having CP load it once avoids 4815 loading it at the beginning of every wavefront. 4816 4817 The kernel prolog code must move it to FLAT_SCRATCH_LO which is SGPRn-3 on 4818 GFX7 and SGPRn-5 on GFX8. FLAT_SCRATCH_LO is used as the FLAT SCRATCH SIZE 4819 in flat memory instructions. 4820 4821* If the *Target Properties* column of :ref:`amdgpu-processor-table` 4822 specifies *Absolute flat scratch*: 4823 4824 If the kernel or any function it calls may use flat operations to access 4825 scratch memory, the prolog code must set up the FLAT_SCRATCH register pair 4826 (FLAT_SCRATCH_LO/FLAT_SCRATCH_HI which are in SGPRn-4/SGPRn-3). Initialization 4827 uses Flat Scratch Init and Scratch Wavefront Offset SGPR registers (see 4828 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`): 4829 4830 The Flat Scratch Init is the 64-bit address of the base of scratch backing 4831 memory being managed by SPI for the queue executing the kernel dispatch. 4832 4833 CP obtains this from the runtime. 4834 4835 The kernel prolog must add the value of the wave's Scratch Wavefront Offset 4836 and move the result as a 64-bit value to the FLAT_SCRATCH SGPR register pair 4837 which is SGPRn-6 and SGPRn-5. It is used as the FLAT SCRATCH BASE in flat 4838 memory instructions. 4839 4840 The Scratch Wavefront Offset must also be used as an offset with Private 4841 segment address when using the Scratch Segment Buffer (see 4842 :ref:`amdgpu-amdhsa-kernel-prolog-private-segment-buffer`). 4843 4844* If the *Target Properties* column of :ref:`amdgpu-processor-table` 4845 specifies *Architected flat scratch*: 4846 4847 If ENABLE_PRIVATE_SEGMENT is enabled in 4848 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table` then the FLAT_SCRATCH 4849 register pair will be initialized to the 64-bit address of the base of scratch 4850 backing memory being managed by SPI for the queue executing the kernel 4851 dispatch plus the value of the wave's Scratch Wavefront Offset for use as the 4852 flat scratch base in flat memory instructions. 4853 4854.. _amdgpu-amdhsa-kernel-prolog-private-segment-buffer: 4855 4856Private Segment Buffer 4857++++++++++++++++++++++ 4858 4859If the *Target Properties* column of :ref:`amdgpu-processor-table` specifies 4860*Architected flat scratch* then a Private Segment Buffer is not supported. 4861Instead the flat SCRATCH instructions are used. 4862 4863Otherwise, Private Segment Buffer SGPR register is used to initialize 4 SGPRs 4864that are used as a V# to access scratch. CP uses the value provided by the 4865runtime. It is used, together with Scratch Wavefront Offset as an offset, to 4866access the private memory space using a segment address. See 4867:ref:`amdgpu-amdhsa-initial-kernel-execution-state`. 4868 4869The scratch V# is a four-aligned SGPR and always selected for the kernel as 4870follows: 4871 4872 - If it is known during instruction selection that there is stack usage, 4873 SGPR0-3 is reserved for use as the scratch V#. Stack usage is assumed if 4874 optimizations are disabled (``-O0``), if stack objects already exist (for 4875 locals, etc.), or if there are any function calls. 4876 4877 - Otherwise, four high numbered SGPRs beginning at a four-aligned SGPR index 4878 are reserved for the tentative scratch V#. These will be used if it is 4879 determined that spilling is needed. 4880 4881 - If no use is made of the tentative scratch V#, then it is unreserved, 4882 and the register count is determined ignoring it. 4883 - If use is made of the tentative scratch V#, then its register numbers 4884 are shifted to the first four-aligned SGPR index after the highest one 4885 allocated by the register allocator, and all uses are updated. The 4886 register count includes them in the shifted location. 4887 - In either case, if the processor has the SGPR allocation bug, the 4888 tentative allocation is not shifted or unreserved in order to ensure 4889 the register count is higher to workaround the bug. 4890 4891 .. note:: 4892 4893 This approach of using a tentative scratch V# and shifting the register 4894 numbers if used avoids having to perform register allocation a second 4895 time if the tentative V# is eliminated. This is more efficient and 4896 avoids the problem that the second register allocation may perform 4897 spilling which will fail as there is no longer a scratch V#. 4898 4899When the kernel prolog code is being emitted it is known whether the scratch V# 4900described above is actually used. If it is, the prolog code must set it up by 4901copying the Private Segment Buffer to the scratch V# registers and then adding 4902the Private Segment Wavefront Offset to the queue base address in the V#. The 4903result is a V# with a base address pointing to the beginning of the wavefront 4904scratch backing memory. 4905 4906The Private Segment Buffer is always requested, but the Private Segment 4907Wavefront Offset is only requested if it is used (see 4908:ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4909 4910.. _amdgpu-amdhsa-memory-model: 4911 4912Memory Model 4913~~~~~~~~~~~~ 4914 4915This section describes the mapping of the LLVM memory model onto AMDGPU machine 4916code (see :ref:`memmodel`). 4917 4918The AMDGPU backend supports the memory synchronization scopes specified in 4919:ref:`amdgpu-memory-scopes`. 4920 4921The code sequences used to implement the memory model specify the order of 4922instructions that a single thread must execute. The ``s_waitcnt`` and cache 4923management instructions such as ``buffer_wbinvl1_vol`` are defined with respect 4924to other memory instructions executed by the same thread. This allows them to be 4925moved earlier or later which can allow them to be combined with other instances 4926of the same instruction, or hoisted/sunk out of loops to improve performance. 4927Only the instructions related to the memory model are given; additional 4928``s_waitcnt`` instructions are required to ensure registers are defined before 4929being used. These may be able to be combined with the memory model ``s_waitcnt`` 4930instructions as described above. 4931 4932The AMDGPU backend supports the following memory models: 4933 4934 HSA Memory Model [HSA]_ 4935 The HSA memory model uses a single happens-before relation for all address 4936 spaces (see :ref:`amdgpu-address-spaces`). 4937 OpenCL Memory Model [OpenCL]_ 4938 The OpenCL memory model which has separate happens-before relations for the 4939 global and local address spaces. Only a fence specifying both global and 4940 local address space, and seq_cst instructions join the relationships. Since 4941 the LLVM ``memfence`` instruction does not allow an address space to be 4942 specified the OpenCL fence has to conservatively assume both local and 4943 global address space was specified. However, optimizations can often be 4944 done to eliminate the additional ``s_waitcnt`` instructions when there are 4945 no intervening memory instructions which access the corresponding address 4946 space. The code sequences in the table indicate what can be omitted for the 4947 OpenCL memory. The target triple environment is used to determine if the 4948 source language is OpenCL (see :ref:`amdgpu-opencl`). 4949 4950``ds/flat_load/store/atomic`` instructions to local memory are termed LDS 4951operations. 4952 4953``buffer/global/flat_load/store/atomic`` instructions to global memory are 4954termed vector memory operations. 4955 4956Private address space uses ``buffer_load/store`` using the scratch V# 4957(GFX6-GFX8), or ``scratch_load/store`` (GFX9-GFX10). Since only a single thread 4958is accessing the memory, atomic memory orderings are not meaningful, and all 4959accesses are treated as non-atomic. 4960 4961Constant address space uses ``buffer/global_load`` instructions (or equivalent 4962scalar memory instructions). Since the constant address space contents do not 4963change during the execution of a kernel dispatch it is not legal to perform 4964stores, and atomic memory orderings are not meaningful, and all accesses are 4965treated as non-atomic. 4966 4967A memory synchronization scope wider than work-group is not meaningful for the 4968group (LDS) address space and is treated as work-group. 4969 4970The memory model does not support the region address space which is treated as 4971non-atomic. 4972 4973Acquire memory ordering is not meaningful on store atomic instructions and is 4974treated as non-atomic. 4975 4976Release memory ordering is not meaningful on load atomic instructions and is 4977treated a non-atomic. 4978 4979Acquire-release memory ordering is not meaningful on load or store atomic 4980instructions and is treated as acquire and release respectively. 4981 4982The memory order also adds the single thread optimization constraints defined in 4983table 4984:ref:`amdgpu-amdhsa-memory-model-single-thread-optimization-constraints-table`. 4985 4986 .. table:: AMDHSA Memory Model Single Thread Optimization Constraints 4987 :name: amdgpu-amdhsa-memory-model-single-thread-optimization-constraints-table 4988 4989 ============ ============================================================== 4990 LLVM Memory Optimization Constraints 4991 Ordering 4992 ============ ============================================================== 4993 unordered *none* 4994 monotonic *none* 4995 acquire - If a load atomic/atomicrmw then no following load/load 4996 atomic/store/store atomic/atomicrmw/fence instruction can be 4997 moved before the acquire. 4998 - If a fence then same as load atomic, plus no preceding 4999 associated fence-paired-atomic can be moved after the fence. 5000 release - If a store atomic/atomicrmw then no preceding load/load 5001 atomic/store/store atomic/atomicrmw/fence instruction can be 5002 moved after the release. 5003 - If a fence then same as store atomic, plus no following 5004 associated fence-paired-atomic can be moved before the 5005 fence. 5006 acq_rel Same constraints as both acquire and release. 5007 seq_cst - If a load atomic then same constraints as acquire, plus no 5008 preceding sequentially consistent load atomic/store 5009 atomic/atomicrmw/fence instruction can be moved after the 5010 seq_cst. 5011 - If a store atomic then the same constraints as release, plus 5012 no following sequentially consistent load atomic/store 5013 atomic/atomicrmw/fence instruction can be moved before the 5014 seq_cst. 5015 - If an atomicrmw/fence then same constraints as acq_rel. 5016 ============ ============================================================== 5017 5018The code sequences used to implement the memory model are defined in the 5019following sections: 5020 5021* :ref:`amdgpu-amdhsa-memory-model-gfx6-gfx9` 5022* :ref:`amdgpu-amdhsa-memory-model-gfx90a` 5023* :ref:`amdgpu-amdhsa-memory-model-gfx940` 5024* :ref:`amdgpu-amdhsa-memory-model-gfx10` 5025 5026.. _amdgpu-amdhsa-memory-model-gfx6-gfx9: 5027 5028Memory Model GFX6-GFX9 5029++++++++++++++++++++++ 5030 5031For GFX6-GFX9: 5032 5033* Each agent has multiple shader arrays (SA). 5034* Each SA has multiple compute units (CU). 5035* Each CU has multiple SIMDs that execute wavefronts. 5036* The wavefronts for a single work-group are executed in the same CU but may be 5037 executed by different SIMDs. 5038* Each CU has a single LDS memory shared by the wavefronts of the work-groups 5039 executing on it. 5040* All LDS operations of a CU are performed as wavefront wide operations in a 5041 global order and involve no caching. Completion is reported to a wavefront in 5042 execution order. 5043* The LDS memory has multiple request queues shared by the SIMDs of a 5044 CU. Therefore, the LDS operations performed by different wavefronts of a 5045 work-group can be reordered relative to each other, which can result in 5046 reordering the visibility of vector memory operations with respect to LDS 5047 operations of other wavefronts in the same work-group. A ``s_waitcnt 5048 lgkmcnt(0)`` is required to ensure synchronization between LDS operations and 5049 vector memory operations between wavefronts of a work-group, but not between 5050 operations performed by the same wavefront. 5051* The vector memory operations are performed as wavefront wide operations and 5052 completion is reported to a wavefront in execution order. The exception is 5053 that for GFX7-GFX9 ``flat_load/store/atomic`` instructions can report out of 5054 vector memory order if they access LDS memory, and out of LDS operation order 5055 if they access global memory. 5056* The vector memory operations access a single vector L1 cache shared by all 5057 SIMDs a CU. Therefore, no special action is required for coherence between the 5058 lanes of a single wavefront, or for coherence between wavefronts in the same 5059 work-group. A ``buffer_wbinvl1_vol`` is required for coherence between 5060 wavefronts executing in different work-groups as they may be executing on 5061 different CUs. 5062* The scalar memory operations access a scalar L1 cache shared by all wavefronts 5063 on a group of CUs. The scalar and vector L1 caches are not coherent. However, 5064 scalar operations are used in a restricted way so do not impact the memory 5065 model. See :ref:`amdgpu-amdhsa-memory-spaces`. 5066* The vector and scalar memory operations use an L2 cache shared by all CUs on 5067 the same agent. 5068* The L2 cache has independent channels to service disjoint ranges of virtual 5069 addresses. 5070* Each CU has a separate request queue per channel. Therefore, the vector and 5071 scalar memory operations performed by wavefronts executing in different 5072 work-groups (which may be executing on different CUs) of an agent can be 5073 reordered relative to each other. A ``s_waitcnt vmcnt(0)`` is required to 5074 ensure synchronization between vector memory operations of different CUs. It 5075 ensures a previous vector memory operation has completed before executing a 5076 subsequent vector memory or LDS operation and so can be used to meet the 5077 requirements of acquire and release. 5078* The L2 cache can be kept coherent with other agents on some targets, or ranges 5079 of virtual addresses can be set up to bypass it to ensure system coherence. 5080 5081Scalar memory operations are only used to access memory that is proven to not 5082change during the execution of the kernel dispatch. This includes constant 5083address space and global address space for program scope ``const`` variables. 5084Therefore, the kernel machine code does not have to maintain the scalar cache to 5085ensure it is coherent with the vector caches. The scalar and vector caches are 5086invalidated between kernel dispatches by CP since constant address space data 5087may change between kernel dispatch executions. See 5088:ref:`amdgpu-amdhsa-memory-spaces`. 5089 5090The one exception is if scalar writes are used to spill SGPR registers. In this 5091case the AMDGPU backend ensures the memory location used to spill is never 5092accessed by vector memory operations at the same time. If scalar writes are used 5093then a ``s_dcache_wb`` is inserted before the ``s_endpgm`` and before a function 5094return since the locations may be used for vector memory instructions by a 5095future wavefront that uses the same scratch area, or a function call that 5096creates a frame at the same address, respectively. There is no need for a 5097``s_dcache_inv`` as all scalar writes are write-before-read in the same thread. 5098 5099For kernarg backing memory: 5100 5101* CP invalidates the L1 cache at the start of each kernel dispatch. 5102* On dGPU the kernarg backing memory is allocated in host memory accessed as 5103 MTYPE UC (uncached) to avoid needing to invalidate the L2 cache. This also 5104 causes it to be treated as non-volatile and so is not invalidated by 5105 ``*_vol``. 5106* On APU the kernarg backing memory it is accessed as MTYPE CC (cache coherent) 5107 and so the L2 cache will be coherent with the CPU and other agents. 5108 5109Scratch backing memory (which is used for the private address space) is accessed 5110with MTYPE NC_NV (non-coherent non-volatile). Since the private address space is 5111only accessed by a single thread, and is always write-before-read, there is 5112never a need to invalidate these entries from the L1 cache. Hence all cache 5113invalidates are done as ``*_vol`` to only invalidate the volatile cache lines. 5114 5115The code sequences used to implement the memory model for GFX6-GFX9 are defined 5116in table :ref:`amdgpu-amdhsa-memory-model-code-sequences-gfx6-gfx9-table`. 5117 5118 .. table:: AMDHSA Memory Model Code Sequences GFX6-GFX9 5119 :name: amdgpu-amdhsa-memory-model-code-sequences-gfx6-gfx9-table 5120 5121 ============ ============ ============== ========== ================================ 5122 LLVM Instr LLVM Memory LLVM Memory AMDGPU AMDGPU Machine Code 5123 Ordering Sync Scope Address GFX6-GFX9 5124 Space 5125 ============ ============ ============== ========== ================================ 5126 **Non-Atomic** 5127 ------------------------------------------------------------------------------------ 5128 load *none* *none* - global - !volatile & !nontemporal 5129 - generic 5130 - private 1. buffer/global/flat_load 5131 - constant 5132 - !volatile & nontemporal 5133 5134 1. buffer/global/flat_load 5135 glc=1 slc=1 5136 5137 - volatile 5138 5139 1. buffer/global/flat_load 5140 glc=1 5141 2. s_waitcnt vmcnt(0) 5142 5143 - Must happen before 5144 any following volatile 5145 global/generic 5146 load/store. 5147 - Ensures that 5148 volatile 5149 operations to 5150 different 5151 addresses will not 5152 be reordered by 5153 hardware. 5154 5155 load *none* *none* - local 1. ds_load 5156 store *none* *none* - global - !volatile & !nontemporal 5157 - generic 5158 - private 1. buffer/global/flat_store 5159 - constant 5160 - !volatile & nontemporal 5161 5162 1. buffer/global/flat_store 5163 glc=1 slc=1 5164 5165 - volatile 5166 5167 1. buffer/global/flat_store 5168 2. s_waitcnt vmcnt(0) 5169 5170 - Must happen before 5171 any following volatile 5172 global/generic 5173 load/store. 5174 - Ensures that 5175 volatile 5176 operations to 5177 different 5178 addresses will not 5179 be reordered by 5180 hardware. 5181 5182 store *none* *none* - local 1. ds_store 5183 **Unordered Atomic** 5184 ------------------------------------------------------------------------------------ 5185 load atomic unordered *any* *any* *Same as non-atomic*. 5186 store atomic unordered *any* *any* *Same as non-atomic*. 5187 atomicrmw unordered *any* *any* *Same as monotonic atomic*. 5188 **Monotonic Atomic** 5189 ------------------------------------------------------------------------------------ 5190 load atomic monotonic - singlethread - global 1. buffer/global/ds/flat_load 5191 - wavefront - local 5192 - workgroup - generic 5193 load atomic monotonic - agent - global 1. buffer/global/flat_load 5194 - system - generic glc=1 5195 store atomic monotonic - singlethread - global 1. buffer/global/flat_store 5196 - wavefront - generic 5197 - workgroup 5198 - agent 5199 - system 5200 store atomic monotonic - singlethread - local 1. ds_store 5201 - wavefront 5202 - workgroup 5203 atomicrmw monotonic - singlethread - global 1. buffer/global/flat_atomic 5204 - wavefront - generic 5205 - workgroup 5206 - agent 5207 - system 5208 atomicrmw monotonic - singlethread - local 1. ds_atomic 5209 - wavefront 5210 - workgroup 5211 **Acquire Atomic** 5212 ------------------------------------------------------------------------------------ 5213 load atomic acquire - singlethread - global 1. buffer/global/ds/flat_load 5214 - wavefront - local 5215 - generic 5216 load atomic acquire - workgroup - global 1. buffer/global_load 5217 load atomic acquire - workgroup - local 1. ds/flat_load 5218 - generic 2. s_waitcnt lgkmcnt(0) 5219 5220 - If OpenCL, omit. 5221 - Must happen before 5222 any following 5223 global/generic 5224 load/load 5225 atomic/store/store 5226 atomic/atomicrmw. 5227 - Ensures any 5228 following global 5229 data read is no 5230 older than a local load 5231 atomic value being 5232 acquired. 5233 5234 load atomic acquire - agent - global 1. buffer/global_load 5235 - system glc=1 5236 2. s_waitcnt vmcnt(0) 5237 5238 - Must happen before 5239 following 5240 buffer_wbinvl1_vol. 5241 - Ensures the load 5242 has completed 5243 before invalidating 5244 the cache. 5245 5246 3. buffer_wbinvl1_vol 5247 5248 - Must happen before 5249 any following 5250 global/generic 5251 load/load 5252 atomic/atomicrmw. 5253 - Ensures that 5254 following 5255 loads will not see 5256 stale global data. 5257 5258 load atomic acquire - agent - generic 1. flat_load glc=1 5259 - system 2. s_waitcnt vmcnt(0) & 5260 lgkmcnt(0) 5261 5262 - If OpenCL omit 5263 lgkmcnt(0). 5264 - Must happen before 5265 following 5266 buffer_wbinvl1_vol. 5267 - Ensures the flat_load 5268 has completed 5269 before invalidating 5270 the cache. 5271 5272 3. buffer_wbinvl1_vol 5273 5274 - Must happen before 5275 any following 5276 global/generic 5277 load/load 5278 atomic/atomicrmw. 5279 - Ensures that 5280 following loads 5281 will not see stale 5282 global data. 5283 5284 atomicrmw acquire - singlethread - global 1. buffer/global/ds/flat_atomic 5285 - wavefront - local 5286 - generic 5287 atomicrmw acquire - workgroup - global 1. buffer/global_atomic 5288 atomicrmw acquire - workgroup - local 1. ds/flat_atomic 5289 - generic 2. s_waitcnt lgkmcnt(0) 5290 5291 - If OpenCL, omit. 5292 - Must happen before 5293 any following 5294 global/generic 5295 load/load 5296 atomic/store/store 5297 atomic/atomicrmw. 5298 - Ensures any 5299 following global 5300 data read is no 5301 older than a local 5302 atomicrmw value 5303 being acquired. 5304 5305 atomicrmw acquire - agent - global 1. buffer/global_atomic 5306 - system 2. s_waitcnt vmcnt(0) 5307 5308 - Must happen before 5309 following 5310 buffer_wbinvl1_vol. 5311 - Ensures the 5312 atomicrmw has 5313 completed before 5314 invalidating the 5315 cache. 5316 5317 3. buffer_wbinvl1_vol 5318 5319 - Must happen before 5320 any following 5321 global/generic 5322 load/load 5323 atomic/atomicrmw. 5324 - Ensures that 5325 following loads 5326 will not see stale 5327 global data. 5328 5329 atomicrmw acquire - agent - generic 1. flat_atomic 5330 - system 2. s_waitcnt vmcnt(0) & 5331 lgkmcnt(0) 5332 5333 - If OpenCL, omit 5334 lgkmcnt(0). 5335 - Must happen before 5336 following 5337 buffer_wbinvl1_vol. 5338 - Ensures the 5339 atomicrmw has 5340 completed before 5341 invalidating the 5342 cache. 5343 5344 3. buffer_wbinvl1_vol 5345 5346 - Must happen before 5347 any following 5348 global/generic 5349 load/load 5350 atomic/atomicrmw. 5351 - Ensures that 5352 following loads 5353 will not see stale 5354 global data. 5355 5356 fence acquire - singlethread *none* *none* 5357 - wavefront 5358 fence acquire - workgroup *none* 1. s_waitcnt lgkmcnt(0) 5359 5360 - If OpenCL and 5361 address space is 5362 not generic, omit. 5363 - However, since LLVM 5364 currently has no 5365 address space on 5366 the fence need to 5367 conservatively 5368 always generate. If 5369 fence had an 5370 address space then 5371 set to address 5372 space of OpenCL 5373 fence flag, or to 5374 generic if both 5375 local and global 5376 flags are 5377 specified. 5378 - Must happen after 5379 any preceding 5380 local/generic load 5381 atomic/atomicrmw 5382 with an equal or 5383 wider sync scope 5384 and memory ordering 5385 stronger than 5386 unordered (this is 5387 termed the 5388 fence-paired-atomic). 5389 - Must happen before 5390 any following 5391 global/generic 5392 load/load 5393 atomic/store/store 5394 atomic/atomicrmw. 5395 - Ensures any 5396 following global 5397 data read is no 5398 older than the 5399 value read by the 5400 fence-paired-atomic. 5401 5402 fence acquire - agent *none* 1. s_waitcnt lgkmcnt(0) & 5403 - system vmcnt(0) 5404 5405 - If OpenCL and 5406 address space is 5407 not generic, omit 5408 lgkmcnt(0). 5409 - However, since LLVM 5410 currently has no 5411 address space on 5412 the fence need to 5413 conservatively 5414 always generate 5415 (see comment for 5416 previous fence). 5417 - Could be split into 5418 separate s_waitcnt 5419 vmcnt(0) and 5420 s_waitcnt 5421 lgkmcnt(0) to allow 5422 them to be 5423 independently moved 5424 according to the 5425 following rules. 5426 - s_waitcnt vmcnt(0) 5427 must happen after 5428 any preceding 5429 global/generic load 5430 atomic/atomicrmw 5431 with an equal or 5432 wider sync scope 5433 and memory ordering 5434 stronger than 5435 unordered (this is 5436 termed the 5437 fence-paired-atomic). 5438 - s_waitcnt lgkmcnt(0) 5439 must happen after 5440 any preceding 5441 local/generic load 5442 atomic/atomicrmw 5443 with an equal or 5444 wider sync scope 5445 and memory ordering 5446 stronger than 5447 unordered (this is 5448 termed the 5449 fence-paired-atomic). 5450 - Must happen before 5451 the following 5452 buffer_wbinvl1_vol. 5453 - Ensures that the 5454 fence-paired atomic 5455 has completed 5456 before invalidating 5457 the 5458 cache. Therefore 5459 any following 5460 locations read must 5461 be no older than 5462 the value read by 5463 the 5464 fence-paired-atomic. 5465 5466 2. buffer_wbinvl1_vol 5467 5468 - Must happen before any 5469 following global/generic 5470 load/load 5471 atomic/store/store 5472 atomic/atomicrmw. 5473 - Ensures that 5474 following loads 5475 will not see stale 5476 global data. 5477 5478 **Release Atomic** 5479 ------------------------------------------------------------------------------------ 5480 store atomic release - singlethread - global 1. buffer/global/ds/flat_store 5481 - wavefront - local 5482 - generic 5483 store atomic release - workgroup - global 1. s_waitcnt lgkmcnt(0) 5484 - generic 5485 - If OpenCL, omit. 5486 - Must happen after 5487 any preceding 5488 local/generic 5489 load/store/load 5490 atomic/store 5491 atomic/atomicrmw. 5492 - Must happen before 5493 the following 5494 store. 5495 - Ensures that all 5496 memory operations 5497 to local have 5498 completed before 5499 performing the 5500 store that is being 5501 released. 5502 5503 2. buffer/global/flat_store 5504 store atomic release - workgroup - local 1. ds_store 5505 store atomic release - agent - global 1. s_waitcnt lgkmcnt(0) & 5506 - system - generic vmcnt(0) 5507 5508 - If OpenCL and 5509 address space is 5510 not generic, omit 5511 lgkmcnt(0). 5512 - Could be split into 5513 separate s_waitcnt 5514 vmcnt(0) and 5515 s_waitcnt 5516 lgkmcnt(0) to allow 5517 them to be 5518 independently moved 5519 according to the 5520 following rules. 5521 - s_waitcnt vmcnt(0) 5522 must happen after 5523 any preceding 5524 global/generic 5525 load/store/load 5526 atomic/store 5527 atomic/atomicrmw. 5528 - s_waitcnt lgkmcnt(0) 5529 must happen after 5530 any preceding 5531 local/generic 5532 load/store/load 5533 atomic/store 5534 atomic/atomicrmw. 5535 - Must happen before 5536 the following 5537 store. 5538 - Ensures that all 5539 memory operations 5540 to memory have 5541 completed before 5542 performing the 5543 store that is being 5544 released. 5545 5546 2. buffer/global/flat_store 5547 atomicrmw release - singlethread - global 1. buffer/global/ds/flat_atomic 5548 - wavefront - local 5549 - generic 5550 atomicrmw release - workgroup - global 1. s_waitcnt lgkmcnt(0) 5551 - generic 5552 - If OpenCL, omit. 5553 - Must happen after 5554 any preceding 5555 local/generic 5556 load/store/load 5557 atomic/store 5558 atomic/atomicrmw. 5559 - Must happen before 5560 the following 5561 atomicrmw. 5562 - Ensures that all 5563 memory operations 5564 to local have 5565 completed before 5566 performing the 5567 atomicrmw that is 5568 being released. 5569 5570 2. buffer/global/flat_atomic 5571 atomicrmw release - workgroup - local 1. ds_atomic 5572 atomicrmw release - agent - global 1. s_waitcnt lgkmcnt(0) & 5573 - system - generic vmcnt(0) 5574 5575 - If OpenCL, omit 5576 lgkmcnt(0). 5577 - Could be split into 5578 separate s_waitcnt 5579 vmcnt(0) and 5580 s_waitcnt 5581 lgkmcnt(0) to allow 5582 them to be 5583 independently moved 5584 according to the 5585 following rules. 5586 - s_waitcnt vmcnt(0) 5587 must happen after 5588 any preceding 5589 global/generic 5590 load/store/load 5591 atomic/store 5592 atomic/atomicrmw. 5593 - s_waitcnt lgkmcnt(0) 5594 must happen after 5595 any preceding 5596 local/generic 5597 load/store/load 5598 atomic/store 5599 atomic/atomicrmw. 5600 - Must happen before 5601 the following 5602 atomicrmw. 5603 - Ensures that all 5604 memory operations 5605 to global and local 5606 have completed 5607 before performing 5608 the atomicrmw that 5609 is being released. 5610 5611 2. buffer/global/flat_atomic 5612 fence release - singlethread *none* *none* 5613 - wavefront 5614 fence release - workgroup *none* 1. s_waitcnt lgkmcnt(0) 5615 5616 - If OpenCL and 5617 address space is 5618 not generic, omit. 5619 - However, since LLVM 5620 currently has no 5621 address space on 5622 the fence need to 5623 conservatively 5624 always generate. If 5625 fence had an 5626 address space then 5627 set to address 5628 space of OpenCL 5629 fence flag, or to 5630 generic if both 5631 local and global 5632 flags are 5633 specified. 5634 - Must happen after 5635 any preceding 5636 local/generic 5637 load/load 5638 atomic/store/store 5639 atomic/atomicrmw. 5640 - Must happen before 5641 any following store 5642 atomic/atomicrmw 5643 with an equal or 5644 wider sync scope 5645 and memory ordering 5646 stronger than 5647 unordered (this is 5648 termed the 5649 fence-paired-atomic). 5650 - Ensures that all 5651 memory operations 5652 to local have 5653 completed before 5654 performing the 5655 following 5656 fence-paired-atomic. 5657 5658 fence release - agent *none* 1. s_waitcnt lgkmcnt(0) & 5659 - system vmcnt(0) 5660 5661 - If OpenCL and 5662 address space is 5663 not generic, omit 5664 lgkmcnt(0). 5665 - If OpenCL and 5666 address space is 5667 local, omit 5668 vmcnt(0). 5669 - However, since LLVM 5670 currently has no 5671 address space on 5672 the fence need to 5673 conservatively 5674 always generate. If 5675 fence had an 5676 address space then 5677 set to address 5678 space of OpenCL 5679 fence flag, or to 5680 generic if both 5681 local and global 5682 flags are 5683 specified. 5684 - Could be split into 5685 separate s_waitcnt 5686 vmcnt(0) and 5687 s_waitcnt 5688 lgkmcnt(0) to allow 5689 them to be 5690 independently moved 5691 according to the 5692 following rules. 5693 - s_waitcnt vmcnt(0) 5694 must happen after 5695 any preceding 5696 global/generic 5697 load/store/load 5698 atomic/store 5699 atomic/atomicrmw. 5700 - s_waitcnt lgkmcnt(0) 5701 must happen after 5702 any preceding 5703 local/generic 5704 load/store/load 5705 atomic/store 5706 atomic/atomicrmw. 5707 - Must happen before 5708 any following store 5709 atomic/atomicrmw 5710 with an equal or 5711 wider sync scope 5712 and memory ordering 5713 stronger than 5714 unordered (this is 5715 termed the 5716 fence-paired-atomic). 5717 - Ensures that all 5718 memory operations 5719 have 5720 completed before 5721 performing the 5722 following 5723 fence-paired-atomic. 5724 5725 **Acquire-Release Atomic** 5726 ------------------------------------------------------------------------------------ 5727 atomicrmw acq_rel - singlethread - global 1. buffer/global/ds/flat_atomic 5728 - wavefront - local 5729 - generic 5730 atomicrmw acq_rel - workgroup - global 1. s_waitcnt lgkmcnt(0) 5731 5732 - If OpenCL, omit. 5733 - Must happen after 5734 any preceding 5735 local/generic 5736 load/store/load 5737 atomic/store 5738 atomic/atomicrmw. 5739 - Must happen before 5740 the following 5741 atomicrmw. 5742 - Ensures that all 5743 memory operations 5744 to local have 5745 completed before 5746 performing the 5747 atomicrmw that is 5748 being released. 5749 5750 2. buffer/global_atomic 5751 5752 atomicrmw acq_rel - workgroup - local 1. ds_atomic 5753 2. s_waitcnt lgkmcnt(0) 5754 5755 - If OpenCL, omit. 5756 - Must happen before 5757 any following 5758 global/generic 5759 load/load 5760 atomic/store/store 5761 atomic/atomicrmw. 5762 - Ensures any 5763 following global 5764 data read is no 5765 older than the local load 5766 atomic value being 5767 acquired. 5768 5769 atomicrmw acq_rel - workgroup - generic 1. s_waitcnt lgkmcnt(0) 5770 5771 - If OpenCL, omit. 5772 - Must happen after 5773 any preceding 5774 local/generic 5775 load/store/load 5776 atomic/store 5777 atomic/atomicrmw. 5778 - Must happen before 5779 the following 5780 atomicrmw. 5781 - Ensures that all 5782 memory operations 5783 to local have 5784 completed before 5785 performing the 5786 atomicrmw that is 5787 being released. 5788 5789 2. flat_atomic 5790 3. s_waitcnt lgkmcnt(0) 5791 5792 - If OpenCL, omit. 5793 - Must happen before 5794 any following 5795 global/generic 5796 load/load 5797 atomic/store/store 5798 atomic/atomicrmw. 5799 - Ensures any 5800 following global 5801 data read is no 5802 older than a local load 5803 atomic value being 5804 acquired. 5805 5806 atomicrmw acq_rel - agent - global 1. s_waitcnt lgkmcnt(0) & 5807 - system vmcnt(0) 5808 5809 - If OpenCL, omit 5810 lgkmcnt(0). 5811 - Could be split into 5812 separate s_waitcnt 5813 vmcnt(0) and 5814 s_waitcnt 5815 lgkmcnt(0) to allow 5816 them to be 5817 independently moved 5818 according to the 5819 following rules. 5820 - s_waitcnt vmcnt(0) 5821 must happen after 5822 any preceding 5823 global/generic 5824 load/store/load 5825 atomic/store 5826 atomic/atomicrmw. 5827 - s_waitcnt lgkmcnt(0) 5828 must happen after 5829 any preceding 5830 local/generic 5831 load/store/load 5832 atomic/store 5833 atomic/atomicrmw. 5834 - Must happen before 5835 the following 5836 atomicrmw. 5837 - Ensures that all 5838 memory operations 5839 to global have 5840 completed before 5841 performing the 5842 atomicrmw that is 5843 being released. 5844 5845 2. buffer/global_atomic 5846 3. s_waitcnt vmcnt(0) 5847 5848 - Must happen before 5849 following 5850 buffer_wbinvl1_vol. 5851 - Ensures the 5852 atomicrmw has 5853 completed before 5854 invalidating the 5855 cache. 5856 5857 4. buffer_wbinvl1_vol 5858 5859 - Must happen before 5860 any following 5861 global/generic 5862 load/load 5863 atomic/atomicrmw. 5864 - Ensures that 5865 following loads 5866 will not see stale 5867 global data. 5868 5869 atomicrmw acq_rel - agent - generic 1. s_waitcnt lgkmcnt(0) & 5870 - system vmcnt(0) 5871 5872 - If OpenCL, omit 5873 lgkmcnt(0). 5874 - Could be split into 5875 separate s_waitcnt 5876 vmcnt(0) and 5877 s_waitcnt 5878 lgkmcnt(0) to allow 5879 them to be 5880 independently moved 5881 according to the 5882 following rules. 5883 - s_waitcnt vmcnt(0) 5884 must happen after 5885 any preceding 5886 global/generic 5887 load/store/load 5888 atomic/store 5889 atomic/atomicrmw. 5890 - s_waitcnt lgkmcnt(0) 5891 must happen after 5892 any preceding 5893 local/generic 5894 load/store/load 5895 atomic/store 5896 atomic/atomicrmw. 5897 - Must happen before 5898 the following 5899 atomicrmw. 5900 - Ensures that all 5901 memory operations 5902 to global have 5903 completed before 5904 performing the 5905 atomicrmw that is 5906 being released. 5907 5908 2. flat_atomic 5909 3. s_waitcnt vmcnt(0) & 5910 lgkmcnt(0) 5911 5912 - If OpenCL, omit 5913 lgkmcnt(0). 5914 - Must happen before 5915 following 5916 buffer_wbinvl1_vol. 5917 - Ensures the 5918 atomicrmw has 5919 completed before 5920 invalidating the 5921 cache. 5922 5923 4. buffer_wbinvl1_vol 5924 5925 - Must happen before 5926 any following 5927 global/generic 5928 load/load 5929 atomic/atomicrmw. 5930 - Ensures that 5931 following loads 5932 will not see stale 5933 global data. 5934 5935 fence acq_rel - singlethread *none* *none* 5936 - wavefront 5937 fence acq_rel - workgroup *none* 1. s_waitcnt lgkmcnt(0) 5938 5939 - If OpenCL and 5940 address space is 5941 not generic, omit. 5942 - However, 5943 since LLVM 5944 currently has no 5945 address space on 5946 the fence need to 5947 conservatively 5948 always generate 5949 (see comment for 5950 previous fence). 5951 - Must happen after 5952 any preceding 5953 local/generic 5954 load/load 5955 atomic/store/store 5956 atomic/atomicrmw. 5957 - Must happen before 5958 any following 5959 global/generic 5960 load/load 5961 atomic/store/store 5962 atomic/atomicrmw. 5963 - Ensures that all 5964 memory operations 5965 to local have 5966 completed before 5967 performing any 5968 following global 5969 memory operations. 5970 - Ensures that the 5971 preceding 5972 local/generic load 5973 atomic/atomicrmw 5974 with an equal or 5975 wider sync scope 5976 and memory ordering 5977 stronger than 5978 unordered (this is 5979 termed the 5980 acquire-fence-paired-atomic) 5981 has completed 5982 before following 5983 global memory 5984 operations. This 5985 satisfies the 5986 requirements of 5987 acquire. 5988 - Ensures that all 5989 previous memory 5990 operations have 5991 completed before a 5992 following 5993 local/generic store 5994 atomic/atomicrmw 5995 with an equal or 5996 wider sync scope 5997 and memory ordering 5998 stronger than 5999 unordered (this is 6000 termed the 6001 release-fence-paired-atomic). 6002 This satisfies the 6003 requirements of 6004 release. 6005 6006 fence acq_rel - agent *none* 1. s_waitcnt lgkmcnt(0) & 6007 - system vmcnt(0) 6008 6009 - If OpenCL and 6010 address space is 6011 not generic, omit 6012 lgkmcnt(0). 6013 - However, since LLVM 6014 currently has no 6015 address space on 6016 the fence need to 6017 conservatively 6018 always generate 6019 (see comment for 6020 previous fence). 6021 - Could be split into 6022 separate s_waitcnt 6023 vmcnt(0) and 6024 s_waitcnt 6025 lgkmcnt(0) to allow 6026 them to be 6027 independently moved 6028 according to the 6029 following rules. 6030 - s_waitcnt vmcnt(0) 6031 must happen after 6032 any preceding 6033 global/generic 6034 load/store/load 6035 atomic/store 6036 atomic/atomicrmw. 6037 - s_waitcnt lgkmcnt(0) 6038 must happen after 6039 any preceding 6040 local/generic 6041 load/store/load 6042 atomic/store 6043 atomic/atomicrmw. 6044 - Must happen before 6045 the following 6046 buffer_wbinvl1_vol. 6047 - Ensures that the 6048 preceding 6049 global/local/generic 6050 load 6051 atomic/atomicrmw 6052 with an equal or 6053 wider sync scope 6054 and memory ordering 6055 stronger than 6056 unordered (this is 6057 termed the 6058 acquire-fence-paired-atomic) 6059 has completed 6060 before invalidating 6061 the cache. This 6062 satisfies the 6063 requirements of 6064 acquire. 6065 - Ensures that all 6066 previous memory 6067 operations have 6068 completed before a 6069 following 6070 global/local/generic 6071 store 6072 atomic/atomicrmw 6073 with an equal or 6074 wider sync scope 6075 and memory ordering 6076 stronger than 6077 unordered (this is 6078 termed the 6079 release-fence-paired-atomic). 6080 This satisfies the 6081 requirements of 6082 release. 6083 6084 2. buffer_wbinvl1_vol 6085 6086 - Must happen before 6087 any following 6088 global/generic 6089 load/load 6090 atomic/store/store 6091 atomic/atomicrmw. 6092 - Ensures that 6093 following loads 6094 will not see stale 6095 global data. This 6096 satisfies the 6097 requirements of 6098 acquire. 6099 6100 **Sequential Consistent Atomic** 6101 ------------------------------------------------------------------------------------ 6102 load atomic seq_cst - singlethread - global *Same as corresponding 6103 - wavefront - local load atomic acquire, 6104 - generic except must generate 6105 all instructions even 6106 for OpenCL.* 6107 load atomic seq_cst - workgroup - global 1. s_waitcnt lgkmcnt(0) 6108 - generic 6109 6110 - Must 6111 happen after 6112 preceding 6113 local/generic load 6114 atomic/store 6115 atomic/atomicrmw 6116 with memory 6117 ordering of seq_cst 6118 and with equal or 6119 wider sync scope. 6120 (Note that seq_cst 6121 fences have their 6122 own s_waitcnt 6123 lgkmcnt(0) and so do 6124 not need to be 6125 considered.) 6126 - Ensures any 6127 preceding 6128 sequential 6129 consistent local 6130 memory instructions 6131 have completed 6132 before executing 6133 this sequentially 6134 consistent 6135 instruction. This 6136 prevents reordering 6137 a seq_cst store 6138 followed by a 6139 seq_cst load. (Note 6140 that seq_cst is 6141 stronger than 6142 acquire/release as 6143 the reordering of 6144 load acquire 6145 followed by a store 6146 release is 6147 prevented by the 6148 s_waitcnt of 6149 the release, but 6150 there is nothing 6151 preventing a store 6152 release followed by 6153 load acquire from 6154 completing out of 6155 order. The s_waitcnt 6156 could be placed after 6157 seq_store or before 6158 the seq_load. We 6159 choose the load to 6160 make the s_waitcnt be 6161 as late as possible 6162 so that the store 6163 may have already 6164 completed.) 6165 6166 2. *Following 6167 instructions same as 6168 corresponding load 6169 atomic acquire, 6170 except must generate 6171 all instructions even 6172 for OpenCL.* 6173 load atomic seq_cst - workgroup - local *Same as corresponding 6174 load atomic acquire, 6175 except must generate 6176 all instructions even 6177 for OpenCL.* 6178 6179 load atomic seq_cst - agent - global 1. s_waitcnt lgkmcnt(0) & 6180 - system - generic vmcnt(0) 6181 6182 - Could be split into 6183 separate s_waitcnt 6184 vmcnt(0) 6185 and s_waitcnt 6186 lgkmcnt(0) to allow 6187 them to be 6188 independently moved 6189 according to the 6190 following rules. 6191 - s_waitcnt lgkmcnt(0) 6192 must happen after 6193 preceding 6194 global/generic load 6195 atomic/store 6196 atomic/atomicrmw 6197 with memory 6198 ordering of seq_cst 6199 and with equal or 6200 wider sync scope. 6201 (Note that seq_cst 6202 fences have their 6203 own s_waitcnt 6204 lgkmcnt(0) and so do 6205 not need to be 6206 considered.) 6207 - s_waitcnt vmcnt(0) 6208 must happen after 6209 preceding 6210 global/generic load 6211 atomic/store 6212 atomic/atomicrmw 6213 with memory 6214 ordering of seq_cst 6215 and with equal or 6216 wider sync scope. 6217 (Note that seq_cst 6218 fences have their 6219 own s_waitcnt 6220 vmcnt(0) and so do 6221 not need to be 6222 considered.) 6223 - Ensures any 6224 preceding 6225 sequential 6226 consistent global 6227 memory instructions 6228 have completed 6229 before executing 6230 this sequentially 6231 consistent 6232 instruction. This 6233 prevents reordering 6234 a seq_cst store 6235 followed by a 6236 seq_cst load. (Note 6237 that seq_cst is 6238 stronger than 6239 acquire/release as 6240 the reordering of 6241 load acquire 6242 followed by a store 6243 release is 6244 prevented by the 6245 s_waitcnt of 6246 the release, but 6247 there is nothing 6248 preventing a store 6249 release followed by 6250 load acquire from 6251 completing out of 6252 order. The s_waitcnt 6253 could be placed after 6254 seq_store or before 6255 the seq_load. We 6256 choose the load to 6257 make the s_waitcnt be 6258 as late as possible 6259 so that the store 6260 may have already 6261 completed.) 6262 6263 2. *Following 6264 instructions same as 6265 corresponding load 6266 atomic acquire, 6267 except must generate 6268 all instructions even 6269 for OpenCL.* 6270 store atomic seq_cst - singlethread - global *Same as corresponding 6271 - wavefront - local store atomic release, 6272 - workgroup - generic except must generate 6273 - agent all instructions even 6274 - system for OpenCL.* 6275 atomicrmw seq_cst - singlethread - global *Same as corresponding 6276 - wavefront - local atomicrmw acq_rel, 6277 - workgroup - generic except must generate 6278 - agent all instructions even 6279 - system for OpenCL.* 6280 fence seq_cst - singlethread *none* *Same as corresponding 6281 - wavefront fence acq_rel, 6282 - workgroup except must generate 6283 - agent all instructions even 6284 - system for OpenCL.* 6285 ============ ============ ============== ========== ================================ 6286 6287.. _amdgpu-amdhsa-memory-model-gfx90a: 6288 6289Memory Model GFX90A 6290+++++++++++++++++++ 6291 6292For GFX90A: 6293 6294* Each agent has multiple shader arrays (SA). 6295* Each SA has multiple compute units (CU). 6296* Each CU has multiple SIMDs that execute wavefronts. 6297* The wavefronts for a single work-group are executed in the same CU but may be 6298 executed by different SIMDs. The exception is when in tgsplit execution mode 6299 when the wavefronts may be executed by different SIMDs in different CUs. 6300* Each CU has a single LDS memory shared by the wavefronts of the work-groups 6301 executing on it. The exception is when in tgsplit execution mode when no LDS 6302 is allocated as wavefronts of the same work-group can be in different CUs. 6303* All LDS operations of a CU are performed as wavefront wide operations in a 6304 global order and involve no caching. Completion is reported to a wavefront in 6305 execution order. 6306* The LDS memory has multiple request queues shared by the SIMDs of a 6307 CU. Therefore, the LDS operations performed by different wavefronts of a 6308 work-group can be reordered relative to each other, which can result in 6309 reordering the visibility of vector memory operations with respect to LDS 6310 operations of other wavefronts in the same work-group. A ``s_waitcnt 6311 lgkmcnt(0)`` is required to ensure synchronization between LDS operations and 6312 vector memory operations between wavefronts of a work-group, but not between 6313 operations performed by the same wavefront. 6314* The vector memory operations are performed as wavefront wide operations and 6315 completion is reported to a wavefront in execution order. The exception is 6316 that ``flat_load/store/atomic`` instructions can report out of vector memory 6317 order if they access LDS memory, and out of LDS operation order if they access 6318 global memory. 6319* The vector memory operations access a single vector L1 cache shared by all 6320 SIMDs a CU. Therefore: 6321 6322 * No special action is required for coherence between the lanes of a single 6323 wavefront. 6324 6325 * No special action is required for coherence between wavefronts in the same 6326 work-group since they execute on the same CU. The exception is when in 6327 tgsplit execution mode as wavefronts of the same work-group can be in 6328 different CUs and so a ``buffer_wbinvl1_vol`` is required as described in 6329 the following item. 6330 6331 * A ``buffer_wbinvl1_vol`` is required for coherence between wavefronts 6332 executing in different work-groups as they may be executing on different 6333 CUs. 6334 6335* The scalar memory operations access a scalar L1 cache shared by all wavefronts 6336 on a group of CUs. The scalar and vector L1 caches are not coherent. However, 6337 scalar operations are used in a restricted way so do not impact the memory 6338 model. See :ref:`amdgpu-amdhsa-memory-spaces`. 6339* The vector and scalar memory operations use an L2 cache shared by all CUs on 6340 the same agent. 6341 6342 * The L2 cache has independent channels to service disjoint ranges of virtual 6343 addresses. 6344 * Each CU has a separate request queue per channel. Therefore, the vector and 6345 scalar memory operations performed by wavefronts executing in different 6346 work-groups (which may be executing on different CUs), or the same 6347 work-group if executing in tgsplit mode, of an agent can be reordered 6348 relative to each other. A ``s_waitcnt vmcnt(0)`` is required to ensure 6349 synchronization between vector memory operations of different CUs. It 6350 ensures a previous vector memory operation has completed before executing a 6351 subsequent vector memory or LDS operation and so can be used to meet the 6352 requirements of acquire and release. 6353 * The L2 cache of one agent can be kept coherent with other agents by: 6354 using the MTYPE RW (read-write) or MTYPE CC (cache-coherent) with the PTE 6355 C-bit for memory local to the L2; and using the MTYPE NC (non-coherent) with 6356 the PTE C-bit set or MTYPE UC (uncached) for memory not local to the L2. 6357 6358 * Any local memory cache lines will be automatically invalidated by writes 6359 from CUs associated with other L2 caches, or writes from the CPU, due to 6360 the cache probe caused by coherent requests. Coherent requests are caused 6361 by GPU accesses to pages with the PTE C-bit set, by CPU accesses over 6362 XGMI, and by PCIe requests that are configured to be coherent requests. 6363 * XGMI accesses from the CPU to local memory may be cached on the CPU. 6364 Subsequent access from the GPU will automatically invalidate or writeback 6365 the CPU cache due to the L2 probe filter and and the PTE C-bit being set. 6366 * Since all work-groups on the same agent share the same L2, no L2 6367 invalidation or writeback is required for coherence. 6368 * To ensure coherence of local and remote memory writes of work-groups in 6369 different agents a ``buffer_wbl2`` is required. It will writeback dirty L2 6370 cache lines of MTYPE RW (used for local coarse grain memory) and MTYPE NC 6371 ()used for remote coarse grain memory). Note that MTYPE CC (used for local 6372 fine grain memory) causes write through to DRAM, and MTYPE UC (used for 6373 remote fine grain memory) bypasses the L2, so both will never result in 6374 dirty L2 cache lines. 6375 * To ensure coherence of local and remote memory reads of work-groups in 6376 different agents a ``buffer_invl2`` is required. It will invalidate L2 6377 cache lines with MTYPE NC (used for remote coarse grain memory). Note that 6378 MTYPE CC (used for local fine grain memory) and MTYPE RW (used for local 6379 coarse memory) cause local reads to be invalidated by remote writes with 6380 with the PTE C-bit so these cache lines are not invalidated. Note that 6381 MTYPE UC (used for remote fine grain memory) bypasses the L2, so will 6382 never result in L2 cache lines that need to be invalidated. 6383 6384 * PCIe access from the GPU to the CPU memory is kept coherent by using the 6385 MTYPE UC (uncached) which bypasses the L2. 6386 6387Scalar memory operations are only used to access memory that is proven to not 6388change during the execution of the kernel dispatch. This includes constant 6389address space and global address space for program scope ``const`` variables. 6390Therefore, the kernel machine code does not have to maintain the scalar cache to 6391ensure it is coherent with the vector caches. The scalar and vector caches are 6392invalidated between kernel dispatches by CP since constant address space data 6393may change between kernel dispatch executions. See 6394:ref:`amdgpu-amdhsa-memory-spaces`. 6395 6396The one exception is if scalar writes are used to spill SGPR registers. In this 6397case the AMDGPU backend ensures the memory location used to spill is never 6398accessed by vector memory operations at the same time. If scalar writes are used 6399then a ``s_dcache_wb`` is inserted before the ``s_endpgm`` and before a function 6400return since the locations may be used for vector memory instructions by a 6401future wavefront that uses the same scratch area, or a function call that 6402creates a frame at the same address, respectively. There is no need for a 6403``s_dcache_inv`` as all scalar writes are write-before-read in the same thread. 6404 6405For kernarg backing memory: 6406 6407* CP invalidates the L1 cache at the start of each kernel dispatch. 6408* On dGPU over XGMI or PCIe the kernarg backing memory is allocated in host 6409 memory accessed as MTYPE UC (uncached) to avoid needing to invalidate the L2 6410 cache. This also causes it to be treated as non-volatile and so is not 6411 invalidated by ``*_vol``. 6412* On APU the kernarg backing memory is accessed as MTYPE CC (cache coherent) and 6413 so the L2 cache will be coherent with the CPU and other agents. 6414 6415Scratch backing memory (which is used for the private address space) is accessed 6416with MTYPE NC_NV (non-coherent non-volatile). Since the private address space is 6417only accessed by a single thread, and is always write-before-read, there is 6418never a need to invalidate these entries from the L1 cache. Hence all cache 6419invalidates are done as ``*_vol`` to only invalidate the volatile cache lines. 6420 6421The code sequences used to implement the memory model for GFX90A are defined 6422in table :ref:`amdgpu-amdhsa-memory-model-code-sequences-gfx90a-table`. 6423 6424 .. table:: AMDHSA Memory Model Code Sequences GFX90A 6425 :name: amdgpu-amdhsa-memory-model-code-sequences-gfx90a-table 6426 6427 ============ ============ ============== ========== ================================ 6428 LLVM Instr LLVM Memory LLVM Memory AMDGPU AMDGPU Machine Code 6429 Ordering Sync Scope Address GFX90A 6430 Space 6431 ============ ============ ============== ========== ================================ 6432 **Non-Atomic** 6433 ------------------------------------------------------------------------------------ 6434 load *none* *none* - global - !volatile & !nontemporal 6435 - generic 6436 - private 1. buffer/global/flat_load 6437 - constant 6438 - !volatile & nontemporal 6439 6440 1. buffer/global/flat_load 6441 glc=1 slc=1 6442 6443 - volatile 6444 6445 1. buffer/global/flat_load 6446 glc=1 6447 2. s_waitcnt vmcnt(0) 6448 6449 - Must happen before 6450 any following volatile 6451 global/generic 6452 load/store. 6453 - Ensures that 6454 volatile 6455 operations to 6456 different 6457 addresses will not 6458 be reordered by 6459 hardware. 6460 6461 load *none* *none* - local 1. ds_load 6462 store *none* *none* - global - !volatile & !nontemporal 6463 - generic 6464 - private 1. buffer/global/flat_store 6465 - constant 6466 - !volatile & nontemporal 6467 6468 1. buffer/global/flat_store 6469 glc=1 slc=1 6470 6471 - volatile 6472 6473 1. buffer/global/flat_store 6474 2. s_waitcnt vmcnt(0) 6475 6476 - Must happen before 6477 any following volatile 6478 global/generic 6479 load/store. 6480 - Ensures that 6481 volatile 6482 operations to 6483 different 6484 addresses will not 6485 be reordered by 6486 hardware. 6487 6488 store *none* *none* - local 1. ds_store 6489 **Unordered Atomic** 6490 ------------------------------------------------------------------------------------ 6491 load atomic unordered *any* *any* *Same as non-atomic*. 6492 store atomic unordered *any* *any* *Same as non-atomic*. 6493 atomicrmw unordered *any* *any* *Same as monotonic atomic*. 6494 **Monotonic Atomic** 6495 ------------------------------------------------------------------------------------ 6496 load atomic monotonic - singlethread - global 1. buffer/global/flat_load 6497 - wavefront - generic 6498 load atomic monotonic - workgroup - global 1. buffer/global/flat_load 6499 - generic glc=1 6500 6501 - If not TgSplit execution 6502 mode, omit glc=1. 6503 6504 load atomic monotonic - singlethread - local *If TgSplit execution mode, 6505 - wavefront local address space cannot 6506 - workgroup be used.* 6507 6508 1. ds_load 6509 load atomic monotonic - agent - global 1. buffer/global/flat_load 6510 - generic glc=1 6511 load atomic monotonic - system - global 1. buffer/global/flat_load 6512 - generic glc=1 6513 store atomic monotonic - singlethread - global 1. buffer/global/flat_store 6514 - wavefront - generic 6515 - workgroup 6516 - agent 6517 store atomic monotonic - system - global 1. buffer/global/flat_store 6518 - generic 6519 store atomic monotonic - singlethread - local *If TgSplit execution mode, 6520 - wavefront local address space cannot 6521 - workgroup be used.* 6522 6523 1. ds_store 6524 atomicrmw monotonic - singlethread - global 1. buffer/global/flat_atomic 6525 - wavefront - generic 6526 - workgroup 6527 - agent 6528 atomicrmw monotonic - system - global 1. buffer/global/flat_atomic 6529 - generic 6530 atomicrmw monotonic - singlethread - local *If TgSplit execution mode, 6531 - wavefront local address space cannot 6532 - workgroup be used.* 6533 6534 1. ds_atomic 6535 **Acquire Atomic** 6536 ------------------------------------------------------------------------------------ 6537 load atomic acquire - singlethread - global 1. buffer/global/ds/flat_load 6538 - wavefront - local 6539 - generic 6540 load atomic acquire - workgroup - global 1. buffer/global_load glc=1 6541 6542 - If not TgSplit execution 6543 mode, omit glc=1. 6544 6545 2. s_waitcnt vmcnt(0) 6546 6547 - If not TgSplit execution 6548 mode, omit. 6549 - Must happen before the 6550 following buffer_wbinvl1_vol. 6551 6552 3. buffer_wbinvl1_vol 6553 6554 - If not TgSplit execution 6555 mode, omit. 6556 - Must happen before 6557 any following 6558 global/generic 6559 load/load 6560 atomic/store/store 6561 atomic/atomicrmw. 6562 - Ensures that 6563 following 6564 loads will not see 6565 stale data. 6566 6567 load atomic acquire - workgroup - local *If TgSplit execution mode, 6568 local address space cannot 6569 be used.* 6570 6571 1. ds_load 6572 2. s_waitcnt lgkmcnt(0) 6573 6574 - If OpenCL, omit. 6575 - Must happen before 6576 any following 6577 global/generic 6578 load/load 6579 atomic/store/store 6580 atomic/atomicrmw. 6581 - Ensures any 6582 following global 6583 data read is no 6584 older than the local load 6585 atomic value being 6586 acquired. 6587 6588 load atomic acquire - workgroup - generic 1. flat_load glc=1 6589 6590 - If not TgSplit execution 6591 mode, omit glc=1. 6592 6593 2. s_waitcnt lgkm/vmcnt(0) 6594 6595 - Use lgkmcnt(0) if not 6596 TgSplit execution mode 6597 and vmcnt(0) if TgSplit 6598 execution mode. 6599 - If OpenCL, omit lgkmcnt(0). 6600 - Must happen before 6601 the following 6602 buffer_wbinvl1_vol and any 6603 following global/generic 6604 load/load 6605 atomic/store/store 6606 atomic/atomicrmw. 6607 - Ensures any 6608 following global 6609 data read is no 6610 older than a local load 6611 atomic value being 6612 acquired. 6613 6614 3. buffer_wbinvl1_vol 6615 6616 - If not TgSplit execution 6617 mode, omit. 6618 - Ensures that 6619 following 6620 loads will not see 6621 stale data. 6622 6623 load atomic acquire - agent - global 1. buffer/global_load 6624 glc=1 6625 2. s_waitcnt vmcnt(0) 6626 6627 - Must happen before 6628 following 6629 buffer_wbinvl1_vol. 6630 - Ensures the load 6631 has completed 6632 before invalidating 6633 the cache. 6634 6635 3. buffer_wbinvl1_vol 6636 6637 - Must happen before 6638 any following 6639 global/generic 6640 load/load 6641 atomic/atomicrmw. 6642 - Ensures that 6643 following 6644 loads will not see 6645 stale global data. 6646 6647 load atomic acquire - system - global 1. buffer/global/flat_load 6648 glc=1 6649 2. s_waitcnt vmcnt(0) 6650 6651 - Must happen before 6652 following buffer_invl2 and 6653 buffer_wbinvl1_vol. 6654 - Ensures the load 6655 has completed 6656 before invalidating 6657 the cache. 6658 6659 3. buffer_invl2; 6660 buffer_wbinvl1_vol 6661 6662 - Must happen before 6663 any following 6664 global/generic 6665 load/load 6666 atomic/atomicrmw. 6667 - Ensures that 6668 following 6669 loads will not see 6670 stale L1 global data, 6671 nor see stale L2 MTYPE 6672 NC global data. 6673 MTYPE RW and CC memory will 6674 never be stale in L2 due to 6675 the memory probes. 6676 6677 load atomic acquire - agent - generic 1. flat_load glc=1 6678 2. s_waitcnt vmcnt(0) & 6679 lgkmcnt(0) 6680 6681 - If TgSplit execution mode, 6682 omit lgkmcnt(0). 6683 - If OpenCL omit 6684 lgkmcnt(0). 6685 - Must happen before 6686 following 6687 buffer_wbinvl1_vol. 6688 - Ensures the flat_load 6689 has completed 6690 before invalidating 6691 the cache. 6692 6693 3. buffer_wbinvl1_vol 6694 6695 - Must happen before 6696 any following 6697 global/generic 6698 load/load 6699 atomic/atomicrmw. 6700 - Ensures that 6701 following loads 6702 will not see stale 6703 global data. 6704 6705 load atomic acquire - system - generic 1. flat_load glc=1 6706 2. s_waitcnt vmcnt(0) & 6707 lgkmcnt(0) 6708 6709 - If TgSplit execution mode, 6710 omit lgkmcnt(0). 6711 - If OpenCL omit 6712 lgkmcnt(0). 6713 - Must happen before 6714 following 6715 buffer_invl2 and 6716 buffer_wbinvl1_vol. 6717 - Ensures the flat_load 6718 has completed 6719 before invalidating 6720 the caches. 6721 6722 3. buffer_invl2; 6723 buffer_wbinvl1_vol 6724 6725 - Must happen before 6726 any following 6727 global/generic 6728 load/load 6729 atomic/atomicrmw. 6730 - Ensures that 6731 following 6732 loads will not see 6733 stale L1 global data, 6734 nor see stale L2 MTYPE 6735 NC global data. 6736 MTYPE RW and CC memory will 6737 never be stale in L2 due to 6738 the memory probes. 6739 6740 atomicrmw acquire - singlethread - global 1. buffer/global/flat_atomic 6741 - wavefront - generic 6742 atomicrmw acquire - singlethread - local *If TgSplit execution mode, 6743 - wavefront local address space cannot 6744 be used.* 6745 6746 1. ds_atomic 6747 atomicrmw acquire - workgroup - global 1. buffer/global_atomic 6748 2. s_waitcnt vmcnt(0) 6749 6750 - If not TgSplit execution 6751 mode, omit. 6752 - Must happen before the 6753 following buffer_wbinvl1_vol. 6754 - Ensures the atomicrmw 6755 has completed 6756 before invalidating 6757 the cache. 6758 6759 3. buffer_wbinvl1_vol 6760 6761 - If not TgSplit execution 6762 mode, omit. 6763 - Must happen before 6764 any following 6765 global/generic 6766 load/load 6767 atomic/atomicrmw. 6768 - Ensures that 6769 following loads 6770 will not see stale 6771 global data. 6772 6773 atomicrmw acquire - workgroup - local *If TgSplit execution mode, 6774 local address space cannot 6775 be used.* 6776 6777 1. ds_atomic 6778 2. s_waitcnt lgkmcnt(0) 6779 6780 - If OpenCL, omit. 6781 - Must happen before 6782 any following 6783 global/generic 6784 load/load 6785 atomic/store/store 6786 atomic/atomicrmw. 6787 - Ensures any 6788 following global 6789 data read is no 6790 older than the local 6791 atomicrmw value 6792 being acquired. 6793 6794 atomicrmw acquire - workgroup - generic 1. flat_atomic 6795 2. s_waitcnt lgkm/vmcnt(0) 6796 6797 - Use lgkmcnt(0) if not 6798 TgSplit execution mode 6799 and vmcnt(0) if TgSplit 6800 execution mode. 6801 - If OpenCL, omit lgkmcnt(0). 6802 - Must happen before 6803 the following 6804 buffer_wbinvl1_vol and 6805 any following 6806 global/generic 6807 load/load 6808 atomic/store/store 6809 atomic/atomicrmw. 6810 - Ensures any 6811 following global 6812 data read is no 6813 older than a local 6814 atomicrmw value 6815 being acquired. 6816 6817 3. buffer_wbinvl1_vol 6818 6819 - If not TgSplit execution 6820 mode, omit. 6821 - Ensures that 6822 following 6823 loads will not see 6824 stale data. 6825 6826 atomicrmw acquire - agent - global 1. buffer/global_atomic 6827 2. s_waitcnt vmcnt(0) 6828 6829 - Must happen before 6830 following 6831 buffer_wbinvl1_vol. 6832 - Ensures the 6833 atomicrmw has 6834 completed before 6835 invalidating the 6836 cache. 6837 6838 3. buffer_wbinvl1_vol 6839 6840 - Must happen before 6841 any following 6842 global/generic 6843 load/load 6844 atomic/atomicrmw. 6845 - Ensures that 6846 following loads 6847 will not see stale 6848 global data. 6849 6850 atomicrmw acquire - system - global 1. buffer/global_atomic 6851 2. s_waitcnt vmcnt(0) 6852 6853 - Must happen before 6854 following buffer_invl2 and 6855 buffer_wbinvl1_vol. 6856 - Ensures the 6857 atomicrmw has 6858 completed before 6859 invalidating the 6860 caches. 6861 6862 3. buffer_invl2; 6863 buffer_wbinvl1_vol 6864 6865 - Must happen before 6866 any following 6867 global/generic 6868 load/load 6869 atomic/atomicrmw. 6870 - Ensures that 6871 following 6872 loads will not see 6873 stale L1 global data, 6874 nor see stale L2 MTYPE 6875 NC global data. 6876 MTYPE RW and CC memory will 6877 never be stale in L2 due to 6878 the memory probes. 6879 6880 atomicrmw acquire - agent - generic 1. flat_atomic 6881 2. s_waitcnt vmcnt(0) & 6882 lgkmcnt(0) 6883 6884 - If TgSplit execution mode, 6885 omit lgkmcnt(0). 6886 - If OpenCL, omit 6887 lgkmcnt(0). 6888 - Must happen before 6889 following 6890 buffer_wbinvl1_vol. 6891 - Ensures the 6892 atomicrmw has 6893 completed before 6894 invalidating the 6895 cache. 6896 6897 3. buffer_wbinvl1_vol 6898 6899 - Must happen before 6900 any following 6901 global/generic 6902 load/load 6903 atomic/atomicrmw. 6904 - Ensures that 6905 following loads 6906 will not see stale 6907 global data. 6908 6909 atomicrmw acquire - system - generic 1. flat_atomic 6910 2. s_waitcnt vmcnt(0) & 6911 lgkmcnt(0) 6912 6913 - If TgSplit execution mode, 6914 omit lgkmcnt(0). 6915 - If OpenCL, omit 6916 lgkmcnt(0). 6917 - Must happen before 6918 following 6919 buffer_invl2 and 6920 buffer_wbinvl1_vol. 6921 - Ensures the 6922 atomicrmw has 6923 completed before 6924 invalidating the 6925 caches. 6926 6927 3. buffer_invl2; 6928 buffer_wbinvl1_vol 6929 6930 - Must happen before 6931 any following 6932 global/generic 6933 load/load 6934 atomic/atomicrmw. 6935 - Ensures that 6936 following 6937 loads will not see 6938 stale L1 global data, 6939 nor see stale L2 MTYPE 6940 NC global data. 6941 MTYPE RW and CC memory will 6942 never be stale in L2 due to 6943 the memory probes. 6944 6945 fence acquire - singlethread *none* *none* 6946 - wavefront 6947 fence acquire - workgroup *none* 1. s_waitcnt lgkm/vmcnt(0) 6948 6949 - Use lgkmcnt(0) if not 6950 TgSplit execution mode 6951 and vmcnt(0) if TgSplit 6952 execution mode. 6953 - If OpenCL and 6954 address space is 6955 not generic, omit 6956 lgkmcnt(0). 6957 - If OpenCL and 6958 address space is 6959 local, omit 6960 vmcnt(0). 6961 - However, since LLVM 6962 currently has no 6963 address space on 6964 the fence need to 6965 conservatively 6966 always generate. If 6967 fence had an 6968 address space then 6969 set to address 6970 space of OpenCL 6971 fence flag, or to 6972 generic if both 6973 local and global 6974 flags are 6975 specified. 6976 - s_waitcnt vmcnt(0) 6977 must happen after 6978 any preceding 6979 global/generic load 6980 atomic/ 6981 atomicrmw 6982 with an equal or 6983 wider sync scope 6984 and memory ordering 6985 stronger than 6986 unordered (this is 6987 termed the 6988 fence-paired-atomic). 6989 - s_waitcnt lgkmcnt(0) 6990 must happen after 6991 any preceding 6992 local/generic load 6993 atomic/atomicrmw 6994 with an equal or 6995 wider sync scope 6996 and memory ordering 6997 stronger than 6998 unordered (this is 6999 termed the 7000 fence-paired-atomic). 7001 - Must happen before 7002 the following 7003 buffer_wbinvl1_vol and 7004 any following 7005 global/generic 7006 load/load 7007 atomic/store/store 7008 atomic/atomicrmw. 7009 - Ensures any 7010 following global 7011 data read is no 7012 older than the 7013 value read by the 7014 fence-paired-atomic. 7015 7016 2. buffer_wbinvl1_vol 7017 7018 - If not TgSplit execution 7019 mode, omit. 7020 - Ensures that 7021 following 7022 loads will not see 7023 stale data. 7024 7025 fence acquire - agent *none* 1. s_waitcnt lgkmcnt(0) & 7026 vmcnt(0) 7027 7028 - If TgSplit execution mode, 7029 omit lgkmcnt(0). 7030 - If OpenCL and 7031 address space is 7032 not generic, omit 7033 lgkmcnt(0). 7034 - However, since LLVM 7035 currently has no 7036 address space on 7037 the fence need to 7038 conservatively 7039 always generate 7040 (see comment for 7041 previous fence). 7042 - Could be split into 7043 separate s_waitcnt 7044 vmcnt(0) and 7045 s_waitcnt 7046 lgkmcnt(0) to allow 7047 them to be 7048 independently moved 7049 according to the 7050 following rules. 7051 - s_waitcnt vmcnt(0) 7052 must happen after 7053 any preceding 7054 global/generic load 7055 atomic/atomicrmw 7056 with an equal or 7057 wider sync scope 7058 and memory ordering 7059 stronger than 7060 unordered (this is 7061 termed the 7062 fence-paired-atomic). 7063 - s_waitcnt lgkmcnt(0) 7064 must happen after 7065 any preceding 7066 local/generic load 7067 atomic/atomicrmw 7068 with an equal or 7069 wider sync scope 7070 and memory ordering 7071 stronger than 7072 unordered (this is 7073 termed the 7074 fence-paired-atomic). 7075 - Must happen before 7076 the following 7077 buffer_wbinvl1_vol. 7078 - Ensures that the 7079 fence-paired atomic 7080 has completed 7081 before invalidating 7082 the 7083 cache. Therefore 7084 any following 7085 locations read must 7086 be no older than 7087 the value read by 7088 the 7089 fence-paired-atomic. 7090 7091 2. buffer_wbinvl1_vol 7092 7093 - Must happen before any 7094 following global/generic 7095 load/load 7096 atomic/store/store 7097 atomic/atomicrmw. 7098 - Ensures that 7099 following loads 7100 will not see stale 7101 global data. 7102 7103 fence acquire - system *none* 1. s_waitcnt lgkmcnt(0) & 7104 vmcnt(0) 7105 7106 - If TgSplit execution mode, 7107 omit lgkmcnt(0). 7108 - If OpenCL and 7109 address space is 7110 not generic, omit 7111 lgkmcnt(0). 7112 - However, since LLVM 7113 currently has no 7114 address space on 7115 the fence need to 7116 conservatively 7117 always generate 7118 (see comment for 7119 previous fence). 7120 - Could be split into 7121 separate s_waitcnt 7122 vmcnt(0) and 7123 s_waitcnt 7124 lgkmcnt(0) to allow 7125 them to be 7126 independently moved 7127 according to the 7128 following rules. 7129 - s_waitcnt vmcnt(0) 7130 must happen after 7131 any preceding 7132 global/generic load 7133 atomic/atomicrmw 7134 with an equal or 7135 wider sync scope 7136 and memory ordering 7137 stronger than 7138 unordered (this is 7139 termed the 7140 fence-paired-atomic). 7141 - s_waitcnt lgkmcnt(0) 7142 must happen after 7143 any preceding 7144 local/generic load 7145 atomic/atomicrmw 7146 with an equal or 7147 wider sync scope 7148 and memory ordering 7149 stronger than 7150 unordered (this is 7151 termed the 7152 fence-paired-atomic). 7153 - Must happen before 7154 the following buffer_invl2 and 7155 buffer_wbinvl1_vol. 7156 - Ensures that the 7157 fence-paired atomic 7158 has completed 7159 before invalidating 7160 the 7161 cache. Therefore 7162 any following 7163 locations read must 7164 be no older than 7165 the value read by 7166 the 7167 fence-paired-atomic. 7168 7169 2. buffer_invl2; 7170 buffer_wbinvl1_vol 7171 7172 - Must happen before any 7173 following global/generic 7174 load/load 7175 atomic/store/store 7176 atomic/atomicrmw. 7177 - Ensures that 7178 following 7179 loads will not see 7180 stale L1 global data, 7181 nor see stale L2 MTYPE 7182 NC global data. 7183 MTYPE RW and CC memory will 7184 never be stale in L2 due to 7185 the memory probes. 7186 **Release Atomic** 7187 ------------------------------------------------------------------------------------ 7188 store atomic release - singlethread - global 1. buffer/global/flat_store 7189 - wavefront - generic 7190 store atomic release - singlethread - local *If TgSplit execution mode, 7191 - wavefront local address space cannot 7192 be used.* 7193 7194 1. ds_store 7195 store atomic release - workgroup - global 1. s_waitcnt lgkm/vmcnt(0) 7196 - generic 7197 - Use lgkmcnt(0) if not 7198 TgSplit execution mode 7199 and vmcnt(0) if TgSplit 7200 execution mode. 7201 - If OpenCL, omit lgkmcnt(0). 7202 - s_waitcnt vmcnt(0) 7203 must happen after 7204 any preceding 7205 global/generic load/store/ 7206 load atomic/store atomic/ 7207 atomicrmw. 7208 - s_waitcnt lgkmcnt(0) 7209 must happen after 7210 any preceding 7211 local/generic 7212 load/store/load 7213 atomic/store 7214 atomic/atomicrmw. 7215 - Must happen before 7216 the following 7217 store. 7218 - Ensures that all 7219 memory operations 7220 have 7221 completed before 7222 performing the 7223 store that is being 7224 released. 7225 7226 2. buffer/global/flat_store 7227 store atomic release - workgroup - local *If TgSplit execution mode, 7228 local address space cannot 7229 be used.* 7230 7231 1. ds_store 7232 store atomic release - agent - global 1. s_waitcnt lgkmcnt(0) & 7233 - generic vmcnt(0) 7234 7235 - If TgSplit execution mode, 7236 omit lgkmcnt(0). 7237 - If OpenCL and 7238 address space is 7239 not generic, omit 7240 lgkmcnt(0). 7241 - Could be split into 7242 separate s_waitcnt 7243 vmcnt(0) and 7244 s_waitcnt 7245 lgkmcnt(0) to allow 7246 them to be 7247 independently moved 7248 according to the 7249 following rules. 7250 - s_waitcnt vmcnt(0) 7251 must happen after 7252 any preceding 7253 global/generic 7254 load/store/load 7255 atomic/store 7256 atomic/atomicrmw. 7257 - s_waitcnt lgkmcnt(0) 7258 must happen after 7259 any preceding 7260 local/generic 7261 load/store/load 7262 atomic/store 7263 atomic/atomicrmw. 7264 - Must happen before 7265 the following 7266 store. 7267 - Ensures that all 7268 memory operations 7269 to memory have 7270 completed before 7271 performing the 7272 store that is being 7273 released. 7274 7275 2. buffer/global/flat_store 7276 store atomic release - system - global 1. buffer_wbl2 7277 - generic 7278 - Must happen before 7279 following s_waitcnt. 7280 - Performs L2 writeback to 7281 ensure previous 7282 global/generic 7283 store/atomicrmw are 7284 visible at system scope. 7285 7286 2. s_waitcnt lgkmcnt(0) & 7287 vmcnt(0) 7288 7289 - If TgSplit execution mode, 7290 omit lgkmcnt(0). 7291 - If OpenCL and 7292 address space is 7293 not generic, omit 7294 lgkmcnt(0). 7295 - Could be split into 7296 separate s_waitcnt 7297 vmcnt(0) and 7298 s_waitcnt 7299 lgkmcnt(0) to allow 7300 them to be 7301 independently moved 7302 according to the 7303 following rules. 7304 - s_waitcnt vmcnt(0) 7305 must happen after any 7306 preceding 7307 global/generic 7308 load/store/load 7309 atomic/store 7310 atomic/atomicrmw. 7311 - s_waitcnt lgkmcnt(0) 7312 must happen after any 7313 preceding 7314 local/generic 7315 load/store/load 7316 atomic/store 7317 atomic/atomicrmw. 7318 - Must happen before 7319 the following 7320 store. 7321 - Ensures that all 7322 memory operations 7323 to memory and the L2 7324 writeback have 7325 completed before 7326 performing the 7327 store that is being 7328 released. 7329 7330 3. buffer/global/flat_store 7331 atomicrmw release - singlethread - global 1. buffer/global/flat_atomic 7332 - wavefront - generic 7333 atomicrmw release - singlethread - local *If TgSplit execution mode, 7334 - wavefront local address space cannot 7335 be used.* 7336 7337 1. ds_atomic 7338 atomicrmw release - workgroup - global 1. s_waitcnt lgkm/vmcnt(0) 7339 - generic 7340 - Use lgkmcnt(0) if not 7341 TgSplit execution mode 7342 and vmcnt(0) if TgSplit 7343 execution mode. 7344 - If OpenCL, omit 7345 lgkmcnt(0). 7346 - s_waitcnt vmcnt(0) 7347 must happen after 7348 any preceding 7349 global/generic load/store/ 7350 load atomic/store atomic/ 7351 atomicrmw. 7352 - s_waitcnt lgkmcnt(0) 7353 must happen after 7354 any preceding 7355 local/generic 7356 load/store/load 7357 atomic/store 7358 atomic/atomicrmw. 7359 - Must happen before 7360 the following 7361 atomicrmw. 7362 - Ensures that all 7363 memory operations 7364 have 7365 completed before 7366 performing the 7367 atomicrmw that is 7368 being released. 7369 7370 2. buffer/global/flat_atomic 7371 atomicrmw release - workgroup - local *If TgSplit execution mode, 7372 local address space cannot 7373 be used.* 7374 7375 1. ds_atomic 7376 atomicrmw release - agent - global 1. s_waitcnt lgkmcnt(0) & 7377 - generic vmcnt(0) 7378 7379 - If TgSplit execution mode, 7380 omit lgkmcnt(0). 7381 - If OpenCL, omit 7382 lgkmcnt(0). 7383 - Could be split into 7384 separate s_waitcnt 7385 vmcnt(0) and 7386 s_waitcnt 7387 lgkmcnt(0) to allow 7388 them to be 7389 independently moved 7390 according to the 7391 following rules. 7392 - s_waitcnt vmcnt(0) 7393 must happen after 7394 any preceding 7395 global/generic 7396 load/store/load 7397 atomic/store 7398 atomic/atomicrmw. 7399 - s_waitcnt lgkmcnt(0) 7400 must happen after 7401 any preceding 7402 local/generic 7403 load/store/load 7404 atomic/store 7405 atomic/atomicrmw. 7406 - Must happen before 7407 the following 7408 atomicrmw. 7409 - Ensures that all 7410 memory operations 7411 to global and local 7412 have completed 7413 before performing 7414 the atomicrmw that 7415 is being released. 7416 7417 2. buffer/global/flat_atomic 7418 atomicrmw release - system - global 1. buffer_wbl2 7419 - generic 7420 - Must happen before 7421 following s_waitcnt. 7422 - Performs L2 writeback to 7423 ensure previous 7424 global/generic 7425 store/atomicrmw are 7426 visible at system scope. 7427 7428 2. s_waitcnt lgkmcnt(0) & 7429 vmcnt(0) 7430 7431 - If TgSplit execution mode, 7432 omit lgkmcnt(0). 7433 - If OpenCL, omit 7434 lgkmcnt(0). 7435 - Could be split into 7436 separate s_waitcnt 7437 vmcnt(0) and 7438 s_waitcnt 7439 lgkmcnt(0) to allow 7440 them to be 7441 independently moved 7442 according to the 7443 following rules. 7444 - s_waitcnt vmcnt(0) 7445 must happen after 7446 any preceding 7447 global/generic 7448 load/store/load 7449 atomic/store 7450 atomic/atomicrmw. 7451 - s_waitcnt lgkmcnt(0) 7452 must happen after 7453 any preceding 7454 local/generic 7455 load/store/load 7456 atomic/store 7457 atomic/atomicrmw. 7458 - Must happen before 7459 the following 7460 atomicrmw. 7461 - Ensures that all 7462 memory operations 7463 to memory and the L2 7464 writeback have 7465 completed before 7466 performing the 7467 store that is being 7468 released. 7469 7470 3. buffer/global/flat_atomic 7471 fence release - singlethread *none* *none* 7472 - wavefront 7473 fence release - workgroup *none* 1. s_waitcnt lgkm/vmcnt(0) 7474 7475 - Use lgkmcnt(0) if not 7476 TgSplit execution mode 7477 and vmcnt(0) if TgSplit 7478 execution mode. 7479 - If OpenCL and 7480 address space is 7481 not generic, omit 7482 lgkmcnt(0). 7483 - If OpenCL and 7484 address space is 7485 local, omit 7486 vmcnt(0). 7487 - However, since LLVM 7488 currently has no 7489 address space on 7490 the fence need to 7491 conservatively 7492 always generate. If 7493 fence had an 7494 address space then 7495 set to address 7496 space of OpenCL 7497 fence flag, or to 7498 generic if both 7499 local and global 7500 flags are 7501 specified. 7502 - s_waitcnt vmcnt(0) 7503 must happen after 7504 any preceding 7505 global/generic 7506 load/store/ 7507 load atomic/store atomic/ 7508 atomicrmw. 7509 - s_waitcnt lgkmcnt(0) 7510 must happen after 7511 any preceding 7512 local/generic 7513 load/load 7514 atomic/store/store 7515 atomic/atomicrmw. 7516 - Must happen before 7517 any following store 7518 atomic/atomicrmw 7519 with an equal or 7520 wider sync scope 7521 and memory ordering 7522 stronger than 7523 unordered (this is 7524 termed the 7525 fence-paired-atomic). 7526 - Ensures that all 7527 memory operations 7528 have 7529 completed before 7530 performing the 7531 following 7532 fence-paired-atomic. 7533 7534 fence release - agent *none* 1. s_waitcnt lgkmcnt(0) & 7535 vmcnt(0) 7536 7537 - If TgSplit execution mode, 7538 omit lgkmcnt(0). 7539 - If OpenCL and 7540 address space is 7541 not generic, omit 7542 lgkmcnt(0). 7543 - If OpenCL and 7544 address space is 7545 local, omit 7546 vmcnt(0). 7547 - However, since LLVM 7548 currently has no 7549 address space on 7550 the fence need to 7551 conservatively 7552 always generate. If 7553 fence had an 7554 address space then 7555 set to address 7556 space of OpenCL 7557 fence flag, or to 7558 generic if both 7559 local and global 7560 flags are 7561 specified. 7562 - Could be split into 7563 separate s_waitcnt 7564 vmcnt(0) and 7565 s_waitcnt 7566 lgkmcnt(0) to allow 7567 them to be 7568 independently moved 7569 according to the 7570 following rules. 7571 - s_waitcnt vmcnt(0) 7572 must happen after 7573 any preceding 7574 global/generic 7575 load/store/load 7576 atomic/store 7577 atomic/atomicrmw. 7578 - s_waitcnt lgkmcnt(0) 7579 must happen after 7580 any preceding 7581 local/generic 7582 load/store/load 7583 atomic/store 7584 atomic/atomicrmw. 7585 - Must happen before 7586 any following store 7587 atomic/atomicrmw 7588 with an equal or 7589 wider sync scope 7590 and memory ordering 7591 stronger than 7592 unordered (this is 7593 termed the 7594 fence-paired-atomic). 7595 - Ensures that all 7596 memory operations 7597 have 7598 completed before 7599 performing the 7600 following 7601 fence-paired-atomic. 7602 7603 fence release - system *none* 1. buffer_wbl2 7604 7605 - If OpenCL and 7606 address space is 7607 local, omit. 7608 - Must happen before 7609 following s_waitcnt. 7610 - Performs L2 writeback to 7611 ensure previous 7612 global/generic 7613 store/atomicrmw are 7614 visible at system scope. 7615 7616 2. s_waitcnt lgkmcnt(0) & 7617 vmcnt(0) 7618 7619 - If TgSplit execution mode, 7620 omit lgkmcnt(0). 7621 - If OpenCL and 7622 address space is 7623 not generic, omit 7624 lgkmcnt(0). 7625 - If OpenCL and 7626 address space is 7627 local, omit 7628 vmcnt(0). 7629 - However, since LLVM 7630 currently has no 7631 address space on 7632 the fence need to 7633 conservatively 7634 always generate. If 7635 fence had an 7636 address space then 7637 set to address 7638 space of OpenCL 7639 fence flag, or to 7640 generic if both 7641 local and global 7642 flags are 7643 specified. 7644 - Could be split into 7645 separate s_waitcnt 7646 vmcnt(0) and 7647 s_waitcnt 7648 lgkmcnt(0) to allow 7649 them to be 7650 independently moved 7651 according to the 7652 following rules. 7653 - s_waitcnt vmcnt(0) 7654 must happen after 7655 any preceding 7656 global/generic 7657 load/store/load 7658 atomic/store 7659 atomic/atomicrmw. 7660 - s_waitcnt lgkmcnt(0) 7661 must happen after 7662 any preceding 7663 local/generic 7664 load/store/load 7665 atomic/store 7666 atomic/atomicrmw. 7667 - Must happen before 7668 any following store 7669 atomic/atomicrmw 7670 with an equal or 7671 wider sync scope 7672 and memory ordering 7673 stronger than 7674 unordered (this is 7675 termed the 7676 fence-paired-atomic). 7677 - Ensures that all 7678 memory operations 7679 have 7680 completed before 7681 performing the 7682 following 7683 fence-paired-atomic. 7684 7685 **Acquire-Release Atomic** 7686 ------------------------------------------------------------------------------------ 7687 atomicrmw acq_rel - singlethread - global 1. buffer/global/flat_atomic 7688 - wavefront - generic 7689 atomicrmw acq_rel - singlethread - local *If TgSplit execution mode, 7690 - wavefront local address space cannot 7691 be used.* 7692 7693 1. ds_atomic 7694 atomicrmw acq_rel - workgroup - global 1. s_waitcnt lgkm/vmcnt(0) 7695 7696 - Use lgkmcnt(0) if not 7697 TgSplit execution mode 7698 and vmcnt(0) if TgSplit 7699 execution mode. 7700 - If OpenCL, omit 7701 lgkmcnt(0). 7702 - Must happen after 7703 any preceding 7704 local/generic 7705 load/store/load 7706 atomic/store 7707 atomic/atomicrmw. 7708 - s_waitcnt vmcnt(0) 7709 must happen after 7710 any preceding 7711 global/generic load/store/ 7712 load atomic/store atomic/ 7713 atomicrmw. 7714 - s_waitcnt lgkmcnt(0) 7715 must happen after 7716 any preceding 7717 local/generic 7718 load/store/load 7719 atomic/store 7720 atomic/atomicrmw. 7721 - Must happen before 7722 the following 7723 atomicrmw. 7724 - Ensures that all 7725 memory operations 7726 have 7727 completed before 7728 performing the 7729 atomicrmw that is 7730 being released. 7731 7732 2. buffer/global_atomic 7733 3. s_waitcnt vmcnt(0) 7734 7735 - If not TgSplit execution 7736 mode, omit. 7737 - Must happen before 7738 the following 7739 buffer_wbinvl1_vol. 7740 - Ensures any 7741 following global 7742 data read is no 7743 older than the 7744 atomicrmw value 7745 being acquired. 7746 7747 4. buffer_wbinvl1_vol 7748 7749 - If not TgSplit execution 7750 mode, omit. 7751 - Ensures that 7752 following 7753 loads will not see 7754 stale data. 7755 7756 atomicrmw acq_rel - workgroup - local *If TgSplit execution mode, 7757 local address space cannot 7758 be used.* 7759 7760 1. ds_atomic 7761 2. s_waitcnt lgkmcnt(0) 7762 7763 - If OpenCL, omit. 7764 - Must happen before 7765 any following 7766 global/generic 7767 load/load 7768 atomic/store/store 7769 atomic/atomicrmw. 7770 - Ensures any 7771 following global 7772 data read is no 7773 older than the local load 7774 atomic value being 7775 acquired. 7776 7777 atomicrmw acq_rel - workgroup - generic 1. s_waitcnt lgkm/vmcnt(0) 7778 7779 - Use lgkmcnt(0) if not 7780 TgSplit execution mode 7781 and vmcnt(0) if TgSplit 7782 execution mode. 7783 - If OpenCL, omit 7784 lgkmcnt(0). 7785 - s_waitcnt vmcnt(0) 7786 must happen after 7787 any preceding 7788 global/generic load/store/ 7789 load atomic/store atomic/ 7790 atomicrmw. 7791 - s_waitcnt lgkmcnt(0) 7792 must happen after 7793 any preceding 7794 local/generic 7795 load/store/load 7796 atomic/store 7797 atomic/atomicrmw. 7798 - Must happen before 7799 the following 7800 atomicrmw. 7801 - Ensures that all 7802 memory operations 7803 have 7804 completed before 7805 performing the 7806 atomicrmw that is 7807 being released. 7808 7809 2. flat_atomic 7810 3. s_waitcnt lgkmcnt(0) & 7811 vmcnt(0) 7812 7813 - If not TgSplit execution 7814 mode, omit vmcnt(0). 7815 - If OpenCL, omit 7816 lgkmcnt(0). 7817 - Must happen before 7818 the following 7819 buffer_wbinvl1_vol and 7820 any following 7821 global/generic 7822 load/load 7823 atomic/store/store 7824 atomic/atomicrmw. 7825 - Ensures any 7826 following global 7827 data read is no 7828 older than a local load 7829 atomic value being 7830 acquired. 7831 7832 3. buffer_wbinvl1_vol 7833 7834 - If not TgSplit execution 7835 mode, omit. 7836 - Ensures that 7837 following 7838 loads will not see 7839 stale data. 7840 7841 atomicrmw acq_rel - agent - global 1. s_waitcnt lgkmcnt(0) & 7842 vmcnt(0) 7843 7844 - If TgSplit execution mode, 7845 omit lgkmcnt(0). 7846 - If OpenCL, omit 7847 lgkmcnt(0). 7848 - Could be split into 7849 separate s_waitcnt 7850 vmcnt(0) and 7851 s_waitcnt 7852 lgkmcnt(0) to allow 7853 them to be 7854 independently moved 7855 according to the 7856 following rules. 7857 - s_waitcnt vmcnt(0) 7858 must happen after 7859 any preceding 7860 global/generic 7861 load/store/load 7862 atomic/store 7863 atomic/atomicrmw. 7864 - s_waitcnt lgkmcnt(0) 7865 must happen after 7866 any preceding 7867 local/generic 7868 load/store/load 7869 atomic/store 7870 atomic/atomicrmw. 7871 - Must happen before 7872 the following 7873 atomicrmw. 7874 - Ensures that all 7875 memory operations 7876 to global have 7877 completed before 7878 performing the 7879 atomicrmw that is 7880 being released. 7881 7882 2. buffer/global_atomic 7883 3. s_waitcnt vmcnt(0) 7884 7885 - Must happen before 7886 following 7887 buffer_wbinvl1_vol. 7888 - Ensures the 7889 atomicrmw has 7890 completed before 7891 invalidating the 7892 cache. 7893 7894 4. buffer_wbinvl1_vol 7895 7896 - Must happen before 7897 any following 7898 global/generic 7899 load/load 7900 atomic/atomicrmw. 7901 - Ensures that 7902 following loads 7903 will not see stale 7904 global data. 7905 7906 atomicrmw acq_rel - system - global 1. buffer_wbl2 7907 7908 - Must happen before 7909 following s_waitcnt. 7910 - Performs L2 writeback to 7911 ensure previous 7912 global/generic 7913 store/atomicrmw are 7914 visible at system scope. 7915 7916 2. s_waitcnt lgkmcnt(0) & 7917 vmcnt(0) 7918 7919 - If TgSplit execution mode, 7920 omit lgkmcnt(0). 7921 - If OpenCL, omit 7922 lgkmcnt(0). 7923 - Could be split into 7924 separate s_waitcnt 7925 vmcnt(0) and 7926 s_waitcnt 7927 lgkmcnt(0) to allow 7928 them to be 7929 independently moved 7930 according to the 7931 following rules. 7932 - s_waitcnt vmcnt(0) 7933 must happen after 7934 any preceding 7935 global/generic 7936 load/store/load 7937 atomic/store 7938 atomic/atomicrmw. 7939 - s_waitcnt lgkmcnt(0) 7940 must happen after 7941 any preceding 7942 local/generic 7943 load/store/load 7944 atomic/store 7945 atomic/atomicrmw. 7946 - Must happen before 7947 the following 7948 atomicrmw. 7949 - Ensures that all 7950 memory operations 7951 to global and L2 writeback 7952 have completed before 7953 performing the 7954 atomicrmw that is 7955 being released. 7956 7957 3. buffer/global_atomic 7958 4. s_waitcnt vmcnt(0) 7959 7960 - Must happen before 7961 following buffer_invl2 and 7962 buffer_wbinvl1_vol. 7963 - Ensures the 7964 atomicrmw has 7965 completed before 7966 invalidating the 7967 caches. 7968 7969 5. buffer_invl2; 7970 buffer_wbinvl1_vol 7971 7972 - Must happen before 7973 any following 7974 global/generic 7975 load/load 7976 atomic/atomicrmw. 7977 - Ensures that 7978 following 7979 loads will not see 7980 stale L1 global data, 7981 nor see stale L2 MTYPE 7982 NC global data. 7983 MTYPE RW and CC memory will 7984 never be stale in L2 due to 7985 the memory probes. 7986 7987 atomicrmw acq_rel - agent - generic 1. s_waitcnt lgkmcnt(0) & 7988 vmcnt(0) 7989 7990 - If TgSplit execution mode, 7991 omit lgkmcnt(0). 7992 - If OpenCL, omit 7993 lgkmcnt(0). 7994 - Could be split into 7995 separate s_waitcnt 7996 vmcnt(0) and 7997 s_waitcnt 7998 lgkmcnt(0) to allow 7999 them to be 8000 independently moved 8001 according to the 8002 following rules. 8003 - s_waitcnt vmcnt(0) 8004 must happen after 8005 any preceding 8006 global/generic 8007 load/store/load 8008 atomic/store 8009 atomic/atomicrmw. 8010 - s_waitcnt lgkmcnt(0) 8011 must happen after 8012 any preceding 8013 local/generic 8014 load/store/load 8015 atomic/store 8016 atomic/atomicrmw. 8017 - Must happen before 8018 the following 8019 atomicrmw. 8020 - Ensures that all 8021 memory operations 8022 to global have 8023 completed before 8024 performing the 8025 atomicrmw that is 8026 being released. 8027 8028 2. flat_atomic 8029 3. s_waitcnt vmcnt(0) & 8030 lgkmcnt(0) 8031 8032 - If TgSplit execution mode, 8033 omit lgkmcnt(0). 8034 - If OpenCL, omit 8035 lgkmcnt(0). 8036 - Must happen before 8037 following 8038 buffer_wbinvl1_vol. 8039 - Ensures the 8040 atomicrmw has 8041 completed before 8042 invalidating the 8043 cache. 8044 8045 4. buffer_wbinvl1_vol 8046 8047 - Must happen before 8048 any following 8049 global/generic 8050 load/load 8051 atomic/atomicrmw. 8052 - Ensures that 8053 following loads 8054 will not see stale 8055 global data. 8056 8057 atomicrmw acq_rel - system - generic 1. buffer_wbl2 8058 8059 - Must happen before 8060 following s_waitcnt. 8061 - Performs L2 writeback to 8062 ensure previous 8063 global/generic 8064 store/atomicrmw are 8065 visible at system scope. 8066 8067 2. s_waitcnt lgkmcnt(0) & 8068 vmcnt(0) 8069 8070 - If TgSplit execution mode, 8071 omit lgkmcnt(0). 8072 - If OpenCL, omit 8073 lgkmcnt(0). 8074 - Could be split into 8075 separate s_waitcnt 8076 vmcnt(0) and 8077 s_waitcnt 8078 lgkmcnt(0) to allow 8079 them to be 8080 independently moved 8081 according to the 8082 following rules. 8083 - s_waitcnt vmcnt(0) 8084 must happen after 8085 any preceding 8086 global/generic 8087 load/store/load 8088 atomic/store 8089 atomic/atomicrmw. 8090 - s_waitcnt lgkmcnt(0) 8091 must happen after 8092 any preceding 8093 local/generic 8094 load/store/load 8095 atomic/store 8096 atomic/atomicrmw. 8097 - Must happen before 8098 the following 8099 atomicrmw. 8100 - Ensures that all 8101 memory operations 8102 to global and L2 writeback 8103 have completed before 8104 performing the 8105 atomicrmw that is 8106 being released. 8107 8108 3. flat_atomic 8109 4. s_waitcnt vmcnt(0) & 8110 lgkmcnt(0) 8111 8112 - If TgSplit execution mode, 8113 omit lgkmcnt(0). 8114 - If OpenCL, omit 8115 lgkmcnt(0). 8116 - Must happen before 8117 following buffer_invl2 and 8118 buffer_wbinvl1_vol. 8119 - Ensures the 8120 atomicrmw has 8121 completed before 8122 invalidating the 8123 caches. 8124 8125 5. buffer_invl2; 8126 buffer_wbinvl1_vol 8127 8128 - Must happen before 8129 any following 8130 global/generic 8131 load/load 8132 atomic/atomicrmw. 8133 - Ensures that 8134 following 8135 loads will not see 8136 stale L1 global data, 8137 nor see stale L2 MTYPE 8138 NC global data. 8139 MTYPE RW and CC memory will 8140 never be stale in L2 due to 8141 the memory probes. 8142 8143 fence acq_rel - singlethread *none* *none* 8144 - wavefront 8145 fence acq_rel - workgroup *none* 1. s_waitcnt lgkm/vmcnt(0) 8146 8147 - Use lgkmcnt(0) if not 8148 TgSplit execution mode 8149 and vmcnt(0) if TgSplit 8150 execution mode. 8151 - If OpenCL and 8152 address space is 8153 not generic, omit 8154 lgkmcnt(0). 8155 - If OpenCL and 8156 address space is 8157 local, omit 8158 vmcnt(0). 8159 - However, 8160 since LLVM 8161 currently has no 8162 address space on 8163 the fence need to 8164 conservatively 8165 always generate 8166 (see comment for 8167 previous fence). 8168 - s_waitcnt vmcnt(0) 8169 must happen after 8170 any preceding 8171 global/generic 8172 load/store/ 8173 load atomic/store atomic/ 8174 atomicrmw. 8175 - s_waitcnt lgkmcnt(0) 8176 must happen after 8177 any preceding 8178 local/generic 8179 load/load 8180 atomic/store/store 8181 atomic/atomicrmw. 8182 - Must happen before 8183 any following 8184 global/generic 8185 load/load 8186 atomic/store/store 8187 atomic/atomicrmw. 8188 - Ensures that all 8189 memory operations 8190 have 8191 completed before 8192 performing any 8193 following global 8194 memory operations. 8195 - Ensures that the 8196 preceding 8197 local/generic load 8198 atomic/atomicrmw 8199 with an equal or 8200 wider sync scope 8201 and memory ordering 8202 stronger than 8203 unordered (this is 8204 termed the 8205 acquire-fence-paired-atomic) 8206 has completed 8207 before following 8208 global memory 8209 operations. This 8210 satisfies the 8211 requirements of 8212 acquire. 8213 - Ensures that all 8214 previous memory 8215 operations have 8216 completed before a 8217 following 8218 local/generic store 8219 atomic/atomicrmw 8220 with an equal or 8221 wider sync scope 8222 and memory ordering 8223 stronger than 8224 unordered (this is 8225 termed the 8226 release-fence-paired-atomic). 8227 This satisfies the 8228 requirements of 8229 release. 8230 - Must happen before 8231 the following 8232 buffer_wbinvl1_vol. 8233 - Ensures that the 8234 acquire-fence-paired 8235 atomic has completed 8236 before invalidating 8237 the 8238 cache. Therefore 8239 any following 8240 locations read must 8241 be no older than 8242 the value read by 8243 the 8244 acquire-fence-paired-atomic. 8245 8246 2. buffer_wbinvl1_vol 8247 8248 - If not TgSplit execution 8249 mode, omit. 8250 - Ensures that 8251 following 8252 loads will not see 8253 stale data. 8254 8255 fence acq_rel - agent *none* 1. s_waitcnt lgkmcnt(0) & 8256 vmcnt(0) 8257 8258 - If TgSplit execution mode, 8259 omit lgkmcnt(0). 8260 - If OpenCL and 8261 address space is 8262 not generic, omit 8263 lgkmcnt(0). 8264 - However, since LLVM 8265 currently has no 8266 address space on 8267 the fence need to 8268 conservatively 8269 always generate 8270 (see comment for 8271 previous fence). 8272 - Could be split into 8273 separate s_waitcnt 8274 vmcnt(0) and 8275 s_waitcnt 8276 lgkmcnt(0) to allow 8277 them to be 8278 independently moved 8279 according to the 8280 following rules. 8281 - s_waitcnt vmcnt(0) 8282 must happen after 8283 any preceding 8284 global/generic 8285 load/store/load 8286 atomic/store 8287 atomic/atomicrmw. 8288 - s_waitcnt lgkmcnt(0) 8289 must happen after 8290 any preceding 8291 local/generic 8292 load/store/load 8293 atomic/store 8294 atomic/atomicrmw. 8295 - Must happen before 8296 the following 8297 buffer_wbinvl1_vol. 8298 - Ensures that the 8299 preceding 8300 global/local/generic 8301 load 8302 atomic/atomicrmw 8303 with an equal or 8304 wider sync scope 8305 and memory ordering 8306 stronger than 8307 unordered (this is 8308 termed the 8309 acquire-fence-paired-atomic) 8310 has completed 8311 before invalidating 8312 the cache. This 8313 satisfies the 8314 requirements of 8315 acquire. 8316 - Ensures that all 8317 previous memory 8318 operations have 8319 completed before a 8320 following 8321 global/local/generic 8322 store 8323 atomic/atomicrmw 8324 with an equal or 8325 wider sync scope 8326 and memory ordering 8327 stronger than 8328 unordered (this is 8329 termed the 8330 release-fence-paired-atomic). 8331 This satisfies the 8332 requirements of 8333 release. 8334 8335 2. buffer_wbinvl1_vol 8336 8337 - Must happen before 8338 any following 8339 global/generic 8340 load/load 8341 atomic/store/store 8342 atomic/atomicrmw. 8343 - Ensures that 8344 following loads 8345 will not see stale 8346 global data. This 8347 satisfies the 8348 requirements of 8349 acquire. 8350 8351 fence acq_rel - system *none* 1. buffer_wbl2 8352 8353 - If OpenCL and 8354 address space is 8355 local, omit. 8356 - Must happen before 8357 following s_waitcnt. 8358 - Performs L2 writeback to 8359 ensure previous 8360 global/generic 8361 store/atomicrmw are 8362 visible at system scope. 8363 8364 2. s_waitcnt lgkmcnt(0) & 8365 vmcnt(0) 8366 8367 - If TgSplit execution mode, 8368 omit lgkmcnt(0). 8369 - If OpenCL and 8370 address space is 8371 not generic, omit 8372 lgkmcnt(0). 8373 - However, since LLVM 8374 currently has no 8375 address space on 8376 the fence need to 8377 conservatively 8378 always generate 8379 (see comment for 8380 previous fence). 8381 - Could be split into 8382 separate s_waitcnt 8383 vmcnt(0) and 8384 s_waitcnt 8385 lgkmcnt(0) to allow 8386 them to be 8387 independently moved 8388 according to the 8389 following rules. 8390 - s_waitcnt vmcnt(0) 8391 must happen after 8392 any preceding 8393 global/generic 8394 load/store/load 8395 atomic/store 8396 atomic/atomicrmw. 8397 - s_waitcnt lgkmcnt(0) 8398 must happen after 8399 any preceding 8400 local/generic 8401 load/store/load 8402 atomic/store 8403 atomic/atomicrmw. 8404 - Must happen before 8405 the following buffer_invl2 and 8406 buffer_wbinvl1_vol. 8407 - Ensures that the 8408 preceding 8409 global/local/generic 8410 load 8411 atomic/atomicrmw 8412 with an equal or 8413 wider sync scope 8414 and memory ordering 8415 stronger than 8416 unordered (this is 8417 termed the 8418 acquire-fence-paired-atomic) 8419 has completed 8420 before invalidating 8421 the cache. This 8422 satisfies the 8423 requirements of 8424 acquire. 8425 - Ensures that all 8426 previous memory 8427 operations have 8428 completed before a 8429 following 8430 global/local/generic 8431 store 8432 atomic/atomicrmw 8433 with an equal or 8434 wider sync scope 8435 and memory ordering 8436 stronger than 8437 unordered (this is 8438 termed the 8439 release-fence-paired-atomic). 8440 This satisfies the 8441 requirements of 8442 release. 8443 8444 3. buffer_invl2; 8445 buffer_wbinvl1_vol 8446 8447 - Must happen before 8448 any following 8449 global/generic 8450 load/load 8451 atomic/store/store 8452 atomic/atomicrmw. 8453 - Ensures that 8454 following 8455 loads will not see 8456 stale L1 global data, 8457 nor see stale L2 MTYPE 8458 NC global data. 8459 MTYPE RW and CC memory will 8460 never be stale in L2 due to 8461 the memory probes. 8462 8463 **Sequential Consistent Atomic** 8464 ------------------------------------------------------------------------------------ 8465 load atomic seq_cst - singlethread - global *Same as corresponding 8466 - wavefront - local load atomic acquire, 8467 - generic except must generate 8468 all instructions even 8469 for OpenCL.* 8470 load atomic seq_cst - workgroup - global 1. s_waitcnt lgkm/vmcnt(0) 8471 - generic 8472 - Use lgkmcnt(0) if not 8473 TgSplit execution mode 8474 and vmcnt(0) if TgSplit 8475 execution mode. 8476 - s_waitcnt lgkmcnt(0) must 8477 happen after 8478 preceding 8479 local/generic load 8480 atomic/store 8481 atomic/atomicrmw 8482 with memory 8483 ordering of seq_cst 8484 and with equal or 8485 wider sync scope. 8486 (Note that seq_cst 8487 fences have their 8488 own s_waitcnt 8489 lgkmcnt(0) and so do 8490 not need to be 8491 considered.) 8492 - s_waitcnt vmcnt(0) 8493 must happen after 8494 preceding 8495 global/generic load 8496 atomic/store 8497 atomic/atomicrmw 8498 with memory 8499 ordering of seq_cst 8500 and with equal or 8501 wider sync scope. 8502 (Note that seq_cst 8503 fences have their 8504 own s_waitcnt 8505 vmcnt(0) and so do 8506 not need to be 8507 considered.) 8508 - Ensures any 8509 preceding 8510 sequential 8511 consistent global/local 8512 memory instructions 8513 have completed 8514 before executing 8515 this sequentially 8516 consistent 8517 instruction. This 8518 prevents reordering 8519 a seq_cst store 8520 followed by a 8521 seq_cst load. (Note 8522 that seq_cst is 8523 stronger than 8524 acquire/release as 8525 the reordering of 8526 load acquire 8527 followed by a store 8528 release is 8529 prevented by the 8530 s_waitcnt of 8531 the release, but 8532 there is nothing 8533 preventing a store 8534 release followed by 8535 load acquire from 8536 completing out of 8537 order. The s_waitcnt 8538 could be placed after 8539 seq_store or before 8540 the seq_load. We 8541 choose the load to 8542 make the s_waitcnt be 8543 as late as possible 8544 so that the store 8545 may have already 8546 completed.) 8547 8548 2. *Following 8549 instructions same as 8550 corresponding load 8551 atomic acquire, 8552 except must generate 8553 all instructions even 8554 for OpenCL.* 8555 load atomic seq_cst - workgroup - local *If TgSplit execution mode, 8556 local address space cannot 8557 be used.* 8558 8559 *Same as corresponding 8560 load atomic acquire, 8561 except must generate 8562 all instructions even 8563 for OpenCL.* 8564 8565 load atomic seq_cst - agent - global 1. s_waitcnt lgkmcnt(0) & 8566 - system - generic vmcnt(0) 8567 8568 - If TgSplit execution mode, 8569 omit lgkmcnt(0). 8570 - Could be split into 8571 separate s_waitcnt 8572 vmcnt(0) 8573 and s_waitcnt 8574 lgkmcnt(0) to allow 8575 them to be 8576 independently moved 8577 according to the 8578 following rules. 8579 - s_waitcnt lgkmcnt(0) 8580 must happen after 8581 preceding 8582 global/generic load 8583 atomic/store 8584 atomic/atomicrmw 8585 with memory 8586 ordering of seq_cst 8587 and with equal or 8588 wider sync scope. 8589 (Note that seq_cst 8590 fences have their 8591 own s_waitcnt 8592 lgkmcnt(0) and so do 8593 not need to be 8594 considered.) 8595 - s_waitcnt vmcnt(0) 8596 must happen after 8597 preceding 8598 global/generic load 8599 atomic/store 8600 atomic/atomicrmw 8601 with memory 8602 ordering of seq_cst 8603 and with equal or 8604 wider sync scope. 8605 (Note that seq_cst 8606 fences have their 8607 own s_waitcnt 8608 vmcnt(0) and so do 8609 not need to be 8610 considered.) 8611 - Ensures any 8612 preceding 8613 sequential 8614 consistent global 8615 memory instructions 8616 have completed 8617 before executing 8618 this sequentially 8619 consistent 8620 instruction. This 8621 prevents reordering 8622 a seq_cst store 8623 followed by a 8624 seq_cst load. (Note 8625 that seq_cst is 8626 stronger than 8627 acquire/release as 8628 the reordering of 8629 load acquire 8630 followed by a store 8631 release is 8632 prevented by the 8633 s_waitcnt of 8634 the release, but 8635 there is nothing 8636 preventing a store 8637 release followed by 8638 load acquire from 8639 completing out of 8640 order. The s_waitcnt 8641 could be placed after 8642 seq_store or before 8643 the seq_load. We 8644 choose the load to 8645 make the s_waitcnt be 8646 as late as possible 8647 so that the store 8648 may have already 8649 completed.) 8650 8651 2. *Following 8652 instructions same as 8653 corresponding load 8654 atomic acquire, 8655 except must generate 8656 all instructions even 8657 for OpenCL.* 8658 store atomic seq_cst - singlethread - global *Same as corresponding 8659 - wavefront - local store atomic release, 8660 - workgroup - generic except must generate 8661 - agent all instructions even 8662 - system for OpenCL.* 8663 atomicrmw seq_cst - singlethread - global *Same as corresponding 8664 - wavefront - local atomicrmw acq_rel, 8665 - workgroup - generic except must generate 8666 - agent all instructions even 8667 - system for OpenCL.* 8668 fence seq_cst - singlethread *none* *Same as corresponding 8669 - wavefront fence acq_rel, 8670 - workgroup except must generate 8671 - agent all instructions even 8672 - system for OpenCL.* 8673 ============ ============ ============== ========== ================================ 8674 8675.. _amdgpu-amdhsa-memory-model-gfx940: 8676 8677Memory Model GFX940 8678+++++++++++++++++++ 8679 8680For GFX940: 8681 8682* Each agent has multiple shader arrays (SA). 8683* Each SA has multiple compute units (CU). 8684* Each CU has multiple SIMDs that execute wavefronts. 8685* The wavefronts for a single work-group are executed in the same CU but may be 8686 executed by different SIMDs. The exception is when in tgsplit execution mode 8687 when the wavefronts may be executed by different SIMDs in different CUs. 8688* Each CU has a single LDS memory shared by the wavefronts of the work-groups 8689 executing on it. The exception is when in tgsplit execution mode when no LDS 8690 is allocated as wavefronts of the same work-group can be in different CUs. 8691* All LDS operations of a CU are performed as wavefront wide operations in a 8692 global order and involve no caching. Completion is reported to a wavefront in 8693 execution order. 8694* The LDS memory has multiple request queues shared by the SIMDs of a 8695 CU. Therefore, the LDS operations performed by different wavefronts of a 8696 work-group can be reordered relative to each other, which can result in 8697 reordering the visibility of vector memory operations with respect to LDS 8698 operations of other wavefronts in the same work-group. A ``s_waitcnt 8699 lgkmcnt(0)`` is required to ensure synchronization between LDS operations and 8700 vector memory operations between wavefronts of a work-group, but not between 8701 operations performed by the same wavefront. 8702* The vector memory operations are performed as wavefront wide operations and 8703 completion is reported to a wavefront in execution order. The exception is 8704 that ``flat_load/store/atomic`` instructions can report out of vector memory 8705 order if they access LDS memory, and out of LDS operation order if they access 8706 global memory. 8707* The vector memory operations access a single vector L1 cache shared by all 8708 SIMDs a CU. Therefore: 8709 8710 * No special action is required for coherence between the lanes of a single 8711 wavefront. 8712 8713 * No special action is required for coherence between wavefronts in the same 8714 work-group since they execute on the same CU. The exception is when in 8715 tgsplit execution mode as wavefronts of the same work-group can be in 8716 different CUs and so a ``buffer_inv sc0`` is required which will invalidate 8717 the L1 cache. 8718 8719 * A ``buffer_inv sc0`` is required to invalidate the L1 cache for coherence 8720 between wavefronts executing in different work-groups as they may be 8721 executing on different CUs. 8722 8723 * Atomic read-modify-write instructions implicitly bypass the L1 cache. 8724 Therefore, they do not use the sc0 bit for coherence and instead use it to 8725 indicate if the instruction returns the original value being updated. They 8726 do use sc1 to indicate system or agent scope coherence. 8727 8728* The scalar memory operations access a scalar L1 cache shared by all wavefronts 8729 on a group of CUs. The scalar and vector L1 caches are not coherent. However, 8730 scalar operations are used in a restricted way so do not impact the memory 8731 model. See :ref:`amdgpu-amdhsa-memory-spaces`. 8732* The vector and scalar memory operations use an L2 cache. 8733 8734 * The gfx940 can be configured as a number of smaller agents with each having 8735 a single L2 shared by all CUs on the same agent, or as fewer (possibly one) 8736 larger agents with groups of CUs on each agent each sharing separate L2 8737 caches. 8738 * The L2 cache has independent channels to service disjoint ranges of virtual 8739 addresses. 8740 * Each CU has a separate request queue per channel for its associated L2. 8741 Therefore, the vector and scalar memory operations performed by wavefronts 8742 executing with different L1 caches and the same L2 cache can be reordered 8743 relative to each other. 8744 * A ``s_waitcnt vmcnt(0)`` is required to ensure synchronization between 8745 vector memory operations of different CUs. It ensures a previous vector 8746 memory operation has completed before executing a subsequent vector memory 8747 or LDS operation and so can be used to meet the requirements of acquire and 8748 release. 8749 * An L2 cache can be kept coherent with other L2 caches by using the MTYPE RW 8750 (read-write) for memory local to the L2, and MTYPE NC (non-coherent) with 8751 the PTE C-bit set for memory not local to the L2. 8752 8753 * Any local memory cache lines will be automatically invalidated by writes 8754 from CUs associated with other L2 caches, or writes from the CPU, due to 8755 the cache probe caused by the PTE C-bit. 8756 * XGMI accesses from the CPU to local memory may be cached on the CPU. 8757 Subsequent access from the GPU will automatically invalidate or writeback 8758 the CPU cache due to the L2 probe filter. 8759 * To ensure coherence of local memory writes of CUs with different L1 caches 8760 in the same agent a ``buffer_wbl2`` is required. It does nothing if the 8761 agent is configured to have a single L2, or will writeback dirty L2 cache 8762 lines if configured to have multiple L2 caches. 8763 * To ensure coherence of local memory writes of CUs in different agents a 8764 ``buffer_wbl2 sc1`` is required. It will writeback dirty L2 cache lines. 8765 * To ensure coherence of local memory reads of CUs with different L1 caches 8766 in the same agent a ``buffer_inv sc1`` is required. It does nothing if the 8767 agent is configured to have a single L2, or will invalidate non-local L2 8768 cache lines if configured to have multiple L2 caches. 8769 * To ensure coherence of local memory reads of CUs in different agents a 8770 ``buffer_inv sc0 sc1`` is required. It will invalidate non-local L2 cache 8771 lines if configured to have multiple L2 caches. 8772 8773 * PCIe access from the GPU to the CPU can be kept coherent by using the MTYPE 8774 UC (uncached) which bypasses the L2. 8775 8776Scalar memory operations are only used to access memory that is proven to not 8777change during the execution of the kernel dispatch. This includes constant 8778address space and global address space for program scope ``const`` variables. 8779Therefore, the kernel machine code does not have to maintain the scalar cache to 8780ensure it is coherent with the vector caches. The scalar and vector caches are 8781invalidated between kernel dispatches by CP since constant address space data 8782may change between kernel dispatch executions. See 8783:ref:`amdgpu-amdhsa-memory-spaces`. 8784 8785The one exception is if scalar writes are used to spill SGPR registers. In this 8786case the AMDGPU backend ensures the memory location used to spill is never 8787accessed by vector memory operations at the same time. If scalar writes are used 8788then a ``s_dcache_wb`` is inserted before the ``s_endpgm`` and before a function 8789return since the locations may be used for vector memory instructions by a 8790future wavefront that uses the same scratch area, or a function call that 8791creates a frame at the same address, respectively. There is no need for a 8792``s_dcache_inv`` as all scalar writes are write-before-read in the same thread. 8793 8794For kernarg backing memory: 8795 8796* CP invalidates the L1 cache at the start of each kernel dispatch. 8797* On dGPU over XGMI or PCIe the kernarg backing memory is allocated in host 8798 memory accessed as MTYPE UC (uncached) to avoid needing to invalidate the L2 8799 cache. This also causes it to be treated as non-volatile and so is not 8800 invalidated by ``*_vol``. 8801* On APU the kernarg backing memory is accessed as MTYPE CC (cache coherent) and 8802 so the L2 cache will be coherent with the CPU and other agents. 8803 8804Scratch backing memory (which is used for the private address space) is accessed 8805with MTYPE NC_NV (non-coherent non-volatile). Since the private address space is 8806only accessed by a single thread, and is always write-before-read, there is 8807never a need to invalidate these entries from the L1 cache. Hence all cache 8808invalidates are done as ``*_vol`` to only invalidate the volatile cache lines. 8809 8810The code sequences used to implement the memory model for GFX940 are defined 8811in table :ref:`amdgpu-amdhsa-memory-model-code-sequences-gfx940-table`. 8812 8813 .. table:: AMDHSA Memory Model Code Sequences GFX940 8814 :name: amdgpu-amdhsa-memory-model-code-sequences-gfx940-table 8815 8816 ============ ============ ============== ========== ================================ 8817 LLVM Instr LLVM Memory LLVM Memory AMDGPU AMDGPU Machine Code 8818 Ordering Sync Scope Address GFX940 8819 Space 8820 ============ ============ ============== ========== ================================ 8821 **Non-Atomic** 8822 ------------------------------------------------------------------------------------ 8823 load *none* *none* - global - !volatile & !nontemporal 8824 - generic 8825 - private 1. buffer/global/flat_load 8826 - constant 8827 - !volatile & nontemporal 8828 8829 1. buffer/global/flat_load 8830 nt=1 8831 8832 - volatile 8833 8834 1. buffer/global/flat_load 8835 sc0=1 sc1=1 8836 2. s_waitcnt vmcnt(0) 8837 8838 - Must happen before 8839 any following volatile 8840 global/generic 8841 load/store. 8842 - Ensures that 8843 volatile 8844 operations to 8845 different 8846 addresses will not 8847 be reordered by 8848 hardware. 8849 8850 load *none* *none* - local 1. ds_load 8851 store *none* *none* - global - !volatile & !nontemporal 8852 - generic 8853 - private 1. buffer/global/flat_store 8854 - constant 8855 - !volatile & nontemporal 8856 8857 1. buffer/global/flat_store 8858 nt=1 8859 8860 - volatile 8861 8862 1. buffer/global/flat_store 8863 sc0=1 sc1=1 8864 2. s_waitcnt vmcnt(0) 8865 8866 - Must happen before 8867 any following volatile 8868 global/generic 8869 load/store. 8870 - Ensures that 8871 volatile 8872 operations to 8873 different 8874 addresses will not 8875 be reordered by 8876 hardware. 8877 8878 store *none* *none* - local 1. ds_store 8879 **Unordered Atomic** 8880 ------------------------------------------------------------------------------------ 8881 load atomic unordered *any* *any* *Same as non-atomic*. 8882 store atomic unordered *any* *any* *Same as non-atomic*. 8883 atomicrmw unordered *any* *any* *Same as monotonic atomic*. 8884 **Monotonic Atomic** 8885 ------------------------------------------------------------------------------------ 8886 load atomic monotonic - singlethread - global 1. buffer/global/flat_load 8887 - wavefront - generic 8888 load atomic monotonic - workgroup - global 1. buffer/global/flat_load 8889 - generic sc0=1 8890 load atomic monotonic - singlethread - local *If TgSplit execution mode, 8891 - wavefront local address space cannot 8892 - workgroup be used.* 8893 8894 1. ds_load 8895 load atomic monotonic - agent - global 1. buffer/global/flat_load 8896 - generic sc1=1 8897 load atomic monotonic - system - global 1. buffer/global/flat_load 8898 - generic sc0=1 sc1=1 8899 store atomic monotonic - singlethread - global 1. buffer/global/flat_store 8900 - wavefront - generic 8901 store atomic monotonic - workgroup - global 1. buffer/global/flat_store 8902 - generic sc0=1 8903 store atomic monotonic - agent - global 1. buffer/global/flat_store 8904 - generic sc1=1 8905 store atomic monotonic - system - global 1. buffer/global/flat_store 8906 - generic sc0=1 sc1=1 8907 store atomic monotonic - singlethread - local *If TgSplit execution mode, 8908 - wavefront local address space cannot 8909 - workgroup be used.* 8910 8911 1. ds_store 8912 atomicrmw monotonic - singlethread - global 1. buffer/global/flat_atomic 8913 - wavefront - generic 8914 - workgroup 8915 - agent 8916 atomicrmw monotonic - system - global 1. buffer/global/flat_atomic 8917 - generic sc1=1 8918 atomicrmw monotonic - singlethread - local *If TgSplit execution mode, 8919 - wavefront local address space cannot 8920 - workgroup be used.* 8921 8922 1. ds_atomic 8923 **Acquire Atomic** 8924 ------------------------------------------------------------------------------------ 8925 load atomic acquire - singlethread - global 1. buffer/global/ds/flat_load 8926 - wavefront - local 8927 - generic 8928 load atomic acquire - workgroup - global 1. buffer/global_load sc0=1 8929 2. s_waitcnt vmcnt(0) 8930 8931 - If not TgSplit execution 8932 mode, omit. 8933 - Must happen before the 8934 following buffer_inv. 8935 8936 3. buffer_inv sc0=1 8937 8938 - If not TgSplit execution 8939 mode, omit. 8940 - Must happen before 8941 any following 8942 global/generic 8943 load/load 8944 atomic/store/store 8945 atomic/atomicrmw. 8946 - Ensures that 8947 following 8948 loads will not see 8949 stale data. 8950 8951 load atomic acquire - workgroup - local *If TgSplit execution mode, 8952 local address space cannot 8953 be used.* 8954 8955 1. ds_load 8956 2. s_waitcnt lgkmcnt(0) 8957 8958 - If OpenCL, omit. 8959 - Must happen before 8960 any following 8961 global/generic 8962 load/load 8963 atomic/store/store 8964 atomic/atomicrmw. 8965 - Ensures any 8966 following global 8967 data read is no 8968 older than the local load 8969 atomic value being 8970 acquired. 8971 8972 load atomic acquire - workgroup - generic 1. flat_load sc0=1 8973 2. s_waitcnt lgkm/vmcnt(0) 8974 8975 - Use lgkmcnt(0) if not 8976 TgSplit execution mode 8977 and vmcnt(0) if TgSplit 8978 execution mode. 8979 - If OpenCL, omit lgkmcnt(0). 8980 - Must happen before 8981 the following 8982 buffer_inv and any 8983 following global/generic 8984 load/load 8985 atomic/store/store 8986 atomic/atomicrmw. 8987 - Ensures any 8988 following global 8989 data read is no 8990 older than a local load 8991 atomic value being 8992 acquired. 8993 8994 3. buffer_inv sc0=1 8995 8996 - If not TgSplit execution 8997 mode, omit. 8998 - Ensures that 8999 following 9000 loads will not see 9001 stale data. 9002 9003 load atomic acquire - agent - global 1. buffer/global_load 9004 sc1=1 9005 2. s_waitcnt vmcnt(0) 9006 9007 - Must happen before 9008 following 9009 buffer_inv. 9010 - Ensures the load 9011 has completed 9012 before invalidating 9013 the cache. 9014 9015 3. buffer_inv sc1=1 9016 9017 - Must happen before 9018 any following 9019 global/generic 9020 load/load 9021 atomic/atomicrmw. 9022 - Ensures that 9023 following 9024 loads will not see 9025 stale global data. 9026 9027 load atomic acquire - system - global 1. buffer/global/flat_load 9028 sc0=1 sc1=1 9029 2. s_waitcnt vmcnt(0) 9030 9031 - Must happen before 9032 following 9033 buffer_inv. 9034 - Ensures the load 9035 has completed 9036 before invalidating 9037 the cache. 9038 9039 3. buffer_inv sc0=1 sc1=1 9040 9041 - Must happen before 9042 any following 9043 global/generic 9044 load/load 9045 atomic/atomicrmw. 9046 - Ensures that 9047 following 9048 loads will not see 9049 stale MTYPE NC global data. 9050 MTYPE RW and CC memory will 9051 never be stale due to the 9052 memory probes. 9053 9054 load atomic acquire - agent - generic 1. flat_load sc1=1 9055 2. s_waitcnt vmcnt(0) & 9056 lgkmcnt(0) 9057 9058 - If TgSplit execution mode, 9059 omit lgkmcnt(0). 9060 - If OpenCL omit 9061 lgkmcnt(0). 9062 - Must happen before 9063 following 9064 buffer_inv. 9065 - Ensures the flat_load 9066 has completed 9067 before invalidating 9068 the cache. 9069 9070 3. buffer_inv sc1=1 9071 9072 - Must happen before 9073 any following 9074 global/generic 9075 load/load 9076 atomic/atomicrmw. 9077 - Ensures that 9078 following loads 9079 will not see stale 9080 global data. 9081 9082 load atomic acquire - system - generic 1. flat_load sc0=1 sc1=1 9083 2. s_waitcnt vmcnt(0) & 9084 lgkmcnt(0) 9085 9086 - If TgSplit execution mode, 9087 omit lgkmcnt(0). 9088 - If OpenCL omit 9089 lgkmcnt(0). 9090 - Must happen before 9091 the following 9092 buffer_inv. 9093 - Ensures the flat_load 9094 has completed 9095 before invalidating 9096 the caches. 9097 9098 3. buffer_inv sc0=1 sc1=1 9099 9100 - Must happen before 9101 any following 9102 global/generic 9103 load/load 9104 atomic/atomicrmw. 9105 - Ensures that 9106 following 9107 loads will not see 9108 stale MTYPE NC global data. 9109 MTYPE RW and CC memory will 9110 never be stale due to the 9111 memory probes. 9112 9113 atomicrmw acquire - singlethread - global 1. buffer/global/flat_atomic 9114 - wavefront - generic 9115 atomicrmw acquire - singlethread - local *If TgSplit execution mode, 9116 - wavefront local address space cannot 9117 be used.* 9118 9119 1. ds_atomic 9120 atomicrmw acquire - workgroup - global 1. buffer/global_atomic 9121 2. s_waitcnt vmcnt(0) 9122 9123 - If not TgSplit execution 9124 mode, omit. 9125 - Must happen before the 9126 following buffer_inv. 9127 - Ensures the atomicrmw 9128 has completed 9129 before invalidating 9130 the cache. 9131 9132 3. buffer_inv sc0=1 9133 9134 - If not TgSplit execution 9135 mode, omit. 9136 - Must happen before 9137 any following 9138 global/generic 9139 load/load 9140 atomic/atomicrmw. 9141 - Ensures that 9142 following loads 9143 will not see stale 9144 global data. 9145 9146 atomicrmw acquire - workgroup - local *If TgSplit execution mode, 9147 local address space cannot 9148 be used.* 9149 9150 1. ds_atomic 9151 2. s_waitcnt lgkmcnt(0) 9152 9153 - If OpenCL, omit. 9154 - Must happen before 9155 any following 9156 global/generic 9157 load/load 9158 atomic/store/store 9159 atomic/atomicrmw. 9160 - Ensures any 9161 following global 9162 data read is no 9163 older than the local 9164 atomicrmw value 9165 being acquired. 9166 9167 atomicrmw acquire - workgroup - generic 1. flat_atomic 9168 2. s_waitcnt lgkm/vmcnt(0) 9169 9170 - Use lgkmcnt(0) if not 9171 TgSplit execution mode 9172 and vmcnt(0) if TgSplit 9173 execution mode. 9174 - If OpenCL, omit lgkmcnt(0). 9175 - Must happen before 9176 the following 9177 buffer_inv and 9178 any following 9179 global/generic 9180 load/load 9181 atomic/store/store 9182 atomic/atomicrmw. 9183 - Ensures any 9184 following global 9185 data read is no 9186 older than a local 9187 atomicrmw value 9188 being acquired. 9189 9190 3. buffer_inv sc0=1 9191 9192 - If not TgSplit execution 9193 mode, omit. 9194 - Ensures that 9195 following 9196 loads will not see 9197 stale data. 9198 9199 atomicrmw acquire - agent - global 1. buffer/global_atomic 9200 2. s_waitcnt vmcnt(0) 9201 9202 - Must happen before 9203 following 9204 buffer_inv. 9205 - Ensures the 9206 atomicrmw has 9207 completed before 9208 invalidating the 9209 cache. 9210 9211 3. buffer_inv sc1=1 9212 9213 - Must happen before 9214 any following 9215 global/generic 9216 load/load 9217 atomic/atomicrmw. 9218 - Ensures that 9219 following loads 9220 will not see stale 9221 global data. 9222 9223 atomicrmw acquire - system - global 1. buffer/global_atomic 9224 sc1=1 9225 2. s_waitcnt vmcnt(0) 9226 9227 - Must happen before 9228 following 9229 buffer_inv. 9230 - Ensures the 9231 atomicrmw has 9232 completed before 9233 invalidating the 9234 caches. 9235 9236 3. buffer_inv sc0=1 sc1=1 9237 9238 - Must happen before 9239 any following 9240 global/generic 9241 load/load 9242 atomic/atomicrmw. 9243 - Ensures that 9244 following 9245 loads will not see 9246 stale MTYPE NC global data. 9247 MTYPE RW and CC memory will 9248 never be stale due to the 9249 memory probes. 9250 9251 atomicrmw acquire - agent - generic 1. flat_atomic 9252 2. s_waitcnt vmcnt(0) & 9253 lgkmcnt(0) 9254 9255 - If TgSplit execution mode, 9256 omit lgkmcnt(0). 9257 - If OpenCL, omit 9258 lgkmcnt(0). 9259 - Must happen before 9260 following 9261 buffer_inv. 9262 - Ensures the 9263 atomicrmw has 9264 completed before 9265 invalidating the 9266 cache. 9267 9268 3. buffer_inv sc1=1 9269 9270 - Must happen before 9271 any following 9272 global/generic 9273 load/load 9274 atomic/atomicrmw. 9275 - Ensures that 9276 following loads 9277 will not see stale 9278 global data. 9279 9280 atomicrmw acquire - system - generic 1. flat_atomic sc1=1 9281 2. s_waitcnt vmcnt(0) & 9282 lgkmcnt(0) 9283 9284 - If TgSplit execution mode, 9285 omit lgkmcnt(0). 9286 - If OpenCL, omit 9287 lgkmcnt(0). 9288 - Must happen before 9289 following 9290 buffer_inv. 9291 - Ensures the 9292 atomicrmw has 9293 completed before 9294 invalidating the 9295 caches. 9296 9297 3. buffer_inv sc0=1 sc1=1 9298 9299 - Must happen before 9300 any following 9301 global/generic 9302 load/load 9303 atomic/atomicrmw. 9304 - Ensures that 9305 following 9306 loads will not see 9307 stale MTYPE NC global data. 9308 MTYPE RW and CC memory will 9309 never be stale due to the 9310 memory probes. 9311 9312 fence acquire - singlethread *none* *none* 9313 - wavefront 9314 fence acquire - workgroup *none* 1. s_waitcnt lgkm/vmcnt(0) 9315 9316 - Use lgkmcnt(0) if not 9317 TgSplit execution mode 9318 and vmcnt(0) if TgSplit 9319 execution mode. 9320 - If OpenCL and 9321 address space is 9322 not generic, omit 9323 lgkmcnt(0). 9324 - If OpenCL and 9325 address space is 9326 local, omit 9327 vmcnt(0). 9328 - However, since LLVM 9329 currently has no 9330 address space on 9331 the fence need to 9332 conservatively 9333 always generate. If 9334 fence had an 9335 address space then 9336 set to address 9337 space of OpenCL 9338 fence flag, or to 9339 generic if both 9340 local and global 9341 flags are 9342 specified. 9343 - s_waitcnt vmcnt(0) 9344 must happen after 9345 any preceding 9346 global/generic load 9347 atomic/ 9348 atomicrmw 9349 with an equal or 9350 wider sync scope 9351 and memory ordering 9352 stronger than 9353 unordered (this is 9354 termed the 9355 fence-paired-atomic). 9356 - s_waitcnt lgkmcnt(0) 9357 must happen after 9358 any preceding 9359 local/generic load 9360 atomic/atomicrmw 9361 with an equal or 9362 wider sync scope 9363 and memory ordering 9364 stronger than 9365 unordered (this is 9366 termed the 9367 fence-paired-atomic). 9368 - Must happen before 9369 the following 9370 buffer_inv and 9371 any following 9372 global/generic 9373 load/load 9374 atomic/store/store 9375 atomic/atomicrmw. 9376 - Ensures any 9377 following global 9378 data read is no 9379 older than the 9380 value read by the 9381 fence-paired-atomic. 9382 9383 3. buffer_inv sc0=1 9384 9385 - If not TgSplit execution 9386 mode, omit. 9387 - Ensures that 9388 following 9389 loads will not see 9390 stale data. 9391 9392 fence acquire - agent *none* 1. s_waitcnt lgkmcnt(0) & 9393 vmcnt(0) 9394 9395 - If TgSplit execution mode, 9396 omit lgkmcnt(0). 9397 - If OpenCL and 9398 address space is 9399 not generic, omit 9400 lgkmcnt(0). 9401 - However, since LLVM 9402 currently has no 9403 address space on 9404 the fence need to 9405 conservatively 9406 always generate 9407 (see comment for 9408 previous fence). 9409 - Could be split into 9410 separate s_waitcnt 9411 vmcnt(0) and 9412 s_waitcnt 9413 lgkmcnt(0) to allow 9414 them to be 9415 independently moved 9416 according to the 9417 following rules. 9418 - s_waitcnt vmcnt(0) 9419 must happen after 9420 any preceding 9421 global/generic load 9422 atomic/atomicrmw 9423 with an equal or 9424 wider sync scope 9425 and memory ordering 9426 stronger than 9427 unordered (this is 9428 termed the 9429 fence-paired-atomic). 9430 - s_waitcnt lgkmcnt(0) 9431 must happen after 9432 any preceding 9433 local/generic load 9434 atomic/atomicrmw 9435 with an equal or 9436 wider sync scope 9437 and memory ordering 9438 stronger than 9439 unordered (this is 9440 termed the 9441 fence-paired-atomic). 9442 - Must happen before 9443 the following 9444 buffer_inv. 9445 - Ensures that the 9446 fence-paired atomic 9447 has completed 9448 before invalidating 9449 the 9450 cache. Therefore 9451 any following 9452 locations read must 9453 be no older than 9454 the value read by 9455 the 9456 fence-paired-atomic. 9457 9458 2. buffer_inv sc1=1 9459 9460 - Must happen before any 9461 following global/generic 9462 load/load 9463 atomic/store/store 9464 atomic/atomicrmw. 9465 - Ensures that 9466 following loads 9467 will not see stale 9468 global data. 9469 9470 fence acquire - system *none* 1. s_waitcnt lgkmcnt(0) & 9471 vmcnt(0) 9472 9473 - If TgSplit execution mode, 9474 omit lgkmcnt(0). 9475 - If OpenCL and 9476 address space is 9477 not generic, omit 9478 lgkmcnt(0). 9479 - However, since LLVM 9480 currently has no 9481 address space on 9482 the fence need to 9483 conservatively 9484 always generate 9485 (see comment for 9486 previous fence). 9487 - Could be split into 9488 separate s_waitcnt 9489 vmcnt(0) and 9490 s_waitcnt 9491 lgkmcnt(0) to allow 9492 them to be 9493 independently moved 9494 according to the 9495 following rules. 9496 - s_waitcnt vmcnt(0) 9497 must happen after 9498 any preceding 9499 global/generic load 9500 atomic/atomicrmw 9501 with an equal or 9502 wider sync scope 9503 and memory ordering 9504 stronger than 9505 unordered (this is 9506 termed the 9507 fence-paired-atomic). 9508 - s_waitcnt lgkmcnt(0) 9509 must happen after 9510 any preceding 9511 local/generic load 9512 atomic/atomicrmw 9513 with an equal or 9514 wider sync scope 9515 and memory ordering 9516 stronger than 9517 unordered (this is 9518 termed the 9519 fence-paired-atomic). 9520 - Must happen before 9521 the following 9522 buffer_inv. 9523 - Ensures that the 9524 fence-paired atomic 9525 has completed 9526 before invalidating 9527 the 9528 cache. Therefore 9529 any following 9530 locations read must 9531 be no older than 9532 the value read by 9533 the 9534 fence-paired-atomic. 9535 9536 2. buffer_inv sc0=1 sc1=1 9537 9538 - Must happen before any 9539 following global/generic 9540 load/load 9541 atomic/store/store 9542 atomic/atomicrmw. 9543 - Ensures that 9544 following loads 9545 will not see stale 9546 global data. 9547 9548 **Release Atomic** 9549 ------------------------------------------------------------------------------------ 9550 store atomic release - singlethread - global 1. buffer/global/flat_store 9551 - wavefront - generic 9552 store atomic release - singlethread - local *If TgSplit execution mode, 9553 - wavefront local address space cannot 9554 be used.* 9555 9556 1. ds_store 9557 store atomic release - workgroup - global 1. s_waitcnt lgkm/vmcnt(0) 9558 - generic 9559 - Use lgkmcnt(0) if not 9560 TgSplit execution mode 9561 and vmcnt(0) if TgSplit 9562 execution mode. 9563 - If OpenCL, omit lgkmcnt(0). 9564 - s_waitcnt vmcnt(0) 9565 must happen after 9566 any preceding 9567 global/generic load/store/ 9568 load atomic/store atomic/ 9569 atomicrmw. 9570 - s_waitcnt lgkmcnt(0) 9571 must happen after 9572 any preceding 9573 local/generic 9574 load/store/load 9575 atomic/store 9576 atomic/atomicrmw. 9577 - Must happen before 9578 the following 9579 store. 9580 - Ensures that all 9581 memory operations 9582 have 9583 completed before 9584 performing the 9585 store that is being 9586 released. 9587 9588 2. buffer/global/flat_store sc0=1 9589 store atomic release - workgroup - local *If TgSplit execution mode, 9590 local address space cannot 9591 be used.* 9592 9593 1. ds_store 9594 store atomic release - agent - global 1. buffer_wbl2 sc1=1 9595 - generic 9596 - Must happen before 9597 following s_waitcnt. 9598 - Performs L2 writeback to 9599 ensure previous 9600 global/generic 9601 store/atomicrmw are 9602 visible at agent scope. 9603 9604 2. s_waitcnt lgkmcnt(0) & 9605 vmcnt(0) 9606 9607 - If TgSplit execution mode, 9608 omit lgkmcnt(0). 9609 - If OpenCL and 9610 address space is 9611 not generic, omit 9612 lgkmcnt(0). 9613 - Could be split into 9614 separate s_waitcnt 9615 vmcnt(0) and 9616 s_waitcnt 9617 lgkmcnt(0) to allow 9618 them to be 9619 independently moved 9620 according to the 9621 following rules. 9622 - s_waitcnt vmcnt(0) 9623 must happen after 9624 any preceding 9625 global/generic 9626 load/store/load 9627 atomic/store 9628 atomic/atomicrmw. 9629 - s_waitcnt lgkmcnt(0) 9630 must happen after 9631 any preceding 9632 local/generic 9633 load/store/load 9634 atomic/store 9635 atomic/atomicrmw. 9636 - Must happen before 9637 the following 9638 store. 9639 - Ensures that all 9640 memory operations 9641 to memory have 9642 completed before 9643 performing the 9644 store that is being 9645 released. 9646 9647 3. buffer/global/flat_store sc1=1 9648 store atomic release - system - global 1. buffer_wbl2 sc0=1 sc1=1 9649 - generic 9650 - Must happen before 9651 following s_waitcnt. 9652 - Performs L2 writeback to 9653 ensure previous 9654 global/generic 9655 store/atomicrmw are 9656 visible at system scope. 9657 9658 2. s_waitcnt lgkmcnt(0) & 9659 vmcnt(0) 9660 9661 - If TgSplit execution mode, 9662 omit lgkmcnt(0). 9663 - If OpenCL and 9664 address space is 9665 not generic, omit 9666 lgkmcnt(0). 9667 - Could be split into 9668 separate s_waitcnt 9669 vmcnt(0) and 9670 s_waitcnt 9671 lgkmcnt(0) to allow 9672 them to be 9673 independently moved 9674 according to the 9675 following rules. 9676 - s_waitcnt vmcnt(0) 9677 must happen after any 9678 preceding 9679 global/generic 9680 load/store/load 9681 atomic/store 9682 atomic/atomicrmw. 9683 - s_waitcnt lgkmcnt(0) 9684 must happen after any 9685 preceding 9686 local/generic 9687 load/store/load 9688 atomic/store 9689 atomic/atomicrmw. 9690 - Must happen before 9691 the following 9692 store. 9693 - Ensures that all 9694 memory operations 9695 to memory and the L2 9696 writeback have 9697 completed before 9698 performing the 9699 store that is being 9700 released. 9701 9702 3. buffer/global/flat_store 9703 sc0=1 sc1=1 9704 atomicrmw release - singlethread - global 1. buffer/global/flat_atomic 9705 - wavefront - generic 9706 atomicrmw release - singlethread - local *If TgSplit execution mode, 9707 - wavefront local address space cannot 9708 be used.* 9709 9710 1. ds_atomic 9711 atomicrmw release - workgroup - global 1. s_waitcnt lgkm/vmcnt(0) 9712 - generic 9713 - Use lgkmcnt(0) if not 9714 TgSplit execution mode 9715 and vmcnt(0) if TgSplit 9716 execution mode. 9717 - If OpenCL, omit 9718 lgkmcnt(0). 9719 - s_waitcnt vmcnt(0) 9720 must happen after 9721 any preceding 9722 global/generic load/store/ 9723 load atomic/store atomic/ 9724 atomicrmw. 9725 - s_waitcnt lgkmcnt(0) 9726 must happen after 9727 any preceding 9728 local/generic 9729 load/store/load 9730 atomic/store 9731 atomic/atomicrmw. 9732 - Must happen before 9733 the following 9734 atomicrmw. 9735 - Ensures that all 9736 memory operations 9737 have 9738 completed before 9739 performing the 9740 atomicrmw that is 9741 being released. 9742 9743 2. buffer/global/flat_atomic sc0=1 9744 atomicrmw release - workgroup - local *If TgSplit execution mode, 9745 local address space cannot 9746 be used.* 9747 9748 1. ds_atomic 9749 atomicrmw release - agent - global 1. buffer_wbl2 sc1=1 9750 - generic 9751 - Must happen before 9752 following s_waitcnt. 9753 - Performs L2 writeback to 9754 ensure previous 9755 global/generic 9756 store/atomicrmw are 9757 visible at agent scope. 9758 9759 2. s_waitcnt lgkmcnt(0) & 9760 vmcnt(0) 9761 9762 - If TgSplit execution mode, 9763 omit lgkmcnt(0). 9764 - If OpenCL, omit 9765 lgkmcnt(0). 9766 - Could be split into 9767 separate s_waitcnt 9768 vmcnt(0) and 9769 s_waitcnt 9770 lgkmcnt(0) to allow 9771 them to be 9772 independently moved 9773 according to the 9774 following rules. 9775 - s_waitcnt vmcnt(0) 9776 must happen after 9777 any preceding 9778 global/generic 9779 load/store/load 9780 atomic/store 9781 atomic/atomicrmw. 9782 - s_waitcnt lgkmcnt(0) 9783 must happen after 9784 any preceding 9785 local/generic 9786 load/store/load 9787 atomic/store 9788 atomic/atomicrmw. 9789 - Must happen before 9790 the following 9791 atomicrmw. 9792 - Ensures that all 9793 memory operations 9794 to global and local 9795 have completed 9796 before performing 9797 the atomicrmw that 9798 is being released. 9799 9800 3. buffer/global/flat_atomic sc1=1 9801 atomicrmw release - system - global 1. buffer_wbl2 sc0=1 sc1=1 9802 - generic 9803 - Must happen before 9804 following s_waitcnt. 9805 - Performs L2 writeback to 9806 ensure previous 9807 global/generic 9808 store/atomicrmw are 9809 visible at system scope. 9810 9811 2. s_waitcnt lgkmcnt(0) & 9812 vmcnt(0) 9813 9814 - If TgSplit execution mode, 9815 omit lgkmcnt(0). 9816 - If OpenCL, omit 9817 lgkmcnt(0). 9818 - Could be split into 9819 separate s_waitcnt 9820 vmcnt(0) and 9821 s_waitcnt 9822 lgkmcnt(0) to allow 9823 them to be 9824 independently moved 9825 according to the 9826 following rules. 9827 - s_waitcnt vmcnt(0) 9828 must happen after 9829 any preceding 9830 global/generic 9831 load/store/load 9832 atomic/store 9833 atomic/atomicrmw. 9834 - s_waitcnt lgkmcnt(0) 9835 must happen after 9836 any preceding 9837 local/generic 9838 load/store/load 9839 atomic/store 9840 atomic/atomicrmw. 9841 - Must happen before 9842 the following 9843 atomicrmw. 9844 - Ensures that all 9845 memory operations 9846 to memory and the L2 9847 writeback have 9848 completed before 9849 performing the 9850 store that is being 9851 released. 9852 9853 3. buffer/global/flat_atomic 9854 sc0=1 sc1=1 9855 fence release - singlethread *none* *none* 9856 - wavefront 9857 fence release - workgroup *none* 1. s_waitcnt lgkm/vmcnt(0) 9858 9859 - Use lgkmcnt(0) if not 9860 TgSplit execution mode 9861 and vmcnt(0) if TgSplit 9862 execution mode. 9863 - If OpenCL and 9864 address space is 9865 not generic, omit 9866 lgkmcnt(0). 9867 - If OpenCL and 9868 address space is 9869 local, omit 9870 vmcnt(0). 9871 - However, since LLVM 9872 currently has no 9873 address space on 9874 the fence need to 9875 conservatively 9876 always generate. If 9877 fence had an 9878 address space then 9879 set to address 9880 space of OpenCL 9881 fence flag, or to 9882 generic if both 9883 local and global 9884 flags are 9885 specified. 9886 - s_waitcnt vmcnt(0) 9887 must happen after 9888 any preceding 9889 global/generic 9890 load/store/ 9891 load atomic/store atomic/ 9892 atomicrmw. 9893 - s_waitcnt lgkmcnt(0) 9894 must happen after 9895 any preceding 9896 local/generic 9897 load/load 9898 atomic/store/store 9899 atomic/atomicrmw. 9900 - Must happen before 9901 any following store 9902 atomic/atomicrmw 9903 with an equal or 9904 wider sync scope 9905 and memory ordering 9906 stronger than 9907 unordered (this is 9908 termed the 9909 fence-paired-atomic). 9910 - Ensures that all 9911 memory operations 9912 have 9913 completed before 9914 performing the 9915 following 9916 fence-paired-atomic. 9917 9918 fence release - agent *none* 1. buffer_wbl2 sc1=1 9919 9920 - If OpenCL and 9921 address space is 9922 local, omit. 9923 - Must happen before 9924 following s_waitcnt. 9925 - Performs L2 writeback to 9926 ensure previous 9927 global/generic 9928 store/atomicrmw are 9929 visible at agent scope. 9930 9931 2. s_waitcnt lgkmcnt(0) & 9932 vmcnt(0) 9933 9934 - If TgSplit execution mode, 9935 omit lgkmcnt(0). 9936 - If OpenCL and 9937 address space is 9938 not generic, omit 9939 lgkmcnt(0). 9940 - If OpenCL and 9941 address space is 9942 local, omit 9943 vmcnt(0). 9944 - However, since LLVM 9945 currently has no 9946 address space on 9947 the fence need to 9948 conservatively 9949 always generate. If 9950 fence had an 9951 address space then 9952 set to address 9953 space of OpenCL 9954 fence flag, or to 9955 generic if both 9956 local and global 9957 flags are 9958 specified. 9959 - Could be split into 9960 separate s_waitcnt 9961 vmcnt(0) and 9962 s_waitcnt 9963 lgkmcnt(0) to allow 9964 them to be 9965 independently moved 9966 according to the 9967 following rules. 9968 - s_waitcnt vmcnt(0) 9969 must happen after 9970 any preceding 9971 global/generic 9972 load/store/load 9973 atomic/store 9974 atomic/atomicrmw. 9975 - s_waitcnt lgkmcnt(0) 9976 must happen after 9977 any preceding 9978 local/generic 9979 load/store/load 9980 atomic/store 9981 atomic/atomicrmw. 9982 - Must happen before 9983 any following store 9984 atomic/atomicrmw 9985 with an equal or 9986 wider sync scope 9987 and memory ordering 9988 stronger than 9989 unordered (this is 9990 termed the 9991 fence-paired-atomic). 9992 - Ensures that all 9993 memory operations 9994 have 9995 completed before 9996 performing the 9997 following 9998 fence-paired-atomic. 9999 10000 fence release - system *none* 1. buffer_wbl2 sc0=1 sc1=1 10001 10002 - Must happen before 10003 following s_waitcnt. 10004 - Performs L2 writeback to 10005 ensure previous 10006 global/generic 10007 store/atomicrmw are 10008 visible at system scope. 10009 10010 2. s_waitcnt lgkmcnt(0) & 10011 vmcnt(0) 10012 10013 - If TgSplit execution mode, 10014 omit lgkmcnt(0). 10015 - If OpenCL and 10016 address space is 10017 not generic, omit 10018 lgkmcnt(0). 10019 - If OpenCL and 10020 address space is 10021 local, omit 10022 vmcnt(0). 10023 - However, since LLVM 10024 currently has no 10025 address space on 10026 the fence need to 10027 conservatively 10028 always generate. If 10029 fence had an 10030 address space then 10031 set to address 10032 space of OpenCL 10033 fence flag, or to 10034 generic if both 10035 local and global 10036 flags are 10037 specified. 10038 - Could be split into 10039 separate s_waitcnt 10040 vmcnt(0) and 10041 s_waitcnt 10042 lgkmcnt(0) to allow 10043 them to be 10044 independently moved 10045 according to the 10046 following rules. 10047 - s_waitcnt vmcnt(0) 10048 must happen after 10049 any preceding 10050 global/generic 10051 load/store/load 10052 atomic/store 10053 atomic/atomicrmw. 10054 - s_waitcnt lgkmcnt(0) 10055 must happen after 10056 any preceding 10057 local/generic 10058 load/store/load 10059 atomic/store 10060 atomic/atomicrmw. 10061 - Must happen before 10062 any following store 10063 atomic/atomicrmw 10064 with an equal or 10065 wider sync scope 10066 and memory ordering 10067 stronger than 10068 unordered (this is 10069 termed the 10070 fence-paired-atomic). 10071 - Ensures that all 10072 memory operations 10073 have 10074 completed before 10075 performing the 10076 following 10077 fence-paired-atomic. 10078 10079 **Acquire-Release Atomic** 10080 ------------------------------------------------------------------------------------ 10081 atomicrmw acq_rel - singlethread - global 1. buffer/global/flat_atomic 10082 - wavefront - generic 10083 atomicrmw acq_rel - singlethread - local *If TgSplit execution mode, 10084 - wavefront local address space cannot 10085 be used.* 10086 10087 1. ds_atomic 10088 atomicrmw acq_rel - workgroup - global 1. s_waitcnt lgkm/vmcnt(0) 10089 10090 - Use lgkmcnt(0) if not 10091 TgSplit execution mode 10092 and vmcnt(0) if TgSplit 10093 execution mode. 10094 - If OpenCL, omit 10095 lgkmcnt(0). 10096 - Must happen after 10097 any preceding 10098 local/generic 10099 load/store/load 10100 atomic/store 10101 atomic/atomicrmw. 10102 - s_waitcnt vmcnt(0) 10103 must happen after 10104 any preceding 10105 global/generic load/store/ 10106 load atomic/store atomic/ 10107 atomicrmw. 10108 - s_waitcnt lgkmcnt(0) 10109 must happen after 10110 any preceding 10111 local/generic 10112 load/store/load 10113 atomic/store 10114 atomic/atomicrmw. 10115 - Must happen before 10116 the following 10117 atomicrmw. 10118 - Ensures that all 10119 memory operations 10120 have 10121 completed before 10122 performing the 10123 atomicrmw that is 10124 being released. 10125 10126 2. buffer/global_atomic 10127 3. s_waitcnt vmcnt(0) 10128 10129 - If not TgSplit execution 10130 mode, omit. 10131 - Must happen before 10132 the following 10133 buffer_inv. 10134 - Ensures any 10135 following global 10136 data read is no 10137 older than the 10138 atomicrmw value 10139 being acquired. 10140 10141 4. buffer_inv sc0=1 10142 10143 - If not TgSplit execution 10144 mode, omit. 10145 - Ensures that 10146 following 10147 loads will not see 10148 stale data. 10149 10150 atomicrmw acq_rel - workgroup - local *If TgSplit execution mode, 10151 local address space cannot 10152 be used.* 10153 10154 1. ds_atomic 10155 2. s_waitcnt lgkmcnt(0) 10156 10157 - If OpenCL, omit. 10158 - Must happen before 10159 any following 10160 global/generic 10161 load/load 10162 atomic/store/store 10163 atomic/atomicrmw. 10164 - Ensures any 10165 following global 10166 data read is no 10167 older than the local load 10168 atomic value being 10169 acquired. 10170 10171 atomicrmw acq_rel - workgroup - generic 1. s_waitcnt lgkm/vmcnt(0) 10172 10173 - Use lgkmcnt(0) if not 10174 TgSplit execution mode 10175 and vmcnt(0) if TgSplit 10176 execution mode. 10177 - If OpenCL, omit 10178 lgkmcnt(0). 10179 - s_waitcnt vmcnt(0) 10180 must happen after 10181 any preceding 10182 global/generic load/store/ 10183 load atomic/store atomic/ 10184 atomicrmw. 10185 - s_waitcnt lgkmcnt(0) 10186 must happen after 10187 any preceding 10188 local/generic 10189 load/store/load 10190 atomic/store 10191 atomic/atomicrmw. 10192 - Must happen before 10193 the following 10194 atomicrmw. 10195 - Ensures that all 10196 memory operations 10197 have 10198 completed before 10199 performing the 10200 atomicrmw that is 10201 being released. 10202 10203 2. flat_atomic 10204 3. s_waitcnt lgkmcnt(0) & 10205 vmcnt(0) 10206 10207 - If not TgSplit execution 10208 mode, omit vmcnt(0). 10209 - If OpenCL, omit 10210 lgkmcnt(0). 10211 - Must happen before 10212 the following 10213 buffer_inv and 10214 any following 10215 global/generic 10216 load/load 10217 atomic/store/store 10218 atomic/atomicrmw. 10219 - Ensures any 10220 following global 10221 data read is no 10222 older than a local load 10223 atomic value being 10224 acquired. 10225 10226 3. buffer_inv sc0=1 10227 10228 - If not TgSplit execution 10229 mode, omit. 10230 - Ensures that 10231 following 10232 loads will not see 10233 stale data. 10234 10235 atomicrmw acq_rel - agent - global 1. buffer_wbl2 sc1=1 10236 10237 - Must happen before 10238 following s_waitcnt. 10239 - Performs L2 writeback to 10240 ensure previous 10241 global/generic 10242 store/atomicrmw are 10243 visible at agent scope. 10244 10245 2. s_waitcnt lgkmcnt(0) & 10246 vmcnt(0) 10247 10248 - If TgSplit execution mode, 10249 omit lgkmcnt(0). 10250 - If OpenCL, omit 10251 lgkmcnt(0). 10252 - Could be split into 10253 separate s_waitcnt 10254 vmcnt(0) and 10255 s_waitcnt 10256 lgkmcnt(0) to allow 10257 them to be 10258 independently moved 10259 according to the 10260 following rules. 10261 - s_waitcnt vmcnt(0) 10262 must happen after 10263 any preceding 10264 global/generic 10265 load/store/load 10266 atomic/store 10267 atomic/atomicrmw. 10268 - s_waitcnt lgkmcnt(0) 10269 must happen after 10270 any preceding 10271 local/generic 10272 load/store/load 10273 atomic/store 10274 atomic/atomicrmw. 10275 - Must happen before 10276 the following 10277 atomicrmw. 10278 - Ensures that all 10279 memory operations 10280 to global have 10281 completed before 10282 performing the 10283 atomicrmw that is 10284 being released. 10285 10286 3. buffer/global_atomic 10287 4. s_waitcnt vmcnt(0) 10288 10289 - Must happen before 10290 following 10291 buffer_inv. 10292 - Ensures the 10293 atomicrmw has 10294 completed before 10295 invalidating the 10296 cache. 10297 10298 5. buffer_inv sc1=1 10299 10300 - Must happen before 10301 any following 10302 global/generic 10303 load/load 10304 atomic/atomicrmw. 10305 - Ensures that 10306 following loads 10307 will not see stale 10308 global data. 10309 10310 atomicrmw acq_rel - system - global 1. buffer_wbl2 sc0=1 sc1=1 10311 10312 - Must happen before 10313 following s_waitcnt. 10314 - Performs L2 writeback to 10315 ensure previous 10316 global/generic 10317 store/atomicrmw are 10318 visible at system scope. 10319 10320 2. s_waitcnt lgkmcnt(0) & 10321 vmcnt(0) 10322 10323 - If TgSplit execution mode, 10324 omit lgkmcnt(0). 10325 - If OpenCL, omit 10326 lgkmcnt(0). 10327 - Could be split into 10328 separate s_waitcnt 10329 vmcnt(0) and 10330 s_waitcnt 10331 lgkmcnt(0) to allow 10332 them to be 10333 independently moved 10334 according to the 10335 following rules. 10336 - s_waitcnt vmcnt(0) 10337 must happen after 10338 any preceding 10339 global/generic 10340 load/store/load 10341 atomic/store 10342 atomic/atomicrmw. 10343 - s_waitcnt lgkmcnt(0) 10344 must happen after 10345 any preceding 10346 local/generic 10347 load/store/load 10348 atomic/store 10349 atomic/atomicrmw. 10350 - Must happen before 10351 the following 10352 atomicrmw. 10353 - Ensures that all 10354 memory operations 10355 to global and L2 writeback 10356 have completed before 10357 performing the 10358 atomicrmw that is 10359 being released. 10360 10361 3. buffer/global_atomic 10362 sc1=1 10363 4. s_waitcnt vmcnt(0) 10364 10365 - Must happen before 10366 following 10367 buffer_inv. 10368 - Ensures the 10369 atomicrmw has 10370 completed before 10371 invalidating the 10372 caches. 10373 10374 5. buffer_inv sc0=1 sc1=1 10375 10376 - Must happen before 10377 any following 10378 global/generic 10379 load/load 10380 atomic/atomicrmw. 10381 - Ensures that 10382 following loads 10383 will not see stale 10384 MTYPE NC global data. 10385 MTYPE RW and CC memory will 10386 never be stale due to the 10387 memory probes. 10388 10389 atomicrmw acq_rel - agent - generic 1. buffer_wbl2 sc1=1 10390 10391 - Must happen before 10392 following s_waitcnt. 10393 - Performs L2 writeback to 10394 ensure previous 10395 global/generic 10396 store/atomicrmw are 10397 visible at agent scope. 10398 10399 2. s_waitcnt lgkmcnt(0) & 10400 vmcnt(0) 10401 10402 - If TgSplit execution mode, 10403 omit lgkmcnt(0). 10404 - If OpenCL, omit 10405 lgkmcnt(0). 10406 - Could be split into 10407 separate s_waitcnt 10408 vmcnt(0) and 10409 s_waitcnt 10410 lgkmcnt(0) to allow 10411 them to be 10412 independently moved 10413 according to the 10414 following rules. 10415 - s_waitcnt vmcnt(0) 10416 must happen after 10417 any preceding 10418 global/generic 10419 load/store/load 10420 atomic/store 10421 atomic/atomicrmw. 10422 - s_waitcnt lgkmcnt(0) 10423 must happen after 10424 any preceding 10425 local/generic 10426 load/store/load 10427 atomic/store 10428 atomic/atomicrmw. 10429 - Must happen before 10430 the following 10431 atomicrmw. 10432 - Ensures that all 10433 memory operations 10434 to global have 10435 completed before 10436 performing the 10437 atomicrmw that is 10438 being released. 10439 10440 3. flat_atomic 10441 4. s_waitcnt vmcnt(0) & 10442 lgkmcnt(0) 10443 10444 - If TgSplit execution mode, 10445 omit lgkmcnt(0). 10446 - If OpenCL, omit 10447 lgkmcnt(0). 10448 - Must happen before 10449 following 10450 buffer_inv. 10451 - Ensures the 10452 atomicrmw has 10453 completed before 10454 invalidating the 10455 cache. 10456 10457 5. buffer_inv sc1=1 10458 10459 - Must happen before 10460 any following 10461 global/generic 10462 load/load 10463 atomic/atomicrmw. 10464 - Ensures that 10465 following loads 10466 will not see stale 10467 global data. 10468 10469 atomicrmw acq_rel - system - generic 1. buffer_wbl2 sc0=1 sc1=1 10470 10471 - Must happen before 10472 following s_waitcnt. 10473 - Performs L2 writeback to 10474 ensure previous 10475 global/generic 10476 store/atomicrmw are 10477 visible at system scope. 10478 10479 2. s_waitcnt lgkmcnt(0) & 10480 vmcnt(0) 10481 10482 - If TgSplit execution mode, 10483 omit lgkmcnt(0). 10484 - If OpenCL, omit 10485 lgkmcnt(0). 10486 - Could be split into 10487 separate s_waitcnt 10488 vmcnt(0) and 10489 s_waitcnt 10490 lgkmcnt(0) to allow 10491 them to be 10492 independently moved 10493 according to the 10494 following rules. 10495 - s_waitcnt vmcnt(0) 10496 must happen after 10497 any preceding 10498 global/generic 10499 load/store/load 10500 atomic/store 10501 atomic/atomicrmw. 10502 - s_waitcnt lgkmcnt(0) 10503 must happen after 10504 any preceding 10505 local/generic 10506 load/store/load 10507 atomic/store 10508 atomic/atomicrmw. 10509 - Must happen before 10510 the following 10511 atomicrmw. 10512 - Ensures that all 10513 memory operations 10514 to global and L2 writeback 10515 have completed before 10516 performing the 10517 atomicrmw that is 10518 being released. 10519 10520 3. flat_atomic sc1=1 10521 4. s_waitcnt vmcnt(0) & 10522 lgkmcnt(0) 10523 10524 - If TgSplit execution mode, 10525 omit lgkmcnt(0). 10526 - If OpenCL, omit 10527 lgkmcnt(0). 10528 - Must happen before 10529 following 10530 buffer_inv. 10531 - Ensures the 10532 atomicrmw has 10533 completed before 10534 invalidating the 10535 caches. 10536 10537 5. buffer_inv sc0=1 sc1=1 10538 10539 - Must happen before 10540 any following 10541 global/generic 10542 load/load 10543 atomic/atomicrmw. 10544 - Ensures that 10545 following loads 10546 will not see stale 10547 MTYPE NC global data. 10548 MTYPE RW and CC memory will 10549 never be stale due to the 10550 memory probes. 10551 10552 fence acq_rel - singlethread *none* *none* 10553 - wavefront 10554 fence acq_rel - workgroup *none* 1. s_waitcnt lgkm/vmcnt(0) 10555 10556 - Use lgkmcnt(0) if not 10557 TgSplit execution mode 10558 and vmcnt(0) if TgSplit 10559 execution mode. 10560 - If OpenCL and 10561 address space is 10562 not generic, omit 10563 lgkmcnt(0). 10564 - If OpenCL and 10565 address space is 10566 local, omit 10567 vmcnt(0). 10568 - However, 10569 since LLVM 10570 currently has no 10571 address space on 10572 the fence need to 10573 conservatively 10574 always generate 10575 (see comment for 10576 previous fence). 10577 - s_waitcnt vmcnt(0) 10578 must happen after 10579 any preceding 10580 global/generic 10581 load/store/ 10582 load atomic/store atomic/ 10583 atomicrmw. 10584 - s_waitcnt lgkmcnt(0) 10585 must happen after 10586 any preceding 10587 local/generic 10588 load/load 10589 atomic/store/store 10590 atomic/atomicrmw. 10591 - Must happen before 10592 any following 10593 global/generic 10594 load/load 10595 atomic/store/store 10596 atomic/atomicrmw. 10597 - Ensures that all 10598 memory operations 10599 have 10600 completed before 10601 performing any 10602 following global 10603 memory operations. 10604 - Ensures that the 10605 preceding 10606 local/generic load 10607 atomic/atomicrmw 10608 with an equal or 10609 wider sync scope 10610 and memory ordering 10611 stronger than 10612 unordered (this is 10613 termed the 10614 acquire-fence-paired-atomic) 10615 has completed 10616 before following 10617 global memory 10618 operations. This 10619 satisfies the 10620 requirements of 10621 acquire. 10622 - Ensures that all 10623 previous memory 10624 operations have 10625 completed before a 10626 following 10627 local/generic store 10628 atomic/atomicrmw 10629 with an equal or 10630 wider sync scope 10631 and memory ordering 10632 stronger than 10633 unordered (this is 10634 termed the 10635 release-fence-paired-atomic). 10636 This satisfies the 10637 requirements of 10638 release. 10639 - Must happen before 10640 the following 10641 buffer_inv. 10642 - Ensures that the 10643 acquire-fence-paired 10644 atomic has completed 10645 before invalidating 10646 the 10647 cache. Therefore 10648 any following 10649 locations read must 10650 be no older than 10651 the value read by 10652 the 10653 acquire-fence-paired-atomic. 10654 10655 3. buffer_inv sc0=1 10656 10657 - If not TgSplit execution 10658 mode, omit. 10659 - Ensures that 10660 following 10661 loads will not see 10662 stale data. 10663 10664 fence acq_rel - agent *none* 1. buffer_wbl2 sc1=1 10665 10666 - If OpenCL and 10667 address space is 10668 local, omit. 10669 - Must happen before 10670 following s_waitcnt. 10671 - Performs L2 writeback to 10672 ensure previous 10673 global/generic 10674 store/atomicrmw are 10675 visible at agent scope. 10676 10677 2. s_waitcnt lgkmcnt(0) & 10678 vmcnt(0) 10679 10680 - If TgSplit execution mode, 10681 omit lgkmcnt(0). 10682 - If OpenCL and 10683 address space is 10684 not generic, omit 10685 lgkmcnt(0). 10686 - However, since LLVM 10687 currently has no 10688 address space on 10689 the fence need to 10690 conservatively 10691 always generate 10692 (see comment for 10693 previous fence). 10694 - Could be split into 10695 separate s_waitcnt 10696 vmcnt(0) and 10697 s_waitcnt 10698 lgkmcnt(0) to allow 10699 them to be 10700 independently moved 10701 according to the 10702 following rules. 10703 - s_waitcnt vmcnt(0) 10704 must happen after 10705 any preceding 10706 global/generic 10707 load/store/load 10708 atomic/store 10709 atomic/atomicrmw. 10710 - s_waitcnt lgkmcnt(0) 10711 must happen after 10712 any preceding 10713 local/generic 10714 load/store/load 10715 atomic/store 10716 atomic/atomicrmw. 10717 - Must happen before 10718 the following 10719 buffer_inv. 10720 - Ensures that the 10721 preceding 10722 global/local/generic 10723 load 10724 atomic/atomicrmw 10725 with an equal or 10726 wider sync scope 10727 and memory ordering 10728 stronger than 10729 unordered (this is 10730 termed the 10731 acquire-fence-paired-atomic) 10732 has completed 10733 before invalidating 10734 the cache. This 10735 satisfies the 10736 requirements of 10737 acquire. 10738 - Ensures that all 10739 previous memory 10740 operations have 10741 completed before a 10742 following 10743 global/local/generic 10744 store 10745 atomic/atomicrmw 10746 with an equal or 10747 wider sync scope 10748 and memory ordering 10749 stronger than 10750 unordered (this is 10751 termed the 10752 release-fence-paired-atomic). 10753 This satisfies the 10754 requirements of 10755 release. 10756 10757 3. buffer_inv sc1=1 10758 10759 - Must happen before 10760 any following 10761 global/generic 10762 load/load 10763 atomic/store/store 10764 atomic/atomicrmw. 10765 - Ensures that 10766 following loads 10767 will not see stale 10768 global data. This 10769 satisfies the 10770 requirements of 10771 acquire. 10772 10773 fence acq_rel - system *none* 1. buffer_wbl2 sc0=1 sc1=1 10774 10775 - If OpenCL and 10776 address space is 10777 local, omit. 10778 - Must happen before 10779 following s_waitcnt. 10780 - Performs L2 writeback to 10781 ensure previous 10782 global/generic 10783 store/atomicrmw are 10784 visible at system scope. 10785 10786 1. s_waitcnt lgkmcnt(0) & 10787 vmcnt(0) 10788 10789 - If TgSplit execution mode, 10790 omit lgkmcnt(0). 10791 - If OpenCL and 10792 address space is 10793 not generic, omit 10794 lgkmcnt(0). 10795 - However, since LLVM 10796 currently has no 10797 address space on 10798 the fence need to 10799 conservatively 10800 always generate 10801 (see comment for 10802 previous fence). 10803 - Could be split into 10804 separate s_waitcnt 10805 vmcnt(0) and 10806 s_waitcnt 10807 lgkmcnt(0) to allow 10808 them to be 10809 independently moved 10810 according to the 10811 following rules. 10812 - s_waitcnt vmcnt(0) 10813 must happen after 10814 any preceding 10815 global/generic 10816 load/store/load 10817 atomic/store 10818 atomic/atomicrmw. 10819 - s_waitcnt lgkmcnt(0) 10820 must happen after 10821 any preceding 10822 local/generic 10823 load/store/load 10824 atomic/store 10825 atomic/atomicrmw. 10826 - Must happen before 10827 the following 10828 buffer_inv. 10829 - Ensures that the 10830 preceding 10831 global/local/generic 10832 load 10833 atomic/atomicrmw 10834 with an equal or 10835 wider sync scope 10836 and memory ordering 10837 stronger than 10838 unordered (this is 10839 termed the 10840 acquire-fence-paired-atomic) 10841 has completed 10842 before invalidating 10843 the cache. This 10844 satisfies the 10845 requirements of 10846 acquire. 10847 - Ensures that all 10848 previous memory 10849 operations have 10850 completed before a 10851 following 10852 global/local/generic 10853 store 10854 atomic/atomicrmw 10855 with an equal or 10856 wider sync scope 10857 and memory ordering 10858 stronger than 10859 unordered (this is 10860 termed the 10861 release-fence-paired-atomic). 10862 This satisfies the 10863 requirements of 10864 release. 10865 10866 2. buffer_inv sc0=1 sc1=1 10867 10868 - Must happen before 10869 any following 10870 global/generic 10871 load/load 10872 atomic/store/store 10873 atomic/atomicrmw. 10874 - Ensures that 10875 following loads 10876 will not see stale 10877 MTYPE NC global data. 10878 MTYPE RW and CC memory will 10879 never be stale due to the 10880 memory probes. 10881 10882 **Sequential Consistent Atomic** 10883 ------------------------------------------------------------------------------------ 10884 load atomic seq_cst - singlethread - global *Same as corresponding 10885 - wavefront - local load atomic acquire, 10886 - generic except must generate 10887 all instructions even 10888 for OpenCL.* 10889 load atomic seq_cst - workgroup - global 1. s_waitcnt lgkm/vmcnt(0) 10890 - generic 10891 - Use lgkmcnt(0) if not 10892 TgSplit execution mode 10893 and vmcnt(0) if TgSplit 10894 execution mode. 10895 - s_waitcnt lgkmcnt(0) must 10896 happen after 10897 preceding 10898 local/generic load 10899 atomic/store 10900 atomic/atomicrmw 10901 with memory 10902 ordering of seq_cst 10903 and with equal or 10904 wider sync scope. 10905 (Note that seq_cst 10906 fences have their 10907 own s_waitcnt 10908 lgkmcnt(0) and so do 10909 not need to be 10910 considered.) 10911 - s_waitcnt vmcnt(0) 10912 must happen after 10913 preceding 10914 global/generic load 10915 atomic/store 10916 atomic/atomicrmw 10917 with memory 10918 ordering of seq_cst 10919 and with equal or 10920 wider sync scope. 10921 (Note that seq_cst 10922 fences have their 10923 own s_waitcnt 10924 vmcnt(0) and so do 10925 not need to be 10926 considered.) 10927 - Ensures any 10928 preceding 10929 sequential 10930 consistent global/local 10931 memory instructions 10932 have completed 10933 before executing 10934 this sequentially 10935 consistent 10936 instruction. This 10937 prevents reordering 10938 a seq_cst store 10939 followed by a 10940 seq_cst load. (Note 10941 that seq_cst is 10942 stronger than 10943 acquire/release as 10944 the reordering of 10945 load acquire 10946 followed by a store 10947 release is 10948 prevented by the 10949 s_waitcnt of 10950 the release, but 10951 there is nothing 10952 preventing a store 10953 release followed by 10954 load acquire from 10955 completing out of 10956 order. The s_waitcnt 10957 could be placed after 10958 seq_store or before 10959 the seq_load. We 10960 choose the load to 10961 make the s_waitcnt be 10962 as late as possible 10963 so that the store 10964 may have already 10965 completed.) 10966 10967 2. *Following 10968 instructions same as 10969 corresponding load 10970 atomic acquire, 10971 except must generate 10972 all instructions even 10973 for OpenCL.* 10974 load atomic seq_cst - workgroup - local *If TgSplit execution mode, 10975 local address space cannot 10976 be used.* 10977 10978 *Same as corresponding 10979 load atomic acquire, 10980 except must generate 10981 all instructions even 10982 for OpenCL.* 10983 10984 load atomic seq_cst - agent - global 1. s_waitcnt lgkmcnt(0) & 10985 - system - generic vmcnt(0) 10986 10987 - If TgSplit execution mode, 10988 omit lgkmcnt(0). 10989 - Could be split into 10990 separate s_waitcnt 10991 vmcnt(0) 10992 and s_waitcnt 10993 lgkmcnt(0) to allow 10994 them to be 10995 independently moved 10996 according to the 10997 following rules. 10998 - s_waitcnt lgkmcnt(0) 10999 must happen after 11000 preceding 11001 global/generic load 11002 atomic/store 11003 atomic/atomicrmw 11004 with memory 11005 ordering of seq_cst 11006 and with equal or 11007 wider sync scope. 11008 (Note that seq_cst 11009 fences have their 11010 own s_waitcnt 11011 lgkmcnt(0) and so do 11012 not need to be 11013 considered.) 11014 - s_waitcnt vmcnt(0) 11015 must happen after 11016 preceding 11017 global/generic load 11018 atomic/store 11019 atomic/atomicrmw 11020 with memory 11021 ordering of seq_cst 11022 and with equal or 11023 wider sync scope. 11024 (Note that seq_cst 11025 fences have their 11026 own s_waitcnt 11027 vmcnt(0) and so do 11028 not need to be 11029 considered.) 11030 - Ensures any 11031 preceding 11032 sequential 11033 consistent global 11034 memory instructions 11035 have completed 11036 before executing 11037 this sequentially 11038 consistent 11039 instruction. This 11040 prevents reordering 11041 a seq_cst store 11042 followed by a 11043 seq_cst load. (Note 11044 that seq_cst is 11045 stronger than 11046 acquire/release as 11047 the reordering of 11048 load acquire 11049 followed by a store 11050 release is 11051 prevented by the 11052 s_waitcnt of 11053 the release, but 11054 there is nothing 11055 preventing a store 11056 release followed by 11057 load acquire from 11058 completing out of 11059 order. The s_waitcnt 11060 could be placed after 11061 seq_store or before 11062 the seq_load. We 11063 choose the load to 11064 make the s_waitcnt be 11065 as late as possible 11066 so that the store 11067 may have already 11068 completed.) 11069 11070 2. *Following 11071 instructions same as 11072 corresponding load 11073 atomic acquire, 11074 except must generate 11075 all instructions even 11076 for OpenCL.* 11077 store atomic seq_cst - singlethread - global *Same as corresponding 11078 - wavefront - local store atomic release, 11079 - workgroup - generic except must generate 11080 - agent all instructions even 11081 - system for OpenCL.* 11082 atomicrmw seq_cst - singlethread - global *Same as corresponding 11083 - wavefront - local atomicrmw acq_rel, 11084 - workgroup - generic except must generate 11085 - agent all instructions even 11086 - system for OpenCL.* 11087 fence seq_cst - singlethread *none* *Same as corresponding 11088 - wavefront fence acq_rel, 11089 - workgroup except must generate 11090 - agent all instructions even 11091 - system for OpenCL.* 11092 ============ ============ ============== ========== ================================ 11093 11094.. _amdgpu-amdhsa-memory-model-gfx10: 11095 11096Memory Model GFX10 11097++++++++++++++++++ 11098 11099For GFX10: 11100 11101* Each agent has multiple shader arrays (SA). 11102* Each SA has multiple work-group processors (WGP). 11103* Each WGP has multiple compute units (CU). 11104* Each CU has multiple SIMDs that execute wavefronts. 11105* The wavefronts for a single work-group are executed in the same 11106 WGP. In CU wavefront execution mode the wavefronts may be executed by 11107 different SIMDs in the same CU. In WGP wavefront execution mode the 11108 wavefronts may be executed by different SIMDs in different CUs in the same 11109 WGP. 11110* Each WGP has a single LDS memory shared by the wavefronts of the work-groups 11111 executing on it. 11112* All LDS operations of a WGP are performed as wavefront wide operations in a 11113 global order and involve no caching. Completion is reported to a wavefront in 11114 execution order. 11115* The LDS memory has multiple request queues shared by the SIMDs of a 11116 WGP. Therefore, the LDS operations performed by different wavefronts of a 11117 work-group can be reordered relative to each other, which can result in 11118 reordering the visibility of vector memory operations with respect to LDS 11119 operations of other wavefronts in the same work-group. A ``s_waitcnt 11120 lgkmcnt(0)`` is required to ensure synchronization between LDS operations and 11121 vector memory operations between wavefronts of a work-group, but not between 11122 operations performed by the same wavefront. 11123* The vector memory operations are performed as wavefront wide operations. 11124 Completion of load/store/sample operations are reported to a wavefront in 11125 execution order of other load/store/sample operations performed by that 11126 wavefront. 11127* The vector memory operations access a vector L0 cache. There is a single L0 11128 cache per CU. Each SIMD of a CU accesses the same L0 cache. Therefore, no 11129 special action is required for coherence between the lanes of a single 11130 wavefront. However, a ``buffer_gl0_inv`` is required for coherence between 11131 wavefronts executing in the same work-group as they may be executing on SIMDs 11132 of different CUs that access different L0s. A ``buffer_gl0_inv`` is also 11133 required for coherence between wavefronts executing in different work-groups 11134 as they may be executing on different WGPs. 11135* The scalar memory operations access a scalar L0 cache shared by all wavefronts 11136 on a WGP. The scalar and vector L0 caches are not coherent. However, scalar 11137 operations are used in a restricted way so do not impact the memory model. See 11138 :ref:`amdgpu-amdhsa-memory-spaces`. 11139* The vector and scalar memory L0 caches use an L1 cache shared by all WGPs on 11140 the same SA. Therefore, no special action is required for coherence between 11141 the wavefronts of a single work-group. However, a ``buffer_gl1_inv`` is 11142 required for coherence between wavefronts executing in different work-groups 11143 as they may be executing on different SAs that access different L1s. 11144* The L1 caches have independent quadrants to service disjoint ranges of virtual 11145 addresses. 11146* Each L0 cache has a separate request queue per L1 quadrant. Therefore, the 11147 vector and scalar memory operations performed by different wavefronts, whether 11148 executing in the same or different work-groups (which may be executing on 11149 different CUs accessing different L0s), can be reordered relative to each 11150 other. A ``s_waitcnt vmcnt(0) & vscnt(0)`` is required to ensure 11151 synchronization between vector memory operations of different wavefronts. It 11152 ensures a previous vector memory operation has completed before executing a 11153 subsequent vector memory or LDS operation and so can be used to meet the 11154 requirements of acquire, release and sequential consistency. 11155* The L1 caches use an L2 cache shared by all SAs on the same agent. 11156* The L2 cache has independent channels to service disjoint ranges of virtual 11157 addresses. 11158* Each L1 quadrant of a single SA accesses a different L2 channel. Each L1 11159 quadrant has a separate request queue per L2 channel. Therefore, the vector 11160 and scalar memory operations performed by wavefronts executing in different 11161 work-groups (which may be executing on different SAs) of an agent can be 11162 reordered relative to each other. A ``s_waitcnt vmcnt(0) & vscnt(0)`` is 11163 required to ensure synchronization between vector memory operations of 11164 different SAs. It ensures a previous vector memory operation has completed 11165 before executing a subsequent vector memory and so can be used to meet the 11166 requirements of acquire, release and sequential consistency. 11167* The L2 cache can be kept coherent with other agents on some targets, or ranges 11168 of virtual addresses can be set up to bypass it to ensure system coherence. 11169* On GFX10.3 a memory attached last level (MALL) cache exists for GPU memory. 11170 The MALL cache is fully coherent with GPU memory and has no impact on system 11171 coherence. All agents (GPU and CPU) access GPU memory through the MALL cache. 11172 11173Scalar memory operations are only used to access memory that is proven to not 11174change during the execution of the kernel dispatch. This includes constant 11175address space and global address space for program scope ``const`` variables. 11176Therefore, the kernel machine code does not have to maintain the scalar cache to 11177ensure it is coherent with the vector caches. The scalar and vector caches are 11178invalidated between kernel dispatches by CP since constant address space data 11179may change between kernel dispatch executions. See 11180:ref:`amdgpu-amdhsa-memory-spaces`. 11181 11182The one exception is if scalar writes are used to spill SGPR registers. In this 11183case the AMDGPU backend ensures the memory location used to spill is never 11184accessed by vector memory operations at the same time. If scalar writes are used 11185then a ``s_dcache_wb`` is inserted before the ``s_endpgm`` and before a function 11186return since the locations may be used for vector memory instructions by a 11187future wavefront that uses the same scratch area, or a function call that 11188creates a frame at the same address, respectively. There is no need for a 11189``s_dcache_inv`` as all scalar writes are write-before-read in the same thread. 11190 11191For kernarg backing memory: 11192 11193* CP invalidates the L0 and L1 caches at the start of each kernel dispatch. 11194* On dGPU the kernarg backing memory is accessed as MTYPE UC (uncached) to avoid 11195 needing to invalidate the L2 cache. 11196* On APU the kernarg backing memory is accessed as MTYPE CC (cache coherent) and 11197 so the L2 cache will be coherent with the CPU and other agents. 11198 11199Scratch backing memory (which is used for the private address space) is accessed 11200with MTYPE NC (non-coherent). Since the private address space is only accessed 11201by a single thread, and is always write-before-read, there is never a need to 11202invalidate these entries from the L0 or L1 caches. 11203 11204Wavefronts are executed in native mode with in-order reporting of loads and 11205sample instructions. In this mode vmcnt reports completion of load, atomic with 11206return and sample instructions in order, and the vscnt reports the completion of 11207store and atomic without return in order. See ``MEM_ORDERED`` field in 11208:ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 11209 11210Wavefronts can be executed in WGP or CU wavefront execution mode: 11211 11212* In WGP wavefront execution mode the wavefronts of a work-group are executed 11213 on the SIMDs of both CUs of the WGP. Therefore, explicit management of the per 11214 CU L0 caches is required for work-group synchronization. Also accesses to L1 11215 at work-group scope need to be explicitly ordered as the accesses from 11216 different CUs are not ordered. 11217* In CU wavefront execution mode the wavefronts of a work-group are executed on 11218 the SIMDs of a single CU of the WGP. Therefore, all global memory access by 11219 the work-group access the same L0 which in turn ensures L1 accesses are 11220 ordered and so do not require explicit management of the caches for 11221 work-group synchronization. 11222 11223See ``WGP_MODE`` field in 11224:ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table` and 11225:ref:`amdgpu-target-features`. 11226 11227The code sequences used to implement the memory model for GFX10 are defined in 11228table :ref:`amdgpu-amdhsa-memory-model-code-sequences-gfx10-table`. 11229 11230 .. table:: AMDHSA Memory Model Code Sequences GFX10 11231 :name: amdgpu-amdhsa-memory-model-code-sequences-gfx10-table 11232 11233 ============ ============ ============== ========== ================================ 11234 LLVM Instr LLVM Memory LLVM Memory AMDGPU AMDGPU Machine Code 11235 Ordering Sync Scope Address GFX10 11236 Space 11237 ============ ============ ============== ========== ================================ 11238 **Non-Atomic** 11239 ------------------------------------------------------------------------------------ 11240 load *none* *none* - global - !volatile & !nontemporal 11241 - generic 11242 - private 1. buffer/global/flat_load 11243 - constant 11244 - !volatile & nontemporal 11245 11246 1. buffer/global/flat_load 11247 slc=1 11248 11249 - volatile 11250 11251 1. buffer/global/flat_load 11252 glc=1 dlc=1 11253 2. s_waitcnt vmcnt(0) 11254 11255 - Must happen before 11256 any following volatile 11257 global/generic 11258 load/store. 11259 - Ensures that 11260 volatile 11261 operations to 11262 different 11263 addresses will not 11264 be reordered by 11265 hardware. 11266 11267 load *none* *none* - local 1. ds_load 11268 store *none* *none* - global - !volatile & !nontemporal 11269 - generic 11270 - private 1. buffer/global/flat_store 11271 - constant 11272 - !volatile & nontemporal 11273 11274 1. buffer/global/flat_store 11275 glc=1 slc=1 11276 11277 - volatile 11278 11279 1. buffer/global/flat_store 11280 2. s_waitcnt vscnt(0) 11281 11282 - Must happen before 11283 any following volatile 11284 global/generic 11285 load/store. 11286 - Ensures that 11287 volatile 11288 operations to 11289 different 11290 addresses will not 11291 be reordered by 11292 hardware. 11293 11294 store *none* *none* - local 1. ds_store 11295 **Unordered Atomic** 11296 ------------------------------------------------------------------------------------ 11297 load atomic unordered *any* *any* *Same as non-atomic*. 11298 store atomic unordered *any* *any* *Same as non-atomic*. 11299 atomicrmw unordered *any* *any* *Same as monotonic atomic*. 11300 **Monotonic Atomic** 11301 ------------------------------------------------------------------------------------ 11302 load atomic monotonic - singlethread - global 1. buffer/global/flat_load 11303 - wavefront - generic 11304 load atomic monotonic - workgroup - global 1. buffer/global/flat_load 11305 - generic glc=1 11306 11307 - If CU wavefront execution 11308 mode, omit glc=1. 11309 11310 load atomic monotonic - singlethread - local 1. ds_load 11311 - wavefront 11312 - workgroup 11313 load atomic monotonic - agent - global 1. buffer/global/flat_load 11314 - system - generic glc=1 dlc=1 11315 store atomic monotonic - singlethread - global 1. buffer/global/flat_store 11316 - wavefront - generic 11317 - workgroup 11318 - agent 11319 - system 11320 store atomic monotonic - singlethread - local 1. ds_store 11321 - wavefront 11322 - workgroup 11323 atomicrmw monotonic - singlethread - global 1. buffer/global/flat_atomic 11324 - wavefront - generic 11325 - workgroup 11326 - agent 11327 - system 11328 atomicrmw monotonic - singlethread - local 1. ds_atomic 11329 - wavefront 11330 - workgroup 11331 **Acquire Atomic** 11332 ------------------------------------------------------------------------------------ 11333 load atomic acquire - singlethread - global 1. buffer/global/ds/flat_load 11334 - wavefront - local 11335 - generic 11336 load atomic acquire - workgroup - global 1. buffer/global_load glc=1 11337 11338 - If CU wavefront execution 11339 mode, omit glc=1. 11340 11341 2. s_waitcnt vmcnt(0) 11342 11343 - If CU wavefront execution 11344 mode, omit. 11345 - Must happen before 11346 the following buffer_gl0_inv 11347 and before any following 11348 global/generic 11349 load/load 11350 atomic/store/store 11351 atomic/atomicrmw. 11352 11353 3. buffer_gl0_inv 11354 11355 - If CU wavefront execution 11356 mode, omit. 11357 - Ensures that 11358 following 11359 loads will not see 11360 stale data. 11361 11362 load atomic acquire - workgroup - local 1. ds_load 11363 2. s_waitcnt lgkmcnt(0) 11364 11365 - If OpenCL, omit. 11366 - Must happen before 11367 the following buffer_gl0_inv 11368 and before any following 11369 global/generic load/load 11370 atomic/store/store 11371 atomic/atomicrmw. 11372 - Ensures any 11373 following global 11374 data read is no 11375 older than the local load 11376 atomic value being 11377 acquired. 11378 11379 3. buffer_gl0_inv 11380 11381 - If CU wavefront execution 11382 mode, omit. 11383 - If OpenCL, omit. 11384 - Ensures that 11385 following 11386 loads will not see 11387 stale data. 11388 11389 load atomic acquire - workgroup - generic 1. flat_load glc=1 11390 11391 - If CU wavefront execution 11392 mode, omit glc=1. 11393 11394 2. s_waitcnt lgkmcnt(0) & 11395 vmcnt(0) 11396 11397 - If CU wavefront execution 11398 mode, omit vmcnt(0). 11399 - If OpenCL, omit 11400 lgkmcnt(0). 11401 - Must happen before 11402 the following 11403 buffer_gl0_inv and any 11404 following global/generic 11405 load/load 11406 atomic/store/store 11407 atomic/atomicrmw. 11408 - Ensures any 11409 following global 11410 data read is no 11411 older than a local load 11412 atomic value being 11413 acquired. 11414 11415 3. buffer_gl0_inv 11416 11417 - If CU wavefront execution 11418 mode, omit. 11419 - Ensures that 11420 following 11421 loads will not see 11422 stale data. 11423 11424 load atomic acquire - agent - global 1. buffer/global_load 11425 - system glc=1 dlc=1 11426 2. s_waitcnt vmcnt(0) 11427 11428 - Must happen before 11429 following 11430 buffer_gl*_inv. 11431 - Ensures the load 11432 has completed 11433 before invalidating 11434 the caches. 11435 11436 3. buffer_gl0_inv; 11437 buffer_gl1_inv 11438 11439 - Must happen before 11440 any following 11441 global/generic 11442 load/load 11443 atomic/atomicrmw. 11444 - Ensures that 11445 following 11446 loads will not see 11447 stale global data. 11448 11449 load atomic acquire - agent - generic 1. flat_load glc=1 dlc=1 11450 - system 2. s_waitcnt vmcnt(0) & 11451 lgkmcnt(0) 11452 11453 - If OpenCL omit 11454 lgkmcnt(0). 11455 - Must happen before 11456 following 11457 buffer_gl*_invl. 11458 - Ensures the flat_load 11459 has completed 11460 before invalidating 11461 the caches. 11462 11463 3. buffer_gl0_inv; 11464 buffer_gl1_inv 11465 11466 - Must happen before 11467 any following 11468 global/generic 11469 load/load 11470 atomic/atomicrmw. 11471 - Ensures that 11472 following loads 11473 will not see stale 11474 global data. 11475 11476 atomicrmw acquire - singlethread - global 1. buffer/global/ds/flat_atomic 11477 - wavefront - local 11478 - generic 11479 atomicrmw acquire - workgroup - global 1. buffer/global_atomic 11480 2. s_waitcnt vm/vscnt(0) 11481 11482 - If CU wavefront execution 11483 mode, omit. 11484 - Use vmcnt(0) if atomic with 11485 return and vscnt(0) if 11486 atomic with no-return. 11487 - Must happen before 11488 the following buffer_gl0_inv 11489 and before any following 11490 global/generic 11491 load/load 11492 atomic/store/store 11493 atomic/atomicrmw. 11494 11495 3. buffer_gl0_inv 11496 11497 - If CU wavefront execution 11498 mode, omit. 11499 - Ensures that 11500 following 11501 loads will not see 11502 stale data. 11503 11504 atomicrmw acquire - workgroup - local 1. ds_atomic 11505 2. s_waitcnt lgkmcnt(0) 11506 11507 - If OpenCL, omit. 11508 - Must happen before 11509 the following 11510 buffer_gl0_inv. 11511 - Ensures any 11512 following global 11513 data read is no 11514 older than the local 11515 atomicrmw value 11516 being acquired. 11517 11518 3. buffer_gl0_inv 11519 11520 - If OpenCL omit. 11521 - Ensures that 11522 following 11523 loads will not see 11524 stale data. 11525 11526 atomicrmw acquire - workgroup - generic 1. flat_atomic 11527 2. s_waitcnt lgkmcnt(0) & 11528 vm/vscnt(0) 11529 11530 - If CU wavefront execution 11531 mode, omit vm/vscnt(0). 11532 - If OpenCL, omit lgkmcnt(0). 11533 - Use vmcnt(0) if atomic with 11534 return and vscnt(0) if 11535 atomic with no-return. 11536 - Must happen before 11537 the following 11538 buffer_gl0_inv. 11539 - Ensures any 11540 following global 11541 data read is no 11542 older than a local 11543 atomicrmw value 11544 being acquired. 11545 11546 3. buffer_gl0_inv 11547 11548 - If CU wavefront execution 11549 mode, omit. 11550 - Ensures that 11551 following 11552 loads will not see 11553 stale data. 11554 11555 atomicrmw acquire - agent - global 1. buffer/global_atomic 11556 - system 2. s_waitcnt vm/vscnt(0) 11557 11558 - Use vmcnt(0) if atomic with 11559 return and vscnt(0) if 11560 atomic with no-return. 11561 - Must happen before 11562 following 11563 buffer_gl*_inv. 11564 - Ensures the 11565 atomicrmw has 11566 completed before 11567 invalidating the 11568 caches. 11569 11570 3. buffer_gl0_inv; 11571 buffer_gl1_inv 11572 11573 - Must happen before 11574 any following 11575 global/generic 11576 load/load 11577 atomic/atomicrmw. 11578 - Ensures that 11579 following loads 11580 will not see stale 11581 global data. 11582 11583 atomicrmw acquire - agent - generic 1. flat_atomic 11584 - system 2. s_waitcnt vm/vscnt(0) & 11585 lgkmcnt(0) 11586 11587 - If OpenCL, omit 11588 lgkmcnt(0). 11589 - Use vmcnt(0) if atomic with 11590 return and vscnt(0) if 11591 atomic with no-return. 11592 - Must happen before 11593 following 11594 buffer_gl*_inv. 11595 - Ensures the 11596 atomicrmw has 11597 completed before 11598 invalidating the 11599 caches. 11600 11601 3. buffer_gl0_inv; 11602 buffer_gl1_inv 11603 11604 - Must happen before 11605 any following 11606 global/generic 11607 load/load 11608 atomic/atomicrmw. 11609 - Ensures that 11610 following loads 11611 will not see stale 11612 global data. 11613 11614 fence acquire - singlethread *none* *none* 11615 - wavefront 11616 fence acquire - workgroup *none* 1. s_waitcnt lgkmcnt(0) & 11617 vmcnt(0) & vscnt(0) 11618 11619 - If CU wavefront execution 11620 mode, omit vmcnt(0) and 11621 vscnt(0). 11622 - If OpenCL and 11623 address space is 11624 not generic, omit 11625 lgkmcnt(0). 11626 - If OpenCL and 11627 address space is 11628 local, omit 11629 vmcnt(0) and vscnt(0). 11630 - However, since LLVM 11631 currently has no 11632 address space on 11633 the fence need to 11634 conservatively 11635 always generate. If 11636 fence had an 11637 address space then 11638 set to address 11639 space of OpenCL 11640 fence flag, or to 11641 generic if both 11642 local and global 11643 flags are 11644 specified. 11645 - Could be split into 11646 separate s_waitcnt 11647 vmcnt(0), s_waitcnt 11648 vscnt(0) and s_waitcnt 11649 lgkmcnt(0) to allow 11650 them to be 11651 independently moved 11652 according to the 11653 following rules. 11654 - s_waitcnt vmcnt(0) 11655 must happen after 11656 any preceding 11657 global/generic load 11658 atomic/ 11659 atomicrmw-with-return-value 11660 with an equal or 11661 wider sync scope 11662 and memory ordering 11663 stronger than 11664 unordered (this is 11665 termed the 11666 fence-paired-atomic). 11667 - s_waitcnt vscnt(0) 11668 must happen after 11669 any preceding 11670 global/generic 11671 atomicrmw-no-return-value 11672 with an equal or 11673 wider sync scope 11674 and memory ordering 11675 stronger than 11676 unordered (this is 11677 termed the 11678 fence-paired-atomic). 11679 - s_waitcnt lgkmcnt(0) 11680 must happen after 11681 any preceding 11682 local/generic load 11683 atomic/atomicrmw 11684 with an equal or 11685 wider sync scope 11686 and memory ordering 11687 stronger than 11688 unordered (this is 11689 termed the 11690 fence-paired-atomic). 11691 - Must happen before 11692 the following 11693 buffer_gl0_inv. 11694 - Ensures that the 11695 fence-paired atomic 11696 has completed 11697 before invalidating 11698 the 11699 cache. Therefore 11700 any following 11701 locations read must 11702 be no older than 11703 the value read by 11704 the 11705 fence-paired-atomic. 11706 11707 3. buffer_gl0_inv 11708 11709 - If CU wavefront execution 11710 mode, omit. 11711 - Ensures that 11712 following 11713 loads will not see 11714 stale data. 11715 11716 fence acquire - agent *none* 1. s_waitcnt lgkmcnt(0) & 11717 - system vmcnt(0) & vscnt(0) 11718 11719 - If OpenCL and 11720 address space is 11721 not generic, omit 11722 lgkmcnt(0). 11723 - If OpenCL and 11724 address space is 11725 local, omit 11726 vmcnt(0) and vscnt(0). 11727 - However, since LLVM 11728 currently has no 11729 address space on 11730 the fence need to 11731 conservatively 11732 always generate 11733 (see comment for 11734 previous fence). 11735 - Could be split into 11736 separate s_waitcnt 11737 vmcnt(0), s_waitcnt 11738 vscnt(0) and s_waitcnt 11739 lgkmcnt(0) to allow 11740 them to be 11741 independently moved 11742 according to the 11743 following rules. 11744 - s_waitcnt vmcnt(0) 11745 must happen after 11746 any preceding 11747 global/generic load 11748 atomic/ 11749 atomicrmw-with-return-value 11750 with an equal or 11751 wider sync scope 11752 and memory ordering 11753 stronger than 11754 unordered (this is 11755 termed the 11756 fence-paired-atomic). 11757 - s_waitcnt vscnt(0) 11758 must happen after 11759 any preceding 11760 global/generic 11761 atomicrmw-no-return-value 11762 with an equal or 11763 wider sync scope 11764 and memory ordering 11765 stronger than 11766 unordered (this is 11767 termed the 11768 fence-paired-atomic). 11769 - s_waitcnt lgkmcnt(0) 11770 must happen after 11771 any preceding 11772 local/generic load 11773 atomic/atomicrmw 11774 with an equal or 11775 wider sync scope 11776 and memory ordering 11777 stronger than 11778 unordered (this is 11779 termed the 11780 fence-paired-atomic). 11781 - Must happen before 11782 the following 11783 buffer_gl*_inv. 11784 - Ensures that the 11785 fence-paired atomic 11786 has completed 11787 before invalidating 11788 the 11789 caches. Therefore 11790 any following 11791 locations read must 11792 be no older than 11793 the value read by 11794 the 11795 fence-paired-atomic. 11796 11797 2. buffer_gl0_inv; 11798 buffer_gl1_inv 11799 11800 - Must happen before any 11801 following global/generic 11802 load/load 11803 atomic/store/store 11804 atomic/atomicrmw. 11805 - Ensures that 11806 following loads 11807 will not see stale 11808 global data. 11809 11810 **Release Atomic** 11811 ------------------------------------------------------------------------------------ 11812 store atomic release - singlethread - global 1. buffer/global/ds/flat_store 11813 - wavefront - local 11814 - generic 11815 store atomic release - workgroup - global 1. s_waitcnt lgkmcnt(0) & 11816 - generic vmcnt(0) & vscnt(0) 11817 11818 - If CU wavefront execution 11819 mode, omit vmcnt(0) and 11820 vscnt(0). 11821 - If OpenCL, omit 11822 lgkmcnt(0). 11823 - Could be split into 11824 separate s_waitcnt 11825 vmcnt(0), s_waitcnt 11826 vscnt(0) and s_waitcnt 11827 lgkmcnt(0) to allow 11828 them to be 11829 independently moved 11830 according to the 11831 following rules. 11832 - s_waitcnt vmcnt(0) 11833 must happen after 11834 any preceding 11835 global/generic load/load 11836 atomic/ 11837 atomicrmw-with-return-value. 11838 - s_waitcnt vscnt(0) 11839 must happen after 11840 any preceding 11841 global/generic 11842 store/store 11843 atomic/ 11844 atomicrmw-no-return-value. 11845 - s_waitcnt lgkmcnt(0) 11846 must happen after 11847 any preceding 11848 local/generic 11849 load/store/load 11850 atomic/store 11851 atomic/atomicrmw. 11852 - Must happen before 11853 the following 11854 store. 11855 - Ensures that all 11856 memory operations 11857 have 11858 completed before 11859 performing the 11860 store that is being 11861 released. 11862 11863 2. buffer/global/flat_store 11864 store atomic release - workgroup - local 1. s_waitcnt vmcnt(0) & vscnt(0) 11865 11866 - If CU wavefront execution 11867 mode, omit. 11868 - If OpenCL, omit. 11869 - Could be split into 11870 separate s_waitcnt 11871 vmcnt(0) and s_waitcnt 11872 vscnt(0) to allow 11873 them to be 11874 independently moved 11875 according to the 11876 following rules. 11877 - s_waitcnt vmcnt(0) 11878 must happen after 11879 any preceding 11880 global/generic load/load 11881 atomic/ 11882 atomicrmw-with-return-value. 11883 - s_waitcnt vscnt(0) 11884 must happen after 11885 any preceding 11886 global/generic 11887 store/store atomic/ 11888 atomicrmw-no-return-value. 11889 - Must happen before 11890 the following 11891 store. 11892 - Ensures that all 11893 global memory 11894 operations have 11895 completed before 11896 performing the 11897 store that is being 11898 released. 11899 11900 2. ds_store 11901 store atomic release - agent - global 1. s_waitcnt lgkmcnt(0) & 11902 - system - generic vmcnt(0) & vscnt(0) 11903 11904 - If OpenCL and 11905 address space is 11906 not generic, omit 11907 lgkmcnt(0). 11908 - Could be split into 11909 separate s_waitcnt 11910 vmcnt(0), s_waitcnt vscnt(0) 11911 and s_waitcnt 11912 lgkmcnt(0) to allow 11913 them to be 11914 independently moved 11915 according to the 11916 following rules. 11917 - s_waitcnt vmcnt(0) 11918 must happen after 11919 any preceding 11920 global/generic 11921 load/load 11922 atomic/ 11923 atomicrmw-with-return-value. 11924 - s_waitcnt vscnt(0) 11925 must happen after 11926 any preceding 11927 global/generic 11928 store/store atomic/ 11929 atomicrmw-no-return-value. 11930 - s_waitcnt lgkmcnt(0) 11931 must happen after 11932 any preceding 11933 local/generic 11934 load/store/load 11935 atomic/store 11936 atomic/atomicrmw. 11937 - Must happen before 11938 the following 11939 store. 11940 - Ensures that all 11941 memory operations 11942 have 11943 completed before 11944 performing the 11945 store that is being 11946 released. 11947 11948 2. buffer/global/flat_store 11949 atomicrmw release - singlethread - global 1. buffer/global/ds/flat_atomic 11950 - wavefront - local 11951 - generic 11952 atomicrmw release - workgroup - global 1. s_waitcnt lgkmcnt(0) & 11953 - generic vmcnt(0) & vscnt(0) 11954 11955 - If CU wavefront execution 11956 mode, omit vmcnt(0) and 11957 vscnt(0). 11958 - If OpenCL, omit lgkmcnt(0). 11959 - Could be split into 11960 separate s_waitcnt 11961 vmcnt(0), s_waitcnt 11962 vscnt(0) and s_waitcnt 11963 lgkmcnt(0) to allow 11964 them to be 11965 independently moved 11966 according to the 11967 following rules. 11968 - s_waitcnt vmcnt(0) 11969 must happen after 11970 any preceding 11971 global/generic load/load 11972 atomic/ 11973 atomicrmw-with-return-value. 11974 - s_waitcnt vscnt(0) 11975 must happen after 11976 any preceding 11977 global/generic 11978 store/store 11979 atomic/ 11980 atomicrmw-no-return-value. 11981 - s_waitcnt lgkmcnt(0) 11982 must happen after 11983 any preceding 11984 local/generic 11985 load/store/load 11986 atomic/store 11987 atomic/atomicrmw. 11988 - Must happen before 11989 the following 11990 atomicrmw. 11991 - Ensures that all 11992 memory operations 11993 have 11994 completed before 11995 performing the 11996 atomicrmw that is 11997 being released. 11998 11999 2. buffer/global/flat_atomic 12000 atomicrmw release - workgroup - local 1. s_waitcnt vmcnt(0) & vscnt(0) 12001 12002 - If CU wavefront execution 12003 mode, omit. 12004 - If OpenCL, omit. 12005 - Could be split into 12006 separate s_waitcnt 12007 vmcnt(0) and s_waitcnt 12008 vscnt(0) to allow 12009 them to be 12010 independently moved 12011 according to the 12012 following rules. 12013 - s_waitcnt vmcnt(0) 12014 must happen after 12015 any preceding 12016 global/generic load/load 12017 atomic/ 12018 atomicrmw-with-return-value. 12019 - s_waitcnt vscnt(0) 12020 must happen after 12021 any preceding 12022 global/generic 12023 store/store atomic/ 12024 atomicrmw-no-return-value. 12025 - Must happen before 12026 the following 12027 store. 12028 - Ensures that all 12029 global memory 12030 operations have 12031 completed before 12032 performing the 12033 store that is being 12034 released. 12035 12036 2. ds_atomic 12037 atomicrmw release - agent - global 1. s_waitcnt lgkmcnt(0) & 12038 - system - generic vmcnt(0) & vscnt(0) 12039 12040 - If OpenCL, omit 12041 lgkmcnt(0). 12042 - Could be split into 12043 separate s_waitcnt 12044 vmcnt(0), s_waitcnt 12045 vscnt(0) and s_waitcnt 12046 lgkmcnt(0) to allow 12047 them to be 12048 independently moved 12049 according to the 12050 following rules. 12051 - s_waitcnt vmcnt(0) 12052 must happen after 12053 any preceding 12054 global/generic 12055 load/load atomic/ 12056 atomicrmw-with-return-value. 12057 - s_waitcnt vscnt(0) 12058 must happen after 12059 any preceding 12060 global/generic 12061 store/store atomic/ 12062 atomicrmw-no-return-value. 12063 - s_waitcnt lgkmcnt(0) 12064 must happen after 12065 any preceding 12066 local/generic 12067 load/store/load 12068 atomic/store 12069 atomic/atomicrmw. 12070 - Must happen before 12071 the following 12072 atomicrmw. 12073 - Ensures that all 12074 memory operations 12075 to global and local 12076 have completed 12077 before performing 12078 the atomicrmw that 12079 is being released. 12080 12081 2. buffer/global/flat_atomic 12082 fence release - singlethread *none* *none* 12083 - wavefront 12084 fence release - workgroup *none* 1. s_waitcnt lgkmcnt(0) & 12085 vmcnt(0) & vscnt(0) 12086 12087 - If CU wavefront execution 12088 mode, omit vmcnt(0) and 12089 vscnt(0). 12090 - If OpenCL and 12091 address space is 12092 not generic, omit 12093 lgkmcnt(0). 12094 - If OpenCL and 12095 address space is 12096 local, omit 12097 vmcnt(0) and vscnt(0). 12098 - However, since LLVM 12099 currently has no 12100 address space on 12101 the fence need to 12102 conservatively 12103 always generate. If 12104 fence had an 12105 address space then 12106 set to address 12107 space of OpenCL 12108 fence flag, or to 12109 generic if both 12110 local and global 12111 flags are 12112 specified. 12113 - Could be split into 12114 separate s_waitcnt 12115 vmcnt(0), s_waitcnt 12116 vscnt(0) and s_waitcnt 12117 lgkmcnt(0) to allow 12118 them to be 12119 independently moved 12120 according to the 12121 following rules. 12122 - s_waitcnt vmcnt(0) 12123 must happen after 12124 any preceding 12125 global/generic 12126 load/load 12127 atomic/ 12128 atomicrmw-with-return-value. 12129 - s_waitcnt vscnt(0) 12130 must happen after 12131 any preceding 12132 global/generic 12133 store/store atomic/ 12134 atomicrmw-no-return-value. 12135 - s_waitcnt lgkmcnt(0) 12136 must happen after 12137 any preceding 12138 local/generic 12139 load/store/load 12140 atomic/store atomic/ 12141 atomicrmw. 12142 - Must happen before 12143 any following store 12144 atomic/atomicrmw 12145 with an equal or 12146 wider sync scope 12147 and memory ordering 12148 stronger than 12149 unordered (this is 12150 termed the 12151 fence-paired-atomic). 12152 - Ensures that all 12153 memory operations 12154 have 12155 completed before 12156 performing the 12157 following 12158 fence-paired-atomic. 12159 12160 fence release - agent *none* 1. s_waitcnt lgkmcnt(0) & 12161 - system vmcnt(0) & vscnt(0) 12162 12163 - If OpenCL and 12164 address space is 12165 not generic, omit 12166 lgkmcnt(0). 12167 - If OpenCL and 12168 address space is 12169 local, omit 12170 vmcnt(0) and vscnt(0). 12171 - However, since LLVM 12172 currently has no 12173 address space on 12174 the fence need to 12175 conservatively 12176 always generate. If 12177 fence had an 12178 address space then 12179 set to address 12180 space of OpenCL 12181 fence flag, or to 12182 generic if both 12183 local and global 12184 flags are 12185 specified. 12186 - Could be split into 12187 separate s_waitcnt 12188 vmcnt(0), s_waitcnt 12189 vscnt(0) and s_waitcnt 12190 lgkmcnt(0) to allow 12191 them to be 12192 independently moved 12193 according to the 12194 following rules. 12195 - s_waitcnt vmcnt(0) 12196 must happen after 12197 any preceding 12198 global/generic 12199 load/load atomic/ 12200 atomicrmw-with-return-value. 12201 - s_waitcnt vscnt(0) 12202 must happen after 12203 any preceding 12204 global/generic 12205 store/store atomic/ 12206 atomicrmw-no-return-value. 12207 - s_waitcnt lgkmcnt(0) 12208 must happen after 12209 any preceding 12210 local/generic 12211 load/store/load 12212 atomic/store 12213 atomic/atomicrmw. 12214 - Must happen before 12215 any following store 12216 atomic/atomicrmw 12217 with an equal or 12218 wider sync scope 12219 and memory ordering 12220 stronger than 12221 unordered (this is 12222 termed the 12223 fence-paired-atomic). 12224 - Ensures that all 12225 memory operations 12226 have 12227 completed before 12228 performing the 12229 following 12230 fence-paired-atomic. 12231 12232 **Acquire-Release Atomic** 12233 ------------------------------------------------------------------------------------ 12234 atomicrmw acq_rel - singlethread - global 1. buffer/global/ds/flat_atomic 12235 - wavefront - local 12236 - generic 12237 atomicrmw acq_rel - workgroup - global 1. s_waitcnt lgkmcnt(0) & 12238 vmcnt(0) & vscnt(0) 12239 12240 - If CU wavefront execution 12241 mode, omit vmcnt(0) and 12242 vscnt(0). 12243 - If OpenCL, omit 12244 lgkmcnt(0). 12245 - Must happen after 12246 any preceding 12247 local/generic 12248 load/store/load 12249 atomic/store 12250 atomic/atomicrmw. 12251 - Could be split into 12252 separate s_waitcnt 12253 vmcnt(0), s_waitcnt 12254 vscnt(0), and s_waitcnt 12255 lgkmcnt(0) to allow 12256 them to be 12257 independently moved 12258 according to the 12259 following rules. 12260 - s_waitcnt vmcnt(0) 12261 must happen after 12262 any preceding 12263 global/generic load/load 12264 atomic/ 12265 atomicrmw-with-return-value. 12266 - s_waitcnt vscnt(0) 12267 must happen after 12268 any preceding 12269 global/generic 12270 store/store 12271 atomic/ 12272 atomicrmw-no-return-value. 12273 - s_waitcnt lgkmcnt(0) 12274 must happen after 12275 any preceding 12276 local/generic 12277 load/store/load 12278 atomic/store 12279 atomic/atomicrmw. 12280 - Must happen before 12281 the following 12282 atomicrmw. 12283 - Ensures that all 12284 memory operations 12285 have 12286 completed before 12287 performing the 12288 atomicrmw that is 12289 being released. 12290 12291 2. buffer/global_atomic 12292 3. s_waitcnt vm/vscnt(0) 12293 12294 - If CU wavefront execution 12295 mode, omit. 12296 - Use vmcnt(0) if atomic with 12297 return and vscnt(0) if 12298 atomic with no-return. 12299 - Must happen before 12300 the following 12301 buffer_gl0_inv. 12302 - Ensures any 12303 following global 12304 data read is no 12305 older than the 12306 atomicrmw value 12307 being acquired. 12308 12309 4. buffer_gl0_inv 12310 12311 - If CU wavefront execution 12312 mode, omit. 12313 - Ensures that 12314 following 12315 loads will not see 12316 stale data. 12317 12318 atomicrmw acq_rel - workgroup - local 1. s_waitcnt vmcnt(0) & vscnt(0) 12319 12320 - If CU wavefront execution 12321 mode, omit. 12322 - If OpenCL, omit. 12323 - Could be split into 12324 separate s_waitcnt 12325 vmcnt(0) and s_waitcnt 12326 vscnt(0) to allow 12327 them to be 12328 independently moved 12329 according to the 12330 following rules. 12331 - s_waitcnt vmcnt(0) 12332 must happen after 12333 any preceding 12334 global/generic load/load 12335 atomic/ 12336 atomicrmw-with-return-value. 12337 - s_waitcnt vscnt(0) 12338 must happen after 12339 any preceding 12340 global/generic 12341 store/store atomic/ 12342 atomicrmw-no-return-value. 12343 - Must happen before 12344 the following 12345 store. 12346 - Ensures that all 12347 global memory 12348 operations have 12349 completed before 12350 performing the 12351 store that is being 12352 released. 12353 12354 2. ds_atomic 12355 3. s_waitcnt lgkmcnt(0) 12356 12357 - If OpenCL, omit. 12358 - Must happen before 12359 the following 12360 buffer_gl0_inv. 12361 - Ensures any 12362 following global 12363 data read is no 12364 older than the local load 12365 atomic value being 12366 acquired. 12367 12368 4. buffer_gl0_inv 12369 12370 - If CU wavefront execution 12371 mode, omit. 12372 - If OpenCL omit. 12373 - Ensures that 12374 following 12375 loads will not see 12376 stale data. 12377 12378 atomicrmw acq_rel - workgroup - generic 1. s_waitcnt lgkmcnt(0) & 12379 vmcnt(0) & vscnt(0) 12380 12381 - If CU wavefront execution 12382 mode, omit vmcnt(0) and 12383 vscnt(0). 12384 - If OpenCL, omit lgkmcnt(0). 12385 - Could be split into 12386 separate s_waitcnt 12387 vmcnt(0), s_waitcnt 12388 vscnt(0) and s_waitcnt 12389 lgkmcnt(0) to allow 12390 them to be 12391 independently moved 12392 according to the 12393 following rules. 12394 - s_waitcnt vmcnt(0) 12395 must happen after 12396 any preceding 12397 global/generic load/load 12398 atomic/ 12399 atomicrmw-with-return-value. 12400 - s_waitcnt vscnt(0) 12401 must happen after 12402 any preceding 12403 global/generic 12404 store/store 12405 atomic/ 12406 atomicrmw-no-return-value. 12407 - s_waitcnt lgkmcnt(0) 12408 must happen after 12409 any preceding 12410 local/generic 12411 load/store/load 12412 atomic/store 12413 atomic/atomicrmw. 12414 - Must happen before 12415 the following 12416 atomicrmw. 12417 - Ensures that all 12418 memory operations 12419 have 12420 completed before 12421 performing the 12422 atomicrmw that is 12423 being released. 12424 12425 2. flat_atomic 12426 3. s_waitcnt lgkmcnt(0) & 12427 vmcnt(0) & vscnt(0) 12428 12429 - If CU wavefront execution 12430 mode, omit vmcnt(0) and 12431 vscnt(0). 12432 - If OpenCL, omit lgkmcnt(0). 12433 - Must happen before 12434 the following 12435 buffer_gl0_inv. 12436 - Ensures any 12437 following global 12438 data read is no 12439 older than the load 12440 atomic value being 12441 acquired. 12442 12443 3. buffer_gl0_inv 12444 12445 - If CU wavefront execution 12446 mode, omit. 12447 - Ensures that 12448 following 12449 loads will not see 12450 stale data. 12451 12452 atomicrmw acq_rel - agent - global 1. s_waitcnt lgkmcnt(0) & 12453 - system vmcnt(0) & vscnt(0) 12454 12455 - If OpenCL, omit 12456 lgkmcnt(0). 12457 - Could be split into 12458 separate s_waitcnt 12459 vmcnt(0), s_waitcnt 12460 vscnt(0) and s_waitcnt 12461 lgkmcnt(0) to allow 12462 them to be 12463 independently moved 12464 according to the 12465 following rules. 12466 - s_waitcnt vmcnt(0) 12467 must happen after 12468 any preceding 12469 global/generic 12470 load/load atomic/ 12471 atomicrmw-with-return-value. 12472 - s_waitcnt vscnt(0) 12473 must happen after 12474 any preceding 12475 global/generic 12476 store/store atomic/ 12477 atomicrmw-no-return-value. 12478 - s_waitcnt lgkmcnt(0) 12479 must happen after 12480 any preceding 12481 local/generic 12482 load/store/load 12483 atomic/store 12484 atomic/atomicrmw. 12485 - Must happen before 12486 the following 12487 atomicrmw. 12488 - Ensures that all 12489 memory operations 12490 to global have 12491 completed before 12492 performing the 12493 atomicrmw that is 12494 being released. 12495 12496 2. buffer/global_atomic 12497 3. s_waitcnt vm/vscnt(0) 12498 12499 - Use vmcnt(0) if atomic with 12500 return and vscnt(0) if 12501 atomic with no-return. 12502 - Must happen before 12503 following 12504 buffer_gl*_inv. 12505 - Ensures the 12506 atomicrmw has 12507 completed before 12508 invalidating the 12509 caches. 12510 12511 4. buffer_gl0_inv; 12512 buffer_gl1_inv 12513 12514 - Must happen before 12515 any following 12516 global/generic 12517 load/load 12518 atomic/atomicrmw. 12519 - Ensures that 12520 following loads 12521 will not see stale 12522 global data. 12523 12524 atomicrmw acq_rel - agent - generic 1. s_waitcnt lgkmcnt(0) & 12525 - system vmcnt(0) & vscnt(0) 12526 12527 - If OpenCL, omit 12528 lgkmcnt(0). 12529 - Could be split into 12530 separate s_waitcnt 12531 vmcnt(0), s_waitcnt 12532 vscnt(0), and s_waitcnt 12533 lgkmcnt(0) to allow 12534 them to be 12535 independently moved 12536 according to the 12537 following rules. 12538 - s_waitcnt vmcnt(0) 12539 must happen after 12540 any preceding 12541 global/generic 12542 load/load atomic 12543 atomicrmw-with-return-value. 12544 - s_waitcnt vscnt(0) 12545 must happen after 12546 any preceding 12547 global/generic 12548 store/store atomic/ 12549 atomicrmw-no-return-value. 12550 - s_waitcnt lgkmcnt(0) 12551 must happen after 12552 any preceding 12553 local/generic 12554 load/store/load 12555 atomic/store 12556 atomic/atomicrmw. 12557 - Must happen before 12558 the following 12559 atomicrmw. 12560 - Ensures that all 12561 memory operations 12562 have 12563 completed before 12564 performing the 12565 atomicrmw that is 12566 being released. 12567 12568 2. flat_atomic 12569 3. s_waitcnt vm/vscnt(0) & 12570 lgkmcnt(0) 12571 12572 - If OpenCL, omit 12573 lgkmcnt(0). 12574 - Use vmcnt(0) if atomic with 12575 return and vscnt(0) if 12576 atomic with no-return. 12577 - Must happen before 12578 following 12579 buffer_gl*_inv. 12580 - Ensures the 12581 atomicrmw has 12582 completed before 12583 invalidating the 12584 caches. 12585 12586 4. buffer_gl0_inv; 12587 buffer_gl1_inv 12588 12589 - Must happen before 12590 any following 12591 global/generic 12592 load/load 12593 atomic/atomicrmw. 12594 - Ensures that 12595 following loads 12596 will not see stale 12597 global data. 12598 12599 fence acq_rel - singlethread *none* *none* 12600 - wavefront 12601 fence acq_rel - workgroup *none* 1. s_waitcnt lgkmcnt(0) & 12602 vmcnt(0) & vscnt(0) 12603 12604 - If CU wavefront execution 12605 mode, omit vmcnt(0) and 12606 vscnt(0). 12607 - If OpenCL and 12608 address space is 12609 not generic, omit 12610 lgkmcnt(0). 12611 - If OpenCL and 12612 address space is 12613 local, omit 12614 vmcnt(0) and vscnt(0). 12615 - However, 12616 since LLVM 12617 currently has no 12618 address space on 12619 the fence need to 12620 conservatively 12621 always generate 12622 (see comment for 12623 previous fence). 12624 - Could be split into 12625 separate s_waitcnt 12626 vmcnt(0), s_waitcnt 12627 vscnt(0) and s_waitcnt 12628 lgkmcnt(0) to allow 12629 them to be 12630 independently moved 12631 according to the 12632 following rules. 12633 - s_waitcnt vmcnt(0) 12634 must happen after 12635 any preceding 12636 global/generic 12637 load/load 12638 atomic/ 12639 atomicrmw-with-return-value. 12640 - s_waitcnt vscnt(0) 12641 must happen after 12642 any preceding 12643 global/generic 12644 store/store atomic/ 12645 atomicrmw-no-return-value. 12646 - s_waitcnt lgkmcnt(0) 12647 must happen after 12648 any preceding 12649 local/generic 12650 load/store/load 12651 atomic/store atomic/ 12652 atomicrmw. 12653 - Must happen before 12654 any following 12655 global/generic 12656 load/load 12657 atomic/store/store 12658 atomic/atomicrmw. 12659 - Ensures that all 12660 memory operations 12661 have 12662 completed before 12663 performing any 12664 following global 12665 memory operations. 12666 - Ensures that the 12667 preceding 12668 local/generic load 12669 atomic/atomicrmw 12670 with an equal or 12671 wider sync scope 12672 and memory ordering 12673 stronger than 12674 unordered (this is 12675 termed the 12676 acquire-fence-paired-atomic) 12677 has completed 12678 before following 12679 global memory 12680 operations. This 12681 satisfies the 12682 requirements of 12683 acquire. 12684 - Ensures that all 12685 previous memory 12686 operations have 12687 completed before a 12688 following 12689 local/generic store 12690 atomic/atomicrmw 12691 with an equal or 12692 wider sync scope 12693 and memory ordering 12694 stronger than 12695 unordered (this is 12696 termed the 12697 release-fence-paired-atomic). 12698 This satisfies the 12699 requirements of 12700 release. 12701 - Must happen before 12702 the following 12703 buffer_gl0_inv. 12704 - Ensures that the 12705 acquire-fence-paired 12706 atomic has completed 12707 before invalidating 12708 the 12709 cache. Therefore 12710 any following 12711 locations read must 12712 be no older than 12713 the value read by 12714 the 12715 acquire-fence-paired-atomic. 12716 12717 3. buffer_gl0_inv 12718 12719 - If CU wavefront execution 12720 mode, omit. 12721 - Ensures that 12722 following 12723 loads will not see 12724 stale data. 12725 12726 fence acq_rel - agent *none* 1. s_waitcnt lgkmcnt(0) & 12727 - system vmcnt(0) & vscnt(0) 12728 12729 - If OpenCL and 12730 address space is 12731 not generic, omit 12732 lgkmcnt(0). 12733 - If OpenCL and 12734 address space is 12735 local, omit 12736 vmcnt(0) and vscnt(0). 12737 - However, since LLVM 12738 currently has no 12739 address space on 12740 the fence need to 12741 conservatively 12742 always generate 12743 (see comment for 12744 previous fence). 12745 - Could be split into 12746 separate s_waitcnt 12747 vmcnt(0), s_waitcnt 12748 vscnt(0) and s_waitcnt 12749 lgkmcnt(0) to allow 12750 them to be 12751 independently moved 12752 according to the 12753 following rules. 12754 - s_waitcnt vmcnt(0) 12755 must happen after 12756 any preceding 12757 global/generic 12758 load/load 12759 atomic/ 12760 atomicrmw-with-return-value. 12761 - s_waitcnt vscnt(0) 12762 must happen after 12763 any preceding 12764 global/generic 12765 store/store atomic/ 12766 atomicrmw-no-return-value. 12767 - s_waitcnt lgkmcnt(0) 12768 must happen after 12769 any preceding 12770 local/generic 12771 load/store/load 12772 atomic/store 12773 atomic/atomicrmw. 12774 - Must happen before 12775 the following 12776 buffer_gl*_inv. 12777 - Ensures that the 12778 preceding 12779 global/local/generic 12780 load 12781 atomic/atomicrmw 12782 with an equal or 12783 wider sync scope 12784 and memory ordering 12785 stronger than 12786 unordered (this is 12787 termed the 12788 acquire-fence-paired-atomic) 12789 has completed 12790 before invalidating 12791 the caches. This 12792 satisfies the 12793 requirements of 12794 acquire. 12795 - Ensures that all 12796 previous memory 12797 operations have 12798 completed before a 12799 following 12800 global/local/generic 12801 store 12802 atomic/atomicrmw 12803 with an equal or 12804 wider sync scope 12805 and memory ordering 12806 stronger than 12807 unordered (this is 12808 termed the 12809 release-fence-paired-atomic). 12810 This satisfies the 12811 requirements of 12812 release. 12813 12814 2. buffer_gl0_inv; 12815 buffer_gl1_inv 12816 12817 - Must happen before 12818 any following 12819 global/generic 12820 load/load 12821 atomic/store/store 12822 atomic/atomicrmw. 12823 - Ensures that 12824 following loads 12825 will not see stale 12826 global data. This 12827 satisfies the 12828 requirements of 12829 acquire. 12830 12831 **Sequential Consistent Atomic** 12832 ------------------------------------------------------------------------------------ 12833 load atomic seq_cst - singlethread - global *Same as corresponding 12834 - wavefront - local load atomic acquire, 12835 - generic except must generate 12836 all instructions even 12837 for OpenCL.* 12838 load atomic seq_cst - workgroup - global 1. s_waitcnt lgkmcnt(0) & 12839 - generic vmcnt(0) & vscnt(0) 12840 12841 - If CU wavefront execution 12842 mode, omit vmcnt(0) and 12843 vscnt(0). 12844 - Could be split into 12845 separate s_waitcnt 12846 vmcnt(0), s_waitcnt 12847 vscnt(0), and s_waitcnt 12848 lgkmcnt(0) to allow 12849 them to be 12850 independently moved 12851 according to the 12852 following rules. 12853 - s_waitcnt lgkmcnt(0) must 12854 happen after 12855 preceding 12856 local/generic load 12857 atomic/store 12858 atomic/atomicrmw 12859 with memory 12860 ordering of seq_cst 12861 and with equal or 12862 wider sync scope. 12863 (Note that seq_cst 12864 fences have their 12865 own s_waitcnt 12866 lgkmcnt(0) and so do 12867 not need to be 12868 considered.) 12869 - s_waitcnt vmcnt(0) 12870 must happen after 12871 preceding 12872 global/generic load 12873 atomic/ 12874 atomicrmw-with-return-value 12875 with memory 12876 ordering of seq_cst 12877 and with equal or 12878 wider sync scope. 12879 (Note that seq_cst 12880 fences have their 12881 own s_waitcnt 12882 vmcnt(0) and so do 12883 not need to be 12884 considered.) 12885 - s_waitcnt vscnt(0) 12886 Must happen after 12887 preceding 12888 global/generic store 12889 atomic/ 12890 atomicrmw-no-return-value 12891 with memory 12892 ordering of seq_cst 12893 and with equal or 12894 wider sync scope. 12895 (Note that seq_cst 12896 fences have their 12897 own s_waitcnt 12898 vscnt(0) and so do 12899 not need to be 12900 considered.) 12901 - Ensures any 12902 preceding 12903 sequential 12904 consistent global/local 12905 memory instructions 12906 have completed 12907 before executing 12908 this sequentially 12909 consistent 12910 instruction. This 12911 prevents reordering 12912 a seq_cst store 12913 followed by a 12914 seq_cst load. (Note 12915 that seq_cst is 12916 stronger than 12917 acquire/release as 12918 the reordering of 12919 load acquire 12920 followed by a store 12921 release is 12922 prevented by the 12923 s_waitcnt of 12924 the release, but 12925 there is nothing 12926 preventing a store 12927 release followed by 12928 load acquire from 12929 completing out of 12930 order. The s_waitcnt 12931 could be placed after 12932 seq_store or before 12933 the seq_load. We 12934 choose the load to 12935 make the s_waitcnt be 12936 as late as possible 12937 so that the store 12938 may have already 12939 completed.) 12940 12941 2. *Following 12942 instructions same as 12943 corresponding load 12944 atomic acquire, 12945 except must generate 12946 all instructions even 12947 for OpenCL.* 12948 load atomic seq_cst - workgroup - local 12949 12950 1. s_waitcnt vmcnt(0) & vscnt(0) 12951 12952 - If CU wavefront execution 12953 mode, omit. 12954 - Could be split into 12955 separate s_waitcnt 12956 vmcnt(0) and s_waitcnt 12957 vscnt(0) to allow 12958 them to be 12959 independently moved 12960 according to the 12961 following rules. 12962 - s_waitcnt vmcnt(0) 12963 Must happen after 12964 preceding 12965 global/generic load 12966 atomic/ 12967 atomicrmw-with-return-value 12968 with memory 12969 ordering of seq_cst 12970 and with equal or 12971 wider sync scope. 12972 (Note that seq_cst 12973 fences have their 12974 own s_waitcnt 12975 vmcnt(0) and so do 12976 not need to be 12977 considered.) 12978 - s_waitcnt vscnt(0) 12979 Must happen after 12980 preceding 12981 global/generic store 12982 atomic/ 12983 atomicrmw-no-return-value 12984 with memory 12985 ordering of seq_cst 12986 and with equal or 12987 wider sync scope. 12988 (Note that seq_cst 12989 fences have their 12990 own s_waitcnt 12991 vscnt(0) and so do 12992 not need to be 12993 considered.) 12994 - Ensures any 12995 preceding 12996 sequential 12997 consistent global 12998 memory instructions 12999 have completed 13000 before executing 13001 this sequentially 13002 consistent 13003 instruction. This 13004 prevents reordering 13005 a seq_cst store 13006 followed by a 13007 seq_cst load. (Note 13008 that seq_cst is 13009 stronger than 13010 acquire/release as 13011 the reordering of 13012 load acquire 13013 followed by a store 13014 release is 13015 prevented by the 13016 s_waitcnt of 13017 the release, but 13018 there is nothing 13019 preventing a store 13020 release followed by 13021 load acquire from 13022 completing out of 13023 order. The s_waitcnt 13024 could be placed after 13025 seq_store or before 13026 the seq_load. We 13027 choose the load to 13028 make the s_waitcnt be 13029 as late as possible 13030 so that the store 13031 may have already 13032 completed.) 13033 13034 2. *Following 13035 instructions same as 13036 corresponding load 13037 atomic acquire, 13038 except must generate 13039 all instructions even 13040 for OpenCL.* 13041 13042 load atomic seq_cst - agent - global 1. s_waitcnt lgkmcnt(0) & 13043 - system - generic vmcnt(0) & vscnt(0) 13044 13045 - Could be split into 13046 separate s_waitcnt 13047 vmcnt(0), s_waitcnt 13048 vscnt(0) and s_waitcnt 13049 lgkmcnt(0) to allow 13050 them to be 13051 independently moved 13052 according to the 13053 following rules. 13054 - s_waitcnt lgkmcnt(0) 13055 must happen after 13056 preceding 13057 local load 13058 atomic/store 13059 atomic/atomicrmw 13060 with memory 13061 ordering of seq_cst 13062 and with equal or 13063 wider sync scope. 13064 (Note that seq_cst 13065 fences have their 13066 own s_waitcnt 13067 lgkmcnt(0) and so do 13068 not need to be 13069 considered.) 13070 - s_waitcnt vmcnt(0) 13071 must happen after 13072 preceding 13073 global/generic load 13074 atomic/ 13075 atomicrmw-with-return-value 13076 with memory 13077 ordering of seq_cst 13078 and with equal or 13079 wider sync scope. 13080 (Note that seq_cst 13081 fences have their 13082 own s_waitcnt 13083 vmcnt(0) and so do 13084 not need to be 13085 considered.) 13086 - s_waitcnt vscnt(0) 13087 Must happen after 13088 preceding 13089 global/generic store 13090 atomic/ 13091 atomicrmw-no-return-value 13092 with memory 13093 ordering of seq_cst 13094 and with equal or 13095 wider sync scope. 13096 (Note that seq_cst 13097 fences have their 13098 own s_waitcnt 13099 vscnt(0) and so do 13100 not need to be 13101 considered.) 13102 - Ensures any 13103 preceding 13104 sequential 13105 consistent global 13106 memory instructions 13107 have completed 13108 before executing 13109 this sequentially 13110 consistent 13111 instruction. This 13112 prevents reordering 13113 a seq_cst store 13114 followed by a 13115 seq_cst load. (Note 13116 that seq_cst is 13117 stronger than 13118 acquire/release as 13119 the reordering of 13120 load acquire 13121 followed by a store 13122 release is 13123 prevented by the 13124 s_waitcnt of 13125 the release, but 13126 there is nothing 13127 preventing a store 13128 release followed by 13129 load acquire from 13130 completing out of 13131 order. The s_waitcnt 13132 could be placed after 13133 seq_store or before 13134 the seq_load. We 13135 choose the load to 13136 make the s_waitcnt be 13137 as late as possible 13138 so that the store 13139 may have already 13140 completed.) 13141 13142 2. *Following 13143 instructions same as 13144 corresponding load 13145 atomic acquire, 13146 except must generate 13147 all instructions even 13148 for OpenCL.* 13149 store atomic seq_cst - singlethread - global *Same as corresponding 13150 - wavefront - local store atomic release, 13151 - workgroup - generic except must generate 13152 - agent all instructions even 13153 - system for OpenCL.* 13154 atomicrmw seq_cst - singlethread - global *Same as corresponding 13155 - wavefront - local atomicrmw acq_rel, 13156 - workgroup - generic except must generate 13157 - agent all instructions even 13158 - system for OpenCL.* 13159 fence seq_cst - singlethread *none* *Same as corresponding 13160 - wavefront fence acq_rel, 13161 - workgroup except must generate 13162 - agent all instructions even 13163 - system for OpenCL.* 13164 ============ ============ ============== ========== ================================ 13165 13166.. _amdgpu-amdhsa-trap-handler-abi: 13167 13168Trap Handler ABI 13169~~~~~~~~~~~~~~~~ 13170 13171For code objects generated by the AMDGPU backend for HSA [HSA]_ compatible 13172runtimes (see :ref:`amdgpu-os`), the runtime installs a trap handler that 13173supports the ``s_trap`` instruction. For usage see: 13174 13175- :ref:`amdgpu-trap-handler-for-amdhsa-os-v2-table` 13176- :ref:`amdgpu-trap-handler-for-amdhsa-os-v3-table` 13177- :ref:`amdgpu-trap-handler-for-amdhsa-os-v4-onwards-table` 13178 13179 .. table:: AMDGPU Trap Handler for AMDHSA OS Code Object V2 13180 :name: amdgpu-trap-handler-for-amdhsa-os-v2-table 13181 13182 =================== =============== =============== ======================================= 13183 Usage Code Sequence Trap Handler Description 13184 Inputs 13185 =================== =============== =============== ======================================= 13186 reserved ``s_trap 0x00`` Reserved by hardware. 13187 ``debugtrap(arg)`` ``s_trap 0x01`` ``SGPR0-1``: Reserved for Finalizer HSA ``debugtrap`` 13188 ``queue_ptr`` intrinsic (not implemented). 13189 ``VGPR0``: 13190 ``arg`` 13191 ``llvm.trap`` ``s_trap 0x02`` ``SGPR0-1``: Causes wave to be halted with the PC at 13192 ``queue_ptr`` the trap instruction. The associated 13193 queue is signalled to put it into the 13194 error state. When the queue is put in 13195 the error state, the waves executing 13196 dispatches on the queue will be 13197 terminated. 13198 ``llvm.debugtrap`` ``s_trap 0x03`` *none* - If debugger not enabled then behaves 13199 as a no-operation. The trap handler 13200 is entered and immediately returns to 13201 continue execution of the wavefront. 13202 - If the debugger is enabled, causes 13203 the debug trap to be reported by the 13204 debugger and the wavefront is put in 13205 the halt state with the PC at the 13206 instruction. The debugger must 13207 increment the PC and resume the wave. 13208 reserved ``s_trap 0x04`` Reserved. 13209 reserved ``s_trap 0x05`` Reserved. 13210 reserved ``s_trap 0x06`` Reserved. 13211 reserved ``s_trap 0x07`` Reserved. 13212 reserved ``s_trap 0x08`` Reserved. 13213 reserved ``s_trap 0xfe`` Reserved. 13214 reserved ``s_trap 0xff`` Reserved. 13215 =================== =============== =============== ======================================= 13216 13217.. 13218 13219 .. table:: AMDGPU Trap Handler for AMDHSA OS Code Object V3 13220 :name: amdgpu-trap-handler-for-amdhsa-os-v3-table 13221 13222 =================== =============== =============== ======================================= 13223 Usage Code Sequence Trap Handler Description 13224 Inputs 13225 =================== =============== =============== ======================================= 13226 reserved ``s_trap 0x00`` Reserved by hardware. 13227 debugger breakpoint ``s_trap 0x01`` *none* Reserved for debugger to use for 13228 breakpoints. Causes wave to be halted 13229 with the PC at the trap instruction. 13230 The debugger is responsible to resume 13231 the wave, including the instruction 13232 that the breakpoint overwrote. 13233 ``llvm.trap`` ``s_trap 0x02`` ``SGPR0-1``: Causes wave to be halted with the PC at 13234 ``queue_ptr`` the trap instruction. The associated 13235 queue is signalled to put it into the 13236 error state. When the queue is put in 13237 the error state, the waves executing 13238 dispatches on the queue will be 13239 terminated. 13240 ``llvm.debugtrap`` ``s_trap 0x03`` *none* - If debugger not enabled then behaves 13241 as a no-operation. The trap handler 13242 is entered and immediately returns to 13243 continue execution of the wavefront. 13244 - If the debugger is enabled, causes 13245 the debug trap to be reported by the 13246 debugger and the wavefront is put in 13247 the halt state with the PC at the 13248 instruction. The debugger must 13249 increment the PC and resume the wave. 13250 reserved ``s_trap 0x04`` Reserved. 13251 reserved ``s_trap 0x05`` Reserved. 13252 reserved ``s_trap 0x06`` Reserved. 13253 reserved ``s_trap 0x07`` Reserved. 13254 reserved ``s_trap 0x08`` Reserved. 13255 reserved ``s_trap 0xfe`` Reserved. 13256 reserved ``s_trap 0xff`` Reserved. 13257 =================== =============== =============== ======================================= 13258 13259.. 13260 13261 .. table:: AMDGPU Trap Handler for AMDHSA OS Code Object V4 and Above 13262 :name: amdgpu-trap-handler-for-amdhsa-os-v4-onwards-table 13263 13264 =================== =============== ================ ================= ======================================= 13265 Usage Code Sequence GFX6-GFX8 Inputs GFX9-GFX10 Inputs Description 13266 =================== =============== ================ ================= ======================================= 13267 reserved ``s_trap 0x00`` Reserved by hardware. 13268 debugger breakpoint ``s_trap 0x01`` *none* *none* Reserved for debugger to use for 13269 breakpoints. Causes wave to be halted 13270 with the PC at the trap instruction. 13271 The debugger is responsible to resume 13272 the wave, including the instruction 13273 that the breakpoint overwrote. 13274 ``llvm.trap`` ``s_trap 0x02`` ``SGPR0-1``: *none* Causes wave to be halted with the PC at 13275 ``queue_ptr`` the trap instruction. The associated 13276 queue is signalled to put it into the 13277 error state. When the queue is put in 13278 the error state, the waves executing 13279 dispatches on the queue will be 13280 terminated. 13281 ``llvm.debugtrap`` ``s_trap 0x03`` *none* *none* - If debugger not enabled then behaves 13282 as a no-operation. The trap handler 13283 is entered and immediately returns to 13284 continue execution of the wavefront. 13285 - If the debugger is enabled, causes 13286 the debug trap to be reported by the 13287 debugger and the wavefront is put in 13288 the halt state with the PC at the 13289 instruction. The debugger must 13290 increment the PC and resume the wave. 13291 reserved ``s_trap 0x04`` Reserved. 13292 reserved ``s_trap 0x05`` Reserved. 13293 reserved ``s_trap 0x06`` Reserved. 13294 reserved ``s_trap 0x07`` Reserved. 13295 reserved ``s_trap 0x08`` Reserved. 13296 reserved ``s_trap 0xfe`` Reserved. 13297 reserved ``s_trap 0xff`` Reserved. 13298 =================== =============== ================ ================= ======================================= 13299 13300.. _amdgpu-amdhsa-function-call-convention: 13301 13302Call Convention 13303~~~~~~~~~~~~~~~ 13304 13305.. note:: 13306 13307 This section is currently incomplete and has inaccuracies. It is WIP that will 13308 be updated as information is determined. 13309 13310See :ref:`amdgpu-dwarf-address-space-identifier` for information on swizzled 13311addresses. Unswizzled addresses are normal linear addresses. 13312 13313.. _amdgpu-amdhsa-function-call-convention-kernel-functions: 13314 13315Kernel Functions 13316++++++++++++++++ 13317 13318This section describes the call convention ABI for the outer kernel function. 13319 13320See :ref:`amdgpu-amdhsa-initial-kernel-execution-state` for the kernel call 13321convention. 13322 13323The following is not part of the AMDGPU kernel calling convention but describes 13324how the AMDGPU implements function calls: 13325 133261. Clang decides the kernarg layout to match the *HSA Programmer's Language 13327 Reference* [HSA]_. 13328 13329 - All structs are passed directly. 13330 - Lambda values are passed *TBA*. 13331 13332 .. TODO:: 13333 13334 - Does this really follow HSA rules? Or are structs >16 bytes passed 13335 by-value struct? 13336 - What is ABI for lambda values? 13337 133384. The kernel performs certain setup in its prolog, as described in 13339 :ref:`amdgpu-amdhsa-kernel-prolog`. 13340 13341.. _amdgpu-amdhsa-function-call-convention-non-kernel-functions: 13342 13343Non-Kernel Functions 13344++++++++++++++++++++ 13345 13346This section describes the call convention ABI for functions other than the 13347outer kernel function. 13348 13349If a kernel has function calls then scratch is always allocated and used for 13350the call stack which grows from low address to high address using the swizzled 13351scratch address space. 13352 13353On entry to a function: 13354 133551. SGPR0-3 contain a V# with the following properties (see 13356 :ref:`amdgpu-amdhsa-kernel-prolog-private-segment-buffer`): 13357 13358 * Base address pointing to the beginning of the wavefront scratch backing 13359 memory. 13360 * Swizzled with dword element size and stride of wavefront size elements. 13361 133622. The FLAT_SCRATCH register pair is setup. See 13363 :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`. 133643. GFX6-GFX8: M0 register set to the size of LDS in bytes. See 13365 :ref:`amdgpu-amdhsa-kernel-prolog-m0`. 133664. The EXEC register is set to the lanes active on entry to the function. 133675. MODE register: *TBD* 133686. VGPR0-31 and SGPR4-29 are used to pass function input arguments as described 13369 below. 133707. SGPR30-31 return address (RA). The code address that the function must 13371 return to when it completes. The value is undefined if the function is *no 13372 return*. 133738. SGPR32 is used for the stack pointer (SP). It is an unswizzled scratch 13374 offset relative to the beginning of the wavefront scratch backing memory. 13375 13376 The unswizzled SP can be used with buffer instructions as an unswizzled SGPR 13377 offset with the scratch V# in SGPR0-3 to access the stack in a swizzled 13378 manner. 13379 13380 The unswizzled SP value can be converted into the swizzled SP value by: 13381 13382 | swizzled SP = unswizzled SP / wavefront size 13383 13384 This may be used to obtain the private address space address of stack 13385 objects and to convert this address to a flat address by adding the flat 13386 scratch aperture base address. 13387 13388 The swizzled SP value is always 4 bytes aligned for the ``r600`` 13389 architecture and 16 byte aligned for the ``amdgcn`` architecture. 13390 13391 .. note:: 13392 13393 The ``amdgcn`` value is selected to avoid dynamic stack alignment for the 13394 OpenCL language which has the largest base type defined as 16 bytes. 13395 13396 On entry, the swizzled SP value is the address of the first function 13397 argument passed on the stack. Other stack passed arguments are positive 13398 offsets from the entry swizzled SP value. 13399 13400 The function may use positive offsets beyond the last stack passed argument 13401 for stack allocated local variables and register spill slots. If necessary, 13402 the function may align these to greater alignment than 16 bytes. After these 13403 the function may dynamically allocate space for such things as runtime sized 13404 ``alloca`` local allocations. 13405 13406 If the function calls another function, it will place any stack allocated 13407 arguments after the last local allocation and adjust SGPR32 to the address 13408 after the last local allocation. 13409 134109. All other registers are unspecified. 1341110. Any necessary ``s_waitcnt`` has been performed to ensure memory is available 13412 to the function. 13413 13414On exit from a function: 13415 134161. VGPR0-31 and SGPR4-29 are used to pass function result arguments as 13417 described below. Any registers used are considered clobbered registers. 134182. The following registers are preserved and have the same value as on entry: 13419 13420 * FLAT_SCRATCH 13421 * EXEC 13422 * GFX6-GFX8: M0 13423 * All SGPR registers except the clobbered registers of SGPR4-31. 13424 * VGPR40-47 13425 * VGPR56-63 13426 * VGPR72-79 13427 * VGPR88-95 13428 * VGPR104-111 13429 * VGPR120-127 13430 * VGPR136-143 13431 * VGPR152-159 13432 * VGPR168-175 13433 * VGPR184-191 13434 * VGPR200-207 13435 * VGPR216-223 13436 * VGPR232-239 13437 * VGPR248-255 13438 13439 .. note:: 13440 13441 Except the argument registers, the VGPRs clobbered and the preserved 13442 registers are intermixed at regular intervals in order to keep a 13443 similar ratio independent of the number of allocated VGPRs. 13444 13445 * GFX90A: All AGPR registers except the clobbered registers AGPR0-31. 13446 * Lanes of all VGPRs that are inactive at the call site. 13447 13448 For the AMDGPU backend, an inter-procedural register allocation (IPRA) 13449 optimization may mark some of clobbered SGPR and VGPR registers as 13450 preserved if it can be determined that the called function does not change 13451 their value. 13452 134532. The PC is set to the RA provided on entry. 134543. MODE register: *TBD*. 134554. All other registers are clobbered. 134565. Any necessary ``s_waitcnt`` has been performed to ensure memory accessed by 13457 function is available to the caller. 13458 13459.. TODO:: 13460 13461 - How are function results returned? The address of structured types is passed 13462 by reference, but what about other types? 13463 13464The function input arguments are made up of the formal arguments explicitly 13465declared by the source language function plus the implicit input arguments used 13466by the implementation. 13467 13468The source language input arguments are: 13469 134701. Any source language implicit ``this`` or ``self`` argument comes first as a 13471 pointer type. 134722. Followed by the function formal arguments in left to right source order. 13473 13474The source language result arguments are: 13475 134761. The function result argument. 13477 13478The source language input or result struct type arguments that are less than or 13479equal to 16 bytes, are decomposed recursively into their base type fields, and 13480each field is passed as if a separate argument. For input arguments, if the 13481called function requires the struct to be in memory, for example because its 13482address is taken, then the function body is responsible for allocating a stack 13483location and copying the field arguments into it. Clang terms this *direct 13484struct*. 13485 13486The source language input struct type arguments that are greater than 16 bytes, 13487are passed by reference. The caller is responsible for allocating a stack 13488location to make a copy of the struct value and pass the address as the input 13489argument. The called function is responsible to perform the dereference when 13490accessing the input argument. Clang terms this *by-value struct*. 13491 13492A source language result struct type argument that is greater than 16 bytes, is 13493returned by reference. The caller is responsible for allocating a stack location 13494to hold the result value and passes the address as the last input argument 13495(before the implicit input arguments). In this case there are no result 13496arguments. The called function is responsible to perform the dereference when 13497storing the result value. Clang terms this *structured return (sret)*. 13498 13499*TODO: correct the ``sret`` definition.* 13500 13501.. TODO:: 13502 13503 Is this definition correct? Or is ``sret`` only used if passing in registers, and 13504 pass as non-decomposed struct as stack argument? Or something else? Is the 13505 memory location in the caller stack frame, or a stack memory argument and so 13506 no address is passed as the caller can directly write to the argument stack 13507 location? But then the stack location is still live after return. If an 13508 argument stack location is it the first stack argument or the last one? 13509 13510Lambda argument types are treated as struct types with an implementation defined 13511set of fields. 13512 13513.. TODO:: 13514 13515 Need to specify the ABI for lambda types for AMDGPU. 13516 13517For AMDGPU backend all source language arguments (including the decomposed 13518struct type arguments) are passed in VGPRs unless marked ``inreg`` in which case 13519they are passed in SGPRs. 13520 13521The AMDGPU backend walks the function call graph from the leaves to determine 13522which implicit input arguments are used, propagating to each caller of the 13523function. The used implicit arguments are appended to the function arguments 13524after the source language arguments in the following order: 13525 13526.. TODO:: 13527 13528 Is recursion or external functions supported? 13529 135301. Work-Item ID (1 VGPR) 13531 13532 The X, Y and Z work-item ID are packed into a single VGRP with the following 13533 layout. Only fields actually used by the function are set. The other bits 13534 are undefined. 13535 13536 The values come from the initial kernel execution state. See 13537 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`. 13538 13539 .. table:: Work-item implicit argument layout 13540 :name: amdgpu-amdhsa-workitem-implicit-argument-layout-table 13541 13542 ======= ======= ============== 13543 Bits Size Field Name 13544 ======= ======= ============== 13545 9:0 10 bits X Work-Item ID 13546 19:10 10 bits Y Work-Item ID 13547 29:20 10 bits Z Work-Item ID 13548 31:30 2 bits Unused 13549 ======= ======= ============== 13550 135512. Dispatch Ptr (2 SGPRs) 13552 13553 The value comes from the initial kernel execution state. See 13554 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 13555 135563. Queue Ptr (2 SGPRs) 13557 13558 The value comes from the initial kernel execution state. See 13559 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 13560 135614. Kernarg Segment Ptr (2 SGPRs) 13562 13563 The value comes from the initial kernel execution state. See 13564 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 13565 135665. Dispatch id (2 SGPRs) 13567 13568 The value comes from the initial kernel execution state. See 13569 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 13570 135716. Work-Group ID X (1 SGPR) 13572 13573 The value comes from the initial kernel execution state. See 13574 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 13575 135767. Work-Group ID Y (1 SGPR) 13577 13578 The value comes from the initial kernel execution state. See 13579 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 13580 135818. Work-Group ID Z (1 SGPR) 13582 13583 The value comes from the initial kernel execution state. See 13584 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 13585 135869. Implicit Argument Ptr (2 SGPRs) 13587 13588 The value is computed by adding an offset to Kernarg Segment Ptr to get the 13589 global address space pointer to the first kernarg implicit argument. 13590 13591The input and result arguments are assigned in order in the following manner: 13592 13593.. note:: 13594 13595 There are likely some errors and omissions in the following description that 13596 need correction. 13597 13598 .. TODO:: 13599 13600 Check the Clang source code to decipher how function arguments and return 13601 results are handled. Also see the AMDGPU specific values used. 13602 13603* VGPR arguments are assigned to consecutive VGPRs starting at VGPR0 up to 13604 VGPR31. 13605 13606 If there are more arguments than will fit in these registers, the remaining 13607 arguments are allocated on the stack in order on naturally aligned 13608 addresses. 13609 13610 .. TODO:: 13611 13612 How are overly aligned structures allocated on the stack? 13613 13614* SGPR arguments are assigned to consecutive SGPRs starting at SGPR0 up to 13615 SGPR29. 13616 13617 If there are more arguments than will fit in these registers, the remaining 13618 arguments are allocated on the stack in order on naturally aligned 13619 addresses. 13620 13621Note that decomposed struct type arguments may have some fields passed in 13622registers and some in memory. 13623 13624.. TODO:: 13625 13626 So, a struct which can pass some fields as decomposed register arguments, will 13627 pass the rest as decomposed stack elements? But an argument that will not start 13628 in registers will not be decomposed and will be passed as a non-decomposed 13629 stack value? 13630 13631The following is not part of the AMDGPU function calling convention but 13632describes how the AMDGPU implements function calls: 13633 136341. SGPR33 is used as a frame pointer (FP) if necessary. Like the SP it is an 13635 unswizzled scratch address. It is only needed if runtime sized ``alloca`` 13636 are used, or for the reasons defined in ``SIFrameLowering``. 136372. Runtime stack alignment is supported. SGPR34 is used as a base pointer (BP) 13638 to access the incoming stack arguments in the function. The BP is needed 13639 only when the function requires the runtime stack alignment. 13640 136413. Allocating SGPR arguments on the stack are not supported. 13642 136434. No CFI is currently generated. See 13644 :ref:`amdgpu-dwarf-call-frame-information`. 13645 13646 .. note:: 13647 13648 CFI will be generated that defines the CFA as the unswizzled address 13649 relative to the wave scratch base in the unswizzled private address space 13650 of the lowest address stack allocated local variable. 13651 13652 ``DW_AT_frame_base`` will be defined as the swizzled address in the 13653 swizzled private address space by dividing the CFA by the wavefront size 13654 (since CFA is always at least dword aligned which matches the scratch 13655 swizzle element size). 13656 13657 If no dynamic stack alignment was performed, the stack allocated arguments 13658 are accessed as negative offsets relative to ``DW_AT_frame_base``, and the 13659 local variables and register spill slots are accessed as positive offsets 13660 relative to ``DW_AT_frame_base``. 13661 136625. Function argument passing is implemented by copying the input physical 13663 registers to virtual registers on entry. The register allocator can spill if 13664 necessary. These are copied back to physical registers at call sites. The 13665 net effect is that each function call can have these values in entirely 13666 distinct locations. The IPRA can help avoid shuffling argument registers. 136676. Call sites are implemented by setting up the arguments at positive offsets 13668 from SP. Then SP is incremented to account for the known frame size before 13669 the call and decremented after the call. 13670 13671 .. note:: 13672 13673 The CFI will reflect the changed calculation needed to compute the CFA 13674 from SP. 13675 136767. 4 byte spill slots are used in the stack frame. One slot is allocated for an 13677 emergency spill slot. Buffer instructions are used for stack accesses and 13678 not the ``flat_scratch`` instruction. 13679 13680 .. TODO:: 13681 13682 Explain when the emergency spill slot is used. 13683 13684.. TODO:: 13685 13686 Possible broken issues: 13687 13688 - Stack arguments must be aligned to required alignment. 13689 - Stack is aligned to max(16, max formal argument alignment) 13690 - Direct argument < 64 bits should check register budget. 13691 - Register budget calculation should respect ``inreg`` for SGPR. 13692 - SGPR overflow is not handled. 13693 - struct with 1 member unpeeling is not checking size of member. 13694 - ``sret`` is after ``this`` pointer. 13695 - Caller is not implementing stack realignment: need an extra pointer. 13696 - Should say AMDGPU passes FP rather than SP. 13697 - Should CFI define CFA as address of locals or arguments. Difference is 13698 apparent when have implemented dynamic alignment. 13699 - If ``SCRATCH`` instruction could allow negative offsets, then can make FP be 13700 highest address of stack frame and use negative offset for locals. Would 13701 allow SP to be the same as FP and could support signal-handler-like as now 13702 have a real SP for the top of the stack. 13703 - How is ``sret`` passed on the stack? In argument stack area? Can it overlay 13704 arguments? 13705 13706AMDPAL 13707------ 13708 13709This section provides code conventions used when the target triple OS is 13710``amdpal`` (see :ref:`amdgpu-target-triples`). 13711 13712.. _amdgpu-amdpal-code-object-metadata-section: 13713 13714Code Object Metadata 13715~~~~~~~~~~~~~~~~~~~~ 13716 13717.. note:: 13718 13719 The metadata is currently in development and is subject to major 13720 changes. Only the current version is supported. *When this document 13721 was generated the version was 2.6.* 13722 13723Code object metadata is specified by the ``NT_AMDGPU_METADATA`` note 13724record (see :ref:`amdgpu-note-records-v3-onwards`). 13725 13726The metadata is represented as Message Pack formatted binary data (see 13727[MsgPack]_). The top level is a Message Pack map that includes the keys 13728defined in table :ref:`amdgpu-amdpal-code-object-metadata-map-table` 13729and referenced tables. 13730 13731Additional information can be added to the maps. To avoid conflicts, any 13732key names should be prefixed by "*vendor-name*." where ``vendor-name`` 13733can be the name of the vendor and specific vendor tool that generates the 13734information. The prefix is abbreviated to simply "." when it appears 13735within a map that has been added by the same *vendor-name*. 13736 13737 .. table:: AMDPAL Code Object Metadata Map 13738 :name: amdgpu-amdpal-code-object-metadata-map-table 13739 13740 =================== ============== ========= ====================================================================== 13741 String Key Value Type Required? Description 13742 =================== ============== ========= ====================================================================== 13743 "amdpal.version" sequence of Required PAL code object metadata (major, minor) version. The current values 13744 2 integers are defined by *Util::Abi::PipelineMetadata(Major|Minor)Version*. 13745 "amdpal.pipelines" sequence of Required Per-pipeline metadata. See 13746 map :ref:`amdgpu-amdpal-code-object-pipeline-metadata-map-table` for the 13747 definition of the keys included in that map. 13748 =================== ============== ========= ====================================================================== 13749 13750.. 13751 13752 .. table:: AMDPAL Code Object Pipeline Metadata Map 13753 :name: amdgpu-amdpal-code-object-pipeline-metadata-map-table 13754 13755 ====================================== ============== ========= =================================================== 13756 String Key Value Type Required? Description 13757 ====================================== ============== ========= =================================================== 13758 ".name" string Source name of the pipeline. 13759 ".type" string Pipeline type, e.g. VsPs. Values include: 13760 13761 - "VsPs" 13762 - "Gs" 13763 - "Cs" 13764 - "Ngg" 13765 - "Tess" 13766 - "GsTess" 13767 - "NggTess" 13768 13769 ".internal_pipeline_hash" sequence of Required Internal compiler hash for this pipeline. Lower 13770 2 integers 64 bits is the "stable" portion of the hash, used 13771 for e.g. shader replacement lookup. Upper 64 bits 13772 is the "unique" portion of the hash, used for 13773 e.g. pipeline cache lookup. The value is 13774 implementation defined, and can not be relied on 13775 between different builds of the compiler. 13776 ".shaders" map Per-API shader metadata. See 13777 :ref:`amdgpu-amdpal-code-object-shader-map-table` 13778 for the definition of the keys included in that 13779 map. 13780 ".hardware_stages" map Per-hardware stage metadata. See 13781 :ref:`amdgpu-amdpal-code-object-hardware-stage-map-table` 13782 for the definition of the keys included in that 13783 map. 13784 ".shader_functions" map Per-shader function metadata. See 13785 :ref:`amdgpu-amdpal-code-object-shader-function-map-table` 13786 for the definition of the keys included in that 13787 map. 13788 ".registers" map Required Hardware register configuration. See 13789 :ref:`amdgpu-amdpal-code-object-register-map-table` 13790 for the definition of the keys included in that 13791 map. 13792 ".user_data_limit" integer Number of user data entries accessed by this 13793 pipeline. 13794 ".spill_threshold" integer The user data spill threshold. 0xFFFF for 13795 NoUserDataSpilling. 13796 ".uses_viewport_array_index" boolean Indicates whether or not the pipeline uses the 13797 viewport array index feature. Pipelines which use 13798 this feature can render into all 16 viewports, 13799 whereas pipelines which do not use it are 13800 restricted to viewport #0. 13801 ".es_gs_lds_size" integer Size in bytes of LDS space used internally for 13802 handling data-passing between the ES and GS 13803 shader stages. This can be zero if the data is 13804 passed using off-chip buffers. This value should 13805 be used to program all user-SGPRs which have been 13806 marked with "UserDataMapping::EsGsLdsSize" 13807 (typically only the GS and VS HW stages will ever 13808 have a user-SGPR so marked). 13809 ".nggSubgroupSize" integer Explicit maximum subgroup size for NGG shaders 13810 (maximum number of threads in a subgroup). 13811 ".num_interpolants" integer Graphics only. Number of PS interpolants. 13812 ".mesh_scratch_memory_size" integer Max mesh shader scratch memory used. 13813 ".api" string Name of the client graphics API. 13814 ".api_create_info" binary Graphics API shader create info binary blob. Can 13815 be defined by the driver using the compiler if 13816 they want to be able to correlate API-specific 13817 information used during creation at a later time. 13818 ====================================== ============== ========= =================================================== 13819 13820.. 13821 13822 .. table:: AMDPAL Code Object Shader Map 13823 :name: amdgpu-amdpal-code-object-shader-map-table 13824 13825 13826 +-------------+--------------+-------------------------------------------------------------------+ 13827 |String Key |Value Type |Description | 13828 +=============+==============+===================================================================+ 13829 |- ".compute" |map |See :ref:`amdgpu-amdpal-code-object-api-shader-metadata-map-table` | 13830 |- ".vertex" | |for the definition of the keys included in that map. | 13831 |- ".hull" | | | 13832 |- ".domain" | | | 13833 |- ".geometry"| | | 13834 |- ".pixel" | | | 13835 +-------------+--------------+-------------------------------------------------------------------+ 13836 13837.. 13838 13839 .. table:: AMDPAL Code Object API Shader Metadata Map 13840 :name: amdgpu-amdpal-code-object-api-shader-metadata-map-table 13841 13842 ==================== ============== ========= ===================================================================== 13843 String Key Value Type Required? Description 13844 ==================== ============== ========= ===================================================================== 13845 ".api_shader_hash" sequence of Required Input shader hash, typically passed in from the client. The value 13846 2 integers is implementation defined, and can not be relied on between 13847 different builds of the compiler. 13848 ".hardware_mapping" sequence of Required Flags indicating the HW stages this API shader maps to. Values 13849 string include: 13850 13851 - ".ls" 13852 - ".hs" 13853 - ".es" 13854 - ".gs" 13855 - ".vs" 13856 - ".ps" 13857 - ".cs" 13858 13859 ==================== ============== ========= ===================================================================== 13860 13861.. 13862 13863 .. table:: AMDPAL Code Object Hardware Stage Map 13864 :name: amdgpu-amdpal-code-object-hardware-stage-map-table 13865 13866 +-------------+--------------+-----------------------------------------------------------------------+ 13867 |String Key |Value Type |Description | 13868 +=============+==============+=======================================================================+ 13869 |- ".ls" |map |See :ref:`amdgpu-amdpal-code-object-hardware-stage-metadata-map-table` | 13870 |- ".hs" | |for the definition of the keys included in that map. | 13871 |- ".es" | | | 13872 |- ".gs" | | | 13873 |- ".vs" | | | 13874 |- ".ps" | | | 13875 |- ".cs" | | | 13876 +-------------+--------------+-----------------------------------------------------------------------+ 13877 13878.. 13879 13880 .. table:: AMDPAL Code Object Hardware Stage Metadata Map 13881 :name: amdgpu-amdpal-code-object-hardware-stage-metadata-map-table 13882 13883 ========================== ============== ========= =============================================================== 13884 String Key Value Type Required? Description 13885 ========================== ============== ========= =============================================================== 13886 ".entry_point" string The ELF symbol pointing to this pipeline's stage entry point. 13887 ".scratch_memory_size" integer Scratch memory size in bytes. 13888 ".lds_size" integer Local Data Share size in bytes. 13889 ".perf_data_buffer_size" integer Performance data buffer size in bytes. 13890 ".vgpr_count" integer Number of VGPRs used. 13891 ".agpr_count" integer Number of AGPRs used. 13892 ".sgpr_count" integer Number of SGPRs used. 13893 ".vgpr_limit" integer If non-zero, indicates the shader was compiled with a 13894 directive to instruct the compiler to limit the VGPR usage to 13895 be less than or equal to the specified value (only set if 13896 different from HW default). 13897 ".sgpr_limit" integer SGPR count upper limit (only set if different from HW 13898 default). 13899 ".threadgroup_dimensions" sequence of Thread-group X/Y/Z dimensions (Compute only). 13900 3 integers 13901 ".wavefront_size" integer Wavefront size (only set if different from HW default). 13902 ".uses_uavs" boolean The shader reads or writes UAVs. 13903 ".uses_rovs" boolean The shader reads or writes ROVs. 13904 ".writes_uavs" boolean The shader writes to one or more UAVs. 13905 ".writes_depth" boolean The shader writes out a depth value. 13906 ".uses_append_consume" boolean The shader uses append and/or consume operations, either 13907 memory or GDS. 13908 ".uses_prim_id" boolean The shader uses PrimID. 13909 ========================== ============== ========= =============================================================== 13910 13911.. 13912 13913 .. table:: AMDPAL Code Object Shader Function Map 13914 :name: amdgpu-amdpal-code-object-shader-function-map-table 13915 13916 =============== ============== ==================================================================== 13917 String Key Value Type Description 13918 =============== ============== ==================================================================== 13919 *symbol name* map *symbol name* is the ELF symbol name of the shader function code 13920 entry address. The value is the function's metadata. See 13921 :ref:`amdgpu-amdpal-code-object-shader-function-metadata-map-table`. 13922 =============== ============== ==================================================================== 13923 13924.. 13925 13926 .. table:: AMDPAL Code Object Shader Function Metadata Map 13927 :name: amdgpu-amdpal-code-object-shader-function-metadata-map-table 13928 13929 ============================= ============== ================================================================= 13930 String Key Value Type Description 13931 ============================= ============== ================================================================= 13932 ".api_shader_hash" sequence of Input shader hash, typically passed in from the client. The value 13933 2 integers is implementation defined, and can not be relied on between 13934 different builds of the compiler. 13935 ".scratch_memory_size" integer Size in bytes of scratch memory used by the shader. 13936 ".lds_size" integer Size in bytes of LDS memory. 13937 ".vgpr_count" integer Number of VGPRs used by the shader. 13938 ".sgpr_count" integer Number of SGPRs used by the shader. 13939 ".stack_frame_size_in_bytes" integer Amount of stack size used by the shader. 13940 ".shader_subtype" string Shader subtype/kind. Values include: 13941 13942 - "Unknown" 13943 13944 ============================= ============== ================================================================= 13945 13946.. 13947 13948 .. table:: AMDPAL Code Object Register Map 13949 :name: amdgpu-amdpal-code-object-register-map-table 13950 13951 ========================== ============== ==================================================================== 13952 32-bit Integer Key Value Type Description 13953 ========================== ============== ==================================================================== 13954 ``reg offset`` 32-bit integer ``reg offset`` is the dword offset into the GFXIP register space of 13955 a GRBM register (i.e., driver accessible GPU register number, not 13956 shader GPR register number). The driver is required to program each 13957 specified register to the corresponding specified value when 13958 executing this pipeline. Typically, the ``reg offsets`` are the 13959 ``uint16_t`` offsets to each register as defined by the hardware 13960 chip headers. The register is set to the provided value. However, a 13961 ``reg offset`` that specifies a user data register (e.g., 13962 COMPUTE_USER_DATA_0) needs special treatment. See 13963 :ref:`amdgpu-amdpal-code-object-user-data-section` section for more 13964 information. 13965 ========================== ============== ==================================================================== 13966 13967.. _amdgpu-amdpal-code-object-user-data-section: 13968 13969User Data 13970+++++++++ 13971 13972Each hardware stage has a set of 32-bit physical SPI *user data registers* 13973(either 16 or 32 based on graphics IP and the stage) which can be 13974written from a command buffer and then loaded into SGPRs when waves are 13975launched via a subsequent dispatch or draw operation. This is the way 13976most arguments are passed from the application/runtime to a hardware 13977shader. 13978 13979PAL abstracts this functionality by exposing a set of 128 *user data 13980entries* per pipeline a client can use to pass arguments from a command 13981buffer to one or more shaders in that pipeline. The ELF code object must 13982specify a mapping from virtualized *user data entries* to physical *user 13983data registers*, and PAL is responsible for implementing that mapping, 13984including spilling overflow *user data entries* to memory if needed. 13985 13986Since the *user data registers* are GRBM-accessible SPI registers, this 13987mapping is actually embedded in the ``.registers`` metadata entry. For 13988most registers, the value in that map is a literal 32-bit value that 13989should be written to the register by the driver. However, when the 13990register is a *user data register* (any USER_DATA register e.g., 13991SPI_SHADER_USER_DATA_PS_5), the value is instead an encoding that tells 13992the driver to write either a *user data entry* value or one of several 13993driver-internal values to the register. This encoding is described in 13994the following table: 13995 13996.. note:: 13997 13998 Currently, *user data registers* 0 and 1 (e.g., SPI_SHADER_USER_DATA_PS_0, 13999 and SPI_SHADER_USER_DATA_PS_1) are reserved. *User data register* 0 must 14000 always be programmed to the address of the GlobalTable, and *user data 14001 register* 1 must always be programmed to the address of the PerShaderTable. 14002 14003.. 14004 14005 .. table:: AMDPAL User Data Mapping 14006 :name: amdgpu-amdpal-code-object-metadata-user-data-mapping-table 14007 14008 ========== ================= =============================================================================== 14009 Value Name Description 14010 ========== ================= =============================================================================== 14011 0..127 *User Data Entry* 32-bit value of user_data_entry[N] as specified via *CmdSetUserData()* 14012 0x10000000 GlobalTable 32-bit pointer to GPU memory containing the global internal table (should 14013 always point to *user data register* 0). 14014 0x10000001 PerShaderTable 32-bit pointer to GPU memory containing the per-shader internal table. See 14015 :ref:`amdgpu-amdpal-code-object-metadata-user-data-per-shader-table-section` 14016 for more detail (should always point to *user data register* 1). 14017 0x10000002 SpillTable 32-bit pointer to GPU memory containing the user data spill table. See 14018 :ref:`amdgpu-amdpal-code-object-metadata-user-data-spill-table-section` for 14019 more detail. 14020 0x10000003 BaseVertex Vertex offset (32-bit unsigned integer). Not needed if the pipeline doesn't 14021 reference the draw index in the vertex shader. Only supported by the first 14022 stage in a graphics pipeline. 14023 0x10000004 BaseInstance Instance offset (32-bit unsigned integer). Only supported by the first stage in 14024 a graphics pipeline. 14025 0x10000005 DrawIndex Draw index (32-bit unsigned integer). Only supported by the first stage in a 14026 graphics pipeline. 14027 0x10000006 Workgroup Thread group count (32-bit unsigned integer). Low half of a 64-bit address of 14028 a buffer containing the grid dimensions for a Compute dispatch operation. The 14029 high half of the address is stored in the next sequential user-SGPR. Only 14030 supported by compute pipelines. 14031 0x1000000A EsGsLdsSize Indicates that PAL will program this user-SGPR to contain the amount of LDS 14032 space used for the ES/GS pseudo-ring-buffer for passing data between shader 14033 stages. 14034 0x1000000B ViewId View id (32-bit unsigned integer) identifies a view of graphic 14035 pipeline instancing. 14036 0x1000000C StreamOutTable 32-bit pointer to GPU memory containing the stream out target SRD table. This 14037 can only appear for one shader stage per pipeline. 14038 0x1000000D PerShaderPerfData 32-bit pointer to GPU memory containing the per-shader performance data buffer. 14039 0x1000000F VertexBufferTable 32-bit pointer to GPU memory containing the vertex buffer SRD table. This can 14040 only appear for one shader stage per pipeline. 14041 0x10000010 UavExportTable 32-bit pointer to GPU memory containing the UAV export SRD table. This can 14042 only appear for one shader stage per pipeline (PS). These replace color targets 14043 and are completely separate from any UAVs used by the shader. This is optional, 14044 and only used by the PS when UAV exports are used to replace color-target 14045 exports to optimize specific shaders. 14046 0x10000011 NggCullingData 64-bit pointer to GPU memory containing the hardware register data needed by 14047 some NGG pipelines to perform culling. This value contains the address of the 14048 first of two consecutive registers which provide the full GPU address. 14049 0x10000015 FetchShaderPtr 64-bit pointer to GPU memory containing the fetch shader subroutine. 14050 ========== ================= =============================================================================== 14051 14052.. _amdgpu-amdpal-code-object-metadata-user-data-per-shader-table-section: 14053 14054Per-Shader Table 14055################ 14056 14057Low 32 bits of the GPU address for an optional buffer in the ``.data`` 14058section of the ELF. The high 32 bits of the address match the high 32 bits 14059of the shader's program counter. 14060 14061The buffer can be anything the shader compiler needs it for, and 14062allows each shader to have its own region of the ``.data`` section. 14063Typically, this could be a table of buffer SRD's and the data pointed to 14064by the buffer SRD's, but it could be a flat-address region of memory as 14065well. Its layout and usage are defined by the shader compiler. 14066 14067Each shader's table in the ``.data`` section is referenced by the symbol 14068``_amdgpu_``\ *xs*\ ``_shdr_intrl_data`` where *xs* corresponds with the 14069hardware shader stage the data is for. E.g., 14070``_amdgpu_cs_shdr_intrl_data`` for the compute shader hardware stage. 14071 14072.. _amdgpu-amdpal-code-object-metadata-user-data-spill-table-section: 14073 14074Spill Table 14075########### 14076 14077It is possible for a hardware shader to need access to more *user data 14078entries* than there are slots available in user data registers for one 14079or more hardware shader stages. In that case, the PAL runtime expects 14080the necessary *user data entries* to be spilled to GPU memory and use 14081one user data register to point to the spilled user data memory. The 14082value of the *user data entry* must then represent the location where 14083a shader expects to read the low 32-bits of the table's GPU virtual 14084address. The *spill table* itself represents a set of 32-bit values 14085managed by the PAL runtime in GPU-accessible memory that can be made 14086indirectly accessible to a hardware shader. 14087 14088Unspecified OS 14089-------------- 14090 14091This section provides code conventions used when the target triple OS is 14092empty (see :ref:`amdgpu-target-triples`). 14093 14094Trap Handler ABI 14095~~~~~~~~~~~~~~~~ 14096 14097For code objects generated by AMDGPU backend for non-amdhsa OS, the runtime does 14098not install a trap handler. The ``llvm.trap`` and ``llvm.debugtrap`` 14099instructions are handled as follows: 14100 14101 .. table:: AMDGPU Trap Handler for Non-AMDHSA OS 14102 :name: amdgpu-trap-handler-for-non-amdhsa-os-table 14103 14104 =============== =============== =========================================== 14105 Usage Code Sequence Description 14106 =============== =============== =========================================== 14107 llvm.trap s_endpgm Causes wavefront to be terminated. 14108 llvm.debugtrap *none* Compiler warning given that there is no 14109 trap handler installed. 14110 =============== =============== =========================================== 14111 14112Source Languages 14113================ 14114 14115.. _amdgpu-opencl: 14116 14117OpenCL 14118------ 14119 14120When the language is OpenCL the following differences occur: 14121 141221. The OpenCL memory model is used (see :ref:`amdgpu-amdhsa-memory-model`). 141232. The AMDGPU backend appends additional arguments to the kernel's explicit 14124 arguments for the AMDHSA OS (see 14125 :ref:`opencl-kernel-implicit-arguments-appended-for-amdhsa-os-table`). 141263. Additional metadata is generated 14127 (see :ref:`amdgpu-amdhsa-code-object-metadata`). 14128 14129 .. table:: OpenCL kernel implicit arguments appended for AMDHSA OS 14130 :name: opencl-kernel-implicit-arguments-appended-for-amdhsa-os-table 14131 14132 ======== ==== ========= =========================================== 14133 Position Byte Byte Description 14134 Size Alignment 14135 ======== ==== ========= =========================================== 14136 1 8 8 OpenCL Global Offset X 14137 2 8 8 OpenCL Global Offset Y 14138 3 8 8 OpenCL Global Offset Z 14139 4 8 8 OpenCL address of printf buffer 14140 5 8 8 OpenCL address of virtual queue used by 14141 enqueue_kernel. 14142 6 8 8 OpenCL address of AqlWrap struct used by 14143 enqueue_kernel. 14144 7 8 8 Pointer argument used for Multi-gird 14145 synchronization. 14146 ======== ==== ========= =========================================== 14147 14148.. _amdgpu-hcc: 14149 14150HCC 14151--- 14152 14153When the language is HCC the following differences occur: 14154 141551. The HSA memory model is used (see :ref:`amdgpu-amdhsa-memory-model`). 14156 14157.. _amdgpu-assembler: 14158 14159Assembler 14160--------- 14161 14162AMDGPU backend has LLVM-MC based assembler which is currently in development. 14163It supports AMDGCN GFX6-GFX10. 14164 14165This section describes general syntax for instructions and operands. 14166 14167Instructions 14168~~~~~~~~~~~~ 14169 14170An instruction has the following :doc:`syntax<AMDGPUInstructionSyntax>`: 14171 14172 | ``<``\ *opcode*\ ``> <``\ *operand0*\ ``>, <``\ *operand1*\ ``>,... 14173 <``\ *modifier0*\ ``> <``\ *modifier1*\ ``>...`` 14174 14175:doc:`Operands<AMDGPUOperandSyntax>` are comma-separated while 14176:doc:`modifiers<AMDGPUModifierSyntax>` are space-separated. 14177 14178The order of operands and modifiers is fixed. 14179Most modifiers are optional and may be omitted. 14180 14181Links to detailed instruction syntax description may be found in the following 14182table. Note that features under development are not included 14183in this description. 14184 14185 ============= ============================================= ======================================= 14186 Architecture Core ISA ISA Variants and Extensions 14187 ============= ============================================= ======================================= 14188 GCN 2 :doc:`GFX7<AMDGPU/AMDGPUAsmGFX7>` \- 14189 GCN 3, GCN 4 :doc:`GFX8<AMDGPU/AMDGPUAsmGFX8>` \- 14190 GCN 5 :doc:`GFX9<AMDGPU/AMDGPUAsmGFX9>` :doc:`gfx900<AMDGPU/AMDGPUAsmGFX900>` 14191 14192 :doc:`gfx902<AMDGPU/AMDGPUAsmGFX900>` 14193 14194 :doc:`gfx904<AMDGPU/AMDGPUAsmGFX904>` 14195 14196 :doc:`gfx906<AMDGPU/AMDGPUAsmGFX906>` 14197 14198 :doc:`gfx909<AMDGPU/AMDGPUAsmGFX900>` 14199 14200 CDNA 1 :doc:`GFX9<AMDGPU/AMDGPUAsmGFX9>` :doc:`gfx908<AMDGPU/AMDGPUAsmGFX908>` 14201 CDNA 2 :doc:`GFX9<AMDGPU/AMDGPUAsmGFX9>` :doc:`gfx90a<AMDGPU/AMDGPUAsmGFX90a>` 14202 RDNA 1 :doc:`GFX10 RDNA1<AMDGPU/AMDGPUAsmGFX10>` :doc:`gfx1010<AMDGPU/AMDGPUAsmGFX10>` 14203 14204 :doc:`gfx1011<AMDGPU/AMDGPUAsmGFX1011>` 14205 14206 :doc:`gfx1012<AMDGPU/AMDGPUAsmGFX1011>` 14207 14208 :doc:`gfx1013<AMDGPU/AMDGPUAsmGFX1013>` 14209 14210 RDNA 2 :doc:`GFX10 RDNA2<AMDGPU/AMDGPUAsmGFX1030>` :doc:`gfx1030<AMDGPU/AMDGPUAsmGFX1030>` 14211 14212 :doc:`gfx1031<AMDGPU/AMDGPUAsmGFX1030>` 14213 14214 :doc:`gfx1032<AMDGPU/AMDGPUAsmGFX1030>` 14215 14216 :doc:`gfx1033<AMDGPU/AMDGPUAsmGFX1030>` 14217 14218 :doc:`gfx1034<AMDGPU/AMDGPUAsmGFX1030>` 14219 14220 :doc:`gfx1035<AMDGPU/AMDGPUAsmGFX1030>` 14221 14222 :doc:`gfx1036<AMDGPU/AMDGPUAsmGFX1030>` 14223 ============= ============================================= ======================================= 14224 14225For more information about instructions, their semantics and supported 14226combinations of operands, refer to one of instruction set architecture manuals 14227[AMD-GCN-GFX6]_, [AMD-GCN-GFX7]_, [AMD-GCN-GFX8]_, 14228[AMD-GCN-GFX900-GFX904-VEGA]_, [AMD-GCN-GFX906-VEGA7NM]_, 14229[AMD-GCN-GFX908-CDNA1]_, [AMD-GCN-GFX90A-CDNA2]_, [AMD-GCN-GFX10-RDNA1]_ and 14230[AMD-GCN-GFX10-RDNA2]_. 14231 14232Operands 14233~~~~~~~~ 14234 14235Detailed description of operands may be found :doc:`here<AMDGPUOperandSyntax>`. 14236 14237Modifiers 14238~~~~~~~~~ 14239 14240Detailed description of modifiers may be found 14241:doc:`here<AMDGPUModifierSyntax>`. 14242 14243Instruction Examples 14244~~~~~~~~~~~~~~~~~~~~ 14245 14246DS 14247++ 14248 14249.. code-block:: nasm 14250 14251 ds_add_u32 v2, v4 offset:16 14252 ds_write_src2_b64 v2 offset0:4 offset1:8 14253 ds_cmpst_f32 v2, v4, v6 14254 ds_min_rtn_f64 v[8:9], v2, v[4:5] 14255 14256For full list of supported instructions, refer to "LDS/GDS instructions" in ISA 14257Manual. 14258 14259FLAT 14260++++ 14261 14262.. code-block:: nasm 14263 14264 flat_load_dword v1, v[3:4] 14265 flat_store_dwordx3 v[3:4], v[5:7] 14266 flat_atomic_swap v1, v[3:4], v5 glc 14267 flat_atomic_cmpswap v1, v[3:4], v[5:6] glc slc 14268 flat_atomic_fmax_x2 v[1:2], v[3:4], v[5:6] glc 14269 14270For full list of supported instructions, refer to "FLAT instructions" in ISA 14271Manual. 14272 14273MUBUF 14274+++++ 14275 14276.. code-block:: nasm 14277 14278 buffer_load_dword v1, off, s[4:7], s1 14279 buffer_store_dwordx4 v[1:4], v2, ttmp[4:7], s1 offen offset:4 glc tfe 14280 buffer_store_format_xy v[1:2], off, s[4:7], s1 14281 buffer_wbinvl1 14282 buffer_atomic_inc v1, v2, s[8:11], s4 idxen offset:4 slc 14283 14284For full list of supported instructions, refer to "MUBUF Instructions" in ISA 14285Manual. 14286 14287SMRD/SMEM 14288+++++++++ 14289 14290.. code-block:: nasm 14291 14292 s_load_dword s1, s[2:3], 0xfc 14293 s_load_dwordx8 s[8:15], s[2:3], s4 14294 s_load_dwordx16 s[88:103], s[2:3], s4 14295 s_dcache_inv_vol 14296 s_memtime s[4:5] 14297 14298For full list of supported instructions, refer to "Scalar Memory Operations" in 14299ISA Manual. 14300 14301SOP1 14302++++ 14303 14304.. code-block:: nasm 14305 14306 s_mov_b32 s1, s2 14307 s_mov_b64 s[0:1], 0x80000000 14308 s_cmov_b32 s1, 200 14309 s_wqm_b64 s[2:3], s[4:5] 14310 s_bcnt0_i32_b64 s1, s[2:3] 14311 s_swappc_b64 s[2:3], s[4:5] 14312 s_cbranch_join s[4:5] 14313 14314For full list of supported instructions, refer to "SOP1 Instructions" in ISA 14315Manual. 14316 14317SOP2 14318++++ 14319 14320.. code-block:: nasm 14321 14322 s_add_u32 s1, s2, s3 14323 s_and_b64 s[2:3], s[4:5], s[6:7] 14324 s_cselect_b32 s1, s2, s3 14325 s_andn2_b32 s2, s4, s6 14326 s_lshr_b64 s[2:3], s[4:5], s6 14327 s_ashr_i32 s2, s4, s6 14328 s_bfm_b64 s[2:3], s4, s6 14329 s_bfe_i64 s[2:3], s[4:5], s6 14330 s_cbranch_g_fork s[4:5], s[6:7] 14331 14332For full list of supported instructions, refer to "SOP2 Instructions" in ISA 14333Manual. 14334 14335SOPC 14336++++ 14337 14338.. code-block:: nasm 14339 14340 s_cmp_eq_i32 s1, s2 14341 s_bitcmp1_b32 s1, s2 14342 s_bitcmp0_b64 s[2:3], s4 14343 s_setvskip s3, s5 14344 14345For full list of supported instructions, refer to "SOPC Instructions" in ISA 14346Manual. 14347 14348SOPP 14349++++ 14350 14351.. code-block:: nasm 14352 14353 s_barrier 14354 s_nop 2 14355 s_endpgm 14356 s_waitcnt 0 ; Wait for all counters to be 0 14357 s_waitcnt vmcnt(0) & expcnt(0) & lgkmcnt(0) ; Equivalent to above 14358 s_waitcnt vmcnt(1) ; Wait for vmcnt counter to be 1. 14359 s_sethalt 9 14360 s_sleep 10 14361 s_sendmsg 0x1 14362 s_sendmsg sendmsg(MSG_INTERRUPT) 14363 s_trap 1 14364 14365For full list of supported instructions, refer to "SOPP Instructions" in ISA 14366Manual. 14367 14368Unless otherwise mentioned, little verification is performed on the operands 14369of SOPP Instructions, so it is up to the programmer to be familiar with the 14370range or acceptable values. 14371 14372VALU 14373++++ 14374 14375For vector ALU instruction opcodes (VOP1, VOP2, VOP3, VOPC, VOP_DPP, VOP_SDWA), 14376the assembler will automatically use optimal encoding based on its operands. To 14377force specific encoding, one can add a suffix to the opcode of the instruction: 14378 14379* _e32 for 32-bit VOP1/VOP2/VOPC 14380* _e64 for 64-bit VOP3 14381* _dpp for VOP_DPP 14382* _sdwa for VOP_SDWA 14383 14384VOP1/VOP2/VOP3/VOPC examples: 14385 14386.. code-block:: nasm 14387 14388 v_mov_b32 v1, v2 14389 v_mov_b32_e32 v1, v2 14390 v_nop 14391 v_cvt_f64_i32_e32 v[1:2], v2 14392 v_floor_f32_e32 v1, v2 14393 v_bfrev_b32_e32 v1, v2 14394 v_add_f32_e32 v1, v2, v3 14395 v_mul_i32_i24_e64 v1, v2, 3 14396 v_mul_i32_i24_e32 v1, -3, v3 14397 v_mul_i32_i24_e32 v1, -100, v3 14398 v_addc_u32 v1, s[0:1], v2, v3, s[2:3] 14399 v_max_f16_e32 v1, v2, v3 14400 14401VOP_DPP examples: 14402 14403.. code-block:: nasm 14404 14405 v_mov_b32 v0, v0 quad_perm:[0,2,1,1] 14406 v_sin_f32 v0, v0 row_shl:1 row_mask:0xa bank_mask:0x1 bound_ctrl:0 14407 v_mov_b32 v0, v0 wave_shl:1 14408 v_mov_b32 v0, v0 row_mirror 14409 v_mov_b32 v0, v0 row_bcast:31 14410 v_mov_b32 v0, v0 quad_perm:[1,3,0,1] row_mask:0xa bank_mask:0x1 bound_ctrl:0 14411 v_add_f32 v0, v0, |v0| row_shl:1 row_mask:0xa bank_mask:0x1 bound_ctrl:0 14412 v_max_f16 v1, v2, v3 row_shl:1 row_mask:0xa bank_mask:0x1 bound_ctrl:0 14413 14414VOP_SDWA examples: 14415 14416.. code-block:: nasm 14417 14418 v_mov_b32 v1, v2 dst_sel:BYTE_0 dst_unused:UNUSED_PRESERVE src0_sel:DWORD 14419 v_min_u32 v200, v200, v1 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:BYTE_1 src1_sel:DWORD 14420 v_sin_f32 v0, v0 dst_unused:UNUSED_PAD src0_sel:WORD_1 14421 v_fract_f32 v0, |v0| dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 14422 v_cmpx_le_u32 vcc, v1, v2 src0_sel:BYTE_2 src1_sel:WORD_0 14423 14424For full list of supported instructions, refer to "Vector ALU instructions". 14425 14426.. _amdgpu-amdhsa-assembler-predefined-symbols-v2: 14427 14428Code Object V2 Predefined Symbols 14429~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 14430 14431.. warning:: 14432 Code object V2 is not the default code object version emitted by 14433 this version of LLVM. 14434 14435The AMDGPU assembler defines and updates some symbols automatically. These 14436symbols do not affect code generation. 14437 14438.option.machine_version_major 14439+++++++++++++++++++++++++++++ 14440 14441Set to the GFX major generation number of the target being assembled for. For 14442example, when assembling for a "GFX9" target this will be set to the integer 14443value "9". The possible GFX major generation numbers are presented in 14444:ref:`amdgpu-processors`. 14445 14446.option.machine_version_minor 14447+++++++++++++++++++++++++++++ 14448 14449Set to the GFX minor generation number of the target being assembled for. For 14450example, when assembling for a "GFX810" target this will be set to the integer 14451value "1". The possible GFX minor generation numbers are presented in 14452:ref:`amdgpu-processors`. 14453 14454.option.machine_version_stepping 14455++++++++++++++++++++++++++++++++ 14456 14457Set to the GFX stepping generation number of the target being assembled for. 14458For example, when assembling for a "GFX704" target this will be set to the 14459integer value "4". The possible GFX stepping generation numbers are presented 14460in :ref:`amdgpu-processors`. 14461 14462.kernel.vgpr_count 14463++++++++++++++++++ 14464 14465Set to zero each time a 14466:ref:`amdgpu-amdhsa-assembler-directive-amdgpu_hsa_kernel` directive is 14467encountered. At each instruction, if the current value of this symbol is less 14468than or equal to the maximum VGPR number explicitly referenced within that 14469instruction then the symbol value is updated to equal that VGPR number plus 14470one. 14471 14472.kernel.sgpr_count 14473++++++++++++++++++ 14474 14475Set to zero each time a 14476:ref:`amdgpu-amdhsa-assembler-directive-amdgpu_hsa_kernel` directive is 14477encountered. At each instruction, if the current value of this symbol is less 14478than or equal to the maximum VGPR number explicitly referenced within that 14479instruction then the symbol value is updated to equal that SGPR number plus 14480one. 14481 14482.. _amdgpu-amdhsa-assembler-directives-v2: 14483 14484Code Object V2 Directives 14485~~~~~~~~~~~~~~~~~~~~~~~~~ 14486 14487.. warning:: 14488 Code object V2 is not the default code object version emitted by 14489 this version of LLVM. 14490 14491AMDGPU ABI defines auxiliary data in output code object. In assembly source, 14492one can specify them with assembler directives. 14493 14494.hsa_code_object_version major, minor 14495+++++++++++++++++++++++++++++++++++++ 14496 14497*major* and *minor* are integers that specify the version of the HSA code 14498object that will be generated by the assembler. 14499 14500.hsa_code_object_isa [major, minor, stepping, vendor, arch] 14501+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 14502 14503 14504*major*, *minor*, and *stepping* are all integers that describe the instruction 14505set architecture (ISA) version of the assembly program. 14506 14507*vendor* and *arch* are quoted strings. *vendor* should always be equal to 14508"AMD" and *arch* should always be equal to "AMDGPU". 14509 14510By default, the assembler will derive the ISA version, *vendor*, and *arch* 14511from the value of the -mcpu option that is passed to the assembler. 14512 14513.. _amdgpu-amdhsa-assembler-directive-amdgpu_hsa_kernel: 14514 14515.amdgpu_hsa_kernel (name) 14516+++++++++++++++++++++++++ 14517 14518This directives specifies that the symbol with given name is a kernel entry 14519point (label) and the object should contain corresponding symbol of type 14520STT_AMDGPU_HSA_KERNEL. 14521 14522.amd_kernel_code_t 14523++++++++++++++++++ 14524 14525This directive marks the beginning of a list of key / value pairs that are used 14526to specify the amd_kernel_code_t object that will be emitted by the assembler. 14527The list must be terminated by the *.end_amd_kernel_code_t* directive. For any 14528amd_kernel_code_t values that are unspecified a default value will be used. The 14529default value for all keys is 0, with the following exceptions: 14530 14531- *amd_code_version_major* defaults to 1. 14532- *amd_kernel_code_version_minor* defaults to 2. 14533- *amd_machine_kind* defaults to 1. 14534- *amd_machine_version_major*, *machine_version_minor*, and 14535 *amd_machine_version_stepping* are derived from the value of the -mcpu option 14536 that is passed to the assembler. 14537- *kernel_code_entry_byte_offset* defaults to 256. 14538- *wavefront_size* defaults 6 for all targets before GFX10. For GFX10 onwards 14539 defaults to 6 if target feature ``wavefrontsize64`` is enabled, otherwise 5. 14540 Note that wavefront size is specified as a power of two, so a value of **n** 14541 means a size of 2^ **n**. 14542- *call_convention* defaults to -1. 14543- *kernarg_segment_alignment*, *group_segment_alignment*, and 14544 *private_segment_alignment* default to 4. Note that alignments are specified 14545 as a power of 2, so a value of **n** means an alignment of 2^ **n**. 14546- *enable_tg_split* defaults to 1 if target feature ``tgsplit`` is enabled for 14547 GFX90A onwards. 14548- *enable_wgp_mode* defaults to 1 if target feature ``cumode`` is disabled for 14549 GFX10 onwards. 14550- *enable_mem_ordered* defaults to 1 for GFX10 onwards. 14551 14552The *.amd_kernel_code_t* directive must be placed immediately after the 14553function label and before any instructions. 14554 14555For a full list of amd_kernel_code_t keys, refer to AMDGPU ABI document, 14556comments in lib/Target/AMDGPU/AmdKernelCodeT.h and test/CodeGen/AMDGPU/hsa.s. 14557 14558.. _amdgpu-amdhsa-assembler-example-v2: 14559 14560Code Object V2 Example Source Code 14561~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 14562 14563.. warning:: 14564 Code Object V2 is not the default code object version emitted by 14565 this version of LLVM. 14566 14567Here is an example of a minimal assembly source file, defining one HSA kernel: 14568 14569.. code:: 14570 :number-lines: 14571 14572 .hsa_code_object_version 1,0 14573 .hsa_code_object_isa 14574 14575 .hsatext 14576 .globl hello_world 14577 .p2align 8 14578 .amdgpu_hsa_kernel hello_world 14579 14580 hello_world: 14581 14582 .amd_kernel_code_t 14583 enable_sgpr_kernarg_segment_ptr = 1 14584 is_ptr64 = 1 14585 compute_pgm_rsrc1_vgprs = 0 14586 compute_pgm_rsrc1_sgprs = 0 14587 compute_pgm_rsrc2_user_sgpr = 2 14588 compute_pgm_rsrc1_wgp_mode = 0 14589 compute_pgm_rsrc1_mem_ordered = 0 14590 compute_pgm_rsrc1_fwd_progress = 1 14591 .end_amd_kernel_code_t 14592 14593 s_load_dwordx2 s[0:1], s[0:1] 0x0 14594 v_mov_b32 v0, 3.14159 14595 s_waitcnt lgkmcnt(0) 14596 v_mov_b32 v1, s0 14597 v_mov_b32 v2, s1 14598 flat_store_dword v[1:2], v0 14599 s_endpgm 14600 .Lfunc_end0: 14601 .size hello_world, .Lfunc_end0-hello_world 14602 14603.. _amdgpu-amdhsa-assembler-predefined-symbols-v3-onwards: 14604 14605Code Object V3 and Above Predefined Symbols 14606~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 14607 14608The AMDGPU assembler defines and updates some symbols automatically. These 14609symbols do not affect code generation. 14610 14611.amdgcn.gfx_generation_number 14612+++++++++++++++++++++++++++++ 14613 14614Set to the GFX major generation number of the target being assembled for. For 14615example, when assembling for a "GFX9" target this will be set to the integer 14616value "9". The possible GFX major generation numbers are presented in 14617:ref:`amdgpu-processors`. 14618 14619.amdgcn.gfx_generation_minor 14620++++++++++++++++++++++++++++ 14621 14622Set to the GFX minor generation number of the target being assembled for. For 14623example, when assembling for a "GFX810" target this will be set to the integer 14624value "1". The possible GFX minor generation numbers are presented in 14625:ref:`amdgpu-processors`. 14626 14627.amdgcn.gfx_generation_stepping 14628+++++++++++++++++++++++++++++++ 14629 14630Set to the GFX stepping generation number of the target being assembled for. 14631For example, when assembling for a "GFX704" target this will be set to the 14632integer value "4". The possible GFX stepping generation numbers are presented 14633in :ref:`amdgpu-processors`. 14634 14635.. _amdgpu-amdhsa-assembler-symbol-next_free_vgpr: 14636 14637.amdgcn.next_free_vgpr 14638++++++++++++++++++++++ 14639 14640Set to zero before assembly begins. At each instruction, if the current value 14641of this symbol is less than or equal to the maximum VGPR number explicitly 14642referenced within that instruction then the symbol value is updated to equal 14643that VGPR number plus one. 14644 14645May be used to set the `.amdhsa_next_free_vgpr` directive in 14646:ref:`amdhsa-kernel-directives-table`. 14647 14648May be set at any time, e.g. manually set to zero at the start of each kernel. 14649 14650.. _amdgpu-amdhsa-assembler-symbol-next_free_sgpr: 14651 14652.amdgcn.next_free_sgpr 14653++++++++++++++++++++++ 14654 14655Set to zero before assembly begins. At each instruction, if the current value 14656of this symbol is less than or equal the maximum SGPR number explicitly 14657referenced within that instruction then the symbol value is updated to equal 14658that SGPR number plus one. 14659 14660May be used to set the `.amdhsa_next_free_spgr` directive in 14661:ref:`amdhsa-kernel-directives-table`. 14662 14663May be set at any time, e.g. manually set to zero at the start of each kernel. 14664 14665.. _amdgpu-amdhsa-assembler-directives-v3-onwards: 14666 14667Code Object V3 and Above Directives 14668~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 14669 14670Directives which begin with ``.amdgcn`` are valid for all ``amdgcn`` 14671architecture processors, and are not OS-specific. Directives which begin with 14672``.amdhsa`` are specific to ``amdgcn`` architecture processors when the 14673``amdhsa`` OS is specified. See :ref:`amdgpu-target-triples` and 14674:ref:`amdgpu-processors`. 14675 14676.. _amdgpu-assembler-directive-amdgcn-target: 14677 14678.amdgcn_target <target-triple> "-" <target-id> 14679++++++++++++++++++++++++++++++++++++++++++++++ 14680 14681Optional directive which declares the ``<target-triple>-<target-id>`` supported 14682by the containing assembler source file. Used by the assembler to validate 14683command-line options such as ``-triple``, ``-mcpu``, and 14684``--offload-arch=<target-id>``. A non-canonical target ID is allowed. See 14685:ref:`amdgpu-target-triples` and :ref:`amdgpu-target-id`. 14686 14687.. note:: 14688 14689 The target ID syntax used for code object V2 to V3 for this directive differs 14690 from that used elsewhere. See :ref:`amdgpu-target-id-v2-v3`. 14691 14692.amdhsa_kernel <name> 14693+++++++++++++++++++++ 14694 14695Creates a correctly aligned AMDHSA kernel descriptor and a symbol, 14696``<name>.kd``, in the current location of the current section. Only valid when 14697the OS is ``amdhsa``. ``<name>`` must be a symbol that labels the first 14698instruction to execute, and does not need to be previously defined. 14699 14700Marks the beginning of a list of directives used to generate the bytes of a 14701kernel descriptor, as described in :ref:`amdgpu-amdhsa-kernel-descriptor`. 14702Directives which may appear in this list are described in 14703:ref:`amdhsa-kernel-directives-table`. Directives may appear in any order, must 14704be valid for the target being assembled for, and cannot be repeated. Directives 14705support the range of values specified by the field they reference in 14706:ref:`amdgpu-amdhsa-kernel-descriptor`. If a directive is not specified, it is 14707assumed to have its default value, unless it is marked as "Required", in which 14708case it is an error to omit the directive. This list of directives is 14709terminated by an ``.end_amdhsa_kernel`` directive. 14710 14711 .. table:: AMDHSA Kernel Assembler Directives 14712 :name: amdhsa-kernel-directives-table 14713 14714 ======================================================== =================== ============ =================== 14715 Directive Default Supported On Description 14716 ======================================================== =================== ============ =================== 14717 ``.amdhsa_group_segment_fixed_size`` 0 GFX6-GFX10 Controls GROUP_SEGMENT_FIXED_SIZE in 14718 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 14719 ``.amdhsa_private_segment_fixed_size`` 0 GFX6-GFX10 Controls PRIVATE_SEGMENT_FIXED_SIZE in 14720 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 14721 ``.amdhsa_kernarg_size`` 0 GFX6-GFX10 Controls KERNARG_SIZE in 14722 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 14723 ``.amdhsa_user_sgpr_count`` 0 GFX6-GFX10 Controls USER_SGPR_COUNT in COMPUTE_PGM_RSRC2 14724 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table` 14725 ``.amdhsa_user_sgpr_private_segment_buffer`` 0 GFX6-GFX10 Controls ENABLE_SGPR_PRIVATE_SEGMENT_BUFFER in 14726 (except :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 14727 GFX940) 14728 ``.amdhsa_user_sgpr_dispatch_ptr`` 0 GFX6-GFX10 Controls ENABLE_SGPR_DISPATCH_PTR in 14729 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 14730 ``.amdhsa_user_sgpr_queue_ptr`` 0 GFX6-GFX10 Controls ENABLE_SGPR_QUEUE_PTR in 14731 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 14732 ``.amdhsa_user_sgpr_kernarg_segment_ptr`` 0 GFX6-GFX10 Controls ENABLE_SGPR_KERNARG_SEGMENT_PTR in 14733 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 14734 ``.amdhsa_user_sgpr_dispatch_id`` 0 GFX6-GFX10 Controls ENABLE_SGPR_DISPATCH_ID in 14735 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 14736 ``.amdhsa_user_sgpr_flat_scratch_init`` 0 GFX6-GFX10 Controls ENABLE_SGPR_FLAT_SCRATCH_INIT in 14737 (except :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 14738 GFX940) 14739 ``.amdhsa_user_sgpr_private_segment_size`` 0 GFX6-GFX10 Controls ENABLE_SGPR_PRIVATE_SEGMENT_SIZE in 14740 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 14741 ``.amdhsa_wavefront_size32`` Target GFX10 Controls ENABLE_WAVEFRONT_SIZE32 in 14742 Feature :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 14743 Specific 14744 (wavefrontsize64) 14745 ``.amdhsa_system_sgpr_private_segment_wavefront_offset`` 0 GFX6-GFX10 Controls ENABLE_PRIVATE_SEGMENT in 14746 (except :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 14747 GFX940) 14748 ``.amdhsa_enable_private_segment`` 0 GFX940 Controls ENABLE_PRIVATE_SEGMENT in 14749 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 14750 ``.amdhsa_system_sgpr_workgroup_id_x`` 1 GFX6-GFX10 Controls ENABLE_SGPR_WORKGROUP_ID_X in 14751 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 14752 ``.amdhsa_system_sgpr_workgroup_id_y`` 0 GFX6-GFX10 Controls ENABLE_SGPR_WORKGROUP_ID_Y in 14753 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 14754 ``.amdhsa_system_sgpr_workgroup_id_z`` 0 GFX6-GFX10 Controls ENABLE_SGPR_WORKGROUP_ID_Z in 14755 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 14756 ``.amdhsa_system_sgpr_workgroup_info`` 0 GFX6-GFX10 Controls ENABLE_SGPR_WORKGROUP_INFO in 14757 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 14758 ``.amdhsa_system_vgpr_workitem_id`` 0 GFX6-GFX10 Controls ENABLE_VGPR_WORKITEM_ID in 14759 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 14760 Possible values are defined in 14761 :ref:`amdgpu-amdhsa-system-vgpr-work-item-id-enumeration-values-table`. 14762 ``.amdhsa_next_free_vgpr`` Required GFX6-GFX10 Maximum VGPR number explicitly referenced, plus one. 14763 Used to calculate GRANULATED_WORKITEM_VGPR_COUNT in 14764 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 14765 ``.amdhsa_next_free_sgpr`` Required GFX6-GFX10 Maximum SGPR number explicitly referenced, plus one. 14766 Used to calculate GRANULATED_WAVEFRONT_SGPR_COUNT in 14767 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 14768 ``.amdhsa_accum_offset`` Required GFX90A, Offset of a first AccVGPR in the unified register file. 14769 GFX940 Used to calculate ACCUM_OFFSET in 14770 :ref:`amdgpu-amdhsa-compute_pgm_rsrc3-gfx90a-table`. 14771 ``.amdhsa_reserve_vcc`` 1 GFX6-GFX10 Whether the kernel may use the special VCC SGPR. 14772 Used to calculate GRANULATED_WAVEFRONT_SGPR_COUNT in 14773 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 14774 ``.amdhsa_reserve_flat_scratch`` 1 GFX7-GFX10 Whether the kernel may use flat instructions to access 14775 (except scratch memory. Used to calculate 14776 GFX940) GRANULATED_WAVEFRONT_SGPR_COUNT in 14777 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 14778 ``.amdhsa_reserve_xnack_mask`` Target GFX8-GFX10 Whether the kernel may trigger XNACK replay. 14779 Feature Used to calculate GRANULATED_WAVEFRONT_SGPR_COUNT in 14780 Specific :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 14781 (xnack) 14782 ``.amdhsa_float_round_mode_32`` 0 GFX6-GFX10 Controls FLOAT_ROUND_MODE_32 in 14783 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 14784 Possible values are defined in 14785 :ref:`amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table`. 14786 ``.amdhsa_float_round_mode_16_64`` 0 GFX6-GFX10 Controls FLOAT_ROUND_MODE_16_64 in 14787 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 14788 Possible values are defined in 14789 :ref:`amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table`. 14790 ``.amdhsa_float_denorm_mode_32`` 0 GFX6-GFX10 Controls FLOAT_DENORM_MODE_32 in 14791 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 14792 Possible values are defined in 14793 :ref:`amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table`. 14794 ``.amdhsa_float_denorm_mode_16_64`` 3 GFX6-GFX10 Controls FLOAT_DENORM_MODE_16_64 in 14795 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 14796 Possible values are defined in 14797 :ref:`amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table`. 14798 ``.amdhsa_dx10_clamp`` 1 GFX6-GFX10 Controls ENABLE_DX10_CLAMP in 14799 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 14800 ``.amdhsa_ieee_mode`` 1 GFX6-GFX10 Controls ENABLE_IEEE_MODE in 14801 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 14802 ``.amdhsa_fp16_overflow`` 0 GFX9-GFX10 Controls FP16_OVFL in 14803 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 14804 ``.amdhsa_tg_split`` Target GFX90A, Controls TG_SPLIT in 14805 Feature GFX940 :ref:`amdgpu-amdhsa-compute_pgm_rsrc3-gfx90a-table`. 14806 Specific 14807 (tgsplit) 14808 ``.amdhsa_workgroup_processor_mode`` Target GFX10 Controls ENABLE_WGP_MODE in 14809 Feature :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 14810 Specific 14811 (cumode) 14812 ``.amdhsa_memory_ordered`` 1 GFX10 Controls MEM_ORDERED in 14813 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 14814 ``.amdhsa_forward_progress`` 0 GFX10 Controls FWD_PROGRESS in 14815 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 14816 ``.amdhsa_shared_vgpr_count`` 0 GFX10 Controls SHARED_VGPR_COUNT in 14817 :ref:`amdgpu-amdhsa-compute_pgm_rsrc3-gfx10-table`. 14818 ``.amdhsa_exception_fp_ieee_invalid_op`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_INVALID_OPERATION in 14819 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 14820 ``.amdhsa_exception_fp_denorm_src`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_FP_DENORMAL_SOURCE in 14821 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 14822 ``.amdhsa_exception_fp_ieee_div_zero`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_DIVISION_BY_ZERO in 14823 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 14824 ``.amdhsa_exception_fp_ieee_overflow`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_OVERFLOW in 14825 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 14826 ``.amdhsa_exception_fp_ieee_underflow`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_UNDERFLOW in 14827 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 14828 ``.amdhsa_exception_fp_ieee_inexact`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_INEXACT in 14829 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 14830 ``.amdhsa_exception_int_div_zero`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_INT_DIVIDE_BY_ZERO in 14831 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 14832 ======================================================== =================== ============ =================== 14833 14834.amdgpu_metadata 14835++++++++++++++++ 14836 14837Optional directive which declares the contents of the ``NT_AMDGPU_METADATA`` 14838note record (see :ref:`amdgpu-elf-note-records-table-v3-onwards`). 14839 14840The contents must be in the [YAML]_ markup format, with the same structure and 14841semantics described in :ref:`amdgpu-amdhsa-code-object-metadata-v3`, 14842:ref:`amdgpu-amdhsa-code-object-metadata-v4` or 14843:ref:`amdgpu-amdhsa-code-object-metadata-v5`. 14844 14845This directive is terminated by an ``.end_amdgpu_metadata`` directive. 14846 14847.. _amdgpu-amdhsa-assembler-example-v3-onwards: 14848 14849Code Object V3 and Above Example Source Code 14850~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 14851 14852Here is an example of a minimal assembly source file, defining one HSA kernel: 14853 14854.. code:: 14855 :number-lines: 14856 14857 .amdgcn_target "amdgcn-amd-amdhsa--gfx900+xnack" // optional 14858 14859 .text 14860 .globl hello_world 14861 .p2align 8 14862 .type hello_world,@function 14863 hello_world: 14864 s_load_dwordx2 s[0:1], s[0:1] 0x0 14865 v_mov_b32 v0, 3.14159 14866 s_waitcnt lgkmcnt(0) 14867 v_mov_b32 v1, s0 14868 v_mov_b32 v2, s1 14869 flat_store_dword v[1:2], v0 14870 s_endpgm 14871 .Lfunc_end0: 14872 .size hello_world, .Lfunc_end0-hello_world 14873 14874 .rodata 14875 .p2align 6 14876 .amdhsa_kernel hello_world 14877 .amdhsa_user_sgpr_kernarg_segment_ptr 1 14878 .amdhsa_next_free_vgpr .amdgcn.next_free_vgpr 14879 .amdhsa_next_free_sgpr .amdgcn.next_free_sgpr 14880 .end_amdhsa_kernel 14881 14882 .amdgpu_metadata 14883 --- 14884 amdhsa.version: 14885 - 1 14886 - 0 14887 amdhsa.kernels: 14888 - .name: hello_world 14889 .symbol: hello_world.kd 14890 .kernarg_segment_size: 48 14891 .group_segment_fixed_size: 0 14892 .private_segment_fixed_size: 0 14893 .kernarg_segment_align: 4 14894 .wavefront_size: 64 14895 .sgpr_count: 2 14896 .vgpr_count: 3 14897 .max_flat_workgroup_size: 256 14898 .args: 14899 - .size: 8 14900 .offset: 0 14901 .value_kind: global_buffer 14902 .address_space: global 14903 .actual_access: write_only 14904 //... 14905 .end_amdgpu_metadata 14906 14907This kernel is equivalent to the following HIP program: 14908 14909.. code:: 14910 :number-lines: 14911 14912 __global__ void hello_world(float *p) { 14913 *p = 3.14159f; 14914 } 14915 14916If an assembly source file contains multiple kernels and/or functions, the 14917:ref:`amdgpu-amdhsa-assembler-symbol-next_free_vgpr` and 14918:ref:`amdgpu-amdhsa-assembler-symbol-next_free_sgpr` symbols may be reset using 14919the ``.set <symbol>, <expression>`` directive. For example, in the case of two 14920kernels, where ``function1`` is only called from ``kernel1`` it is sufficient 14921to group the function with the kernel that calls it and reset the symbols 14922between the two connected components: 14923 14924.. code:: 14925 :number-lines: 14926 14927 .amdgcn_target "amdgcn-amd-amdhsa--gfx900+xnack" // optional 14928 14929 // gpr tracking symbols are implicitly set to zero 14930 14931 .text 14932 .globl kern0 14933 .p2align 8 14934 .type kern0,@function 14935 kern0: 14936 // ... 14937 s_endpgm 14938 .Lkern0_end: 14939 .size kern0, .Lkern0_end-kern0 14940 14941 .rodata 14942 .p2align 6 14943 .amdhsa_kernel kern0 14944 // ... 14945 .amdhsa_next_free_vgpr .amdgcn.next_free_vgpr 14946 .amdhsa_next_free_sgpr .amdgcn.next_free_sgpr 14947 .end_amdhsa_kernel 14948 14949 // reset symbols to begin tracking usage in func1 and kern1 14950 .set .amdgcn.next_free_vgpr, 0 14951 .set .amdgcn.next_free_sgpr, 0 14952 14953 .text 14954 .hidden func1 14955 .global func1 14956 .p2align 2 14957 .type func1,@function 14958 func1: 14959 // ... 14960 s_setpc_b64 s[30:31] 14961 .Lfunc1_end: 14962 .size func1, .Lfunc1_end-func1 14963 14964 .globl kern1 14965 .p2align 8 14966 .type kern1,@function 14967 kern1: 14968 // ... 14969 s_getpc_b64 s[4:5] 14970 s_add_u32 s4, s4, func1@rel32@lo+4 14971 s_addc_u32 s5, s5, func1@rel32@lo+4 14972 s_swappc_b64 s[30:31], s[4:5] 14973 // ... 14974 s_endpgm 14975 .Lkern1_end: 14976 .size kern1, .Lkern1_end-kern1 14977 14978 .rodata 14979 .p2align 6 14980 .amdhsa_kernel kern1 14981 // ... 14982 .amdhsa_next_free_vgpr .amdgcn.next_free_vgpr 14983 .amdhsa_next_free_sgpr .amdgcn.next_free_sgpr 14984 .end_amdhsa_kernel 14985 14986These symbols cannot identify connected components in order to automatically 14987track the usage for each kernel. However, in some cases careful organization of 14988the kernels and functions in the source file means there is minimal additional 14989effort required to accurately calculate GPR usage. 14990 14991Additional Documentation 14992======================== 14993 14994.. [AMD-GCN-GFX6] `AMD Southern Islands Series ISA <http://developer.amd.com/wordpress/media/2012/12/AMD_Southern_Islands_Instruction_Set_Architecture.pdf>`__ 14995.. [AMD-GCN-GFX7] `AMD Sea Islands Series ISA <http://developer.amd.com/wordpress/media/2013/07/AMD_Sea_Islands_Instruction_Set_Architecture.pdf>`_ 14996.. [AMD-GCN-GFX8] `AMD GCN3 Instruction Set Architecture <http://amd-dev.wpengine.netdna-cdn.com/wordpress/media/2013/12/AMD_GCN3_Instruction_Set_Architecture_rev1.1.pdf>`__ 14997.. [AMD-GCN-GFX900-GFX904-VEGA] `AMD Vega Instruction Set Architecture <http://developer.amd.com/wordpress/media/2013/12/Vega_Shader_ISA_28July2017.pdf>`__ 14998.. [AMD-GCN-GFX906-VEGA7NM] `AMD Vega 7nm Instruction Set Architecture <https://gpuopen.com/wp-content/uploads/2019/11/Vega_7nm_Shader_ISA_26November2019.pdf>`__ 14999.. [AMD-GCN-GFX908-CDNA1] `AMD Instinct MI100 Instruction Set Architecture <https://developer.amd.com/wp-content/resources/CDNA1_Shader_ISA_14December2020.pdf>`__ 15000.. [AMD-GCN-GFX90A-CDNA2] `AMD Instinct MI200 Instruction Set Architecture <https://developer.amd.com/wp-content/resources/CDNA2_Shader_ISA_4February2022.pdf>`__ 15001.. [AMD-GCN-GFX10-RDNA1] `AMD RDNA 1.0 Instruction Set Architecture <https://gpuopen.com/wp-content/uploads/2019/08/RDNA_Shader_ISA_5August2019.pdf>`__ 15002.. [AMD-GCN-GFX10-RDNA2] `AMD RDNA 2 Instruction Set Architecture <https://developer.amd.com/wp-content/resources/RDNA2_Shader_ISA_November2020.pdf>`__ 15003.. [AMD-RADEON-HD-2000-3000] `AMD R6xx shader ISA <http://developer.amd.com/wordpress/media/2012/10/R600_Instruction_Set_Architecture.pdf>`__ 15004.. [AMD-RADEON-HD-4000] `AMD R7xx shader ISA <http://developer.amd.com/wordpress/media/2012/10/R700-Family_Instruction_Set_Architecture.pdf>`__ 15005.. [AMD-RADEON-HD-5000] `AMD Evergreen shader ISA <http://developer.amd.com/wordpress/media/2012/10/AMD_Evergreen-Family_Instruction_Set_Architecture.pdf>`__ 15006.. [AMD-RADEON-HD-6000] `AMD Cayman/Trinity shader ISA <http://developer.amd.com/wordpress/media/2012/10/AMD_HD_6900_Series_Instruction_Set_Architecture.pdf>`__ 15007.. [AMD-ROCm] `AMD ROCm™ Platform <https://rocmdocs.amd.com/>`__ 15008.. [AMD-ROCm-github] `AMD ROCm™ github <http://github.com/RadeonOpenCompute>`__ 15009.. [AMD-ROCm-Release-Notes] `AMD ROCm Release Notes <https://github.com/RadeonOpenCompute/ROCm>`__ 15010.. [CLANG-ATTR] `Attributes in Clang <https://clang.llvm.org/docs/AttributeReference.html>`__ 15011.. [DWARF] `DWARF Debugging Information Format <http://dwarfstd.org/>`__ 15012.. [ELF] `Executable and Linkable Format (ELF) <http://www.sco.com/developers/gabi/>`__ 15013.. [HRF] `Heterogeneous-race-free Memory Models <http://benedictgaster.org/wp-content/uploads/2014/01/asplos269-FINAL.pdf>`__ 15014.. [HSA] `Heterogeneous System Architecture (HSA) Foundation <http://www.hsafoundation.com/>`__ 15015.. [MsgPack] `Message Pack <http://www.msgpack.org/>`__ 15016.. [OpenCL] `The OpenCL Specification Version 2.0 <http://www.khronos.org/registry/cl/specs/opencl-2.0.pdf>`__ 15017.. [SEMVER] `Semantic Versioning <https://semver.org/>`__ 15018.. [YAML] `YAML Ain't Markup Language (YAML™) Version 1.2 <http://www.yaml.org/spec/1.2/spec.html>`__ 15019