1============================= 2User Guide for AMDGPU Backend 3============================= 4 5.. contents:: 6 :local: 7 8.. toctree:: 9 :hidden: 10 11 AMDGPU/AMDGPUAsmGFX7 12 AMDGPU/AMDGPUAsmGFX8 13 AMDGPU/AMDGPUAsmGFX9 14 AMDGPU/AMDGPUAsmGFX900 15 AMDGPU/AMDGPUAsmGFX904 16 AMDGPU/AMDGPUAsmGFX906 17 AMDGPU/AMDGPUAsmGFX908 18 AMDGPU/AMDGPUAsmGFX90a 19 AMDGPU/AMDGPUAsmGFX10 20 AMDGPU/AMDGPUAsmGFX1011 21 AMDGPU/AMDGPUAsmGFX1013 22 AMDGPU/AMDGPUAsmGFX1030 23 AMDGPUModifierSyntax 24 AMDGPUOperandSyntax 25 AMDGPUInstructionSyntax 26 AMDGPUInstructionNotation 27 AMDGPUDwarfExtensionsForHeterogeneousDebugging 28 AMDGPUDwarfExtensionAllowLocationDescriptionOnTheDwarfExpressionStack/AMDGPUDwarfExtensionAllowLocationDescriptionOnTheDwarfExpressionStack 29 30Introduction 31============ 32 33The AMDGPU backend provides ISA code generation for AMD GPUs, starting with the 34R600 family up until the current GCN families. It lives in the 35``llvm/lib/Target/AMDGPU`` directory. 36 37LLVM 38==== 39 40.. _amdgpu-target-triples: 41 42Target Triples 43-------------- 44 45Use the Clang option ``-target <Architecture>-<Vendor>-<OS>-<Environment>`` 46to specify the target triple: 47 48 .. table:: AMDGPU Architectures 49 :name: amdgpu-architecture-table 50 51 ============ ============================================================== 52 Architecture Description 53 ============ ============================================================== 54 ``r600`` AMD GPUs HD2XXX-HD6XXX for graphics and compute shaders. 55 ``amdgcn`` AMD GPUs GCN GFX6 onwards for graphics and compute shaders. 56 ============ ============================================================== 57 58 .. table:: AMDGPU Vendors 59 :name: amdgpu-vendor-table 60 61 ============ ============================================================== 62 Vendor Description 63 ============ ============================================================== 64 ``amd`` Can be used for all AMD GPU usage. 65 ``mesa3d`` Can be used if the OS is ``mesa3d``. 66 ============ ============================================================== 67 68 .. table:: AMDGPU Operating Systems 69 :name: amdgpu-os 70 71 ============== ============================================================ 72 OS Description 73 ============== ============================================================ 74 *<empty>* Defaults to the *unknown* OS. 75 ``amdhsa`` Compute kernels executed on HSA [HSA]_ compatible runtimes 76 such as: 77 78 - AMD's ROCm™ runtime [AMD-ROCm]_ using the *rocm-amdhsa* 79 loader on Linux. See *AMD ROCm Platform Release Notes* 80 [AMD-ROCm-Release-Notes]_ for supported hardware and 81 software. 82 - AMD's PAL runtime using the *pal-amdhsa* loader on 83 Windows. 84 85 ``amdpal`` Graphic shaders and compute kernels executed on AMD's PAL 86 runtime using the *pal-amdpal* loader on Windows and Linux 87 Pro. 88 ``mesa3d`` Graphic shaders and compute kernels executed on AMD's Mesa 89 3D runtime using the *mesa-mesa3d* loader on Linux. 90 ============== ============================================================ 91 92 .. table:: AMDGPU Environments 93 :name: amdgpu-environment-table 94 95 ============ ============================================================== 96 Environment Description 97 ============ ============================================================== 98 *<empty>* Default. 99 ============ ============================================================== 100 101.. _amdgpu-processors: 102 103Processors 104---------- 105 106Use the Clang options ``-mcpu=<target-id>`` or ``--offload-arch=<target-id>`` to 107specify the AMDGPU processor together with optional target features. See 108:ref:`amdgpu-target-id` and :ref:`amdgpu-target-features` for AMD GPU target 109specific information. 110 111Every processor supports every OS ABI (see :ref:`amdgpu-os`) with the following exceptions: 112 113* ``amdhsa`` is not supported in ``r600`` architecture (see :ref:`amdgpu-architecture-table`). 114 115 116 .. table:: AMDGPU Processors 117 :name: amdgpu-processor-table 118 119 =========== =============== ============ ===== ================= =============== =============== ====================== 120 Processor Alternative Target dGPU/ Target Target OS Support Example 121 Processor Triple APU Features Properties *(see* Products 122 Architecture Supported `amdgpu-os`_ 123 *and 124 corresponding 125 runtime release 126 notes for 127 current 128 information and 129 level of 130 support)* 131 =========== =============== ============ ===== ================= =============== =============== ====================== 132 **Radeon HD 2000/3000 Series (R600)** [AMD-RADEON-HD-2000-3000]_ 133 ----------------------------------------------------------------------------------------------------------------------- 134 ``r600`` ``r600`` dGPU - Does not 135 support 136 generic 137 address 138 space 139 ``r630`` ``r600`` dGPU - Does not 140 support 141 generic 142 address 143 space 144 ``rs880`` ``r600`` dGPU - Does not 145 support 146 generic 147 address 148 space 149 ``rv670`` ``r600`` dGPU - Does not 150 support 151 generic 152 address 153 space 154 **Radeon HD 4000 Series (R700)** [AMD-RADEON-HD-4000]_ 155 ----------------------------------------------------------------------------------------------------------------------- 156 ``rv710`` ``r600`` dGPU - Does not 157 support 158 generic 159 address 160 space 161 ``rv730`` ``r600`` dGPU - Does not 162 support 163 generic 164 address 165 space 166 ``rv770`` ``r600`` dGPU - Does not 167 support 168 generic 169 address 170 space 171 **Radeon HD 5000 Series (Evergreen)** [AMD-RADEON-HD-5000]_ 172 ----------------------------------------------------------------------------------------------------------------------- 173 ``cedar`` ``r600`` dGPU - Does not 174 support 175 generic 176 address 177 space 178 ``cypress`` ``r600`` dGPU - Does not 179 support 180 generic 181 address 182 space 183 ``juniper`` ``r600`` dGPU - Does not 184 support 185 generic 186 address 187 space 188 ``redwood`` ``r600`` dGPU - Does not 189 support 190 generic 191 address 192 space 193 ``sumo`` ``r600`` dGPU - Does not 194 support 195 generic 196 address 197 space 198 **Radeon HD 6000 Series (Northern Islands)** [AMD-RADEON-HD-6000]_ 199 ----------------------------------------------------------------------------------------------------------------------- 200 ``barts`` ``r600`` dGPU - Does not 201 support 202 generic 203 address 204 space 205 ``caicos`` ``r600`` dGPU - Does not 206 support 207 generic 208 address 209 space 210 ``cayman`` ``r600`` dGPU - Does not 211 support 212 generic 213 address 214 space 215 ``turks`` ``r600`` dGPU - Does not 216 support 217 generic 218 address 219 space 220 **GCN GFX6 (Southern Islands (SI))** [AMD-GCN-GFX6]_ 221 ----------------------------------------------------------------------------------------------------------------------- 222 ``gfx600`` - ``tahiti`` ``amdgcn`` dGPU - Does not - *pal-amdpal* 223 support 224 generic 225 address 226 space 227 ``gfx601`` - ``pitcairn`` ``amdgcn`` dGPU - Does not - *pal-amdpal* 228 - ``verde`` support 229 generic 230 address 231 space 232 ``gfx602`` - ``hainan`` ``amdgcn`` dGPU - Does not - *pal-amdpal* 233 - ``oland`` support 234 generic 235 address 236 space 237 **GCN GFX7 (Sea Islands (CI))** [AMD-GCN-GFX7]_ 238 ----------------------------------------------------------------------------------------------------------------------- 239 ``gfx700`` - ``kaveri`` ``amdgcn`` APU - Offset - *rocm-amdhsa* - A6-7000 240 flat - *pal-amdhsa* - A6 Pro-7050B 241 scratch - *pal-amdpal* - A8-7100 242 - A8 Pro-7150B 243 - A10-7300 244 - A10 Pro-7350B 245 - FX-7500 246 - A8-7200P 247 - A10-7400P 248 - FX-7600P 249 ``gfx701`` - ``hawaii`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - FirePro W8100 250 flat - *pal-amdhsa* - FirePro W9100 251 scratch - *pal-amdpal* - FirePro S9150 252 - FirePro S9170 253 ``gfx702`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - Radeon R9 290 254 flat - *pal-amdhsa* - Radeon R9 290x 255 scratch - *pal-amdpal* - Radeon R390 256 - Radeon R390x 257 ``gfx703`` - ``kabini`` ``amdgcn`` APU - Offset - *pal-amdhsa* - E1-2100 258 - ``mullins`` flat - *pal-amdpal* - E1-2200 259 scratch - E1-2500 260 - E2-3000 261 - E2-3800 262 - A4-5000 263 - A4-5100 264 - A6-5200 265 - A4 Pro-3340B 266 ``gfx704`` - ``bonaire`` ``amdgcn`` dGPU - Offset - *pal-amdhsa* - Radeon HD 7790 267 flat - *pal-amdpal* - Radeon HD 8770 268 scratch - R7 260 269 - R7 260X 270 ``gfx705`` ``amdgcn`` APU - Offset - *pal-amdhsa* *TBA* 271 flat - *pal-amdpal* 272 scratch .. TODO:: 273 274 Add product 275 names. 276 277 **GCN GFX8 (Volcanic Islands (VI))** [AMD-GCN-GFX8]_ 278 ----------------------------------------------------------------------------------------------------------------------- 279 ``gfx801`` - ``carrizo`` ``amdgcn`` APU - xnack - Offset - *rocm-amdhsa* - A6-8500P 280 flat - *pal-amdhsa* - Pro A6-8500B 281 scratch - *pal-amdpal* - A8-8600P 282 - Pro A8-8600B 283 - FX-8800P 284 - Pro A12-8800B 285 - A10-8700P 286 - Pro A10-8700B 287 - A10-8780P 288 - A10-9600P 289 - A10-9630P 290 - A12-9700P 291 - A12-9730P 292 - FX-9800P 293 - FX-9830P 294 - E2-9010 295 - A6-9210 296 - A9-9410 297 ``gfx802`` - ``iceland`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - Radeon R9 285 298 - ``tonga`` flat - *pal-amdhsa* - Radeon R9 380 299 scratch - *pal-amdpal* - Radeon R9 385 300 ``gfx803`` - ``fiji`` ``amdgcn`` dGPU - *rocm-amdhsa* - Radeon R9 Nano 301 - *pal-amdhsa* - Radeon R9 Fury 302 - *pal-amdpal* - Radeon R9 FuryX 303 - Radeon Pro Duo 304 - FirePro S9300x2 305 - Radeon Instinct MI8 306 \ - ``polaris10`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - Radeon RX 470 307 flat - *pal-amdhsa* - Radeon RX 480 308 scratch - *pal-amdpal* - Radeon Instinct MI6 309 \ - ``polaris11`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - Radeon RX 460 310 flat - *pal-amdhsa* 311 scratch - *pal-amdpal* 312 ``gfx805`` - ``tongapro`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - FirePro S7150 313 flat - *pal-amdhsa* - FirePro S7100 314 scratch - *pal-amdpal* - FirePro W7100 315 - Mobile FirePro 316 M7170 317 ``gfx810`` - ``stoney`` ``amdgcn`` APU - xnack - Offset - *rocm-amdhsa* *TBA* 318 flat - *pal-amdhsa* 319 scratch - *pal-amdpal* .. TODO:: 320 321 Add product 322 names. 323 324 **GCN GFX9 (Vega)** [AMD-GCN-GFX900-GFX904-VEGA]_ [AMD-GCN-GFX906-VEGA7NM]_ [AMD-GCN-GFX908-CDNA1]_ [AMD-GCN-GFX90A-CDNA2]_ 325 ----------------------------------------------------------------------------------------------------------------------- 326 ``gfx900`` ``amdgcn`` dGPU - xnack - Absolute - *rocm-amdhsa* - Radeon Vega 327 flat - *pal-amdhsa* Frontier Edition 328 scratch - *pal-amdpal* - Radeon RX Vega 56 329 - Radeon RX Vega 64 330 - Radeon RX Vega 64 331 Liquid 332 - Radeon Instinct MI25 333 ``gfx902`` ``amdgcn`` APU - xnack - Absolute - *rocm-amdhsa* - Ryzen 3 2200G 334 flat - *pal-amdhsa* - Ryzen 5 2400G 335 scratch - *pal-amdpal* 336 ``gfx904`` ``amdgcn`` dGPU - xnack - *rocm-amdhsa* *TBA* 337 - *pal-amdhsa* 338 - *pal-amdpal* .. TODO:: 339 340 Add product 341 names. 342 343 ``gfx906`` ``amdgcn`` dGPU - sramecc - Absolute - *rocm-amdhsa* - Radeon Instinct MI50 344 - xnack flat - *pal-amdhsa* - Radeon Instinct MI60 345 scratch - *pal-amdpal* - Radeon VII 346 - Radeon Pro VII 347 ``gfx908`` ``amdgcn`` dGPU - sramecc - *rocm-amdhsa* - AMD Instinct MI100 Accelerator 348 - xnack - Absolute 349 flat 350 scratch 351 ``gfx909`` ``amdgcn`` APU - xnack - Absolute - *pal-amdpal* *TBA* 352 flat 353 scratch .. TODO:: 354 355 Add product 356 names. 357 358 ``gfx90a`` ``amdgcn`` dGPU - sramecc - Absolute - *rocm-amdhsa* *TBA* 359 - tgsplit flat 360 - xnack scratch .. TODO:: 361 - Packed 362 work-item Add product 363 IDs names. 364 365 ``gfx90c`` ``amdgcn`` APU - xnack - Absolute - *pal-amdpal* - Ryzen 7 4700G 366 flat - Ryzen 7 4700GE 367 scratch - Ryzen 5 4600G 368 - Ryzen 5 4600GE 369 - Ryzen 3 4300G 370 - Ryzen 3 4300GE 371 - Ryzen Pro 4000G 372 - Ryzen 7 Pro 4700G 373 - Ryzen 7 Pro 4750GE 374 - Ryzen 5 Pro 4650G 375 - Ryzen 5 Pro 4650GE 376 - Ryzen 3 Pro 4350G 377 - Ryzen 3 Pro 4350GE 378 379 ``gfx940`` ``amdgcn`` dGPU - sramecc - Architected *TBA* 380 - tgsplit flat 381 - xnack scratch .. TODO:: 382 - Packed 383 work-item Add product 384 IDs names. 385 386 **GCN GFX10.1 (RDNA 1)** [AMD-GCN-GFX10-RDNA1]_ 387 ----------------------------------------------------------------------------------------------------------------------- 388 ``gfx1010`` ``amdgcn`` dGPU - cumode - Absolute - *rocm-amdhsa* - Radeon RX 5700 389 - wavefrontsize64 flat - *pal-amdhsa* - Radeon RX 5700 XT 390 - xnack scratch - *pal-amdpal* - Radeon Pro 5600 XT 391 - Radeon Pro 5600M 392 ``gfx1011`` ``amdgcn`` dGPU - cumode - *rocm-amdhsa* - Radeon Pro V520 393 - wavefrontsize64 - Absolute - *pal-amdhsa* 394 - xnack flat - *pal-amdpal* 395 scratch 396 ``gfx1012`` ``amdgcn`` dGPU - cumode - Absolute - *rocm-amdhsa* - Radeon RX 5500 397 - wavefrontsize64 flat - *pal-amdhsa* - Radeon RX 5500 XT 398 - xnack scratch - *pal-amdpal* 399 ``gfx1013`` ``amdgcn`` APU - cumode - Absolute - *rocm-amdhsa* *TBA* 400 - wavefrontsize64 flat - *pal-amdhsa* 401 - xnack scratch - *pal-amdpal* .. TODO:: 402 403 Add product 404 names. 405 406 **GCN GFX10.3 (RDNA 2)** [AMD-GCN-GFX10-RDNA2]_ 407 ----------------------------------------------------------------------------------------------------------------------- 408 ``gfx1030`` ``amdgcn`` dGPU - cumode - Absolute - *rocm-amdhsa* - Radeon RX 6800 409 - wavefrontsize64 flat - *pal-amdhsa* - Radeon RX 6800 XT 410 scratch - *pal-amdpal* - Radeon RX 6900 XT 411 ``gfx1031`` ``amdgcn`` dGPU - cumode - Absolute - *rocm-amdhsa* - Radeon RX 6700 XT 412 - wavefrontsize64 flat - *pal-amdhsa* 413 scratch - *pal-amdpal* 414 ``gfx1032`` ``amdgcn`` dGPU - cumode - Absolute - *rocm-amdhsa* *TBA* 415 - wavefrontsize64 flat - *pal-amdhsa* 416 scratch - *pal-amdpal* .. TODO:: 417 418 Add product 419 names. 420 421 ``gfx1033`` ``amdgcn`` APU - cumode - Absolute - *pal-amdpal* *TBA* 422 - wavefrontsize64 flat 423 scratch .. TODO:: 424 425 Add product 426 names. 427 ``gfx1034`` ``amdgcn`` dGPU - cumode - Absolute - *pal-amdpal* *TBA* 428 - wavefrontsize64 flat 429 scratch .. TODO:: 430 431 Add product 432 names. 433 434 ``gfx1035`` ``amdgcn`` APU - cumode - Absolute - *pal-amdpal* *TBA* 435 - wavefrontsize64 flat 436 scratch .. TODO:: 437 Add product 438 names. 439 440 ``gfx1036`` ``amdgcn`` APU - cumode - Absolute - *pal-amdpal* *TBA* 441 - wavefrontsize64 flat 442 scratch .. TODO:: 443 444 Add product 445 names. 446 447 =========== =============== ============ ===== ================= =============== =============== ====================== 448 449.. _amdgpu-target-features: 450 451Target Features 452--------------- 453 454Target features control how code is generated to support certain 455processor specific features. Not all target features are supported by 456all processors. The runtime must ensure that the features supported by 457the device used to execute the code match the features enabled when 458generating the code. A mismatch of features may result in incorrect 459execution, or a reduction in performance. 460 461The target features supported by each processor is listed in 462:ref:`amdgpu-processor-table`. 463 464Target features are controlled by exactly one of the following Clang 465options: 466 467``-mcpu=<target-id>`` or ``--offload-arch=<target-id>`` 468 469 The ``-mcpu`` and ``--offload-arch`` can specify the target feature as 470 optional components of the target ID. If omitted, the target feature has the 471 ``any`` value. See :ref:`amdgpu-target-id`. 472 473``-m[no-]<target-feature>`` 474 475 Target features not specified by the target ID are specified using a 476 separate option. These target features can have an ``on`` or ``off`` 477 value. ``on`` is specified by omitting the ``no-`` prefix, and 478 ``off`` is specified by including the ``no-`` prefix. The default 479 if not specified is ``off``. 480 481For example: 482 483``-mcpu=gfx908:xnack+`` 484 Enable the ``xnack`` feature. 485``-mcpu=gfx908:xnack-`` 486 Disable the ``xnack`` feature. 487``-mcumode`` 488 Enable the ``cumode`` feature. 489``-mno-cumode`` 490 Disable the ``cumode`` feature. 491 492 .. table:: AMDGPU Target Features 493 :name: amdgpu-target-features-table 494 495 =============== ============================ ================================================== 496 Target Feature Clang Option to Control Description 497 Name 498 =============== ============================ ================================================== 499 cumode - ``-m[no-]cumode`` Control the wavefront execution mode used 500 when generating code for kernels. When disabled 501 native WGP wavefront execution mode is used, 502 when enabled CU wavefront execution mode is used 503 (see :ref:`amdgpu-amdhsa-memory-model`). 504 505 sramecc - ``-mcpu`` If specified, generate code that can only be 506 - ``--offload-arch`` loaded and executed in a process that has a 507 matching setting for SRAMECC. 508 509 If not specified for code object V2 to V3, generate 510 code that can be loaded and executed in a process 511 with SRAMECC enabled. 512 513 If not specified for code object V4 or above, generate 514 code that can be loaded and executed in a process 515 with either setting of SRAMECC. 516 517 tgsplit ``-m[no-]tgsplit`` Enable/disable generating code that assumes 518 work-groups are launched in threadgroup split mode. 519 When enabled the waves of a work-group may be 520 launched in different CUs. 521 522 wavefrontsize64 - ``-m[no-]wavefrontsize64`` Control the wavefront size used when 523 generating code for kernels. When disabled 524 native wavefront size 32 is used, when enabled 525 wavefront size 64 is used. 526 527 xnack - ``-mcpu`` If specified, generate code that can only be 528 - ``--offload-arch`` loaded and executed in a process that has a 529 matching setting for XNACK replay. 530 531 If not specified for code object V2 to V3, generate 532 code that can be loaded and executed in a process 533 with XNACK replay enabled. 534 535 If not specified for code object V4 or above, generate 536 code that can be loaded and executed in a process 537 with either setting of XNACK replay. 538 539 XNACK replay can be used for demand paging and 540 page migration. If enabled in the device, then if 541 a page fault occurs the code may execute 542 incorrectly unless generated with XNACK replay 543 enabled, or generated for code object V4 or above without 544 specifying XNACK replay. Executing code that was 545 generated with XNACK replay enabled, or generated 546 for code object V4 or above without specifying XNACK replay, 547 on a device that does not have XNACK replay 548 enabled will execute correctly but may be less 549 performant than code generated for XNACK replay 550 disabled. 551 =============== ============================ ================================================== 552 553.. _amdgpu-target-id: 554 555Target ID 556--------- 557 558AMDGPU supports target IDs. See `Clang Offload Bundler 559<https://clang.llvm.org/docs/ClangOffloadBundler.html>`_ for a general 560description. The AMDGPU target specific information is: 561 562**processor** 563 Is an AMDGPU processor or alternative processor name specified in 564 :ref:`amdgpu-processor-table`. The non-canonical form target ID allows both 565 the primary processor and alternative processor names. The canonical form 566 target ID only allow the primary processor name. 567 568**target-feature** 569 Is a target feature name specified in :ref:`amdgpu-target-features-table` that 570 is supported by the processor. The target features supported by each processor 571 is specified in :ref:`amdgpu-processor-table`. Those that can be specified in 572 a target ID are marked as being controlled by ``-mcpu`` and 573 ``--offload-arch``. Each target feature must appear at most once in a target 574 ID. The non-canonical form target ID allows the target features to be 575 specified in any order. The canonical form target ID requires the target 576 features to be specified in alphabetic order. 577 578.. _amdgpu-target-id-v2-v3: 579 580Code Object V2 to V3 Target ID 581~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 582 583The target ID syntax for code object V2 to V3 is the same as defined in `Clang 584Offload Bundler <https://clang.llvm.org/docs/ClangOffloadBundler.html>`_ except 585when used in the :ref:`amdgpu-assembler-directive-amdgcn-target` assembler 586directive and the bundle entry ID. In those cases it has the following BNF 587syntax: 588 589.. code:: 590 591 <target-id> ::== <processor> ( "+" <target-feature> )* 592 593Where a target feature is omitted if *Off* and present if *On* or *Any*. 594 595.. note:: 596 597 The code object V2 to V3 cannot represent *Any* and treats it the same as 598 *On*. 599 600.. _amdgpu-embedding-bundled-objects: 601 602Embedding Bundled Code Objects 603------------------------------ 604 605AMDGPU supports the HIP and OpenMP languages that perform code object embedding 606as described in `Clang Offload Bundler 607<https://clang.llvm.org/docs/ClangOffloadBundler.html>`_. 608 609.. note:: 610 611 The target ID syntax used for code object V2 to V3 for a bundle entry ID 612 differs from that used elsewhere. See :ref:`amdgpu-target-id-v2-v3`. 613 614.. _amdgpu-address-spaces: 615 616Address Spaces 617-------------- 618 619The AMDGPU architecture supports a number of memory address spaces. The address 620space names use the OpenCL standard names, with some additions. 621 622The AMDGPU address spaces correspond to target architecture specific LLVM 623address space numbers used in LLVM IR. 624 625The AMDGPU address spaces are described in 626:ref:`amdgpu-address-spaces-table`. Only 64-bit process address spaces are 627supported for the ``amdgcn`` target. 628 629 .. table:: AMDGPU Address Spaces 630 :name: amdgpu-address-spaces-table 631 632 ================================= =============== =========== ================ ======= ============================ 633 .. 64-Bit Process Address Space 634 --------------------------------- --------------- ----------- ---------------- ------------------------------------ 635 Address Space Name LLVM IR Address HSA Segment Hardware Address NULL Value 636 Space Number Name Name Size 637 ================================= =============== =========== ================ ======= ============================ 638 Generic 0 flat flat 64 0x0000000000000000 639 Global 1 global global 64 0x0000000000000000 640 Region 2 N/A GDS 32 *not implemented for AMDHSA* 641 Local 3 group LDS 32 0xFFFFFFFF 642 Constant 4 constant *same as global* 64 0x0000000000000000 643 Private 5 private scratch 32 0xFFFFFFFF 644 Constant 32-bit 6 *TODO* 0x00000000 645 Buffer Fat Pointer (experimental) 7 *TODO* 646 ================================= =============== =========== ================ ======= ============================ 647 648**Generic** 649 The generic address space is supported unless the *Target Properties* column 650 of :ref:`amdgpu-processor-table` specifies *Does not support generic address 651 space*. 652 653 The generic address space uses the hardware flat address support for two fixed 654 ranges of virtual addresses (the private and local apertures), that are 655 outside the range of addressable global memory, to map from a flat address to 656 a private or local address. This uses FLAT instructions that can take a flat 657 address and access global, private (scratch), and group (LDS) memory depending 658 on if the address is within one of the aperture ranges. 659 660 Flat access to scratch requires hardware aperture setup and setup in the 661 kernel prologue (see :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`). Flat 662 access to LDS requires hardware aperture setup and M0 (GFX7-GFX8) register 663 setup (see :ref:`amdgpu-amdhsa-kernel-prolog-m0`). 664 665 To convert between a private or group address space address (termed a segment 666 address) and a flat address the base address of the corresponding aperture 667 can be used. For GFX7-GFX8 these are available in the 668 :ref:`amdgpu-amdhsa-hsa-aql-queue` the address of which can be obtained with 669 Queue Ptr SGPR (see :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). For 670 GFX9-GFX10 the aperture base addresses are directly available as inline 671 constant registers ``SRC_SHARED_BASE/LIMIT`` and ``SRC_PRIVATE_BASE/LIMIT``. 672 In 64-bit address mode the aperture sizes are 2^32 bytes and the base is 673 aligned to 2^32 which makes it easier to convert from flat to segment or 674 segment to flat. 675 676 A global address space address has the same value when used as a flat address 677 so no conversion is needed. 678 679**Global and Constant** 680 The global and constant address spaces both use global virtual addresses, 681 which are the same virtual address space used by the CPU. However, some 682 virtual addresses may only be accessible to the CPU, some only accessible 683 by the GPU, and some by both. 684 685 Using the constant address space indicates that the data will not change 686 during the execution of the kernel. This allows scalar read instructions to 687 be used. As the constant address space could only be modified on the host 688 side, a generic pointer loaded from the constant address space is safe to be 689 assumed as a global pointer since only the device global memory is visible 690 and managed on the host side. The vector and scalar L1 caches are invalidated 691 of volatile data before each kernel dispatch execution to allow constant 692 memory to change values between kernel dispatches. 693 694**Region** 695 The region address space uses the hardware Global Data Store (GDS). All 696 wavefronts executing on the same device will access the same memory for any 697 given region address. However, the same region address accessed by wavefronts 698 executing on different devices will access different memory. It is higher 699 performance than global memory. It is allocated by the runtime. The data 700 store (DS) instructions can be used to access it. 701 702**Local** 703 The local address space uses the hardware Local Data Store (LDS) which is 704 automatically allocated when the hardware creates the wavefronts of a 705 work-group, and freed when all the wavefronts of a work-group have 706 terminated. All wavefronts belonging to the same work-group will access the 707 same memory for any given local address. However, the same local address 708 accessed by wavefronts belonging to different work-groups will access 709 different memory. It is higher performance than global memory. The data store 710 (DS) instructions can be used to access it. 711 712**Private** 713 The private address space uses the hardware scratch memory support which 714 automatically allocates memory when it creates a wavefront and frees it when 715 a wavefronts terminates. The memory accessed by a lane of a wavefront for any 716 given private address will be different to the memory accessed by another lane 717 of the same or different wavefront for the same private address. 718 719 If a kernel dispatch uses scratch, then the hardware allocates memory from a 720 pool of backing memory allocated by the runtime for each wavefront. The lanes 721 of the wavefront access this using dword (4 byte) interleaving. The mapping 722 used from private address to backing memory address is: 723 724 ``wavefront-scratch-base + 725 ((private-address / 4) * wavefront-size * 4) + 726 (wavefront-lane-id * 4) + (private-address % 4)`` 727 728 If each lane of a wavefront accesses the same private address, the 729 interleaving results in adjacent dwords being accessed and hence requires 730 fewer cache lines to be fetched. 731 732 There are different ways that the wavefront scratch base address is 733 determined by a wavefront (see 734 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 735 736 Scratch memory can be accessed in an interleaved manner using buffer 737 instructions with the scratch buffer descriptor and per wavefront scratch 738 offset, by the scratch instructions, or by flat instructions. Multi-dword 739 access is not supported except by flat and scratch instructions in 740 GFX9-GFX10. 741 742**Constant 32-bit** 743 *TODO* 744 745**Buffer Fat Pointer** 746 The buffer fat pointer is an experimental address space that is currently 747 unsupported in the backend. It exposes a non-integral pointer that is in 748 the future intended to support the modelling of 128-bit buffer descriptors 749 plus a 32-bit offset into the buffer (in total encapsulating a 160-bit 750 *pointer*), allowing normal LLVM load/store/atomic operations to be used to 751 model the buffer descriptors used heavily in graphics workloads targeting 752 the backend. 753 754.. _amdgpu-memory-scopes: 755 756Memory Scopes 757------------- 758 759This section provides LLVM memory synchronization scopes supported by the AMDGPU 760backend memory model when the target triple OS is ``amdhsa`` (see 761:ref:`amdgpu-amdhsa-memory-model` and :ref:`amdgpu-target-triples`). 762 763The memory model supported is based on the HSA memory model [HSA]_ which is 764based in turn on HRF-indirect with scope inclusion [HRF]_. The happens-before 765relation is transitive over the synchronizes-with relation independent of scope 766and synchronizes-with allows the memory scope instances to be inclusive (see 767table :ref:`amdgpu-amdhsa-llvm-sync-scopes-table`). 768 769This is different to the OpenCL [OpenCL]_ memory model which does not have scope 770inclusion and requires the memory scopes to exactly match. However, this 771is conservatively correct for OpenCL. 772 773 .. table:: AMDHSA LLVM Sync Scopes 774 :name: amdgpu-amdhsa-llvm-sync-scopes-table 775 776 ======================= =================================================== 777 LLVM Sync Scope Description 778 ======================= =================================================== 779 *none* The default: ``system``. 780 781 Synchronizes with, and participates in modification 782 and seq_cst total orderings with, other operations 783 (except image operations) for all address spaces 784 (except private, or generic that accesses private) 785 provided the other operation's sync scope is: 786 787 - ``system``. 788 - ``agent`` and executed by a thread on the same 789 agent. 790 - ``workgroup`` and executed by a thread in the 791 same work-group. 792 - ``wavefront`` and executed by a thread in the 793 same wavefront. 794 795 ``agent`` Synchronizes with, and participates in modification 796 and seq_cst total orderings with, other operations 797 (except image operations) for all address spaces 798 (except private, or generic that accesses private) 799 provided the other operation's sync scope is: 800 801 - ``system`` or ``agent`` and executed by a thread 802 on the same agent. 803 - ``workgroup`` and executed by a thread in the 804 same work-group. 805 - ``wavefront`` and executed by a thread in the 806 same wavefront. 807 808 ``workgroup`` Synchronizes with, and participates in modification 809 and seq_cst total orderings with, other operations 810 (except image operations) for all address spaces 811 (except private, or generic that accesses private) 812 provided the other operation's sync scope is: 813 814 - ``system``, ``agent`` or ``workgroup`` and 815 executed by a thread in the same work-group. 816 - ``wavefront`` and executed by a thread in the 817 same wavefront. 818 819 ``wavefront`` Synchronizes with, and participates in modification 820 and seq_cst total orderings with, other operations 821 (except image operations) for all address spaces 822 (except private, or generic that accesses private) 823 provided the other operation's sync scope is: 824 825 - ``system``, ``agent``, ``workgroup`` or 826 ``wavefront`` and executed by a thread in the 827 same wavefront. 828 829 ``singlethread`` Only synchronizes with and participates in 830 modification and seq_cst total orderings with, 831 other operations (except image operations) running 832 in the same thread for all address spaces (for 833 example, in signal handlers). 834 835 ``one-as`` Same as ``system`` but only synchronizes with other 836 operations within the same address space. 837 838 ``agent-one-as`` Same as ``agent`` but only synchronizes with other 839 operations within the same address space. 840 841 ``workgroup-one-as`` Same as ``workgroup`` but only synchronizes with 842 other operations within the same address space. 843 844 ``wavefront-one-as`` Same as ``wavefront`` but only synchronizes with 845 other operations within the same address space. 846 847 ``singlethread-one-as`` Same as ``singlethread`` but only synchronizes with 848 other operations within the same address space. 849 ======================= =================================================== 850 851LLVM IR Intrinsics 852------------------ 853 854The AMDGPU backend implements the following LLVM IR intrinsics. 855 856*This section is WIP.* 857 858.. TODO:: 859 860 List AMDGPU intrinsics. 861 862LLVM IR Attributes 863------------------ 864 865The AMDGPU backend supports the following LLVM IR attributes. 866 867 .. table:: AMDGPU LLVM IR Attributes 868 :name: amdgpu-llvm-ir-attributes-table 869 870 ======================================= ========================================================== 871 LLVM Attribute Description 872 ======================================= ========================================================== 873 "amdgpu-flat-work-group-size"="min,max" Specify the minimum and maximum flat work group sizes that 874 will be specified when the kernel is dispatched. Generated 875 by the ``amdgpu_flat_work_group_size`` CLANG attribute [CLANG-ATTR]_. 876 The implied default value is 1,1024. 877 878 "amdgpu-implicitarg-num-bytes"="n" Number of kernel argument bytes to add to the kernel 879 argument block size for the implicit arguments. This 880 varies by OS and language (for OpenCL see 881 :ref:`opencl-kernel-implicit-arguments-appended-for-amdhsa-os-table`). 882 "amdgpu-num-sgpr"="n" Specifies the number of SGPRs to use. Generated by 883 the ``amdgpu_num_sgpr`` CLANG attribute [CLANG-ATTR]_. 884 "amdgpu-num-vgpr"="n" Specifies the number of VGPRs to use. Generated by the 885 ``amdgpu_num_vgpr`` CLANG attribute [CLANG-ATTR]_. 886 "amdgpu-waves-per-eu"="m,n" Specify the minimum and maximum number of waves per 887 execution unit. Generated by the ``amdgpu_waves_per_eu`` 888 CLANG attribute [CLANG-ATTR]_. This is an optimization hint, 889 and the backend may not be able to satisfy the request. If 890 the specified range is incompatible with the function's 891 "amdgpu-flat-work-group-size" value, the implied occupancy 892 bounds by the workgroup size takes precedence. 893 894 "amdgpu-ieee" true/false. Specify whether the function expects the IEEE field of the 895 mode register to be set on entry. Overrides the default for 896 the calling convention. 897 "amdgpu-dx10-clamp" true/false. Specify whether the function expects the DX10_CLAMP field of 898 the mode register to be set on entry. Overrides the default 899 for the calling convention. 900 901 "amdgpu-no-workitem-id-x" Indicates the function does not depend on the value of the 902 llvm.amdgcn.workitem.id.x intrinsic. If a function is marked with this 903 attribute, or reached through a call site marked with this attribute, 904 the value returned by the intrinsic is undefined. The backend can 905 generally infer this during code generation, so typically there is no 906 benefit to frontends marking functions with this. 907 908 "amdgpu-no-workitem-id-y" The same as amdgpu-no-workitem-id-x, except for the 909 llvm.amdgcn.workitem.id.y intrinsic. 910 911 "amdgpu-no-workitem-id-z" The same as amdgpu-no-workitem-id-x, except for the 912 llvm.amdgcn.workitem.id.z intrinsic. 913 914 "amdgpu-no-workgroup-id-x" The same as amdgpu-no-workitem-id-x, except for the 915 llvm.amdgcn.workgroup.id.x intrinsic. 916 917 "amdgpu-no-workgroup-id-y" The same as amdgpu-no-workitem-id-x, except for the 918 llvm.amdgcn.workgroup.id.y intrinsic. 919 920 "amdgpu-no-workgroup-id-z" The same as amdgpu-no-workitem-id-x, except for the 921 llvm.amdgcn.workgroup.id.z intrinsic. 922 923 "amdgpu-no-dispatch-ptr" The same as amdgpu-no-workitem-id-x, except for the 924 llvm.amdgcn.dispatch.ptr intrinsic. 925 926 "amdgpu-no-implicitarg-ptr" The same as amdgpu-no-workitem-id-x, except for the 927 llvm.amdgcn.implicitarg.ptr intrinsic. 928 929 "amdgpu-no-dispatch-id" The same as amdgpu-no-workitem-id-x, except for the 930 llvm.amdgcn.dispatch.id intrinsic. 931 932 "amdgpu-no-queue-ptr" Similar to amdgpu-no-workitem-id-x, except for the 933 llvm.amdgcn.queue.ptr intrinsic. Note that unlike the other ABI hint 934 attributes, the queue pointer may be required in situations where the 935 intrinsic call does not directly appear in the program. Some subtargets 936 require the queue pointer for to handle some addrspacecasts, as well 937 as the llvm.amdgcn.is.shared, llvm.amdgcn.is.private, llvm.trap, and 938 llvm.debug intrinsics. 939 940 "amdgpu-no-hostcall-ptr" Similar to amdgpu-no-implicitarg-ptr, except specific to the implicit 941 kernel argument that holds the pointer to the hostcall buffer. If this 942 attribute is absent, then the amdgpu-no-implicitarg-ptr is also removed. 943 944 "amdgpu-no-heap-ptr" Similar to amdgpu-no-implicitarg-ptr, except specific to the implicit 945 kernel argument that holds the pointer to an initialized memory buffer 946 that conforms to the requirements of the malloc/free device library V1 947 version implementation. If this attribute is absent, then the 948 amdgpu-no-implicitarg-ptr is also removed. 949 950 "amdgpu-no-multigrid-sync-arg" Similar to amdgpu-no-implicitarg-ptr, except specific to the implicit 951 kernel argument that holds the multigrid synchronization pointer. If this 952 attribute is absent, then the amdgpu-no-implicitarg-ptr is also removed. 953 ======================================= ========================================================== 954 955.. _amdgpu-elf-code-object: 956 957ELF Code Object 958=============== 959 960The AMDGPU backend generates a standard ELF [ELF]_ relocatable code object that 961can be linked by ``lld`` to produce a standard ELF shared code object which can 962be loaded and executed on an AMDGPU target. 963 964.. _amdgpu-elf-header: 965 966Header 967------ 968 969The AMDGPU backend uses the following ELF header: 970 971 .. table:: AMDGPU ELF Header 972 :name: amdgpu-elf-header-table 973 974 ========================== =============================== 975 Field Value 976 ========================== =============================== 977 ``e_ident[EI_CLASS]`` ``ELFCLASS64`` 978 ``e_ident[EI_DATA]`` ``ELFDATA2LSB`` 979 ``e_ident[EI_OSABI]`` - ``ELFOSABI_NONE`` 980 - ``ELFOSABI_AMDGPU_HSA`` 981 - ``ELFOSABI_AMDGPU_PAL`` 982 - ``ELFOSABI_AMDGPU_MESA3D`` 983 ``e_ident[EI_ABIVERSION]`` - ``ELFABIVERSION_AMDGPU_HSA_V2`` 984 - ``ELFABIVERSION_AMDGPU_HSA_V3`` 985 - ``ELFABIVERSION_AMDGPU_HSA_V4`` 986 - ``ELFABIVERSION_AMDGPU_HSA_V5`` 987 - ``ELFABIVERSION_AMDGPU_PAL`` 988 - ``ELFABIVERSION_AMDGPU_MESA3D`` 989 ``e_type`` - ``ET_REL`` 990 - ``ET_DYN`` 991 ``e_machine`` ``EM_AMDGPU`` 992 ``e_entry`` 0 993 ``e_flags`` See :ref:`amdgpu-elf-header-e_flags-v2-table`, 994 :ref:`amdgpu-elf-header-e_flags-table-v3`, 995 and :ref:`amdgpu-elf-header-e_flags-table-v4-onwards` 996 ========================== =============================== 997 998.. 999 1000 .. table:: AMDGPU ELF Header Enumeration Values 1001 :name: amdgpu-elf-header-enumeration-values-table 1002 1003 =============================== ===== 1004 Name Value 1005 =============================== ===== 1006 ``EM_AMDGPU`` 224 1007 ``ELFOSABI_NONE`` 0 1008 ``ELFOSABI_AMDGPU_HSA`` 64 1009 ``ELFOSABI_AMDGPU_PAL`` 65 1010 ``ELFOSABI_AMDGPU_MESA3D`` 66 1011 ``ELFABIVERSION_AMDGPU_HSA_V2`` 0 1012 ``ELFABIVERSION_AMDGPU_HSA_V3`` 1 1013 ``ELFABIVERSION_AMDGPU_HSA_V4`` 2 1014 ``ELFABIVERSION_AMDGPU_HSA_V5`` 3 1015 ``ELFABIVERSION_AMDGPU_PAL`` 0 1016 ``ELFABIVERSION_AMDGPU_MESA3D`` 0 1017 =============================== ===== 1018 1019``e_ident[EI_CLASS]`` 1020 The ELF class is: 1021 1022 * ``ELFCLASS32`` for ``r600`` architecture. 1023 1024 * ``ELFCLASS64`` for ``amdgcn`` architecture which only supports 64-bit 1025 process address space applications. 1026 1027``e_ident[EI_DATA]`` 1028 All AMDGPU targets use ``ELFDATA2LSB`` for little-endian byte ordering. 1029 1030``e_ident[EI_OSABI]`` 1031 One of the following AMDGPU target architecture specific OS ABIs 1032 (see :ref:`amdgpu-os`): 1033 1034 * ``ELFOSABI_NONE`` for *unknown* OS. 1035 1036 * ``ELFOSABI_AMDGPU_HSA`` for ``amdhsa`` OS. 1037 1038 * ``ELFOSABI_AMDGPU_PAL`` for ``amdpal`` OS. 1039 1040 * ``ELFOSABI_AMDGPU_MESA3D`` for ``mesa3D`` OS. 1041 1042``e_ident[EI_ABIVERSION]`` 1043 The ABI version of the AMDGPU target architecture specific OS ABI to which the code 1044 object conforms: 1045 1046 * ``ELFABIVERSION_AMDGPU_HSA_V2`` is used to specify the version of AMD HSA 1047 runtime ABI for code object V2. Specify using the Clang option 1048 ``-mcode-object-version=2``. 1049 1050 * ``ELFABIVERSION_AMDGPU_HSA_V3`` is used to specify the version of AMD HSA 1051 runtime ABI for code object V3. Specify using the Clang option 1052 ``-mcode-object-version=3``. 1053 1054 * ``ELFABIVERSION_AMDGPU_HSA_V4`` is used to specify the version of AMD HSA 1055 runtime ABI for code object V4. Specify using the Clang option 1056 ``-mcode-object-version=4``. This is the default code object 1057 version if not specified. 1058 1059 * ``ELFABIVERSION_AMDGPU_HSA_V5`` is used to specify the version of AMD HSA 1060 runtime ABI for code object V5. Specify using the Clang option 1061 ``-mcode-object-version=5``. 1062 1063 * ``ELFABIVERSION_AMDGPU_PAL`` is used to specify the version of AMD PAL 1064 runtime ABI. 1065 1066 * ``ELFABIVERSION_AMDGPU_MESA3D`` is used to specify the version of AMD MESA 1067 3D runtime ABI. 1068 1069``e_type`` 1070 Can be one of the following values: 1071 1072 1073 ``ET_REL`` 1074 The type produced by the AMDGPU backend compiler as it is relocatable code 1075 object. 1076 1077 ``ET_DYN`` 1078 The type produced by the linker as it is a shared code object. 1079 1080 The AMD HSA runtime loader requires a ``ET_DYN`` code object. 1081 1082``e_machine`` 1083 The value ``EM_AMDGPU`` is used for the machine for all processors supported 1084 by the ``r600`` and ``amdgcn`` architectures (see 1085 :ref:`amdgpu-processor-table`). The specific processor is specified in the 1086 ``NT_AMD_HSA_ISA_VERSION`` note record for code object V2 (see 1087 :ref:`amdgpu-note-records-v2`) and in the ``EF_AMDGPU_MACH`` bit field of the 1088 ``e_flags`` for code object V3 and above (see 1089 :ref:`amdgpu-elf-header-e_flags-table-v3` and 1090 :ref:`amdgpu-elf-header-e_flags-table-v4-onwards`). 1091 1092``e_entry`` 1093 The entry point is 0 as the entry points for individual kernels must be 1094 selected in order to invoke them through AQL packets. 1095 1096``e_flags`` 1097 The AMDGPU backend uses the following ELF header flags: 1098 1099 .. table:: AMDGPU ELF Header ``e_flags`` for Code Object V2 1100 :name: amdgpu-elf-header-e_flags-v2-table 1101 1102 ===================================== ===== ============================= 1103 Name Value Description 1104 ===================================== ===== ============================= 1105 ``EF_AMDGPU_FEATURE_XNACK_V2`` 0x01 Indicates if the ``xnack`` 1106 target feature is 1107 enabled for all code 1108 contained in the code object. 1109 If the processor 1110 does not support the 1111 ``xnack`` target 1112 feature then must 1113 be 0. 1114 See 1115 :ref:`amdgpu-target-features`. 1116 ``EF_AMDGPU_FEATURE_TRAP_HANDLER_V2`` 0x02 Indicates if the trap 1117 handler is enabled for all 1118 code contained in the code 1119 object. If the processor 1120 does not support a trap 1121 handler then must be 0. 1122 See 1123 :ref:`amdgpu-target-features`. 1124 ===================================== ===== ============================= 1125 1126 .. table:: AMDGPU ELF Header ``e_flags`` for Code Object V3 1127 :name: amdgpu-elf-header-e_flags-table-v3 1128 1129 ================================= ===== ============================= 1130 Name Value Description 1131 ================================= ===== ============================= 1132 ``EF_AMDGPU_MACH`` 0x0ff AMDGPU processor selection 1133 mask for 1134 ``EF_AMDGPU_MACH_xxx`` values 1135 defined in 1136 :ref:`amdgpu-ef-amdgpu-mach-table`. 1137 ``EF_AMDGPU_FEATURE_XNACK_V3`` 0x100 Indicates if the ``xnack`` 1138 target feature is 1139 enabled for all code 1140 contained in the code object. 1141 If the processor 1142 does not support the 1143 ``xnack`` target 1144 feature then must 1145 be 0. 1146 See 1147 :ref:`amdgpu-target-features`. 1148 ``EF_AMDGPU_FEATURE_SRAMECC_V3`` 0x200 Indicates if the ``sramecc`` 1149 target feature is 1150 enabled for all code 1151 contained in the code object. 1152 If the processor 1153 does not support the 1154 ``sramecc`` target 1155 feature then must 1156 be 0. 1157 See 1158 :ref:`amdgpu-target-features`. 1159 ================================= ===== ============================= 1160 1161 .. table:: AMDGPU ELF Header ``e_flags`` for Code Object V4 and After 1162 :name: amdgpu-elf-header-e_flags-table-v4-onwards 1163 1164 ============================================ ===== =================================== 1165 Name Value Description 1166 ============================================ ===== =================================== 1167 ``EF_AMDGPU_MACH`` 0x0ff AMDGPU processor selection 1168 mask for 1169 ``EF_AMDGPU_MACH_xxx`` values 1170 defined in 1171 :ref:`amdgpu-ef-amdgpu-mach-table`. 1172 ``EF_AMDGPU_FEATURE_XNACK_V4`` 0x300 XNACK selection mask for 1173 ``EF_AMDGPU_FEATURE_XNACK_*_V4`` 1174 values. 1175 ``EF_AMDGPU_FEATURE_XNACK_UNSUPPORTED_V4`` 0x000 XNACK unsuppored. 1176 ``EF_AMDGPU_FEATURE_XNACK_ANY_V4`` 0x100 XNACK can have any value. 1177 ``EF_AMDGPU_FEATURE_XNACK_OFF_V4`` 0x200 XNACK disabled. 1178 ``EF_AMDGPU_FEATURE_XNACK_ON_V4`` 0x300 XNACK enabled. 1179 ``EF_AMDGPU_FEATURE_SRAMECC_V4`` 0xc00 SRAMECC selection mask for 1180 ``EF_AMDGPU_FEATURE_SRAMECC_*_V4`` 1181 values. 1182 ``EF_AMDGPU_FEATURE_SRAMECC_UNSUPPORTED_V4`` 0x000 SRAMECC unsuppored. 1183 ``EF_AMDGPU_FEATURE_SRAMECC_ANY_V4`` 0x400 SRAMECC can have any value. 1184 ``EF_AMDGPU_FEATURE_SRAMECC_OFF_V4`` 0x800 SRAMECC disabled, 1185 ``EF_AMDGPU_FEATURE_SRAMECC_ON_V4`` 0xc00 SRAMECC enabled. 1186 ============================================ ===== =================================== 1187 1188 .. table:: AMDGPU ``EF_AMDGPU_MACH`` Values 1189 :name: amdgpu-ef-amdgpu-mach-table 1190 1191 ==================================== ========== ============================= 1192 Name Value Description (see 1193 :ref:`amdgpu-processor-table`) 1194 ==================================== ========== ============================= 1195 ``EF_AMDGPU_MACH_NONE`` 0x000 *not specified* 1196 ``EF_AMDGPU_MACH_R600_R600`` 0x001 ``r600`` 1197 ``EF_AMDGPU_MACH_R600_R630`` 0x002 ``r630`` 1198 ``EF_AMDGPU_MACH_R600_RS880`` 0x003 ``rs880`` 1199 ``EF_AMDGPU_MACH_R600_RV670`` 0x004 ``rv670`` 1200 ``EF_AMDGPU_MACH_R600_RV710`` 0x005 ``rv710`` 1201 ``EF_AMDGPU_MACH_R600_RV730`` 0x006 ``rv730`` 1202 ``EF_AMDGPU_MACH_R600_RV770`` 0x007 ``rv770`` 1203 ``EF_AMDGPU_MACH_R600_CEDAR`` 0x008 ``cedar`` 1204 ``EF_AMDGPU_MACH_R600_CYPRESS`` 0x009 ``cypress`` 1205 ``EF_AMDGPU_MACH_R600_JUNIPER`` 0x00a ``juniper`` 1206 ``EF_AMDGPU_MACH_R600_REDWOOD`` 0x00b ``redwood`` 1207 ``EF_AMDGPU_MACH_R600_SUMO`` 0x00c ``sumo`` 1208 ``EF_AMDGPU_MACH_R600_BARTS`` 0x00d ``barts`` 1209 ``EF_AMDGPU_MACH_R600_CAICOS`` 0x00e ``caicos`` 1210 ``EF_AMDGPU_MACH_R600_CAYMAN`` 0x00f ``cayman`` 1211 ``EF_AMDGPU_MACH_R600_TURKS`` 0x010 ``turks`` 1212 *reserved* 0x011 - Reserved for ``r600`` 1213 0x01f architecture processors. 1214 ``EF_AMDGPU_MACH_AMDGCN_GFX600`` 0x020 ``gfx600`` 1215 ``EF_AMDGPU_MACH_AMDGCN_GFX601`` 0x021 ``gfx601`` 1216 ``EF_AMDGPU_MACH_AMDGCN_GFX700`` 0x022 ``gfx700`` 1217 ``EF_AMDGPU_MACH_AMDGCN_GFX701`` 0x023 ``gfx701`` 1218 ``EF_AMDGPU_MACH_AMDGCN_GFX702`` 0x024 ``gfx702`` 1219 ``EF_AMDGPU_MACH_AMDGCN_GFX703`` 0x025 ``gfx703`` 1220 ``EF_AMDGPU_MACH_AMDGCN_GFX704`` 0x026 ``gfx704`` 1221 *reserved* 0x027 Reserved. 1222 ``EF_AMDGPU_MACH_AMDGCN_GFX801`` 0x028 ``gfx801`` 1223 ``EF_AMDGPU_MACH_AMDGCN_GFX802`` 0x029 ``gfx802`` 1224 ``EF_AMDGPU_MACH_AMDGCN_GFX803`` 0x02a ``gfx803`` 1225 ``EF_AMDGPU_MACH_AMDGCN_GFX810`` 0x02b ``gfx810`` 1226 ``EF_AMDGPU_MACH_AMDGCN_GFX900`` 0x02c ``gfx900`` 1227 ``EF_AMDGPU_MACH_AMDGCN_GFX902`` 0x02d ``gfx902`` 1228 ``EF_AMDGPU_MACH_AMDGCN_GFX904`` 0x02e ``gfx904`` 1229 ``EF_AMDGPU_MACH_AMDGCN_GFX906`` 0x02f ``gfx906`` 1230 ``EF_AMDGPU_MACH_AMDGCN_GFX908`` 0x030 ``gfx908`` 1231 ``EF_AMDGPU_MACH_AMDGCN_GFX909`` 0x031 ``gfx909`` 1232 ``EF_AMDGPU_MACH_AMDGCN_GFX90C`` 0x032 ``gfx90c`` 1233 ``EF_AMDGPU_MACH_AMDGCN_GFX1010`` 0x033 ``gfx1010`` 1234 ``EF_AMDGPU_MACH_AMDGCN_GFX1011`` 0x034 ``gfx1011`` 1235 ``EF_AMDGPU_MACH_AMDGCN_GFX1012`` 0x035 ``gfx1012`` 1236 ``EF_AMDGPU_MACH_AMDGCN_GFX1030`` 0x036 ``gfx1030`` 1237 ``EF_AMDGPU_MACH_AMDGCN_GFX1031`` 0x037 ``gfx1031`` 1238 ``EF_AMDGPU_MACH_AMDGCN_GFX1032`` 0x038 ``gfx1032`` 1239 ``EF_AMDGPU_MACH_AMDGCN_GFX1033`` 0x039 ``gfx1033`` 1240 ``EF_AMDGPU_MACH_AMDGCN_GFX602`` 0x03a ``gfx602`` 1241 ``EF_AMDGPU_MACH_AMDGCN_GFX705`` 0x03b ``gfx705`` 1242 ``EF_AMDGPU_MACH_AMDGCN_GFX805`` 0x03c ``gfx805`` 1243 ``EF_AMDGPU_MACH_AMDGCN_GFX1035`` 0x03d ``gfx1035`` 1244 ``EF_AMDGPU_MACH_AMDGCN_GFX1034`` 0x03e ``gfx1034`` 1245 ``EF_AMDGPU_MACH_AMDGCN_GFX90A`` 0x03f ``gfx90a`` 1246 ``EF_AMDGPU_MACH_AMDGCN_GFX940`` 0x040 ``gfx940`` 1247 *reserved* 0x041 Reserved. 1248 ``EF_AMDGPU_MACH_AMDGCN_GFX1013`` 0x042 ``gfx1013`` 1249 *reserved* 0x043 Reserved. 1250 *reserved* 0x044 Reserved. 1251 ``EF_AMDGPU_MACH_AMDGCN_GFX1036`` 0x045 ``gfx1036`` 1252 ==================================== ========== ============================= 1253 1254Sections 1255-------- 1256 1257An AMDGPU target ELF code object has the standard ELF sections which include: 1258 1259 .. table:: AMDGPU ELF Sections 1260 :name: amdgpu-elf-sections-table 1261 1262 ================== ================ ================================= 1263 Name Type Attributes 1264 ================== ================ ================================= 1265 ``.bss`` ``SHT_NOBITS`` ``SHF_ALLOC`` + ``SHF_WRITE`` 1266 ``.data`` ``SHT_PROGBITS`` ``SHF_ALLOC`` + ``SHF_WRITE`` 1267 ``.debug_``\ *\** ``SHT_PROGBITS`` *none* 1268 ``.dynamic`` ``SHT_DYNAMIC`` ``SHF_ALLOC`` 1269 ``.dynstr`` ``SHT_PROGBITS`` ``SHF_ALLOC`` 1270 ``.dynsym`` ``SHT_PROGBITS`` ``SHF_ALLOC`` 1271 ``.got`` ``SHT_PROGBITS`` ``SHF_ALLOC`` + ``SHF_WRITE`` 1272 ``.hash`` ``SHT_HASH`` ``SHF_ALLOC`` 1273 ``.note`` ``SHT_NOTE`` *none* 1274 ``.rela``\ *name* ``SHT_RELA`` *none* 1275 ``.rela.dyn`` ``SHT_RELA`` *none* 1276 ``.rodata`` ``SHT_PROGBITS`` ``SHF_ALLOC`` 1277 ``.shstrtab`` ``SHT_STRTAB`` *none* 1278 ``.strtab`` ``SHT_STRTAB`` *none* 1279 ``.symtab`` ``SHT_SYMTAB`` *none* 1280 ``.text`` ``SHT_PROGBITS`` ``SHF_ALLOC`` + ``SHF_EXECINSTR`` 1281 ================== ================ ================================= 1282 1283These sections have their standard meanings (see [ELF]_) and are only generated 1284if needed. 1285 1286``.debug``\ *\** 1287 The standard DWARF sections. See :ref:`amdgpu-dwarf-debug-information` for 1288 information on the DWARF produced by the AMDGPU backend. 1289 1290``.dynamic``, ``.dynstr``, ``.dynsym``, ``.hash`` 1291 The standard sections used by a dynamic loader. 1292 1293``.note`` 1294 See :ref:`amdgpu-note-records` for the note records supported by the AMDGPU 1295 backend. 1296 1297``.rela``\ *name*, ``.rela.dyn`` 1298 For relocatable code objects, *name* is the name of the section that the 1299 relocation records apply. For example, ``.rela.text`` is the section name for 1300 relocation records associated with the ``.text`` section. 1301 1302 For linked shared code objects, ``.rela.dyn`` contains all the relocation 1303 records from each of the relocatable code object's ``.rela``\ *name* sections. 1304 1305 See :ref:`amdgpu-relocation-records` for the relocation records supported by 1306 the AMDGPU backend. 1307 1308``.text`` 1309 The executable machine code for the kernels and functions they call. Generated 1310 as position independent code. See :ref:`amdgpu-code-conventions` for 1311 information on conventions used in the isa generation. 1312 1313.. _amdgpu-note-records: 1314 1315Note Records 1316------------ 1317 1318The AMDGPU backend code object contains ELF note records in the ``.note`` 1319section. The set of generated notes and their semantics depend on the code 1320object version; see :ref:`amdgpu-note-records-v2` and 1321:ref:`amdgpu-note-records-v3-onwards`. 1322 1323As required by ``ELFCLASS32`` and ``ELFCLASS64``, minimal zero-byte padding 1324must be generated after the ``name`` field to ensure the ``desc`` field is 4 1325byte aligned. In addition, minimal zero-byte padding must be generated to 1326ensure the ``desc`` field size is a multiple of 4 bytes. The ``sh_addralign`` 1327field of the ``.note`` section must be at least 4 to indicate at least 8 byte 1328alignment. 1329 1330.. _amdgpu-note-records-v2: 1331 1332Code Object V2 Note Records 1333~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1334 1335.. warning:: 1336 Code object V2 is not the default code object version emitted by 1337 this version of LLVM. 1338 1339The AMDGPU backend code object uses the following ELF note record in the 1340``.note`` section when compiling for code object V2. 1341 1342The note record vendor field is "AMD". 1343 1344Additional note records may be present, but any which are not documented here 1345are deprecated and should not be used. 1346 1347 .. table:: AMDGPU Code Object V2 ELF Note Records 1348 :name: amdgpu-elf-note-records-v2-table 1349 1350 ===== ===================================== ====================================== 1351 Name Type Description 1352 ===== ===================================== ====================================== 1353 "AMD" ``NT_AMD_HSA_CODE_OBJECT_VERSION`` Code object version. 1354 "AMD" ``NT_AMD_HSA_HSAIL`` HSAIL properties generated by the HSAIL 1355 Finalizer and not the LLVM compiler. 1356 "AMD" ``NT_AMD_HSA_ISA_VERSION`` Target ISA version. 1357 "AMD" ``NT_AMD_HSA_METADATA`` Metadata null terminated string in 1358 YAML [YAML]_ textual format. 1359 "AMD" ``NT_AMD_HSA_ISA_NAME`` Target ISA name. 1360 ===== ===================================== ====================================== 1361 1362.. 1363 1364 .. table:: AMDGPU Code Object V2 ELF Note Record Enumeration Values 1365 :name: amdgpu-elf-note-record-enumeration-values-v2-table 1366 1367 ===================================== ===== 1368 Name Value 1369 ===================================== ===== 1370 ``NT_AMD_HSA_CODE_OBJECT_VERSION`` 1 1371 ``NT_AMD_HSA_HSAIL`` 2 1372 ``NT_AMD_HSA_ISA_VERSION`` 3 1373 *reserved* 4-9 1374 ``NT_AMD_HSA_METADATA`` 10 1375 ``NT_AMD_HSA_ISA_NAME`` 11 1376 ===================================== ===== 1377 1378``NT_AMD_HSA_CODE_OBJECT_VERSION`` 1379 Specifies the code object version number. The description field has the 1380 following layout: 1381 1382 .. code:: c 1383 1384 struct amdgpu_hsa_note_code_object_version_s { 1385 uint32_t major_version; 1386 uint32_t minor_version; 1387 }; 1388 1389 The ``major_version`` has a value less than or equal to 2. 1390 1391``NT_AMD_HSA_HSAIL`` 1392 Specifies the HSAIL properties used by the HSAIL Finalizer. The description 1393 field has the following layout: 1394 1395 .. code:: c 1396 1397 struct amdgpu_hsa_note_hsail_s { 1398 uint32_t hsail_major_version; 1399 uint32_t hsail_minor_version; 1400 uint8_t profile; 1401 uint8_t machine_model; 1402 uint8_t default_float_round; 1403 }; 1404 1405``NT_AMD_HSA_ISA_VERSION`` 1406 Specifies the target ISA version. The description field has the following layout: 1407 1408 .. code:: c 1409 1410 struct amdgpu_hsa_note_isa_s { 1411 uint16_t vendor_name_size; 1412 uint16_t architecture_name_size; 1413 uint32_t major; 1414 uint32_t minor; 1415 uint32_t stepping; 1416 char vendor_and_architecture_name[1]; 1417 }; 1418 1419 ``vendor_name_size`` and ``architecture_name_size`` are the length of the 1420 vendor and architecture names respectively, including the NUL character. 1421 1422 ``vendor_and_architecture_name`` contains the NUL terminates string for the 1423 vendor, immediately followed by the NUL terminated string for the 1424 architecture. 1425 1426 This note record is used by the HSA runtime loader. 1427 1428 Code object V2 only supports a limited number of processors and has fixed 1429 settings for target features. See 1430 :ref:`amdgpu-elf-note-record-supported_processors-v2-table` for a list of 1431 processors and the corresponding target ID. In the table the note record ISA 1432 name is a concatenation of the vendor name, architecture name, major, minor, 1433 and stepping separated by a ":". 1434 1435 The target ID column shows the processor name and fixed target features used 1436 by the LLVM compiler. The LLVM compiler does not generate a 1437 ``NT_AMD_HSA_HSAIL`` note record. 1438 1439 A code object generated by the Finalizer also uses code object V2 and always 1440 generates a ``NT_AMD_HSA_HSAIL`` note record. The processor name and 1441 ``sramecc`` target feature is as shown in 1442 :ref:`amdgpu-elf-note-record-supported_processors-v2-table` but the ``xnack`` 1443 target feature is specified by the ``EF_AMDGPU_FEATURE_XNACK_V2`` ``e_flags`` 1444 bit. 1445 1446``NT_AMD_HSA_ISA_NAME`` 1447 Specifies the target ISA name as a non-NUL terminated string. 1448 1449 This note record is not used by the HSA runtime loader. 1450 1451 See the ``NT_AMD_HSA_ISA_VERSION`` note record description of the code object 1452 V2's limited support of processors and fixed settings for target features. 1453 1454 See :ref:`amdgpu-elf-note-record-supported_processors-v2-table` for a mapping 1455 from the string to the corresponding target ID. If the ``xnack`` target 1456 feature is supported and enabled, the string produced by the LLVM compiler 1457 will may have a ``+xnack`` appended. The Finlizer did not do the appending and 1458 instead used the ``EF_AMDGPU_FEATURE_XNACK_V2`` ``e_flags`` bit. 1459 1460``NT_AMD_HSA_METADATA`` 1461 Specifies extensible metadata associated with the code objects executed on HSA 1462 [HSA]_ compatible runtimes (see :ref:`amdgpu-os`). It is required when the 1463 target triple OS is ``amdhsa`` (see :ref:`amdgpu-target-triples`). See 1464 :ref:`amdgpu-amdhsa-code-object-metadata-v2` for the syntax of the code object 1465 metadata string. 1466 1467 .. table:: AMDGPU Code Object V2 Supported Processors and Fixed Target Feature Settings 1468 :name: amdgpu-elf-note-record-supported_processors-v2-table 1469 1470 ===================== ========================== 1471 Note Record ISA Name Target ID 1472 ===================== ========================== 1473 ``AMD:AMDGPU:6:0:0`` ``gfx600`` 1474 ``AMD:AMDGPU:6:0:1`` ``gfx601`` 1475 ``AMD:AMDGPU:6:0:2`` ``gfx602`` 1476 ``AMD:AMDGPU:7:0:0`` ``gfx700`` 1477 ``AMD:AMDGPU:7:0:1`` ``gfx701`` 1478 ``AMD:AMDGPU:7:0:2`` ``gfx702`` 1479 ``AMD:AMDGPU:7:0:3`` ``gfx703`` 1480 ``AMD:AMDGPU:7:0:4`` ``gfx704`` 1481 ``AMD:AMDGPU:7:0:5`` ``gfx705`` 1482 ``AMD:AMDGPU:8:0:0`` ``gfx802`` 1483 ``AMD:AMDGPU:8:0:1`` ``gfx801:xnack+`` 1484 ``AMD:AMDGPU:8:0:2`` ``gfx802`` 1485 ``AMD:AMDGPU:8:0:3`` ``gfx803`` 1486 ``AMD:AMDGPU:8:0:4`` ``gfx803`` 1487 ``AMD:AMDGPU:8:0:5`` ``gfx805`` 1488 ``AMD:AMDGPU:8:1:0`` ``gfx810:xnack+`` 1489 ``AMD:AMDGPU:9:0:0`` ``gfx900:xnack-`` 1490 ``AMD:AMDGPU:9:0:1`` ``gfx900:xnack+`` 1491 ``AMD:AMDGPU:9:0:2`` ``gfx902:xnack-`` 1492 ``AMD:AMDGPU:9:0:3`` ``gfx902:xnack+`` 1493 ``AMD:AMDGPU:9:0:4`` ``gfx904:xnack-`` 1494 ``AMD:AMDGPU:9:0:5`` ``gfx904:xnack+`` 1495 ``AMD:AMDGPU:9:0:6`` ``gfx906:sramecc-:xnack-`` 1496 ``AMD:AMDGPU:9:0:7`` ``gfx906:sramecc-:xnack+`` 1497 ``AMD:AMDGPU:9:0:12`` ``gfx90c:xnack-`` 1498 ===================== ========================== 1499 1500.. _amdgpu-note-records-v3-onwards: 1501 1502Code Object V3 and Above Note Records 1503~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1504 1505The AMDGPU backend code object uses the following ELF note record in the 1506``.note`` section when compiling for code object V3 and above. 1507 1508The note record vendor field is "AMDGPU". 1509 1510Additional note records may be present, but any which are not documented here 1511are deprecated and should not be used. 1512 1513 .. table:: AMDGPU Code Object V3 and Above ELF Note Records 1514 :name: amdgpu-elf-note-records-table-v3-onwards 1515 1516 ======== ============================== ====================================== 1517 Name Type Description 1518 ======== ============================== ====================================== 1519 "AMDGPU" ``NT_AMDGPU_METADATA`` Metadata in Message Pack [MsgPack]_ 1520 binary format. 1521 ======== ============================== ====================================== 1522 1523.. 1524 1525 .. table:: AMDGPU Code Object V3 and Above ELF Note Record Enumeration Values 1526 :name: amdgpu-elf-note-record-enumeration-values-table-v3-onwards 1527 1528 ============================== ===== 1529 Name Value 1530 ============================== ===== 1531 *reserved* 0-31 1532 ``NT_AMDGPU_METADATA`` 32 1533 ============================== ===== 1534 1535``NT_AMDGPU_METADATA`` 1536 Specifies extensible metadata associated with an AMDGPU code object. It is 1537 encoded as a map in the Message Pack [MsgPack]_ binary data format. See 1538 :ref:`amdgpu-amdhsa-code-object-metadata-v3`, 1539 :ref:`amdgpu-amdhsa-code-object-metadata-v4` and 1540 :ref:`amdgpu-amdhsa-code-object-metadata-v5` for the map keys defined for the 1541 ``amdhsa`` OS. 1542 1543.. _amdgpu-symbols: 1544 1545Symbols 1546------- 1547 1548Symbols include the following: 1549 1550 .. table:: AMDGPU ELF Symbols 1551 :name: amdgpu-elf-symbols-table 1552 1553 ===================== ================== ================ ================== 1554 Name Type Section Description 1555 ===================== ================== ================ ================== 1556 *link-name* ``STT_OBJECT`` - ``.data`` Global variable 1557 - ``.rodata`` 1558 - ``.bss`` 1559 *link-name*\ ``.kd`` ``STT_OBJECT`` - ``.rodata`` Kernel descriptor 1560 *link-name* ``STT_FUNC`` - ``.text`` Kernel entry point 1561 *link-name* ``STT_OBJECT`` - SHN_AMDGPU_LDS Global variable in LDS 1562 ===================== ================== ================ ================== 1563 1564Global variable 1565 Global variables both used and defined by the compilation unit. 1566 1567 If the symbol is defined in the compilation unit then it is allocated in the 1568 appropriate section according to if it has initialized data or is readonly. 1569 1570 If the symbol is external then its section is ``STN_UNDEF`` and the loader 1571 will resolve relocations using the definition provided by another code object 1572 or explicitly defined by the runtime. 1573 1574 If the symbol resides in local/group memory (LDS) then its section is the 1575 special processor specific section name ``SHN_AMDGPU_LDS``, and the 1576 ``st_value`` field describes alignment requirements as it does for common 1577 symbols. 1578 1579 .. TODO:: 1580 1581 Add description of linked shared object symbols. Seems undefined symbols 1582 are marked as STT_NOTYPE. 1583 1584Kernel descriptor 1585 Every HSA kernel has an associated kernel descriptor. It is the address of the 1586 kernel descriptor that is used in the AQL dispatch packet used to invoke the 1587 kernel, not the kernel entry point. The layout of the HSA kernel descriptor is 1588 defined in :ref:`amdgpu-amdhsa-kernel-descriptor`. 1589 1590Kernel entry point 1591 Every HSA kernel also has a symbol for its machine code entry point. 1592 1593.. _amdgpu-relocation-records: 1594 1595Relocation Records 1596------------------ 1597 1598AMDGPU backend generates ``Elf64_Rela`` relocation records. Supported 1599relocatable fields are: 1600 1601``word32`` 1602 This specifies a 32-bit field occupying 4 bytes with arbitrary byte 1603 alignment. These values use the same byte order as other word values in the 1604 AMDGPU architecture. 1605 1606``word64`` 1607 This specifies a 64-bit field occupying 8 bytes with arbitrary byte 1608 alignment. These values use the same byte order as other word values in the 1609 AMDGPU architecture. 1610 1611Following notations are used for specifying relocation calculations: 1612 1613**A** 1614 Represents the addend used to compute the value of the relocatable field. 1615 1616**G** 1617 Represents the offset into the global offset table at which the relocation 1618 entry's symbol will reside during execution. 1619 1620**GOT** 1621 Represents the address of the global offset table. 1622 1623**P** 1624 Represents the place (section offset for ``et_rel`` or address for ``et_dyn``) 1625 of the storage unit being relocated (computed using ``r_offset``). 1626 1627**S** 1628 Represents the value of the symbol whose index resides in the relocation 1629 entry. Relocations not using this must specify a symbol index of 1630 ``STN_UNDEF``. 1631 1632**B** 1633 Represents the base address of a loaded executable or shared object which is 1634 the difference between the ELF address and the actual load address. 1635 Relocations using this are only valid in executable or shared objects. 1636 1637The following relocation types are supported: 1638 1639 .. table:: AMDGPU ELF Relocation Records 1640 :name: amdgpu-elf-relocation-records-table 1641 1642 ========================== ======= ===== ========== ============================== 1643 Relocation Type Kind Value Field Calculation 1644 ========================== ======= ===== ========== ============================== 1645 ``R_AMDGPU_NONE`` 0 *none* *none* 1646 ``R_AMDGPU_ABS32_LO`` Static, 1 ``word32`` (S + A) & 0xFFFFFFFF 1647 Dynamic 1648 ``R_AMDGPU_ABS32_HI`` Static, 2 ``word32`` (S + A) >> 32 1649 Dynamic 1650 ``R_AMDGPU_ABS64`` Static, 3 ``word64`` S + A 1651 Dynamic 1652 ``R_AMDGPU_REL32`` Static 4 ``word32`` S + A - P 1653 ``R_AMDGPU_REL64`` Static 5 ``word64`` S + A - P 1654 ``R_AMDGPU_ABS32`` Static, 6 ``word32`` S + A 1655 Dynamic 1656 ``R_AMDGPU_GOTPCREL`` Static 7 ``word32`` G + GOT + A - P 1657 ``R_AMDGPU_GOTPCREL32_LO`` Static 8 ``word32`` (G + GOT + A - P) & 0xFFFFFFFF 1658 ``R_AMDGPU_GOTPCREL32_HI`` Static 9 ``word32`` (G + GOT + A - P) >> 32 1659 ``R_AMDGPU_REL32_LO`` Static 10 ``word32`` (S + A - P) & 0xFFFFFFFF 1660 ``R_AMDGPU_REL32_HI`` Static 11 ``word32`` (S + A - P) >> 32 1661 *reserved* 12 1662 ``R_AMDGPU_RELATIVE64`` Dynamic 13 ``word64`` B + A 1663 ``R_AMDGPU_REL16`` Static 14 ``word16`` ((S + A - P) - 4) / 4 1664 ========================== ======= ===== ========== ============================== 1665 1666``R_AMDGPU_ABS32_LO`` and ``R_AMDGPU_ABS32_HI`` are only supported by 1667the ``mesa3d`` OS, which does not support ``R_AMDGPU_ABS64``. 1668 1669There is no current OS loader support for 32-bit programs and so 1670``R_AMDGPU_ABS32`` is not used. 1671 1672.. _amdgpu-loaded-code-object-path-uniform-resource-identifier: 1673 1674Loaded Code Object Path Uniform Resource Identifier (URI) 1675--------------------------------------------------------- 1676 1677The AMD GPU code object loader represents the path of the ELF shared object from 1678which the code object was loaded as a textual Uniform Resource Identifier (URI). 1679Note that the code object is the in memory loaded relocated form of the ELF 1680shared object. Multiple code objects may be loaded at different memory 1681addresses in the same process from the same ELF shared object. 1682 1683The loaded code object path URI syntax is defined by the following BNF syntax: 1684 1685.. code:: 1686 1687 code_object_uri ::== file_uri | memory_uri 1688 file_uri ::== "file://" file_path [ range_specifier ] 1689 memory_uri ::== "memory://" process_id range_specifier 1690 range_specifier ::== [ "#" | "?" ] "offset=" number "&" "size=" number 1691 file_path ::== URI_ENCODED_OS_FILE_PATH 1692 process_id ::== DECIMAL_NUMBER 1693 number ::== HEX_NUMBER | DECIMAL_NUMBER | OCTAL_NUMBER 1694 1695**number** 1696 Is a C integral literal where hexadecimal values are prefixed by "0x" or "0X", 1697 and octal values by "0". 1698 1699**file_path** 1700 Is the file's path specified as a URI encoded UTF-8 string. In URI encoding, 1701 every character that is not in the regular expression ``[a-zA-Z0-9/_.~-]`` is 1702 encoded as two uppercase hexadecimal digits proceeded by "%". Directories in 1703 the path are separated by "/". 1704 1705**offset** 1706 Is a 0-based byte offset to the start of the code object. For a file URI, it 1707 is from the start of the file specified by the ``file_path``, and if omitted 1708 defaults to 0. For a memory URI, it is the memory address and is required. 1709 1710**size** 1711 Is the number of bytes in the code object. For a file URI, if omitted it 1712 defaults to the size of the file. It is required for a memory URI. 1713 1714**process_id** 1715 Is the identity of the process owning the memory. For Linux it is the C 1716 unsigned integral decimal literal for the process ID (PID). 1717 1718For example: 1719 1720.. code:: 1721 1722 file:///dir1/dir2/file1 1723 file:///dir3/dir4/file2#offset=0x2000&size=3000 1724 memory://1234#offset=0x20000&size=3000 1725 1726.. _amdgpu-dwarf-debug-information: 1727 1728DWARF Debug Information 1729======================= 1730 1731.. warning:: 1732 1733 This section describes **provisional support** for AMDGPU DWARF [DWARF]_ that 1734 is not currently fully implemented and is subject to change. 1735 1736AMDGPU generates DWARF [DWARF]_ debugging information ELF sections (see 1737:ref:`amdgpu-elf-code-object`) which contain information that maps the code 1738object executable code and data to the source language constructs. It can be 1739used by tools such as debuggers and profilers. It uses features defined in 1740:doc:`AMDGPUDwarfExtensionsForHeterogeneousDebugging` that are made available in 1741DWARF Version 4 and DWARF Version 5 as an LLVM vendor extension. 1742 1743This section defines the AMDGPU target architecture specific DWARF mappings. 1744 1745.. _amdgpu-dwarf-register-identifier: 1746 1747Register Identifier 1748------------------- 1749 1750This section defines the AMDGPU target architecture register numbers used in 1751DWARF operation expressions (see DWARF Version 5 section 2.5 and 1752:ref:`amdgpu-dwarf-operation-expressions`) and Call Frame Information 1753instructions (see DWARF Version 5 section 6.4 and 1754:ref:`amdgpu-dwarf-call-frame-information`). 1755 1756A single code object can contain code for kernels that have different wavefront 1757sizes. The vector registers and some scalar registers are based on the wavefront 1758size. AMDGPU defines distinct DWARF registers for each wavefront size. This 1759simplifies the consumer of the DWARF so that each register has a fixed size, 1760rather than being dynamic according to the wavefront size mode. Similarly, 1761distinct DWARF registers are defined for those registers that vary in size 1762according to the process address size. This allows a consumer to treat a 1763specific AMDGPU processor as a single architecture regardless of how it is 1764configured at run time. The compiler explicitly specifies the DWARF registers 1765that match the mode in which the code it is generating will be executed. 1766 1767DWARF registers are encoded as numbers, which are mapped to architecture 1768registers. The mapping for AMDGPU is defined in 1769:ref:`amdgpu-dwarf-register-mapping-table`. All AMDGPU targets use the same 1770mapping. 1771 1772.. table:: AMDGPU DWARF Register Mapping 1773 :name: amdgpu-dwarf-register-mapping-table 1774 1775 ============== ================= ======== ================================== 1776 DWARF Register AMDGPU Register Bit Size Description 1777 ============== ================= ======== ================================== 1778 0 PC_32 32 Program Counter (PC) when 1779 executing in a 32-bit process 1780 address space. Used in the CFI to 1781 describe the PC of the calling 1782 frame. 1783 1 EXEC_MASK_32 32 Execution Mask Register when 1784 executing in wavefront 32 mode. 1785 2-15 *Reserved* *Reserved for highly accessed 1786 registers using DWARF shortcut.* 1787 16 PC_64 64 Program Counter (PC) when 1788 executing in a 64-bit process 1789 address space. Used in the CFI to 1790 describe the PC of the calling 1791 frame. 1792 17 EXEC_MASK_64 64 Execution Mask Register when 1793 executing in wavefront 64 mode. 1794 18-31 *Reserved* *Reserved for highly accessed 1795 registers using DWARF shortcut.* 1796 32-95 SGPR0-SGPR63 32 Scalar General Purpose 1797 Registers. 1798 96-127 *Reserved* *Reserved for frequently accessed 1799 registers using DWARF 1-byte ULEB.* 1800 128 STATUS 32 Status Register. 1801 129-511 *Reserved* *Reserved for future Scalar 1802 Architectural Registers.* 1803 512 VCC_32 32 Vector Condition Code Register 1804 when executing in wavefront 32 1805 mode. 1806 513-767 *Reserved* *Reserved for future Vector 1807 Architectural Registers when 1808 executing in wavefront 32 mode.* 1809 768 VCC_64 64 Vector Condition Code Register 1810 when executing in wavefront 64 1811 mode. 1812 769-1023 *Reserved* *Reserved for future Vector 1813 Architectural Registers when 1814 executing in wavefront 64 mode.* 1815 1024-1087 *Reserved* *Reserved for padding.* 1816 1088-1129 SGPR64-SGPR105 32 Scalar General Purpose Registers. 1817 1130-1535 *Reserved* *Reserved for future Scalar 1818 General Purpose Registers.* 1819 1536-1791 VGPR0-VGPR255 32*32 Vector General Purpose Registers 1820 when executing in wavefront 32 1821 mode. 1822 1792-2047 *Reserved* *Reserved for future Vector 1823 General Purpose Registers when 1824 executing in wavefront 32 mode.* 1825 2048-2303 AGPR0-AGPR255 32*32 Vector Accumulation Registers 1826 when executing in wavefront 32 1827 mode. 1828 2304-2559 *Reserved* *Reserved for future Vector 1829 Accumulation Registers when 1830 executing in wavefront 32 mode.* 1831 2560-2815 VGPR0-VGPR255 64*32 Vector General Purpose Registers 1832 when executing in wavefront 64 1833 mode. 1834 2816-3071 *Reserved* *Reserved for future Vector 1835 General Purpose Registers when 1836 executing in wavefront 64 mode.* 1837 3072-3327 AGPR0-AGPR255 64*32 Vector Accumulation Registers 1838 when executing in wavefront 64 1839 mode. 1840 3328-3583 *Reserved* *Reserved for future Vector 1841 Accumulation Registers when 1842 executing in wavefront 64 mode.* 1843 ============== ================= ======== ================================== 1844 1845The vector registers are represented as the full size for the wavefront. They 1846are organized as consecutive dwords (32-bits), one per lane, with the dword at 1847the least significant bit position corresponding to lane 0 and so forth. DWARF 1848location expressions involving the ``DW_OP_LLVM_offset`` and 1849``DW_OP_LLVM_push_lane`` operations are used to select the part of the vector 1850register corresponding to the lane that is executing the current thread of 1851execution in languages that are implemented using a SIMD or SIMT execution 1852model. 1853 1854If the wavefront size is 32 lanes then the wavefront 32 mode register 1855definitions are used. If the wavefront size is 64 lanes then the wavefront 64 1856mode register definitions are used. Some AMDGPU targets support executing in 1857both wavefront 32 and wavefront 64 mode. The register definitions corresponding 1858to the wavefront mode of the generated code will be used. 1859 1860If code is generated to execute in a 32-bit process address space, then the 186132-bit process address space register definitions are used. If code is generated 1862to execute in a 64-bit process address space, then the 64-bit process address 1863space register definitions are used. The ``amdgcn`` target only supports the 186464-bit process address space. 1865 1866.. _amdgpu-dwarf-address-class-identifier: 1867 1868Address Class Identifier 1869------------------------ 1870 1871The DWARF address class represents the source language memory space. See DWARF 1872Version 5 section 2.12 which is updated by the *DWARF Extensions For 1873Heterogeneous Debugging* section :ref:`amdgpu-dwarf-segment_addresses`. 1874 1875The DWARF address class mapping used for AMDGPU is defined in 1876:ref:`amdgpu-dwarf-address-class-mapping-table`. 1877 1878.. table:: AMDGPU DWARF Address Class Mapping 1879 :name: amdgpu-dwarf-address-class-mapping-table 1880 1881 ========================= ====== ================= 1882 DWARF AMDGPU 1883 -------------------------------- ----------------- 1884 Address Class Name Value Address Space 1885 ========================= ====== ================= 1886 ``DW_ADDR_none`` 0x0000 Generic (Flat) 1887 ``DW_ADDR_LLVM_global`` 0x0001 Global 1888 ``DW_ADDR_LLVM_constant`` 0x0002 Global 1889 ``DW_ADDR_LLVM_group`` 0x0003 Local (group/LDS) 1890 ``DW_ADDR_LLVM_private`` 0x0004 Private (Scratch) 1891 ``DW_ADDR_AMDGPU_region`` 0x8000 Region (GDS) 1892 ========================= ====== ================= 1893 1894The DWARF address class values defined in the *DWARF Extensions For 1895Heterogeneous Debugging* section :ref:`amdgpu-dwarf-segment_addresses` are used. 1896 1897In addition, ``DW_ADDR_AMDGPU_region`` is encoded as a vendor extension. This is 1898available for use for the AMD extension for access to the hardware GDS memory 1899which is scratchpad memory allocated per device. 1900 1901For AMDGPU if no ``DW_AT_address_class`` attribute is present, then the default 1902address class of ``DW_ADDR_none`` is used. 1903 1904See :ref:`amdgpu-dwarf-address-space-identifier` for information on the AMDGPU 1905mapping of DWARF address classes to DWARF address spaces, including address size 1906and NULL value. 1907 1908.. _amdgpu-dwarf-address-space-identifier: 1909 1910Address Space Identifier 1911------------------------ 1912 1913DWARF address spaces correspond to target architecture specific linear 1914addressable memory areas. See DWARF Version 5 section 2.12 and *DWARF Extensions 1915For Heterogeneous Debugging* section :ref:`amdgpu-dwarf-segment_addresses`. 1916 1917The DWARF address space mapping used for AMDGPU is defined in 1918:ref:`amdgpu-dwarf-address-space-mapping-table`. 1919 1920.. table:: AMDGPU DWARF Address Space Mapping 1921 :name: amdgpu-dwarf-address-space-mapping-table 1922 1923 ======================================= ===== ======= ======== ================= ======================= 1924 DWARF AMDGPU Notes 1925 --------------------------------------- ----- ---------------- ----------------- ----------------------- 1926 Address Space Name Value Address Bit Size Address Space 1927 --------------------------------------- ----- ------- -------- ----------------- ----------------------- 1928 .. 64-bit 32-bit 1929 process process 1930 address address 1931 space space 1932 ======================================= ===== ======= ======== ================= ======================= 1933 ``DW_ASPACE_none`` 0x00 64 32 Global *default address space* 1934 ``DW_ASPACE_AMDGPU_generic`` 0x01 64 32 Generic (Flat) 1935 ``DW_ASPACE_AMDGPU_region`` 0x02 32 32 Region (GDS) 1936 ``DW_ASPACE_AMDGPU_local`` 0x03 32 32 Local (group/LDS) 1937 *Reserved* 0x04 1938 ``DW_ASPACE_AMDGPU_private_lane`` 0x05 32 32 Private (Scratch) *focused lane* 1939 ``DW_ASPACE_AMDGPU_private_wave`` 0x06 32 32 Private (Scratch) *unswizzled wavefront* 1940 ======================================= ===== ======= ======== ================= ======================= 1941 1942See :ref:`amdgpu-address-spaces` for information on the AMDGPU address spaces 1943including address size and NULL value. 1944 1945The ``DW_ASPACE_none`` address space is the default target architecture address 1946space used in DWARF operations that do not specify an address space. It 1947therefore has to map to the global address space so that the ``DW_OP_addr*`` and 1948related operations can refer to addresses in the program code. 1949 1950The ``DW_ASPACE_AMDGPU_generic`` address space allows location expressions to 1951specify the flat address space. If the address corresponds to an address in the 1952local address space, then it corresponds to the wavefront that is executing the 1953focused thread of execution. If the address corresponds to an address in the 1954private address space, then it corresponds to the lane that is executing the 1955focused thread of execution for languages that are implemented using a SIMD or 1956SIMT execution model. 1957 1958.. note:: 1959 1960 CUDA-like languages such as HIP that do not have address spaces in the 1961 language type system, but do allow variables to be allocated in different 1962 address spaces, need to explicitly specify the ``DW_ASPACE_AMDGPU_generic`` 1963 address space in the DWARF expression operations as the default address space 1964 is the global address space. 1965 1966The ``DW_ASPACE_AMDGPU_local`` address space allows location expressions to 1967specify the local address space corresponding to the wavefront that is executing 1968the focused thread of execution. 1969 1970The ``DW_ASPACE_AMDGPU_private_lane`` address space allows location expressions 1971to specify the private address space corresponding to the lane that is executing 1972the focused thread of execution for languages that are implemented using a SIMD 1973or SIMT execution model. 1974 1975The ``DW_ASPACE_AMDGPU_private_wave`` address space allows location expressions 1976to specify the unswizzled private address space corresponding to the wavefront 1977that is executing the focused thread of execution. The wavefront view of private 1978memory is the per wavefront unswizzled backing memory layout defined in 1979:ref:`amdgpu-address-spaces`, such that address 0 corresponds to the first 1980location for the backing memory of the wavefront (namely the address is not 1981offset by ``wavefront-scratch-base``). The following formula can be used to 1982convert from a ``DW_ASPACE_AMDGPU_private_lane`` address to a 1983``DW_ASPACE_AMDGPU_private_wave`` address: 1984 1985:: 1986 1987 private-address-wavefront = 1988 ((private-address-lane / 4) * wavefront-size * 4) + 1989 (wavefront-lane-id * 4) + (private-address-lane % 4) 1990 1991If the ``DW_ASPACE_AMDGPU_private_lane`` address is dword aligned, and the start 1992of the dwords for each lane starting with lane 0 is required, then this 1993simplifies to: 1994 1995:: 1996 1997 private-address-wavefront = 1998 private-address-lane * wavefront-size 1999 2000A compiler can use the ``DW_ASPACE_AMDGPU_private_wave`` address space to read a 2001complete spilled vector register back into a complete vector register in the 2002CFI. The frame pointer can be a private lane address which is dword aligned, 2003which can be shifted to multiply by the wavefront size, and then used to form a 2004private wavefront address that gives a location for a contiguous set of dwords, 2005one per lane, where the vector register dwords are spilled. The compiler knows 2006the wavefront size since it generates the code. Note that the type of the 2007address may have to be converted as the size of a 2008``DW_ASPACE_AMDGPU_private_lane`` address may be smaller than the size of a 2009``DW_ASPACE_AMDGPU_private_wave`` address. 2010 2011.. _amdgpu-dwarf-lane-identifier: 2012 2013Lane identifier 2014--------------- 2015 2016DWARF lane identifies specify a target architecture lane position for hardware 2017that executes in a SIMD or SIMT manner, and on which a source language maps its 2018threads of execution onto those lanes. The DWARF lane identifier is pushed by 2019the ``DW_OP_LLVM_push_lane`` DWARF expression operation. See DWARF Version 5 2020section 2.5 which is updated by *DWARF Extensions For Heterogeneous Debugging* 2021section :ref:`amdgpu-dwarf-operation-expressions`. 2022 2023For AMDGPU, the lane identifier corresponds to the hardware lane ID of a 2024wavefront. It is numbered from 0 to the wavefront size minus 1. 2025 2026Operation Expressions 2027--------------------- 2028 2029DWARF expressions are used to compute program values and the locations of 2030program objects. See DWARF Version 5 section 2.5 and 2031:ref:`amdgpu-dwarf-operation-expressions`. 2032 2033DWARF location descriptions describe how to access storage which includes memory 2034and registers. When accessing storage on AMDGPU, bytes are ordered with least 2035significant bytes first, and bits are ordered within bytes with least 2036significant bits first. 2037 2038For AMDGPU CFI expressions, ``DW_OP_LLVM_select_bit_piece`` is used to describe 2039unwinding vector registers that are spilled under the execution mask to memory: 2040the zero-single location description is the vector register, and the one-single 2041location description is the spilled memory location description. The 2042``DW_OP_LLVM_form_aspace_address`` is used to specify the address space of the 2043memory location description. 2044 2045In AMDGPU expressions, ``DW_OP_LLVM_select_bit_piece`` is used by the 2046``DW_AT_LLVM_lane_pc`` attribute expression where divergent control flow is 2047controlled by the execution mask. An undefined location description together 2048with ``DW_OP_LLVM_extend`` is used to indicate the lane was not active on entry 2049to the subprogram. See :ref:`amdgpu-dwarf-dw-at-llvm-lane-pc` for an example. 2050 2051Debugger Information Entry Attributes 2052------------------------------------- 2053 2054This section describes how certain debugger information entry attributes are 2055used by AMDGPU. See the sections in DWARF Version 5 section 3.3.5 and 3.1.1 2056which are updated by *DWARF Extensions For Heterogeneous Debugging* section 2057:ref:`amdgpu-dwarf-low-level-information` and 2058:ref:`amdgpu-dwarf-full-and-partial-compilation-unit-entries`. 2059 2060.. _amdgpu-dwarf-dw-at-llvm-lane-pc: 2061 2062``DW_AT_LLVM_lane_pc`` 2063~~~~~~~~~~~~~~~~~~~~~~ 2064 2065For AMDGPU, the ``DW_AT_LLVM_lane_pc`` attribute is used to specify the program 2066location of the separate lanes of a SIMT thread. 2067 2068If the lane is an active lane then this will be the same as the current program 2069location. 2070 2071If the lane is inactive, but was active on entry to the subprogram, then this is 2072the program location in the subprogram at which execution of the lane is 2073conceptual positioned. 2074 2075If the lane was not active on entry to the subprogram, then this will be the 2076undefined location. A client debugger can check if the lane is part of a valid 2077work-group by checking that the lane is in the range of the associated 2078work-group within the grid, accounting for partial work-groups. If it is not, 2079then the debugger can omit any information for the lane. Otherwise, the debugger 2080may repeatedly unwind the stack and inspect the ``DW_AT_LLVM_lane_pc`` of the 2081calling subprogram until it finds a non-undefined location. Conceptually the 2082lane only has the call frames that it has a non-undefined 2083``DW_AT_LLVM_lane_pc``. 2084 2085The following example illustrates how the AMDGPU backend can generate a DWARF 2086location list expression for the nested ``IF/THEN/ELSE`` structures of the 2087following subprogram pseudo code for a target with 64 lanes per wavefront. 2088 2089.. code:: 2090 :number-lines: 2091 2092 SUBPROGRAM X 2093 BEGIN 2094 a; 2095 IF (c1) THEN 2096 b; 2097 IF (c2) THEN 2098 c; 2099 ELSE 2100 d; 2101 ENDIF 2102 e; 2103 ELSE 2104 f; 2105 ENDIF 2106 g; 2107 END 2108 2109The AMDGPU backend may generate the following pseudo LLVM MIR to manipulate the 2110execution mask (``EXEC``) to linearize the control flow. The condition is 2111evaluated to make a mask of the lanes for which the condition evaluates to true. 2112First the ``THEN`` region is executed by setting the ``EXEC`` mask to the 2113logical ``AND`` of the current ``EXEC`` mask with the condition mask. Then the 2114``ELSE`` region is executed by negating the ``EXEC`` mask and logical ``AND`` of 2115the saved ``EXEC`` mask at the start of the region. After the ``IF/THEN/ELSE`` 2116region the ``EXEC`` mask is restored to the value it had at the beginning of the 2117region. This is shown below. Other approaches are possible, but the basic 2118concept is the same. 2119 2120.. code:: 2121 :number-lines: 2122 2123 $lex_start: 2124 a; 2125 %1 = EXEC 2126 %2 = c1 2127 $lex_1_start: 2128 EXEC = %1 & %2 2129 $if_1_then: 2130 b; 2131 %3 = EXEC 2132 %4 = c2 2133 $lex_1_1_start: 2134 EXEC = %3 & %4 2135 $lex_1_1_then: 2136 c; 2137 EXEC = ~EXEC & %3 2138 $lex_1_1_else: 2139 d; 2140 EXEC = %3 2141 $lex_1_1_end: 2142 e; 2143 EXEC = ~EXEC & %1 2144 $lex_1_else: 2145 f; 2146 EXEC = %1 2147 $lex_1_end: 2148 g; 2149 $lex_end: 2150 2151To create the DWARF location list expression that defines the location 2152description of a vector of lane program locations, the LLVM MIR ``DBG_VALUE`` 2153pseudo instruction can be used to annotate the linearized control flow. This can 2154be done by defining an artificial variable for the lane PC. The DWARF location 2155list expression created for it is used as the value of the 2156``DW_AT_LLVM_lane_pc`` attribute on the subprogram's debugger information entry. 2157 2158A DWARF procedure is defined for each well nested structured control flow region 2159which provides the conceptual lane program location for a lane if it is not 2160active (namely it is divergent). The DWARF operation expression for each region 2161conceptually inherits the value of the immediately enclosing region and modifies 2162it according to the semantics of the region. 2163 2164For an ``IF/THEN/ELSE`` region the divergent program location is at the start of 2165the region for the ``THEN`` region since it is executed first. For the ``ELSE`` 2166region the divergent program location is at the end of the ``IF/THEN/ELSE`` 2167region since the ``THEN`` region has completed. 2168 2169The lane PC artificial variable is assigned at each region transition. It uses 2170the immediately enclosing region's DWARF procedure to compute the program 2171location for each lane assuming they are divergent, and then modifies the result 2172by inserting the current program location for each lane that the ``EXEC`` mask 2173indicates is active. 2174 2175By having separate DWARF procedures for each region, they can be reused to 2176define the value for any nested region. This reduces the total size of the DWARF 2177operation expressions. 2178 2179The following provides an example using pseudo LLVM MIR. 2180 2181.. code:: 2182 :number-lines: 2183 2184 $lex_start: 2185 DEFINE_DWARF %__uint_64 = DW_TAG_base_type[ 2186 DW_AT_name = "__uint64"; 2187 DW_AT_byte_size = 8; 2188 DW_AT_encoding = DW_ATE_unsigned; 2189 ]; 2190 DEFINE_DWARF %__active_lane_pc = DW_TAG_dwarf_procedure[ 2191 DW_AT_name = "__active_lane_pc"; 2192 DW_AT_location = [ 2193 DW_OP_regx PC; 2194 DW_OP_LLVM_extend 64, 64; 2195 DW_OP_regval_type EXEC, %uint_64; 2196 DW_OP_LLVM_select_bit_piece 64, 64; 2197 ]; 2198 ]; 2199 DEFINE_DWARF %__divergent_lane_pc = DW_TAG_dwarf_procedure[ 2200 DW_AT_name = "__divergent_lane_pc"; 2201 DW_AT_location = [ 2202 DW_OP_LLVM_undefined; 2203 DW_OP_LLVM_extend 64, 64; 2204 ]; 2205 ]; 2206 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 2207 DW_OP_call_ref %__divergent_lane_pc; 2208 DW_OP_call_ref %__active_lane_pc; 2209 ]; 2210 a; 2211 %1 = EXEC; 2212 DBG_VALUE %1, $noreg, %__lex_1_save_exec; 2213 %2 = c1; 2214 $lex_1_start: 2215 EXEC = %1 & %2; 2216 $lex_1_then: 2217 DEFINE_DWARF %__divergent_lane_pc_1_then = DW_TAG_dwarf_procedure[ 2218 DW_AT_name = "__divergent_lane_pc_1_then"; 2219 DW_AT_location = DIExpression[ 2220 DW_OP_call_ref %__divergent_lane_pc; 2221 DW_OP_addrx &lex_1_start; 2222 DW_OP_stack_value; 2223 DW_OP_LLVM_extend 64, 64; 2224 DW_OP_call_ref %__lex_1_save_exec; 2225 DW_OP_deref_type 64, %__uint_64; 2226 DW_OP_LLVM_select_bit_piece 64, 64; 2227 ]; 2228 ]; 2229 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 2230 DW_OP_call_ref %__divergent_lane_pc_1_then; 2231 DW_OP_call_ref %__active_lane_pc; 2232 ]; 2233 b; 2234 %3 = EXEC; 2235 DBG_VALUE %3, %__lex_1_1_save_exec; 2236 %4 = c2; 2237 $lex_1_1_start: 2238 EXEC = %3 & %4; 2239 $lex_1_1_then: 2240 DEFINE_DWARF %__divergent_lane_pc_1_1_then = DW_TAG_dwarf_procedure[ 2241 DW_AT_name = "__divergent_lane_pc_1_1_then"; 2242 DW_AT_location = DIExpression[ 2243 DW_OP_call_ref %__divergent_lane_pc_1_then; 2244 DW_OP_addrx &lex_1_1_start; 2245 DW_OP_stack_value; 2246 DW_OP_LLVM_extend 64, 64; 2247 DW_OP_call_ref %__lex_1_1_save_exec; 2248 DW_OP_deref_type 64, %__uint_64; 2249 DW_OP_LLVM_select_bit_piece 64, 64; 2250 ]; 2251 ]; 2252 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 2253 DW_OP_call_ref %__divergent_lane_pc_1_1_then; 2254 DW_OP_call_ref %__active_lane_pc; 2255 ]; 2256 c; 2257 EXEC = ~EXEC & %3; 2258 $lex_1_1_else: 2259 DEFINE_DWARF %__divergent_lane_pc_1_1_else = DW_TAG_dwarf_procedure[ 2260 DW_AT_name = "__divergent_lane_pc_1_1_else"; 2261 DW_AT_location = DIExpression[ 2262 DW_OP_call_ref %__divergent_lane_pc_1_then; 2263 DW_OP_addrx &lex_1_1_end; 2264 DW_OP_stack_value; 2265 DW_OP_LLVM_extend 64, 64; 2266 DW_OP_call_ref %__lex_1_1_save_exec; 2267 DW_OP_deref_type 64, %__uint_64; 2268 DW_OP_LLVM_select_bit_piece 64, 64; 2269 ]; 2270 ]; 2271 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 2272 DW_OP_call_ref %__divergent_lane_pc_1_1_else; 2273 DW_OP_call_ref %__active_lane_pc; 2274 ]; 2275 d; 2276 EXEC = %3; 2277 $lex_1_1_end: 2278 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 2279 DW_OP_call_ref %__divergent_lane_pc; 2280 DW_OP_call_ref %__active_lane_pc; 2281 ]; 2282 e; 2283 EXEC = ~EXEC & %1; 2284 $lex_1_else: 2285 DEFINE_DWARF %__divergent_lane_pc_1_else = DW_TAG_dwarf_procedure[ 2286 DW_AT_name = "__divergent_lane_pc_1_else"; 2287 DW_AT_location = DIExpression[ 2288 DW_OP_call_ref %__divergent_lane_pc; 2289 DW_OP_addrx &lex_1_end; 2290 DW_OP_stack_value; 2291 DW_OP_LLVM_extend 64, 64; 2292 DW_OP_call_ref %__lex_1_save_exec; 2293 DW_OP_deref_type 64, %__uint_64; 2294 DW_OP_LLVM_select_bit_piece 64, 64; 2295 ]; 2296 ]; 2297 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 2298 DW_OP_call_ref %__divergent_lane_pc_1_else; 2299 DW_OP_call_ref %__active_lane_pc; 2300 ]; 2301 f; 2302 EXEC = %1; 2303 $lex_1_end: 2304 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc DIExpression[ 2305 DW_OP_call_ref %__divergent_lane_pc; 2306 DW_OP_call_ref %__active_lane_pc; 2307 ]; 2308 g; 2309 $lex_end: 2310 2311The DWARF procedure ``%__active_lane_pc`` is used to update the lane pc elements 2312that are active, with the current program location. 2313 2314Artificial variables %__lex_1_save_exec and %__lex_1_1_save_exec are created for 2315the execution masks saved on entry to a region. Using the ``DBG_VALUE`` pseudo 2316instruction, location list entries will be created that describe where the 2317artificial variables are allocated at any given program location. The compiler 2318may allocate them to registers or spill them to memory. 2319 2320The DWARF procedures for each region use the values of the saved execution mask 2321artificial variables to only update the lanes that are active on entry to the 2322region. All other lanes retain the value of the enclosing region where they were 2323last active. If they were not active on entry to the subprogram, then will have 2324the undefined location description. 2325 2326Other structured control flow regions can be handled similarly. For example, 2327loops would set the divergent program location for the region at the end of the 2328loop. Any lanes active will be in the loop, and any lanes not active must have 2329exited the loop. 2330 2331An ``IF/THEN/ELSEIF/ELSEIF/...`` region can be treated as a nest of 2332``IF/THEN/ELSE`` regions. 2333 2334The DWARF procedures can use the active lane artificial variable described in 2335:ref:`amdgpu-dwarf-amdgpu-dw-at-llvm-active-lane` rather than the actual 2336``EXEC`` mask in order to support whole or quad wavefront mode. 2337 2338.. _amdgpu-dwarf-amdgpu-dw-at-llvm-active-lane: 2339 2340``DW_AT_LLVM_active_lane`` 2341~~~~~~~~~~~~~~~~~~~~~~~~~~ 2342 2343The ``DW_AT_LLVM_active_lane`` attribute on a subprogram debugger information 2344entry is used to specify the lanes that are conceptually active for a SIMT 2345thread. 2346 2347The execution mask may be modified to implement whole or quad wavefront mode 2348operations. For example, all lanes may need to temporarily be made active to 2349execute a whole wavefront operation. Such regions would save the ``EXEC`` mask, 2350update it to enable the necessary lanes, perform the operations, and then 2351restore the ``EXEC`` mask from the saved value. While executing the whole 2352wavefront region, the conceptual execution mask is the saved value, not the 2353``EXEC`` value. 2354 2355This is handled by defining an artificial variable for the active lane mask. The 2356active lane mask artificial variable would be the actual ``EXEC`` mask for 2357normal regions, and the saved execution mask for regions where the mask is 2358temporarily updated. The location list expression created for this artificial 2359variable is used to define the value of the ``DW_AT_LLVM_active_lane`` 2360attribute. 2361 2362``DW_AT_LLVM_augmentation`` 2363~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2364 2365For AMDGPU, the ``DW_AT_LLVM_augmentation`` attribute of a compilation unit 2366debugger information entry has the following value for the augmentation string: 2367 2368:: 2369 2370 [amdgpu:v0.0] 2371 2372The "vX.Y" specifies the major X and minor Y version number of the AMDGPU 2373extensions used in the DWARF of the compilation unit. The version number 2374conforms to [SEMVER]_. 2375 2376Call Frame Information 2377---------------------- 2378 2379DWARF Call Frame Information (CFI) describes how a consumer can virtually 2380*unwind* call frames in a running process or core dump. See DWARF Version 5 2381section 6.4 and :ref:`amdgpu-dwarf-call-frame-information`. 2382 2383For AMDGPU, the Common Information Entry (CIE) fields have the following values: 2384 23851. ``augmentation`` string contains the following null-terminated UTF-8 string: 2386 2387 :: 2388 2389 [amd:v0.0] 2390 2391 The ``vX.Y`` specifies the major X and minor Y version number of the AMDGPU 2392 extensions used in this CIE or to the FDEs that use it. The version number 2393 conforms to [SEMVER]_. 2394 23952. ``address_size`` for the ``Global`` address space is defined in 2396 :ref:`amdgpu-dwarf-address-space-identifier`. 2397 23983. ``segment_selector_size`` is 0 as AMDGPU does not use a segment selector. 2399 24004. ``code_alignment_factor`` is 4 bytes. 2401 2402 .. TODO:: 2403 2404 Add to :ref:`amdgpu-processor-table` table. 2405 24065. ``data_alignment_factor`` is 4 bytes. 2407 2408 .. TODO:: 2409 2410 Add to :ref:`amdgpu-processor-table` table. 2411 24126. ``return_address_register`` is ``PC_32`` for 32-bit processes and ``PC_64`` 2413 for 64-bit processes defined in :ref:`amdgpu-dwarf-register-identifier`. 2414 24157. ``initial_instructions`` Since a subprogram X with fewer registers can be 2416 called from subprogram Y that has more allocated, X will not change any of 2417 the extra registers as it cannot access them. Therefore, the default rule 2418 for all columns is ``same value``. 2419 2420For AMDGPU the register number follows the numbering defined in 2421:ref:`amdgpu-dwarf-register-identifier`. 2422 2423For AMDGPU the instructions are variable size. A consumer can subtract 1 from 2424the return address to get the address of a byte within the call site 2425instructions. See DWARF Version 5 section 6.4.4. 2426 2427Accelerated Access 2428------------------ 2429 2430See DWARF Version 5 section 6.1. 2431 2432Lookup By Name Section Header 2433~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2434 2435See DWARF Version 5 section 6.1.1.4.1 and :ref:`amdgpu-dwarf-lookup-by-name`. 2436 2437For AMDGPU the lookup by name section header table: 2438 2439``augmentation_string_size`` (uword) 2440 2441 Set to the length of the ``augmentation_string`` value which is always a 2442 multiple of 4. 2443 2444``augmentation_string`` (sequence of UTF-8 characters) 2445 2446 Contains the following UTF-8 string null padded to a multiple of 4 bytes: 2447 2448 :: 2449 2450 [amdgpu:v0.0] 2451 2452 The "vX.Y" specifies the major X and minor Y version number of the AMDGPU 2453 extensions used in the DWARF of this index. The version number conforms to 2454 [SEMVER]_. 2455 2456 .. note:: 2457 2458 This is different to the DWARF Version 5 definition that requires the first 2459 4 characters to be the vendor ID. But this is consistent with the other 2460 augmentation strings and does allow multiple vendor contributions. However, 2461 backwards compatibility may be more desirable. 2462 2463Lookup By Address Section Header 2464~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2465 2466See DWARF Version 5 section 6.1.2. 2467 2468For AMDGPU the lookup by address section header table: 2469 2470``address_size`` (ubyte) 2471 2472 Match the address size for the ``Global`` address space defined in 2473 :ref:`amdgpu-dwarf-address-space-identifier`. 2474 2475``segment_selector_size`` (ubyte) 2476 2477 AMDGPU does not use a segment selector so this is 0. The entries in the 2478 ``.debug_aranges`` do not have a segment selector. 2479 2480Line Number Information 2481----------------------- 2482 2483See DWARF Version 5 section 6.2 and :ref:`amdgpu-dwarf-line-number-information`. 2484 2485AMDGPU does not use the ``isa`` state machine registers and always sets it to 0. 2486The instruction set must be obtained from the ELF file header ``e_flags`` field 2487in the ``EF_AMDGPU_MACH`` bit position (see :ref:`ELF Header 2488<amdgpu-elf-header>`). See DWARF Version 5 section 6.2.2. 2489 2490.. TODO:: 2491 2492 Should the ``isa`` state machine register be used to indicate if the code is 2493 in wavefront32 or wavefront64 mode? Or used to specify the architecture ISA? 2494 2495For AMDGPU the line number program header fields have the following values (see 2496DWARF Version 5 section 6.2.4): 2497 2498``address_size`` (ubyte) 2499 Matches the address size for the ``Global`` address space defined in 2500 :ref:`amdgpu-dwarf-address-space-identifier`. 2501 2502``segment_selector_size`` (ubyte) 2503 AMDGPU does not use a segment selector so this is 0. 2504 2505``minimum_instruction_length`` (ubyte) 2506 For GFX9-GFX10 this is 4. 2507 2508``maximum_operations_per_instruction`` (ubyte) 2509 For GFX9-GFX10 this is 1. 2510 2511Source text for online-compiled programs (for example, those compiled by the 2512OpenCL language runtime) may be embedded into the DWARF Version 5 line table. 2513See DWARF Version 5 section 6.2.4.1 which is updated by *DWARF Extensions For 2514Heterogeneous Debugging* section :ref:`DW_LNCT_LLVM_source 2515<amdgpu-dwarf-line-number-information-dw-lnct-llvm-source>`. 2516 2517The Clang option used to control source embedding in AMDGPU is defined in 2518:ref:`amdgpu-clang-debug-options-table`. 2519 2520 .. table:: AMDGPU Clang Debug Options 2521 :name: amdgpu-clang-debug-options-table 2522 2523 ==================== ================================================== 2524 Debug Flag Description 2525 ==================== ================================================== 2526 -g[no-]embed-source Enable/disable embedding source text in DWARF 2527 debug sections. Useful for environments where 2528 source cannot be written to disk, such as 2529 when performing online compilation. 2530 ==================== ================================================== 2531 2532For example: 2533 2534``-gembed-source`` 2535 Enable the embedded source. 2536 2537``-gno-embed-source`` 2538 Disable the embedded source. 2539 254032-Bit and 64-Bit DWARF Formats 2541------------------------------- 2542 2543See DWARF Version 5 section 7.4 and 2544:ref:`amdgpu-dwarf-32-bit-and-64-bit-dwarf-formats`. 2545 2546For AMDGPU: 2547 2548* For the ``amdgcn`` target architecture only the 64-bit process address space 2549 is supported. 2550 2551* The producer can generate either 32-bit or 64-bit DWARF format. LLVM generates 2552 the 32-bit DWARF format. 2553 2554Unit Headers 2555------------ 2556 2557For AMDGPU the following values apply for each of the unit headers described in 2558DWARF Version 5 sections 7.5.1.1, 7.5.1.2, and 7.5.1.3: 2559 2560``address_size`` (ubyte) 2561 Matches the address size for the ``Global`` address space defined in 2562 :ref:`amdgpu-dwarf-address-space-identifier`. 2563 2564.. _amdgpu-code-conventions: 2565 2566Code Conventions 2567================ 2568 2569This section provides code conventions used for each supported target triple OS 2570(see :ref:`amdgpu-target-triples`). 2571 2572AMDHSA 2573------ 2574 2575This section provides code conventions used when the target triple OS is 2576``amdhsa`` (see :ref:`amdgpu-target-triples`). 2577 2578.. _amdgpu-amdhsa-code-object-metadata: 2579 2580Code Object Metadata 2581~~~~~~~~~~~~~~~~~~~~ 2582 2583The code object metadata specifies extensible metadata associated with the code 2584objects executed on HSA [HSA]_ compatible runtimes (see :ref:`amdgpu-os`). The 2585encoding and semantics of this metadata depends on the code object version; see 2586:ref:`amdgpu-amdhsa-code-object-metadata-v2`, 2587:ref:`amdgpu-amdhsa-code-object-metadata-v3`, 2588:ref:`amdgpu-amdhsa-code-object-metadata-v4` and 2589:ref:`amdgpu-amdhsa-code-object-metadata-v5`. 2590 2591Code object metadata is specified in a note record (see 2592:ref:`amdgpu-note-records`) and is required when the target triple OS is 2593``amdhsa`` (see :ref:`amdgpu-target-triples`). It must contain the minimum 2594information necessary to support the HSA compatible runtime kernel queries. For 2595example, the segment sizes needed in a dispatch packet. In addition, a 2596high-level language runtime may require other information to be included. For 2597example, the AMD OpenCL runtime records kernel argument information. 2598 2599.. _amdgpu-amdhsa-code-object-metadata-v2: 2600 2601Code Object V2 Metadata 2602+++++++++++++++++++++++ 2603 2604.. warning:: 2605 Code object V2 is not the default code object version emitted by this version 2606 of LLVM. 2607 2608Code object V2 metadata is specified by the ``NT_AMD_HSA_METADATA`` note record 2609(see :ref:`amdgpu-note-records-v2`). 2610 2611The metadata is specified as a YAML formatted string (see [YAML]_ and 2612:doc:`YamlIO`). 2613 2614.. TODO:: 2615 2616 Is the string null terminated? It probably should not if YAML allows it to 2617 contain null characters, otherwise it should be. 2618 2619The metadata is represented as a single YAML document comprised of the mapping 2620defined in table :ref:`amdgpu-amdhsa-code-object-metadata-map-v2-table` and 2621referenced tables. 2622 2623For boolean values, the string values of ``false`` and ``true`` are used for 2624false and true respectively. 2625 2626Additional information can be added to the mappings. To avoid conflicts, any 2627non-AMD key names should be prefixed by "*vendor-name*.". 2628 2629 .. table:: AMDHSA Code Object V2 Metadata Map 2630 :name: amdgpu-amdhsa-code-object-metadata-map-v2-table 2631 2632 ========== ============== ========= ======================================= 2633 String Key Value Type Required? Description 2634 ========== ============== ========= ======================================= 2635 "Version" sequence of Required - The first integer is the major 2636 2 integers version. Currently 1. 2637 - The second integer is the minor 2638 version. Currently 0. 2639 "Printf" sequence of Each string is encoded information 2640 strings about a printf function call. The 2641 encoded information is organized as 2642 fields separated by colon (':'): 2643 2644 ``ID:N:S[0]:S[1]:...:S[N-1]:FormatString`` 2645 2646 where: 2647 2648 ``ID`` 2649 A 32-bit integer as a unique id for 2650 each printf function call 2651 2652 ``N`` 2653 A 32-bit integer equal to the number 2654 of arguments of printf function call 2655 minus 1 2656 2657 ``S[i]`` (where i = 0, 1, ... , N-1) 2658 32-bit integers for the size in bytes 2659 of the i-th FormatString argument of 2660 the printf function call 2661 2662 FormatString 2663 The format string passed to the 2664 printf function call. 2665 "Kernels" sequence of Required Sequence of the mappings for each 2666 mapping kernel in the code object. See 2667 :ref:`amdgpu-amdhsa-code-object-kernel-metadata-map-v2-table` 2668 for the definition of the mapping. 2669 ========== ============== ========= ======================================= 2670 2671.. 2672 2673 .. table:: AMDHSA Code Object V2 Kernel Metadata Map 2674 :name: amdgpu-amdhsa-code-object-kernel-metadata-map-v2-table 2675 2676 ================= ============== ========= ================================ 2677 String Key Value Type Required? Description 2678 ================= ============== ========= ================================ 2679 "Name" string Required Source name of the kernel. 2680 "SymbolName" string Required Name of the kernel 2681 descriptor ELF symbol. 2682 "Language" string Source language of the kernel. 2683 Values include: 2684 2685 - "OpenCL C" 2686 - "OpenCL C++" 2687 - "HCC" 2688 - "OpenMP" 2689 2690 "LanguageVersion" sequence of - The first integer is the major 2691 2 integers version. 2692 - The second integer is the 2693 minor version. 2694 "Attrs" mapping Mapping of kernel attributes. 2695 See 2696 :ref:`amdgpu-amdhsa-code-object-kernel-attribute-metadata-map-v2-table` 2697 for the mapping definition. 2698 "Args" sequence of Sequence of mappings of the 2699 mapping kernel arguments. See 2700 :ref:`amdgpu-amdhsa-code-object-kernel-argument-metadata-map-v2-table` 2701 for the definition of the mapping. 2702 "CodeProps" mapping Mapping of properties related to 2703 the kernel code. See 2704 :ref:`amdgpu-amdhsa-code-object-kernel-code-properties-metadata-map-v2-table` 2705 for the mapping definition. 2706 ================= ============== ========= ================================ 2707 2708.. 2709 2710 .. table:: AMDHSA Code Object V2 Kernel Attribute Metadata Map 2711 :name: amdgpu-amdhsa-code-object-kernel-attribute-metadata-map-v2-table 2712 2713 =================== ============== ========= ============================== 2714 String Key Value Type Required? Description 2715 =================== ============== ========= ============================== 2716 "ReqdWorkGroupSize" sequence of If not 0, 0, 0 then all values 2717 3 integers must be >=1 and the dispatch 2718 work-group size X, Y, Z must 2719 correspond to the specified 2720 values. Defaults to 0, 0, 0. 2721 2722 Corresponds to the OpenCL 2723 ``reqd_work_group_size`` 2724 attribute. 2725 "WorkGroupSizeHint" sequence of The dispatch work-group size 2726 3 integers X, Y, Z is likely to be the 2727 specified values. 2728 2729 Corresponds to the OpenCL 2730 ``work_group_size_hint`` 2731 attribute. 2732 "VecTypeHint" string The name of a scalar or vector 2733 type. 2734 2735 Corresponds to the OpenCL 2736 ``vec_type_hint`` attribute. 2737 2738 "RuntimeHandle" string The external symbol name 2739 associated with a kernel. 2740 OpenCL runtime allocates a 2741 global buffer for the symbol 2742 and saves the kernel's address 2743 to it, which is used for 2744 device side enqueueing. Only 2745 available for device side 2746 enqueued kernels. 2747 =================== ============== ========= ============================== 2748 2749.. 2750 2751 .. table:: AMDHSA Code Object V2 Kernel Argument Metadata Map 2752 :name: amdgpu-amdhsa-code-object-kernel-argument-metadata-map-v2-table 2753 2754 ================= ============== ========= ================================ 2755 String Key Value Type Required? Description 2756 ================= ============== ========= ================================ 2757 "Name" string Kernel argument name. 2758 "TypeName" string Kernel argument type name. 2759 "Size" integer Required Kernel argument size in bytes. 2760 "Align" integer Required Kernel argument alignment in 2761 bytes. Must be a power of two. 2762 "ValueKind" string Required Kernel argument kind that 2763 specifies how to set up the 2764 corresponding argument. 2765 Values include: 2766 2767 "ByValue" 2768 The argument is copied 2769 directly into the kernarg. 2770 2771 "GlobalBuffer" 2772 A global address space pointer 2773 to the buffer data is passed 2774 in the kernarg. 2775 2776 "DynamicSharedPointer" 2777 A group address space pointer 2778 to dynamically allocated LDS 2779 is passed in the kernarg. 2780 2781 "Sampler" 2782 A global address space 2783 pointer to a S# is passed in 2784 the kernarg. 2785 2786 "Image" 2787 A global address space 2788 pointer to a T# is passed in 2789 the kernarg. 2790 2791 "Pipe" 2792 A global address space pointer 2793 to an OpenCL pipe is passed in 2794 the kernarg. 2795 2796 "Queue" 2797 A global address space pointer 2798 to an OpenCL device enqueue 2799 queue is passed in the 2800 kernarg. 2801 2802 "HiddenGlobalOffsetX" 2803 The OpenCL grid dispatch 2804 global offset for the X 2805 dimension is passed in the 2806 kernarg. 2807 2808 "HiddenGlobalOffsetY" 2809 The OpenCL grid dispatch 2810 global offset for the Y 2811 dimension is passed in the 2812 kernarg. 2813 2814 "HiddenGlobalOffsetZ" 2815 The OpenCL grid dispatch 2816 global offset for the Z 2817 dimension is passed in the 2818 kernarg. 2819 2820 "HiddenNone" 2821 An argument that is not used 2822 by the kernel. Space needs to 2823 be left for it, but it does 2824 not need to be set up. 2825 2826 "HiddenPrintfBuffer" 2827 A global address space pointer 2828 to the runtime printf buffer 2829 is passed in kernarg. Mutually 2830 exclusive with 2831 "HiddenHostcallBuffer". 2832 2833 "HiddenHostcallBuffer" 2834 A global address space pointer 2835 to the runtime hostcall buffer 2836 is passed in kernarg. Mutually 2837 exclusive with 2838 "HiddenPrintfBuffer". 2839 2840 "HiddenDefaultQueue" 2841 A global address space pointer 2842 to the OpenCL device enqueue 2843 queue that should be used by 2844 the kernel by default is 2845 passed in the kernarg. 2846 2847 "HiddenCompletionAction" 2848 A global address space pointer 2849 to help link enqueued kernels into 2850 the ancestor tree for determining 2851 when the parent kernel has finished. 2852 2853 "HiddenMultiGridSyncArg" 2854 A global address space pointer for 2855 multi-grid synchronization is 2856 passed in the kernarg. 2857 2858 "ValueType" string Unused and deprecated. This should no longer 2859 be emitted, but is accepted for compatibility. 2860 2861 2862 "PointeeAlign" integer Alignment in bytes of pointee 2863 type for pointer type kernel 2864 argument. Must be a power 2865 of 2. Only present if 2866 "ValueKind" is 2867 "DynamicSharedPointer". 2868 "AddrSpaceQual" string Kernel argument address space 2869 qualifier. Only present if 2870 "ValueKind" is "GlobalBuffer" or 2871 "DynamicSharedPointer". Values 2872 are: 2873 2874 - "Private" 2875 - "Global" 2876 - "Constant" 2877 - "Local" 2878 - "Generic" 2879 - "Region" 2880 2881 .. TODO:: 2882 2883 Is GlobalBuffer only Global 2884 or Constant? Is 2885 DynamicSharedPointer always 2886 Local? Can HCC allow Generic? 2887 How can Private or Region 2888 ever happen? 2889 2890 "AccQual" string Kernel argument access 2891 qualifier. Only present if 2892 "ValueKind" is "Image" or 2893 "Pipe". Values 2894 are: 2895 2896 - "ReadOnly" 2897 - "WriteOnly" 2898 - "ReadWrite" 2899 2900 .. TODO:: 2901 2902 Does this apply to 2903 GlobalBuffer? 2904 2905 "ActualAccQual" string The actual memory accesses 2906 performed by the kernel on the 2907 kernel argument. Only present if 2908 "ValueKind" is "GlobalBuffer", 2909 "Image", or "Pipe". This may be 2910 more restrictive than indicated 2911 by "AccQual" to reflect what the 2912 kernel actual does. If not 2913 present then the runtime must 2914 assume what is implied by 2915 "AccQual" and "IsConst". Values 2916 are: 2917 2918 - "ReadOnly" 2919 - "WriteOnly" 2920 - "ReadWrite" 2921 2922 "IsConst" boolean Indicates if the kernel argument 2923 is const qualified. Only present 2924 if "ValueKind" is 2925 "GlobalBuffer". 2926 2927 "IsRestrict" boolean Indicates if the kernel argument 2928 is restrict qualified. Only 2929 present if "ValueKind" is 2930 "GlobalBuffer". 2931 2932 "IsVolatile" boolean Indicates if the kernel argument 2933 is volatile qualified. Only 2934 present if "ValueKind" is 2935 "GlobalBuffer". 2936 2937 "IsPipe" boolean Indicates if the kernel argument 2938 is pipe qualified. Only present 2939 if "ValueKind" is "Pipe". 2940 2941 .. TODO:: 2942 2943 Can GlobalBuffer be pipe 2944 qualified? 2945 2946 ================= ============== ========= ================================ 2947 2948.. 2949 2950 .. table:: AMDHSA Code Object V2 Kernel Code Properties Metadata Map 2951 :name: amdgpu-amdhsa-code-object-kernel-code-properties-metadata-map-v2-table 2952 2953 ============================ ============== ========= ===================== 2954 String Key Value Type Required? Description 2955 ============================ ============== ========= ===================== 2956 "KernargSegmentSize" integer Required The size in bytes of 2957 the kernarg segment 2958 that holds the values 2959 of the arguments to 2960 the kernel. 2961 "GroupSegmentFixedSize" integer Required The amount of group 2962 segment memory 2963 required by a 2964 work-group in 2965 bytes. This does not 2966 include any 2967 dynamically allocated 2968 group segment memory 2969 that may be added 2970 when the kernel is 2971 dispatched. 2972 "PrivateSegmentFixedSize" integer Required The amount of fixed 2973 private address space 2974 memory required for a 2975 work-item in 2976 bytes. If the kernel 2977 uses a dynamic call 2978 stack then additional 2979 space must be added 2980 to this value for the 2981 call stack. 2982 "KernargSegmentAlign" integer Required The maximum byte 2983 alignment of 2984 arguments in the 2985 kernarg segment. Must 2986 be a power of 2. 2987 "WavefrontSize" integer Required Wavefront size. Must 2988 be a power of 2. 2989 "NumSGPRs" integer Required Number of scalar 2990 registers used by a 2991 wavefront for 2992 GFX6-GFX10. This 2993 includes the special 2994 SGPRs for VCC, Flat 2995 Scratch (GFX7-GFX10) 2996 and XNACK (for 2997 GFX8-GFX10). It does 2998 not include the 16 2999 SGPR added if a trap 3000 handler is 3001 enabled. It is not 3002 rounded up to the 3003 allocation 3004 granularity. 3005 "NumVGPRs" integer Required Number of vector 3006 registers used by 3007 each work-item for 3008 GFX6-GFX10 3009 "MaxFlatWorkGroupSize" integer Required Maximum flat 3010 work-group size 3011 supported by the 3012 kernel in work-items. 3013 Must be >=1 and 3014 consistent with 3015 ReqdWorkGroupSize if 3016 not 0, 0, 0. 3017 "NumSpilledSGPRs" integer Number of stores from 3018 a scalar register to 3019 a register allocator 3020 created spill 3021 location. 3022 "NumSpilledVGPRs" integer Number of stores from 3023 a vector register to 3024 a register allocator 3025 created spill 3026 location. 3027 ============================ ============== ========= ===================== 3028 3029.. _amdgpu-amdhsa-code-object-metadata-v3: 3030 3031Code Object V3 Metadata 3032+++++++++++++++++++++++ 3033 3034.. warning:: 3035 Code object V3 is not the default code object version emitted by this version 3036 of LLVM. 3037 3038Code object V3 and above metadata is specified by the ``NT_AMDGPU_METADATA`` note 3039record (see :ref:`amdgpu-note-records-v3-onwards`). 3040 3041The metadata is represented as Message Pack formatted binary data (see 3042[MsgPack]_). The top level is a Message Pack map that includes the 3043keys defined in table 3044:ref:`amdgpu-amdhsa-code-object-metadata-map-table-v3` and referenced 3045tables. 3046 3047Additional information can be added to the maps. To avoid conflicts, 3048any key names should be prefixed by "*vendor-name*." where 3049``vendor-name`` can be the name of the vendor and specific vendor 3050tool that generates the information. The prefix is abbreviated to 3051simply "." when it appears within a map that has been added by the 3052same *vendor-name*. 3053 3054 .. table:: AMDHSA Code Object V3 Metadata Map 3055 :name: amdgpu-amdhsa-code-object-metadata-map-table-v3 3056 3057 ================= ============== ========= ======================================= 3058 String Key Value Type Required? Description 3059 ================= ============== ========= ======================================= 3060 "amdhsa.version" sequence of Required - The first integer is the major 3061 2 integers version. Currently 1. 3062 - The second integer is the minor 3063 version. Currently 0. 3064 "amdhsa.printf" sequence of Each string is encoded information 3065 strings about a printf function call. The 3066 encoded information is organized as 3067 fields separated by colon (':'): 3068 3069 ``ID:N:S[0]:S[1]:...:S[N-1]:FormatString`` 3070 3071 where: 3072 3073 ``ID`` 3074 A 32-bit integer as a unique id for 3075 each printf function call 3076 3077 ``N`` 3078 A 32-bit integer equal to the number 3079 of arguments of printf function call 3080 minus 1 3081 3082 ``S[i]`` (where i = 0, 1, ... , N-1) 3083 32-bit integers for the size in bytes 3084 of the i-th FormatString argument of 3085 the printf function call 3086 3087 FormatString 3088 The format string passed to the 3089 printf function call. 3090 "amdhsa.kernels" sequence of Required Sequence of the maps for each 3091 map kernel in the code object. See 3092 :ref:`amdgpu-amdhsa-code-object-kernel-metadata-map-table-v3` 3093 for the definition of the keys included 3094 in that map. 3095 ================= ============== ========= ======================================= 3096 3097.. 3098 3099 .. table:: AMDHSA Code Object V3 Kernel Metadata Map 3100 :name: amdgpu-amdhsa-code-object-kernel-metadata-map-table-v3 3101 3102 =================================== ============== ========= ================================ 3103 String Key Value Type Required? Description 3104 =================================== ============== ========= ================================ 3105 ".name" string Required Source name of the kernel. 3106 ".symbol" string Required Name of the kernel 3107 descriptor ELF symbol. 3108 ".language" string Source language of the kernel. 3109 Values include: 3110 3111 - "OpenCL C" 3112 - "OpenCL C++" 3113 - "HCC" 3114 - "HIP" 3115 - "OpenMP" 3116 - "Assembler" 3117 3118 ".language_version" sequence of - The first integer is the major 3119 2 integers version. 3120 - The second integer is the 3121 minor version. 3122 ".args" sequence of Sequence of maps of the 3123 map kernel arguments. See 3124 :ref:`amdgpu-amdhsa-code-object-kernel-argument-metadata-map-table-v3` 3125 for the definition of the keys 3126 included in that map. 3127 ".reqd_workgroup_size" sequence of If not 0, 0, 0 then all values 3128 3 integers must be >=1 and the dispatch 3129 work-group size X, Y, Z must 3130 correspond to the specified 3131 values. Defaults to 0, 0, 0. 3132 3133 Corresponds to the OpenCL 3134 ``reqd_work_group_size`` 3135 attribute. 3136 ".workgroup_size_hint" sequence of The dispatch work-group size 3137 3 integers X, Y, Z is likely to be the 3138 specified values. 3139 3140 Corresponds to the OpenCL 3141 ``work_group_size_hint`` 3142 attribute. 3143 ".vec_type_hint" string The name of a scalar or vector 3144 type. 3145 3146 Corresponds to the OpenCL 3147 ``vec_type_hint`` attribute. 3148 3149 ".device_enqueue_symbol" string The external symbol name 3150 associated with a kernel. 3151 OpenCL runtime allocates a 3152 global buffer for the symbol 3153 and saves the kernel's address 3154 to it, which is used for 3155 device side enqueueing. Only 3156 available for device side 3157 enqueued kernels. 3158 ".kernarg_segment_size" integer Required The size in bytes of 3159 the kernarg segment 3160 that holds the values 3161 of the arguments to 3162 the kernel. 3163 ".group_segment_fixed_size" integer Required The amount of group 3164 segment memory 3165 required by a 3166 work-group in 3167 bytes. This does not 3168 include any 3169 dynamically allocated 3170 group segment memory 3171 that may be added 3172 when the kernel is 3173 dispatched. 3174 ".private_segment_fixed_size" integer Required The amount of fixed 3175 private address space 3176 memory required for a 3177 work-item in 3178 bytes. If the kernel 3179 uses a dynamic call 3180 stack then additional 3181 space must be added 3182 to this value for the 3183 call stack. 3184 ".kernarg_segment_align" integer Required The maximum byte 3185 alignment of 3186 arguments in the 3187 kernarg segment. Must 3188 be a power of 2. 3189 ".wavefront_size" integer Required Wavefront size. Must 3190 be a power of 2. 3191 ".sgpr_count" integer Required Number of scalar 3192 registers required by a 3193 wavefront for 3194 GFX6-GFX9. A register 3195 is required if it is 3196 used explicitly, or 3197 if a higher numbered 3198 register is used 3199 explicitly. This 3200 includes the special 3201 SGPRs for VCC, Flat 3202 Scratch (GFX7-GFX9) 3203 and XNACK (for 3204 GFX8-GFX9). It does 3205 not include the 16 3206 SGPR added if a trap 3207 handler is 3208 enabled. It is not 3209 rounded up to the 3210 allocation 3211 granularity. 3212 ".vgpr_count" integer Required Number of vector 3213 registers required by 3214 each work-item for 3215 GFX6-GFX9. A register 3216 is required if it is 3217 used explicitly, or 3218 if a higher numbered 3219 register is used 3220 explicitly. 3221 ".agpr_count" integer Required Number of accumulator 3222 registers required by 3223 each work-item for 3224 GFX90A, GFX908. 3225 ".max_flat_workgroup_size" integer Required Maximum flat 3226 work-group size 3227 supported by the 3228 kernel in work-items. 3229 Must be >=1 and 3230 consistent with 3231 ReqdWorkGroupSize if 3232 not 0, 0, 0. 3233 ".sgpr_spill_count" integer Number of stores from 3234 a scalar register to 3235 a register allocator 3236 created spill 3237 location. 3238 ".vgpr_spill_count" integer Number of stores from 3239 a vector register to 3240 a register allocator 3241 created spill 3242 location. 3243 ".kind" string The kind of the kernel 3244 with the following 3245 values: 3246 3247 "normal" 3248 Regular kernels. 3249 3250 "init" 3251 These kernels must be 3252 invoked after loading 3253 the containing code 3254 object and must 3255 complete before any 3256 normal and fini 3257 kernels in the same 3258 code object are 3259 invoked. 3260 3261 "fini" 3262 These kernels must be 3263 invoked before 3264 unloading the 3265 containing code object 3266 and after all init and 3267 normal kernels in the 3268 same code object have 3269 been invoked and 3270 completed. 3271 3272 If omitted, "normal" is 3273 assumed. 3274 =================================== ============== ========= ================================ 3275 3276.. 3277 3278 .. table:: AMDHSA Code Object V3 Kernel Argument Metadata Map 3279 :name: amdgpu-amdhsa-code-object-kernel-argument-metadata-map-table-v3 3280 3281 ====================== ============== ========= ================================ 3282 String Key Value Type Required? Description 3283 ====================== ============== ========= ================================ 3284 ".name" string Kernel argument name. 3285 ".type_name" string Kernel argument type name. 3286 ".size" integer Required Kernel argument size in bytes. 3287 ".offset" integer Required Kernel argument offset in 3288 bytes. The offset must be a 3289 multiple of the alignment 3290 required by the argument. 3291 ".value_kind" string Required Kernel argument kind that 3292 specifies how to set up the 3293 corresponding argument. 3294 Values include: 3295 3296 "by_value" 3297 The argument is copied 3298 directly into the kernarg. 3299 3300 "global_buffer" 3301 A global address space pointer 3302 to the buffer data is passed 3303 in the kernarg. 3304 3305 "dynamic_shared_pointer" 3306 A group address space pointer 3307 to dynamically allocated LDS 3308 is passed in the kernarg. 3309 3310 "sampler" 3311 A global address space 3312 pointer to a S# is passed in 3313 the kernarg. 3314 3315 "image" 3316 A global address space 3317 pointer to a T# is passed in 3318 the kernarg. 3319 3320 "pipe" 3321 A global address space pointer 3322 to an OpenCL pipe is passed in 3323 the kernarg. 3324 3325 "queue" 3326 A global address space pointer 3327 to an OpenCL device enqueue 3328 queue is passed in the 3329 kernarg. 3330 3331 "hidden_global_offset_x" 3332 The OpenCL grid dispatch 3333 global offset for the X 3334 dimension is passed in the 3335 kernarg. 3336 3337 "hidden_global_offset_y" 3338 The OpenCL grid dispatch 3339 global offset for the Y 3340 dimension is passed in the 3341 kernarg. 3342 3343 "hidden_global_offset_z" 3344 The OpenCL grid dispatch 3345 global offset for the Z 3346 dimension is passed in the 3347 kernarg. 3348 3349 "hidden_none" 3350 An argument that is not used 3351 by the kernel. Space needs to 3352 be left for it, but it does 3353 not need to be set up. 3354 3355 "hidden_printf_buffer" 3356 A global address space pointer 3357 to the runtime printf buffer 3358 is passed in kernarg. Mutually 3359 exclusive with 3360 "hidden_hostcall_buffer" 3361 before Code Object V5. 3362 3363 "hidden_hostcall_buffer" 3364 A global address space pointer 3365 to the runtime hostcall buffer 3366 is passed in kernarg. Mutually 3367 exclusive with 3368 "hidden_printf_buffer" 3369 before Code Object V5. 3370 3371 "hidden_default_queue" 3372 A global address space pointer 3373 to the OpenCL device enqueue 3374 queue that should be used by 3375 the kernel by default is 3376 passed in the kernarg. 3377 3378 "hidden_completion_action" 3379 A global address space pointer 3380 to help link enqueued kernels into 3381 the ancestor tree for determining 3382 when the parent kernel has finished. 3383 3384 "hidden_multigrid_sync_arg" 3385 A global address space pointer for 3386 multi-grid synchronization is 3387 passed in the kernarg. 3388 3389 ".value_type" string Unused and deprecated. This should no longer 3390 be emitted, but is accepted for compatibility. 3391 3392 ".pointee_align" integer Alignment in bytes of pointee 3393 type for pointer type kernel 3394 argument. Must be a power 3395 of 2. Only present if 3396 ".value_kind" is 3397 "dynamic_shared_pointer". 3398 ".address_space" string Kernel argument address space 3399 qualifier. Only present if 3400 ".value_kind" is "global_buffer" or 3401 "dynamic_shared_pointer". Values 3402 are: 3403 3404 - "private" 3405 - "global" 3406 - "constant" 3407 - "local" 3408 - "generic" 3409 - "region" 3410 3411 .. TODO:: 3412 3413 Is "global_buffer" only "global" 3414 or "constant"? Is 3415 "dynamic_shared_pointer" always 3416 "local"? Can HCC allow "generic"? 3417 How can "private" or "region" 3418 ever happen? 3419 3420 ".access" string Kernel argument access 3421 qualifier. Only present if 3422 ".value_kind" is "image" or 3423 "pipe". Values 3424 are: 3425 3426 - "read_only" 3427 - "write_only" 3428 - "read_write" 3429 3430 .. TODO:: 3431 3432 Does this apply to 3433 "global_buffer"? 3434 3435 ".actual_access" string The actual memory accesses 3436 performed by the kernel on the 3437 kernel argument. Only present if 3438 ".value_kind" is "global_buffer", 3439 "image", or "pipe". This may be 3440 more restrictive than indicated 3441 by ".access" to reflect what the 3442 kernel actual does. If not 3443 present then the runtime must 3444 assume what is implied by 3445 ".access" and ".is_const" . Values 3446 are: 3447 3448 - "read_only" 3449 - "write_only" 3450 - "read_write" 3451 3452 ".is_const" boolean Indicates if the kernel argument 3453 is const qualified. Only present 3454 if ".value_kind" is 3455 "global_buffer". 3456 3457 ".is_restrict" boolean Indicates if the kernel argument 3458 is restrict qualified. Only 3459 present if ".value_kind" is 3460 "global_buffer". 3461 3462 ".is_volatile" boolean Indicates if the kernel argument 3463 is volatile qualified. Only 3464 present if ".value_kind" is 3465 "global_buffer". 3466 3467 ".is_pipe" boolean Indicates if the kernel argument 3468 is pipe qualified. Only present 3469 if ".value_kind" is "pipe". 3470 3471 .. TODO:: 3472 3473 Can "global_buffer" be pipe 3474 qualified? 3475 3476 ====================== ============== ========= ================================ 3477 3478.. _amdgpu-amdhsa-code-object-metadata-v4: 3479 3480Code Object V4 Metadata 3481+++++++++++++++++++++++ 3482 3483Code object V4 metadata is the same as 3484:ref:`amdgpu-amdhsa-code-object-metadata-v3` with the changes and additions 3485defined in table :ref:`amdgpu-amdhsa-code-object-metadata-map-table-v4`. 3486 3487 .. table:: AMDHSA Code Object V4 Metadata Map Changes 3488 :name: amdgpu-amdhsa-code-object-metadata-map-table-v4 3489 3490 ================= ============== ========= ======================================= 3491 String Key Value Type Required? Description 3492 ================= ============== ========= ======================================= 3493 "amdhsa.version" sequence of Required - The first integer is the major 3494 2 integers version. Currently 1. 3495 - The second integer is the minor 3496 version. Currently 1. 3497 "amdhsa.target" string Required The target name of the code using the syntax: 3498 3499 .. code:: 3500 3501 <target-triple> [ "-" <target-id> ] 3502 3503 A canonical target ID must be 3504 used. See :ref:`amdgpu-target-triples` 3505 and :ref:`amdgpu-target-id`. 3506 ================= ============== ========= ======================================= 3507 3508.. _amdgpu-amdhsa-code-object-metadata-v5: 3509 3510Code Object V5 Metadata 3511+++++++++++++++++++++++ 3512 3513.. warning:: 3514 Code object V5 is not the default code object version emitted by this version 3515 of LLVM. 3516 3517 3518Code object V5 metadata is the same as 3519:ref:`amdgpu-amdhsa-code-object-metadata-v4` with the changes defined in table 3520:ref:`amdgpu-amdhsa-code-object-metadata-map-table-v5` and table 3521:ref:`amdgpu-amdhsa-code-object-kernel-argument-metadata-map-table-v5`. 3522 3523 .. table:: AMDHSA Code Object V5 Metadata Map Changes 3524 :name: amdgpu-amdhsa-code-object-metadata-map-table-v5 3525 3526 ================= ============== ========= ======================================= 3527 String Key Value Type Required? Description 3528 ================= ============== ========= ======================================= 3529 "amdhsa.version" sequence of Required - The first integer is the major 3530 2 integers version. Currently 1. 3531 - The second integer is the minor 3532 version. Currently 2. 3533 ================= ============== ========= ======================================= 3534 3535.. 3536 3537 .. table:: AMDHSA Code Object V5 Kernel Argument Metadata Map Additions and Changes 3538 :name: amdgpu-amdhsa-code-object-kernel-argument-metadata-map-table-v5 3539 3540 ====================== ============== ========= ================================ 3541 String Key Value Type Required? Description 3542 ====================== ============== ========= ================================ 3543 ".value_kind" string Required Kernel argument kind that 3544 specifies how to set up the 3545 corresponding argument. 3546 Values include: 3547 the same as code object V3 metadata 3548 (see :ref:`amdgpu-amdhsa-code-object-kernel-argument-metadata-map-table-v3`) 3549 with the following additions: 3550 3551 "hidden_block_count_x" 3552 The grid dispatch work-group count for the X dimension 3553 is passed in the kernarg. Some languages, such as OpenCL, 3554 support a last work-group in each dimension being partial. 3555 This count only includes the non-partial work-group count. 3556 This is not the same as the value in the AQL dispatch packet, 3557 which has the grid size in work-items. 3558 3559 "hidden_block_count_y" 3560 The grid dispatch work-group count for the Y dimension 3561 is passed in the kernarg. Some languages, such as OpenCL, 3562 support a last work-group in each dimension being partial. 3563 This count only includes the non-partial work-group count. 3564 This is not the same as the value in the AQL dispatch packet, 3565 which has the grid size in work-items. If the grid dimensionality 3566 is 1, then must be 1. 3567 3568 "hidden_block_count_z" 3569 The grid dispatch work-group count for the Z dimension 3570 is passed in the kernarg. Some languages, such as OpenCL, 3571 support a last work-group in each dimension being partial. 3572 This count only includes the non-partial work-group count. 3573 This is not the same as the value in the AQL dispatch packet, 3574 which has the grid size in work-items. If the grid dimensionality 3575 is 1 or 2, then must be 1. 3576 3577 "hidden_group_size_x" 3578 The grid dispatch work-group size for the X dimension is 3579 passed in the kernarg. This size only applies to the 3580 non-partial work-groups. This is the same value as the AQL 3581 dispatch packet work-group size. 3582 3583 "hidden_group_size_y" 3584 The grid dispatch work-group size for the Y dimension is 3585 passed in the kernarg. This size only applies to the 3586 non-partial work-groups. This is the same value as the AQL 3587 dispatch packet work-group size. If the grid dimensionality 3588 is 1, then must be 1. 3589 3590 "hidden_group_size_z" 3591 The grid dispatch work-group size for the Z dimension is 3592 passed in the kernarg. This size only applies to the 3593 non-partial work-groups. This is the same value as the AQL 3594 dispatch packet work-group size. If the grid dimensionality 3595 is 1 or 2, then must be 1. 3596 3597 "hidden_remainder_x" 3598 The grid dispatch work group size of the the partial work group 3599 of the X dimension, if it exists. Must be zero if a partial 3600 work group does not exist in the X dimension. 3601 3602 "hidden_remainder_y" 3603 The grid dispatch work group size of the the partial work group 3604 of the Y dimension, if it exists. Must be zero if a partial 3605 work group does not exist in the Y dimension. 3606 3607 "hidden_remainder_z" 3608 The grid dispatch work group size of the the partial work group 3609 of the Z dimension, if it exists. Must be zero if a partial 3610 work group does not exist in the Z dimension. 3611 3612 "hidden_grid_dims" 3613 The grid dispatch dimensionality. This is the same value 3614 as the AQL dispatch packet dimensionality. Must be a value 3615 between 1 and 3. 3616 3617 "hidden_heap_v1" 3618 A global address space pointer to an initialized memory 3619 buffer that conforms to the requirements of the malloc/free 3620 device library V1 version implementation. 3621 3622 "hidden_private_base" 3623 The high 32 bits of the flat addressing private aperture base. 3624 Only used by GFX8 to allow conversion between private segment 3625 and flat addresses. See :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`. 3626 3627 "hidden_shared_base" 3628 The high 32 bits of the flat addressing shared aperture base. 3629 Only used by GFX8 to allow conversion between shared segment 3630 and flat addresses. See :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`. 3631 3632 "hidden_queue_ptr" 3633 A global memory address space pointer to the ROCm runtime 3634 ``struct amd_queue_t`` structure for the HSA queue of the 3635 associated dispatch AQL packet. It is only required for pre-GFX9 3636 devices for the trap handler ABI (see :ref:`amdgpu-amdhsa-trap-handler-abi`). 3637 3638 ====================== ============== ========= ================================ 3639 3640.. 3641 3642Kernel Dispatch 3643~~~~~~~~~~~~~~~ 3644 3645The HSA architected queuing language (AQL) defines a user space memory interface 3646that can be used to control the dispatch of kernels, in an agent independent 3647way. An agent can have zero or more AQL queues created for it using an HSA 3648compatible runtime (see :ref:`amdgpu-os`), in which AQL packets (all of which 3649are 64 bytes) can be placed. See the *HSA Platform System Architecture 3650Specification* [HSA]_ for the AQL queue mechanics and packet layouts. 3651 3652The packet processor of a kernel agent is responsible for detecting and 3653dispatching HSA kernels from the AQL queues associated with it. For AMD GPUs the 3654packet processor is implemented by the hardware command processor (CP), 3655asynchronous dispatch controller (ADC) and shader processor input controller 3656(SPI). 3657 3658An HSA compatible runtime can be used to allocate an AQL queue object. It uses 3659the kernel mode driver to initialize and register the AQL queue with CP. 3660 3661To dispatch a kernel the following actions are performed. This can occur in the 3662CPU host program, or from an HSA kernel executing on a GPU. 3663 36641. A pointer to an AQL queue for the kernel agent on which the kernel is to be 3665 executed is obtained. 36662. A pointer to the kernel descriptor (see 3667 :ref:`amdgpu-amdhsa-kernel-descriptor`) of the kernel to execute is obtained. 3668 It must be for a kernel that is contained in a code object that was loaded 3669 by an HSA compatible runtime on the kernel agent with which the AQL queue is 3670 associated. 36713. Space is allocated for the kernel arguments using the HSA compatible runtime 3672 allocator for a memory region with the kernarg property for the kernel agent 3673 that will execute the kernel. It must be at least 16-byte aligned. 36744. Kernel argument values are assigned to the kernel argument memory 3675 allocation. The layout is defined in the *HSA Programmer's Language 3676 Reference* [HSA]_. For AMDGPU the kernel execution directly accesses the 3677 kernel argument memory in the same way constant memory is accessed. (Note 3678 that the HSA specification allows an implementation to copy the kernel 3679 argument contents to another location that is accessed by the kernel.) 36805. An AQL kernel dispatch packet is created on the AQL queue. The HSA compatible 3681 runtime api uses 64-bit atomic operations to reserve space in the AQL queue 3682 for the packet. The packet must be set up, and the final write must use an 3683 atomic store release to set the packet kind to ensure the packet contents are 3684 visible to the kernel agent. AQL defines a doorbell signal mechanism to 3685 notify the kernel agent that the AQL queue has been updated. These rules, and 3686 the layout of the AQL queue and kernel dispatch packet is defined in the *HSA 3687 System Architecture Specification* [HSA]_. 36886. A kernel dispatch packet includes information about the actual dispatch, 3689 such as grid and work-group size, together with information from the code 3690 object about the kernel, such as segment sizes. The HSA compatible runtime 3691 queries on the kernel symbol can be used to obtain the code object values 3692 which are recorded in the :ref:`amdgpu-amdhsa-code-object-metadata`. 36937. CP executes micro-code and is responsible for detecting and setting up the 3694 GPU to execute the wavefronts of a kernel dispatch. 36958. CP ensures that when the a wavefront starts executing the kernel machine 3696 code, the scalar general purpose registers (SGPR) and vector general purpose 3697 registers (VGPR) are set up as required by the machine code. The required 3698 setup is defined in the :ref:`amdgpu-amdhsa-kernel-descriptor`. The initial 3699 register state is defined in 3700 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`. 37019. The prolog of the kernel machine code (see 3702 :ref:`amdgpu-amdhsa-kernel-prolog`) sets up the machine state as necessary 3703 before continuing executing the machine code that corresponds to the kernel. 370410. When the kernel dispatch has completed execution, CP signals the completion 3705 signal specified in the kernel dispatch packet if not 0. 3706 3707.. _amdgpu-amdhsa-memory-spaces: 3708 3709Memory Spaces 3710~~~~~~~~~~~~~ 3711 3712The memory space properties are: 3713 3714 .. table:: AMDHSA Memory Spaces 3715 :name: amdgpu-amdhsa-memory-spaces-table 3716 3717 ================= =========== ======== ======= ================== 3718 Memory Space Name HSA Segment Hardware Address NULL Value 3719 Name Name Size 3720 ================= =========== ======== ======= ================== 3721 Private private scratch 32 0x00000000 3722 Local group LDS 32 0xFFFFFFFF 3723 Global global global 64 0x0000000000000000 3724 Constant constant *same as 64 0x0000000000000000 3725 global* 3726 Generic flat flat 64 0x0000000000000000 3727 Region N/A GDS 32 *not implemented 3728 for AMDHSA* 3729 ================= =========== ======== ======= ================== 3730 3731The global and constant memory spaces both use global virtual addresses, which 3732are the same virtual address space used by the CPU. However, some virtual 3733addresses may only be accessible to the CPU, some only accessible by the GPU, 3734and some by both. 3735 3736Using the constant memory space indicates that the data will not change during 3737the execution of the kernel. This allows scalar read instructions to be 3738used. The vector and scalar L1 caches are invalidated of volatile data before 3739each kernel dispatch execution to allow constant memory to change values between 3740kernel dispatches. 3741 3742The local memory space uses the hardware Local Data Store (LDS) which is 3743automatically allocated when the hardware creates work-groups of wavefronts, and 3744freed when all the wavefronts of a work-group have terminated. The data store 3745(DS) instructions can be used to access it. 3746 3747The private memory space uses the hardware scratch memory support. If the kernel 3748uses scratch, then the hardware allocates memory that is accessed using 3749wavefront lane dword (4 byte) interleaving. The mapping used from private 3750address to physical address is: 3751 3752 ``wavefront-scratch-base + 3753 (private-address * wavefront-size * 4) + 3754 (wavefront-lane-id * 4)`` 3755 3756There are different ways that the wavefront scratch base address is determined 3757by a wavefront (see :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). This 3758memory can be accessed in an interleaved manner using buffer instruction with 3759the scratch buffer descriptor and per wavefront scratch offset, by the scratch 3760instructions, or by flat instructions. If each lane of a wavefront accesses the 3761same private address, the interleaving results in adjacent dwords being accessed 3762and hence requires fewer cache lines to be fetched. Multi-dword access is not 3763supported except by flat and scratch instructions in GFX9-GFX10. 3764 3765The generic address space uses the hardware flat address support available in 3766GFX7-GFX10. This uses two fixed ranges of virtual addresses (the private and 3767local apertures), that are outside the range of addressible global memory, to 3768map from a flat address to a private or local address. 3769 3770FLAT instructions can take a flat address and access global, private (scratch) 3771and group (LDS) memory depending on if the address is within one of the 3772aperture ranges. Flat access to scratch requires hardware aperture setup and 3773setup in the kernel prologue (see 3774:ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`). Flat access to LDS requires 3775hardware aperture setup and M0 (GFX7-GFX8) register setup (see 3776:ref:`amdgpu-amdhsa-kernel-prolog-m0`). 3777 3778To convert between a segment address and a flat address the base address of the 3779apertures address can be used. For GFX7-GFX8 these are available in the 3780:ref:`amdgpu-amdhsa-hsa-aql-queue` the address of which can be obtained with 3781Queue Ptr SGPR (see :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). For 3782GFX9-GFX10 the aperture base addresses are directly available as inline constant 3783registers ``SRC_SHARED_BASE/LIMIT`` and ``SRC_PRIVATE_BASE/LIMIT``. In 64 bit 3784address mode the aperture sizes are 2^32 bytes and the base is aligned to 2^32 3785which makes it easier to convert from flat to segment or segment to flat. 3786 3787Image and Samplers 3788~~~~~~~~~~~~~~~~~~ 3789 3790Image and sample handles created by an HSA compatible runtime (see 3791:ref:`amdgpu-os`) are 64-bit addresses of a hardware 32-byte V# and 48 byte S# 3792object respectively. In order to support the HSA ``query_sampler`` operations 3793two extra dwords are used to store the HSA BRIG enumeration values for the 3794queries that are not trivially deducible from the S# representation. 3795 3796HSA Signals 3797~~~~~~~~~~~ 3798 3799HSA signal handles created by an HSA compatible runtime (see :ref:`amdgpu-os`) 3800are 64-bit addresses of a structure allocated in memory accessible from both the 3801CPU and GPU. The structure is defined by the runtime and subject to change 3802between releases. For example, see [AMD-ROCm-github]_. 3803 3804.. _amdgpu-amdhsa-hsa-aql-queue: 3805 3806HSA AQL Queue 3807~~~~~~~~~~~~~ 3808 3809The HSA AQL queue structure is defined by an HSA compatible runtime (see 3810:ref:`amdgpu-os`) and subject to change between releases. For example, see 3811[AMD-ROCm-github]_. For some processors it contains fields needed to implement 3812certain language features such as the flat address aperture bases. It also 3813contains fields used by CP such as managing the allocation of scratch memory. 3814 3815.. _amdgpu-amdhsa-kernel-descriptor: 3816 3817Kernel Descriptor 3818~~~~~~~~~~~~~~~~~ 3819 3820A kernel descriptor consists of the information needed by CP to initiate the 3821execution of a kernel, including the entry point address of the machine code 3822that implements the kernel. 3823 3824Code Object V3 Kernel Descriptor 3825++++++++++++++++++++++++++++++++ 3826 3827CP microcode requires the Kernel descriptor to be allocated on 64-byte 3828alignment. 3829 3830The fields used by CP for code objects before V3 also match those specified in 3831:ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 3832 3833 .. table:: Code Object V3 Kernel Descriptor 3834 :name: amdgpu-amdhsa-kernel-descriptor-v3-table 3835 3836 ======= ======= =============================== ============================ 3837 Bits Size Field Name Description 3838 ======= ======= =============================== ============================ 3839 31:0 4 bytes GROUP_SEGMENT_FIXED_SIZE The amount of fixed local 3840 address space memory 3841 required for a work-group 3842 in bytes. This does not 3843 include any dynamically 3844 allocated local address 3845 space memory that may be 3846 added when the kernel is 3847 dispatched. 3848 63:32 4 bytes PRIVATE_SEGMENT_FIXED_SIZE The amount of fixed 3849 private address space 3850 memory required for a 3851 work-item in bytes. 3852 Additional space may need to 3853 be added to this value if 3854 the call stack has 3855 non-inlined function calls. 3856 95:64 4 bytes KERNARG_SIZE The size of the kernarg 3857 memory pointed to by the 3858 AQL dispatch packet. The 3859 kernarg memory is used to 3860 pass arguments to the 3861 kernel. 3862 3863 * If the kernarg pointer in 3864 the dispatch packet is NULL 3865 then there are no kernel 3866 arguments. 3867 * If the kernarg pointer in 3868 the dispatch packet is 3869 not NULL and this value 3870 is 0 then the kernarg 3871 memory size is 3872 unspecified. 3873 * If the kernarg pointer in 3874 the dispatch packet is 3875 not NULL and this value 3876 is not 0 then the value 3877 specifies the kernarg 3878 memory size in bytes. It 3879 is recommended to provide 3880 a value as it may be used 3881 by CP to optimize making 3882 the kernarg memory 3883 visible to the kernel 3884 code. 3885 3886 127:96 4 bytes Reserved, must be 0. 3887 191:128 8 bytes KERNEL_CODE_ENTRY_BYTE_OFFSET Byte offset (possibly 3888 negative) from base 3889 address of kernel 3890 descriptor to kernel's 3891 entry point instruction 3892 which must be 256 byte 3893 aligned. 3894 351:272 20 Reserved, must be 0. 3895 bytes 3896 383:352 4 bytes COMPUTE_PGM_RSRC3 GFX6-GFX9 3897 Reserved, must be 0. 3898 GFX90A, GFX940 3899 Compute Shader (CS) 3900 program settings used by 3901 CP to set up 3902 ``COMPUTE_PGM_RSRC3`` 3903 configuration 3904 register. See 3905 :ref:`amdgpu-amdhsa-compute_pgm_rsrc3-gfx90a-table`. 3906 GFX10 3907 Compute Shader (CS) 3908 program settings used by 3909 CP to set up 3910 ``COMPUTE_PGM_RSRC3`` 3911 configuration 3912 register. See 3913 :ref:`amdgpu-amdhsa-compute_pgm_rsrc3-gfx10-table`. 3914 415:384 4 bytes COMPUTE_PGM_RSRC1 Compute Shader (CS) 3915 program settings used by 3916 CP to set up 3917 ``COMPUTE_PGM_RSRC1`` 3918 configuration 3919 register. See 3920 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 3921 447:416 4 bytes COMPUTE_PGM_RSRC2 Compute Shader (CS) 3922 program settings used by 3923 CP to set up 3924 ``COMPUTE_PGM_RSRC2`` 3925 configuration 3926 register. See 3927 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 3928 458:448 7 bits *See separate bits below.* Enable the setup of the 3929 SGPR user data registers 3930 (see 3931 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 3932 3933 The total number of SGPR 3934 user data registers 3935 requested must not exceed 3936 16 and match value in 3937 ``compute_pgm_rsrc2.user_sgpr.user_sgpr_count``. 3938 Any requests beyond 16 3939 will be ignored. 3940 >448 1 bit ENABLE_SGPR_PRIVATE_SEGMENT If the *Target Properties* 3941 _BUFFER column of 3942 :ref:`amdgpu-processor-table` 3943 specifies *Architected flat 3944 scratch* then not supported 3945 and must be 0, 3946 >449 1 bit ENABLE_SGPR_DISPATCH_PTR 3947 >450 1 bit ENABLE_SGPR_QUEUE_PTR 3948 >451 1 bit ENABLE_SGPR_KERNARG_SEGMENT_PTR 3949 >452 1 bit ENABLE_SGPR_DISPATCH_ID 3950 >453 1 bit ENABLE_SGPR_FLAT_SCRATCH_INIT If the *Target Properties* 3951 column of 3952 :ref:`amdgpu-processor-table` 3953 specifies *Architected flat 3954 scratch* then not supported 3955 and must be 0, 3956 >454 1 bit ENABLE_SGPR_PRIVATE_SEGMENT 3957 _SIZE 3958 457:455 3 bits Reserved, must be 0. 3959 458 1 bit ENABLE_WAVEFRONT_SIZE32 GFX6-GFX9 3960 Reserved, must be 0. 3961 GFX10 3962 - If 0 execute in 3963 wavefront size 64 mode. 3964 - If 1 execute in 3965 native wavefront size 3966 32 mode. 3967 463:459 1 bit Reserved, must be 0. 3968 464 1 bit RESERVED_464 Deprecated, must be 0. 3969 467:465 3 bits Reserved, must be 0. 3970 468 1 bit RESERVED_468 Deprecated, must be 0. 3971 469:471 3 bits Reserved, must be 0. 3972 511:472 5 bytes Reserved, must be 0. 3973 512 **Total size 64 bytes.** 3974 ======= ==================================================================== 3975 3976.. 3977 3978 .. table:: compute_pgm_rsrc1 for GFX6-GFX10 3979 :name: amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table 3980 3981 ======= ======= =============================== =========================================================================== 3982 Bits Size Field Name Description 3983 ======= ======= =============================== =========================================================================== 3984 5:0 6 bits GRANULATED_WORKITEM_VGPR_COUNT Number of vector register 3985 blocks used by each work-item; 3986 granularity is device 3987 specific: 3988 3989 GFX6-GFX9 3990 - vgprs_used 0..256 3991 - max(0, ceil(vgprs_used / 4) - 1) 3992 GFX90A, GFX940 3993 - vgprs_used 0..512 3994 - vgprs_used = align(arch_vgprs, 4) 3995 + acc_vgprs 3996 - max(0, ceil(vgprs_used / 8) - 1) 3997 GFX10 (wavefront size 64) 3998 - max_vgpr 1..256 3999 - max(0, ceil(vgprs_used / 4) - 1) 4000 GFX10 (wavefront size 32) 4001 - max_vgpr 1..256 4002 - max(0, ceil(vgprs_used / 8) - 1) 4003 4004 Where vgprs_used is defined 4005 as the highest VGPR number 4006 explicitly referenced plus 4007 one. 4008 4009 Used by CP to set up 4010 ``COMPUTE_PGM_RSRC1.VGPRS``. 4011 4012 The 4013 :ref:`amdgpu-assembler` 4014 calculates this 4015 automatically for the 4016 selected processor from 4017 values provided to the 4018 `.amdhsa_kernel` directive 4019 by the 4020 `.amdhsa_next_free_vgpr` 4021 nested directive (see 4022 :ref:`amdhsa-kernel-directives-table`). 4023 9:6 4 bits GRANULATED_WAVEFRONT_SGPR_COUNT Number of scalar register 4024 blocks used by a wavefront; 4025 granularity is device 4026 specific: 4027 4028 GFX6-GFX8 4029 - sgprs_used 0..112 4030 - max(0, ceil(sgprs_used / 8) - 1) 4031 GFX9 4032 - sgprs_used 0..112 4033 - 2 * max(0, ceil(sgprs_used / 16) - 1) 4034 GFX10 4035 Reserved, must be 0. 4036 (128 SGPRs always 4037 allocated.) 4038 4039 Where sgprs_used is 4040 defined as the highest 4041 SGPR number explicitly 4042 referenced plus one, plus 4043 a target specific number 4044 of additional special 4045 SGPRs for VCC, 4046 FLAT_SCRATCH (GFX7+) and 4047 XNACK_MASK (GFX8+), and 4048 any additional 4049 target specific 4050 limitations. It does not 4051 include the 16 SGPRs added 4052 if a trap handler is 4053 enabled. 4054 4055 The target specific 4056 limitations and special 4057 SGPR layout are defined in 4058 the hardware 4059 documentation, which can 4060 be found in the 4061 :ref:`amdgpu-processors` 4062 table. 4063 4064 Used by CP to set up 4065 ``COMPUTE_PGM_RSRC1.SGPRS``. 4066 4067 The 4068 :ref:`amdgpu-assembler` 4069 calculates this 4070 automatically for the 4071 selected processor from 4072 values provided to the 4073 `.amdhsa_kernel` directive 4074 by the 4075 `.amdhsa_next_free_sgpr` 4076 and `.amdhsa_reserve_*` 4077 nested directives (see 4078 :ref:`amdhsa-kernel-directives-table`). 4079 11:10 2 bits PRIORITY Must be 0. 4080 4081 Start executing wavefront 4082 at the specified priority. 4083 4084 CP is responsible for 4085 filling in 4086 ``COMPUTE_PGM_RSRC1.PRIORITY``. 4087 13:12 2 bits FLOAT_ROUND_MODE_32 Wavefront starts execution 4088 with specified rounding 4089 mode for single (32 4090 bit) floating point 4091 precision floating point 4092 operations. 4093 4094 Floating point rounding 4095 mode values are defined in 4096 :ref:`amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table`. 4097 4098 Used by CP to set up 4099 ``COMPUTE_PGM_RSRC1.FLOAT_MODE``. 4100 15:14 2 bits FLOAT_ROUND_MODE_16_64 Wavefront starts execution 4101 with specified rounding 4102 denorm mode for half/double (16 4103 and 64-bit) floating point 4104 precision floating point 4105 operations. 4106 4107 Floating point rounding 4108 mode values are defined in 4109 :ref:`amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table`. 4110 4111 Used by CP to set up 4112 ``COMPUTE_PGM_RSRC1.FLOAT_MODE``. 4113 17:16 2 bits FLOAT_DENORM_MODE_32 Wavefront starts execution 4114 with specified denorm mode 4115 for single (32 4116 bit) floating point 4117 precision floating point 4118 operations. 4119 4120 Floating point denorm mode 4121 values are defined in 4122 :ref:`amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table`. 4123 4124 Used by CP to set up 4125 ``COMPUTE_PGM_RSRC1.FLOAT_MODE``. 4126 19:18 2 bits FLOAT_DENORM_MODE_16_64 Wavefront starts execution 4127 with specified denorm mode 4128 for half/double (16 4129 and 64-bit) floating point 4130 precision floating point 4131 operations. 4132 4133 Floating point denorm mode 4134 values are defined in 4135 :ref:`amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table`. 4136 4137 Used by CP to set up 4138 ``COMPUTE_PGM_RSRC1.FLOAT_MODE``. 4139 20 1 bit PRIV Must be 0. 4140 4141 Start executing wavefront 4142 in privilege trap handler 4143 mode. 4144 4145 CP is responsible for 4146 filling in 4147 ``COMPUTE_PGM_RSRC1.PRIV``. 4148 21 1 bit ENABLE_DX10_CLAMP Wavefront starts execution 4149 with DX10 clamp mode 4150 enabled. Used by the vector 4151 ALU to force DX10 style 4152 treatment of NaN's (when 4153 set, clamp NaN to zero, 4154 otherwise pass NaN 4155 through). 4156 4157 Used by CP to set up 4158 ``COMPUTE_PGM_RSRC1.DX10_CLAMP``. 4159 22 1 bit DEBUG_MODE Must be 0. 4160 4161 Start executing wavefront 4162 in single step mode. 4163 4164 CP is responsible for 4165 filling in 4166 ``COMPUTE_PGM_RSRC1.DEBUG_MODE``. 4167 23 1 bit ENABLE_IEEE_MODE Wavefront starts execution 4168 with IEEE mode 4169 enabled. Floating point 4170 opcodes that support 4171 exception flag gathering 4172 will quiet and propagate 4173 signaling-NaN inputs per 4174 IEEE 754-2008. Min_dx10 and 4175 max_dx10 become IEEE 4176 754-2008 compliant due to 4177 signaling-NaN propagation 4178 and quieting. 4179 4180 Used by CP to set up 4181 ``COMPUTE_PGM_RSRC1.IEEE_MODE``. 4182 24 1 bit BULKY Must be 0. 4183 4184 Only one work-group allowed 4185 to execute on a compute 4186 unit. 4187 4188 CP is responsible for 4189 filling in 4190 ``COMPUTE_PGM_RSRC1.BULKY``. 4191 25 1 bit CDBG_USER Must be 0. 4192 4193 Flag that can be used to 4194 control debugging code. 4195 4196 CP is responsible for 4197 filling in 4198 ``COMPUTE_PGM_RSRC1.CDBG_USER``. 4199 26 1 bit FP16_OVFL GFX6-GFX8 4200 Reserved, must be 0. 4201 GFX9-GFX10 4202 Wavefront starts execution 4203 with specified fp16 overflow 4204 mode. 4205 4206 - If 0, fp16 overflow generates 4207 +/-INF values. 4208 - If 1, fp16 overflow that is the 4209 result of an +/-INF input value 4210 or divide by 0 produces a +/-INF, 4211 otherwise clamps computed 4212 overflow to +/-MAX_FP16 as 4213 appropriate. 4214 4215 Used by CP to set up 4216 ``COMPUTE_PGM_RSRC1.FP16_OVFL``. 4217 28:27 2 bits Reserved, must be 0. 4218 29 1 bit WGP_MODE GFX6-GFX9 4219 Reserved, must be 0. 4220 GFX10 4221 - If 0 execute work-groups in 4222 CU wavefront execution mode. 4223 - If 1 execute work-groups on 4224 in WGP wavefront execution mode. 4225 4226 See :ref:`amdgpu-amdhsa-memory-model`. 4227 4228 Used by CP to set up 4229 ``COMPUTE_PGM_RSRC1.WGP_MODE``. 4230 30 1 bit MEM_ORDERED GFX6-GFX9 4231 Reserved, must be 0. 4232 GFX10 4233 Controls the behavior of the 4234 s_waitcnt's vmcnt and vscnt 4235 counters. 4236 4237 - If 0 vmcnt reports completion 4238 of load and atomic with return 4239 out of order with sample 4240 instructions, and the vscnt 4241 reports the completion of 4242 store and atomic without 4243 return in order. 4244 - If 1 vmcnt reports completion 4245 of load, atomic with return 4246 and sample instructions in 4247 order, and the vscnt reports 4248 the completion of store and 4249 atomic without return in order. 4250 4251 Used by CP to set up 4252 ``COMPUTE_PGM_RSRC1.MEM_ORDERED``. 4253 31 1 bit FWD_PROGRESS GFX6-GFX9 4254 Reserved, must be 0. 4255 GFX10 4256 - If 0 execute SIMD wavefronts 4257 using oldest first policy. 4258 - If 1 execute SIMD wavefronts to 4259 ensure wavefronts will make some 4260 forward progress. 4261 4262 Used by CP to set up 4263 ``COMPUTE_PGM_RSRC1.FWD_PROGRESS``. 4264 32 **Total size 4 bytes** 4265 ======= =================================================================================================================== 4266 4267.. 4268 4269 .. table:: compute_pgm_rsrc2 for GFX6-GFX10 4270 :name: amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table 4271 4272 ======= ======= =============================== =========================================================================== 4273 Bits Size Field Name Description 4274 ======= ======= =============================== =========================================================================== 4275 0 1 bit ENABLE_PRIVATE_SEGMENT * Enable the setup of the 4276 private segment. 4277 * If the *Target Properties* 4278 column of 4279 :ref:`amdgpu-processor-table` 4280 does not specify 4281 *Architected flat 4282 scratch* then enable the 4283 setup of the SGPR 4284 wavefront scratch offset 4285 system register (see 4286 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4287 * If the *Target Properties* 4288 column of 4289 :ref:`amdgpu-processor-table` 4290 specifies *Architected 4291 flat scratch* then enable 4292 the setup of the 4293 FLAT_SCRATCH register 4294 pair (see 4295 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4296 4297 Used by CP to set up 4298 ``COMPUTE_PGM_RSRC2.SCRATCH_EN``. 4299 5:1 5 bits USER_SGPR_COUNT The total number of SGPR 4300 user data 4301 registers requested. This 4302 number must be greater than 4303 or equal to the number of user 4304 data registers enabled. 4305 4306 Used by CP to set up 4307 ``COMPUTE_PGM_RSRC2.USER_SGPR``. 4308 6 1 bit ENABLE_TRAP_HANDLER Must be 0. 4309 4310 This bit represents 4311 ``COMPUTE_PGM_RSRC2.TRAP_PRESENT``, 4312 which is set by the CP if 4313 the runtime has installed a 4314 trap handler. 4315 7 1 bit ENABLE_SGPR_WORKGROUP_ID_X Enable the setup of the 4316 system SGPR register for 4317 the work-group id in the X 4318 dimension (see 4319 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4320 4321 Used by CP to set up 4322 ``COMPUTE_PGM_RSRC2.TGID_X_EN``. 4323 8 1 bit ENABLE_SGPR_WORKGROUP_ID_Y Enable the setup of the 4324 system SGPR register for 4325 the work-group id in the Y 4326 dimension (see 4327 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4328 4329 Used by CP to set up 4330 ``COMPUTE_PGM_RSRC2.TGID_Y_EN``. 4331 9 1 bit ENABLE_SGPR_WORKGROUP_ID_Z Enable the setup of the 4332 system SGPR register for 4333 the work-group id in the Z 4334 dimension (see 4335 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4336 4337 Used by CP to set up 4338 ``COMPUTE_PGM_RSRC2.TGID_Z_EN``. 4339 10 1 bit ENABLE_SGPR_WORKGROUP_INFO Enable the setup of the 4340 system SGPR register for 4341 work-group information (see 4342 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4343 4344 Used by CP to set up 4345 ``COMPUTE_PGM_RSRC2.TGID_SIZE_EN``. 4346 12:11 2 bits ENABLE_VGPR_WORKITEM_ID Enable the setup of the 4347 VGPR system registers used 4348 for the work-item ID. 4349 :ref:`amdgpu-amdhsa-system-vgpr-work-item-id-enumeration-values-table` 4350 defines the values. 4351 4352 Used by CP to set up 4353 ``COMPUTE_PGM_RSRC2.TIDIG_CMP_CNT``. 4354 13 1 bit ENABLE_EXCEPTION_ADDRESS_WATCH Must be 0. 4355 4356 Wavefront starts execution 4357 with address watch 4358 exceptions enabled which 4359 are generated when L1 has 4360 witnessed a thread access 4361 an *address of 4362 interest*. 4363 4364 CP is responsible for 4365 filling in the address 4366 watch bit in 4367 ``COMPUTE_PGM_RSRC2.EXCP_EN_MSB`` 4368 according to what the 4369 runtime requests. 4370 14 1 bit ENABLE_EXCEPTION_MEMORY Must be 0. 4371 4372 Wavefront starts execution 4373 with memory violation 4374 exceptions exceptions 4375 enabled which are generated 4376 when a memory violation has 4377 occurred for this wavefront from 4378 L1 or LDS 4379 (write-to-read-only-memory, 4380 mis-aligned atomic, LDS 4381 address out of range, 4382 illegal address, etc.). 4383 4384 CP sets the memory 4385 violation bit in 4386 ``COMPUTE_PGM_RSRC2.EXCP_EN_MSB`` 4387 according to what the 4388 runtime requests. 4389 23:15 9 bits GRANULATED_LDS_SIZE Must be 0. 4390 4391 CP uses the rounded value 4392 from the dispatch packet, 4393 not this value, as the 4394 dispatch may contain 4395 dynamically allocated group 4396 segment memory. CP writes 4397 directly to 4398 ``COMPUTE_PGM_RSRC2.LDS_SIZE``. 4399 4400 Amount of group segment 4401 (LDS) to allocate for each 4402 work-group. Granularity is 4403 device specific: 4404 4405 GFX6 4406 roundup(lds-size / (64 * 4)) 4407 GFX7-GFX10 4408 roundup(lds-size / (128 * 4)) 4409 4410 24 1 bit ENABLE_EXCEPTION_IEEE_754_FP Wavefront starts execution 4411 _INVALID_OPERATION with specified exceptions 4412 enabled. 4413 4414 Used by CP to set up 4415 ``COMPUTE_PGM_RSRC2.EXCP_EN`` 4416 (set from bits 0..6). 4417 4418 IEEE 754 FP Invalid 4419 Operation 4420 25 1 bit ENABLE_EXCEPTION_FP_DENORMAL FP Denormal one or more 4421 _SOURCE input operands is a 4422 denormal number 4423 26 1 bit ENABLE_EXCEPTION_IEEE_754_FP IEEE 754 FP Division by 4424 _DIVISION_BY_ZERO Zero 4425 27 1 bit ENABLE_EXCEPTION_IEEE_754_FP IEEE 754 FP FP Overflow 4426 _OVERFLOW 4427 28 1 bit ENABLE_EXCEPTION_IEEE_754_FP IEEE 754 FP Underflow 4428 _UNDERFLOW 4429 29 1 bit ENABLE_EXCEPTION_IEEE_754_FP IEEE 754 FP Inexact 4430 _INEXACT 4431 30 1 bit ENABLE_EXCEPTION_INT_DIVIDE_BY Integer Division by Zero 4432 _ZERO (rcp_iflag_f32 instruction 4433 only) 4434 31 1 bit Reserved, must be 0. 4435 32 **Total size 4 bytes.** 4436 ======= =================================================================================================================== 4437 4438.. 4439 4440 .. table:: compute_pgm_rsrc3 for GFX90A, GFX940 4441 :name: amdgpu-amdhsa-compute_pgm_rsrc3-gfx90a-table 4442 4443 ======= ======= =============================== =========================================================================== 4444 Bits Size Field Name Description 4445 ======= ======= =============================== =========================================================================== 4446 5:0 6 bits ACCUM_OFFSET Offset of a first AccVGPR in the unified register file. Granularity 4. 4447 Value 0-63. 0 - accum-offset = 4, 1 - accum-offset = 8, ..., 4448 63 - accum-offset = 256. 4449 6:15 10 Reserved, must be 0. 4450 bits 4451 16 1 bit TG_SPLIT - If 0 the waves of a work-group are 4452 launched in the same CU. 4453 - If 1 the waves of a work-group can be 4454 launched in different CUs. The waves 4455 cannot use S_BARRIER or LDS. 4456 17:31 15 Reserved, must be 0. 4457 bits 4458 32 **Total size 4 bytes.** 4459 ======= =================================================================================================================== 4460 4461.. 4462 4463 .. table:: compute_pgm_rsrc3 for GFX10 4464 :name: amdgpu-amdhsa-compute_pgm_rsrc3-gfx10-table 4465 4466 ======= ======= =============================== =========================================================================== 4467 Bits Size Field Name Description 4468 ======= ======= =============================== =========================================================================== 4469 3:0 4 bits SHARED_VGPR_COUNT Number of shared VGPR blocks when executing in subvector mode. For 4470 wavefront size 64 the value is 0-15, representing 0-120 VGPRs (granularity 4471 of 8), such that (compute_pgm_rsrc1.vgprs +1)*4 + shared_vgpr_count*8 does 4472 not exceed 256. For wavefront size 32 shared_vgpr_count must be 0. 4473 31:4 28 Reserved, must be 0. 4474 bits 4475 32 **Total size 4 bytes.** 4476 ======= =================================================================================================================== 4477 4478.. 4479 4480 .. table:: Floating Point Rounding Mode Enumeration Values 4481 :name: amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table 4482 4483 ====================================== ===== ============================== 4484 Enumeration Name Value Description 4485 ====================================== ===== ============================== 4486 FLOAT_ROUND_MODE_NEAR_EVEN 0 Round Ties To Even 4487 FLOAT_ROUND_MODE_PLUS_INFINITY 1 Round Toward +infinity 4488 FLOAT_ROUND_MODE_MINUS_INFINITY 2 Round Toward -infinity 4489 FLOAT_ROUND_MODE_ZERO 3 Round Toward 0 4490 ====================================== ===== ============================== 4491 4492.. 4493 4494 .. table:: Floating Point Denorm Mode Enumeration Values 4495 :name: amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table 4496 4497 ====================================== ===== ============================== 4498 Enumeration Name Value Description 4499 ====================================== ===== ============================== 4500 FLOAT_DENORM_MODE_FLUSH_SRC_DST 0 Flush Source and Destination 4501 Denorms 4502 FLOAT_DENORM_MODE_FLUSH_DST 1 Flush Output Denorms 4503 FLOAT_DENORM_MODE_FLUSH_SRC 2 Flush Source Denorms 4504 FLOAT_DENORM_MODE_FLUSH_NONE 3 No Flush 4505 ====================================== ===== ============================== 4506 4507.. 4508 4509 .. table:: System VGPR Work-Item ID Enumeration Values 4510 :name: amdgpu-amdhsa-system-vgpr-work-item-id-enumeration-values-table 4511 4512 ======================================== ===== ============================ 4513 Enumeration Name Value Description 4514 ======================================== ===== ============================ 4515 SYSTEM_VGPR_WORKITEM_ID_X 0 Set work-item X dimension 4516 ID. 4517 SYSTEM_VGPR_WORKITEM_ID_X_Y 1 Set work-item X and Y 4518 dimensions ID. 4519 SYSTEM_VGPR_WORKITEM_ID_X_Y_Z 2 Set work-item X, Y and Z 4520 dimensions ID. 4521 SYSTEM_VGPR_WORKITEM_ID_UNDEFINED 3 Undefined. 4522 ======================================== ===== ============================ 4523 4524.. _amdgpu-amdhsa-initial-kernel-execution-state: 4525 4526Initial Kernel Execution State 4527~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 4528 4529This section defines the register state that will be set up by the packet 4530processor prior to the start of execution of every wavefront. This is limited by 4531the constraints of the hardware controllers of CP/ADC/SPI. 4532 4533The order of the SGPR registers is defined, but the compiler can specify which 4534ones are actually setup in the kernel descriptor using the ``enable_sgpr_*`` bit 4535fields (see :ref:`amdgpu-amdhsa-kernel-descriptor`). The register numbers used 4536for enabled registers are dense starting at SGPR0: the first enabled register is 4537SGPR0, the next enabled register is SGPR1 etc.; disabled registers do not have 4538an SGPR number. 4539 4540The initial SGPRs comprise up to 16 User SRGPs that are set by CP and apply to 4541all wavefronts of the grid. It is possible to specify more than 16 User SGPRs 4542using the ``enable_sgpr_*`` bit fields, in which case only the first 16 are 4543actually initialized. These are then immediately followed by the System SGPRs 4544that are set up by ADC/SPI and can have different values for each wavefront of 4545the grid dispatch. 4546 4547SGPR register initial state is defined in 4548:ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 4549 4550 .. table:: SGPR Register Set Up Order 4551 :name: amdgpu-amdhsa-sgpr-register-set-up-order-table 4552 4553 ========== ========================== ====== ============================== 4554 SGPR Order Name Number Description 4555 (kernel descriptor enable of 4556 field) SGPRs 4557 ========== ========================== ====== ============================== 4558 First Private Segment Buffer 4 See 4559 (enable_sgpr_private :ref:`amdgpu-amdhsa-kernel-prolog-private-segment-buffer`. 4560 _segment_buffer) 4561 then Dispatch Ptr 2 64-bit address of AQL dispatch 4562 (enable_sgpr_dispatch_ptr) packet for kernel dispatch 4563 actually executing. 4564 then Queue Ptr 2 64-bit address of amd_queue_t 4565 (enable_sgpr_queue_ptr) object for AQL queue on which 4566 the dispatch packet was 4567 queued. 4568 then Kernarg Segment Ptr 2 64-bit address of Kernarg 4569 (enable_sgpr_kernarg segment. This is directly 4570 _segment_ptr) copied from the 4571 kernarg_address in the kernel 4572 dispatch packet. 4573 4574 Having CP load it once avoids 4575 loading it at the beginning of 4576 every wavefront. 4577 then Dispatch Id 2 64-bit Dispatch ID of the 4578 (enable_sgpr_dispatch_id) dispatch packet being 4579 executed. 4580 then Flat Scratch Init 2 See 4581 (enable_sgpr_flat_scratch :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`. 4582 _init) 4583 then Private Segment Size 1 The 32-bit byte size of a 4584 (enable_sgpr_private single work-item's memory 4585 _segment_size) allocation. This is the 4586 value from the kernel 4587 dispatch packet Private 4588 Segment Byte Size rounded up 4589 by CP to a multiple of 4590 DWORD. 4591 4592 Having CP load it once avoids 4593 loading it at the beginning of 4594 every wavefront. 4595 4596 This is not used for 4597 GFX7-GFX8 since it is the same 4598 value as the second SGPR of 4599 Flat Scratch Init. However, it 4600 may be needed for GFX9-GFX10 which 4601 changes the meaning of the 4602 Flat Scratch Init value. 4603 then Work-Group Id X 1 32-bit work-group id in X 4604 (enable_sgpr_workgroup_id dimension of grid for 4605 _X) wavefront. 4606 then Work-Group Id Y 1 32-bit work-group id in Y 4607 (enable_sgpr_workgroup_id dimension of grid for 4608 _Y) wavefront. 4609 then Work-Group Id Z 1 32-bit work-group id in Z 4610 (enable_sgpr_workgroup_id dimension of grid for 4611 _Z) wavefront. 4612 then Work-Group Info 1 {first_wavefront, 14'b0000, 4613 (enable_sgpr_workgroup ordered_append_term[10:0], 4614 _info) threadgroup_size_in_wavefronts[5:0]} 4615 then Scratch Wavefront Offset 1 See 4616 (enable_sgpr_private :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`. 4617 _segment_wavefront_offset) and 4618 :ref:`amdgpu-amdhsa-kernel-prolog-private-segment-buffer`. 4619 ========== ========================== ====== ============================== 4620 4621The order of the VGPR registers is defined, but the compiler can specify which 4622ones are actually setup in the kernel descriptor using the ``enable_vgpr*`` bit 4623fields (see :ref:`amdgpu-amdhsa-kernel-descriptor`). The register numbers used 4624for enabled registers are dense starting at VGPR0: the first enabled register is 4625VGPR0, the next enabled register is VGPR1 etc.; disabled registers do not have a 4626VGPR number. 4627 4628There are different methods used for the VGPR initial state: 4629 4630* Unless the *Target Properties* column of :ref:`amdgpu-processor-table` 4631 specifies otherwise, a separate VGPR register is used per work-item ID. The 4632 VGPR register initial state for this method is defined in 4633 :ref:`amdgpu-amdhsa-vgpr-register-set-up-order-for-unpacked-work-item-id-method-table`. 4634* If *Target Properties* column of :ref:`amdgpu-processor-table` 4635 specifies *Packed work-item IDs*, the initial value of VGPR0 register is used 4636 for all work-item IDs. The register layout for this method is defined in 4637 :ref:`amdgpu-amdhsa-register-layout-for-packed-work-item-id-method-table`. 4638 4639 .. table:: VGPR Register Set Up Order for Unpacked Work-Item ID Method 4640 :name: amdgpu-amdhsa-vgpr-register-set-up-order-for-unpacked-work-item-id-method-table 4641 4642 ========== ========================== ====== ============================== 4643 VGPR Order Name Number Description 4644 (kernel descriptor enable of 4645 field) VGPRs 4646 ========== ========================== ====== ============================== 4647 First Work-Item Id X 1 32-bit work-item id in X 4648 (Always initialized) dimension of work-group for 4649 wavefront lane. 4650 then Work-Item Id Y 1 32-bit work-item id in Y 4651 (enable_vgpr_workitem_id dimension of work-group for 4652 > 0) wavefront lane. 4653 then Work-Item Id Z 1 32-bit work-item id in Z 4654 (enable_vgpr_workitem_id dimension of work-group for 4655 > 1) wavefront lane. 4656 ========== ========================== ====== ============================== 4657 4658.. 4659 4660 .. table:: Register Layout for Packed Work-Item ID Method 4661 :name: amdgpu-amdhsa-register-layout-for-packed-work-item-id-method-table 4662 4663 ======= ======= ================ ========================================= 4664 Bits Size Field Name Description 4665 ======= ======= ================ ========================================= 4666 0:9 10 bits Work-Item Id X Work-item id in X 4667 dimension of work-group for 4668 wavefront lane. 4669 4670 Always initialized. 4671 4672 10:19 10 bits Work-Item Id Y Work-item id in Y 4673 dimension of work-group for 4674 wavefront lane. 4675 4676 Initialized if enable_vgpr_workitem_id > 4677 0, otherwise set to 0. 4678 20:29 10 bits Work-Item Id Z Work-item id in Z 4679 dimension of work-group for 4680 wavefront lane. 4681 4682 Initialized if enable_vgpr_workitem_id > 4683 1, otherwise set to 0. 4684 30:31 2 bits Reserved, set to 0. 4685 ======= ======= ================ ========================================= 4686 4687The setting of registers is done by GPU CP/ADC/SPI hardware as follows: 4688 46891. SGPRs before the Work-Group Ids are set by CP using the 16 User Data 4690 registers. 46912. Work-group Id registers X, Y, Z are set by ADC which supports any 4692 combination including none. 46933. Scratch Wavefront Offset is set by SPI in a per wavefront basis which is why 4694 its value cannot be included with the flat scratch init value which is per 4695 queue (see :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`). 46964. The VGPRs are set by SPI which only supports specifying either (X), (X, Y) 4697 or (X, Y, Z). 46985. Flat Scratch register pair initialization is described in 4699 :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`. 4700 4701The global segment can be accessed either using buffer instructions (GFX6 which 4702has V# 64-bit address support), flat instructions (GFX7-GFX10), or global 4703instructions (GFX9-GFX10). 4704 4705If buffer operations are used, then the compiler can generate a V# with the 4706following properties: 4707 4708* base address of 0 4709* no swizzle 4710* ATC: 1 if IOMMU present (such as APU) 4711* ptr64: 1 4712* MTYPE set to support memory coherence that matches the runtime (such as CC for 4713 APU and NC for dGPU). 4714 4715.. _amdgpu-amdhsa-kernel-prolog: 4716 4717Kernel Prolog 4718~~~~~~~~~~~~~ 4719 4720The compiler performs initialization in the kernel prologue depending on the 4721target and information about things like stack usage in the kernel and called 4722functions. Some of this initialization requires the compiler to request certain 4723User and System SGPRs be present in the 4724:ref:`amdgpu-amdhsa-initial-kernel-execution-state` via the 4725:ref:`amdgpu-amdhsa-kernel-descriptor`. 4726 4727.. _amdgpu-amdhsa-kernel-prolog-cfi: 4728 4729CFI 4730+++ 4731 47321. The CFI return address is undefined. 4733 47342. The CFI CFA is defined using an expression which evaluates to a location 4735 description that comprises one memory location description for the 4736 ``DW_ASPACE_AMDGPU_private_lane`` address space address ``0``. 4737 4738.. _amdgpu-amdhsa-kernel-prolog-m0: 4739 4740M0 4741++ 4742 4743GFX6-GFX8 4744 The M0 register must be initialized with a value at least the total LDS size 4745 if the kernel may access LDS via DS or flat operations. Total LDS size is 4746 available in dispatch packet. For M0, it is also possible to use maximum 4747 possible value of LDS for given target (0x7FFF for GFX6 and 0xFFFF for 4748 GFX7-GFX8). 4749GFX9-GFX10 4750 The M0 register is not used for range checking LDS accesses and so does not 4751 need to be initialized in the prolog. 4752 4753.. _amdgpu-amdhsa-kernel-prolog-stack-pointer: 4754 4755Stack Pointer 4756+++++++++++++ 4757 4758If the kernel has function calls it must set up the ABI stack pointer described 4759in :ref:`amdgpu-amdhsa-function-call-convention-non-kernel-functions` by setting 4760SGPR32 to the unswizzled scratch offset of the address past the last local 4761allocation. 4762 4763.. _amdgpu-amdhsa-kernel-prolog-frame-pointer: 4764 4765Frame Pointer 4766+++++++++++++ 4767 4768If the kernel needs a frame pointer for the reasons defined in 4769``SIFrameLowering`` then SGPR33 is used and is always set to ``0`` in the 4770kernel prolog. If a frame pointer is not required then all uses of the frame 4771pointer are replaced with immediate ``0`` offsets. 4772 4773.. _amdgpu-amdhsa-kernel-prolog-flat-scratch: 4774 4775Flat Scratch 4776++++++++++++ 4777 4778There are different methods used for initializing flat scratch: 4779 4780* If the *Target Properties* column of :ref:`amdgpu-processor-table` 4781 specifies *Does not support generic address space*: 4782 4783 Flat scratch is not supported and there is no flat scratch register pair. 4784 4785* If the *Target Properties* column of :ref:`amdgpu-processor-table` 4786 specifies *Offset flat scratch*: 4787 4788 If the kernel or any function it calls may use flat operations to access 4789 scratch memory, the prolog code must set up the FLAT_SCRATCH register pair 4790 (FLAT_SCRATCH_LO/FLAT_SCRATCH_HI). Initialization uses Flat Scratch Init and 4791 Scratch Wavefront Offset SGPR registers (see 4792 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`): 4793 4794 1. The low word of Flat Scratch Init is the 32-bit byte offset from 4795 ``SH_HIDDEN_PRIVATE_BASE_VIMID`` to the base of scratch backing memory 4796 being managed by SPI for the queue executing the kernel dispatch. This is 4797 the same value used in the Scratch Segment Buffer V# base address. 4798 4799 CP obtains this from the runtime. (The Scratch Segment Buffer base address 4800 is ``SH_HIDDEN_PRIVATE_BASE_VIMID`` plus this offset.) 4801 4802 The prolog must add the value of Scratch Wavefront Offset to get the 4803 wavefront's byte scratch backing memory offset from 4804 ``SH_HIDDEN_PRIVATE_BASE_VIMID``. 4805 4806 The Scratch Wavefront Offset must also be used as an offset with Private 4807 segment address when using the Scratch Segment Buffer. 4808 4809 Since FLAT_SCRATCH_LO is in units of 256 bytes, the offset must be right 4810 shifted by 8 before moving into FLAT_SCRATCH_HI. 4811 4812 FLAT_SCRATCH_HI corresponds to SGPRn-4 on GFX7, and SGPRn-6 on GFX8 (where 4813 SGPRn is the highest numbered SGPR allocated to the wavefront). 4814 FLAT_SCRATCH_HI is multiplied by 256 (as it is in units of 256 bytes) and 4815 added to ``SH_HIDDEN_PRIVATE_BASE_VIMID`` to calculate the per wavefront 4816 FLAT SCRATCH BASE in flat memory instructions that access the scratch 4817 aperture. 4818 2. The second word of Flat Scratch Init is 32-bit byte size of a single 4819 work-items scratch memory usage. 4820 4821 CP obtains this from the runtime, and it is always a multiple of DWORD. CP 4822 checks that the value in the kernel dispatch packet Private Segment Byte 4823 Size is not larger and requests the runtime to increase the queue's scratch 4824 size if necessary. 4825 4826 CP directly loads from the kernel dispatch packet Private Segment Byte Size 4827 field and rounds up to a multiple of DWORD. Having CP load it once avoids 4828 loading it at the beginning of every wavefront. 4829 4830 The kernel prolog code must move it to FLAT_SCRATCH_LO which is SGPRn-3 on 4831 GFX7 and SGPRn-5 on GFX8. FLAT_SCRATCH_LO is used as the FLAT SCRATCH SIZE 4832 in flat memory instructions. 4833 4834* If the *Target Properties* column of :ref:`amdgpu-processor-table` 4835 specifies *Absolute flat scratch*: 4836 4837 If the kernel or any function it calls may use flat operations to access 4838 scratch memory, the prolog code must set up the FLAT_SCRATCH register pair 4839 (FLAT_SCRATCH_LO/FLAT_SCRATCH_HI which are in SGPRn-4/SGPRn-3). Initialization 4840 uses Flat Scratch Init and Scratch Wavefront Offset SGPR registers (see 4841 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`): 4842 4843 The Flat Scratch Init is the 64-bit address of the base of scratch backing 4844 memory being managed by SPI for the queue executing the kernel dispatch. 4845 4846 CP obtains this from the runtime. 4847 4848 The kernel prolog must add the value of the wave's Scratch Wavefront Offset 4849 and move the result as a 64-bit value to the FLAT_SCRATCH SGPR register pair 4850 which is SGPRn-6 and SGPRn-5. It is used as the FLAT SCRATCH BASE in flat 4851 memory instructions. 4852 4853 The Scratch Wavefront Offset must also be used as an offset with Private 4854 segment address when using the Scratch Segment Buffer (see 4855 :ref:`amdgpu-amdhsa-kernel-prolog-private-segment-buffer`). 4856 4857* If the *Target Properties* column of :ref:`amdgpu-processor-table` 4858 specifies *Architected flat scratch*: 4859 4860 If ENABLE_PRIVATE_SEGMENT is enabled in 4861 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table` then the FLAT_SCRATCH 4862 register pair will be initialized to the 64-bit address of the base of scratch 4863 backing memory being managed by SPI for the queue executing the kernel 4864 dispatch plus the value of the wave's Scratch Wavefront Offset for use as the 4865 flat scratch base in flat memory instructions. 4866 4867.. _amdgpu-amdhsa-kernel-prolog-private-segment-buffer: 4868 4869Private Segment Buffer 4870++++++++++++++++++++++ 4871 4872If the *Target Properties* column of :ref:`amdgpu-processor-table` specifies 4873*Architected flat scratch* then a Private Segment Buffer is not supported. 4874Instead the flat SCRATCH instructions are used. 4875 4876Otherwise, Private Segment Buffer SGPR register is used to initialize 4 SGPRs 4877that are used as a V# to access scratch. CP uses the value provided by the 4878runtime. It is used, together with Scratch Wavefront Offset as an offset, to 4879access the private memory space using a segment address. See 4880:ref:`amdgpu-amdhsa-initial-kernel-execution-state`. 4881 4882The scratch V# is a four-aligned SGPR and always selected for the kernel as 4883follows: 4884 4885 - If it is known during instruction selection that there is stack usage, 4886 SGPR0-3 is reserved for use as the scratch V#. Stack usage is assumed if 4887 optimizations are disabled (``-O0``), if stack objects already exist (for 4888 locals, etc.), or if there are any function calls. 4889 4890 - Otherwise, four high numbered SGPRs beginning at a four-aligned SGPR index 4891 are reserved for the tentative scratch V#. These will be used if it is 4892 determined that spilling is needed. 4893 4894 - If no use is made of the tentative scratch V#, then it is unreserved, 4895 and the register count is determined ignoring it. 4896 - If use is made of the tentative scratch V#, then its register numbers 4897 are shifted to the first four-aligned SGPR index after the highest one 4898 allocated by the register allocator, and all uses are updated. The 4899 register count includes them in the shifted location. 4900 - In either case, if the processor has the SGPR allocation bug, the 4901 tentative allocation is not shifted or unreserved in order to ensure 4902 the register count is higher to workaround the bug. 4903 4904 .. note:: 4905 4906 This approach of using a tentative scratch V# and shifting the register 4907 numbers if used avoids having to perform register allocation a second 4908 time if the tentative V# is eliminated. This is more efficient and 4909 avoids the problem that the second register allocation may perform 4910 spilling which will fail as there is no longer a scratch V#. 4911 4912When the kernel prolog code is being emitted it is known whether the scratch V# 4913described above is actually used. If it is, the prolog code must set it up by 4914copying the Private Segment Buffer to the scratch V# registers and then adding 4915the Private Segment Wavefront Offset to the queue base address in the V#. The 4916result is a V# with a base address pointing to the beginning of the wavefront 4917scratch backing memory. 4918 4919The Private Segment Buffer is always requested, but the Private Segment 4920Wavefront Offset is only requested if it is used (see 4921:ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4922 4923.. _amdgpu-amdhsa-memory-model: 4924 4925Memory Model 4926~~~~~~~~~~~~ 4927 4928This section describes the mapping of the LLVM memory model onto AMDGPU machine 4929code (see :ref:`memmodel`). 4930 4931The AMDGPU backend supports the memory synchronization scopes specified in 4932:ref:`amdgpu-memory-scopes`. 4933 4934The code sequences used to implement the memory model specify the order of 4935instructions that a single thread must execute. The ``s_waitcnt`` and cache 4936management instructions such as ``buffer_wbinvl1_vol`` are defined with respect 4937to other memory instructions executed by the same thread. This allows them to be 4938moved earlier or later which can allow them to be combined with other instances 4939of the same instruction, or hoisted/sunk out of loops to improve performance. 4940Only the instructions related to the memory model are given; additional 4941``s_waitcnt`` instructions are required to ensure registers are defined before 4942being used. These may be able to be combined with the memory model ``s_waitcnt`` 4943instructions as described above. 4944 4945The AMDGPU backend supports the following memory models: 4946 4947 HSA Memory Model [HSA]_ 4948 The HSA memory model uses a single happens-before relation for all address 4949 spaces (see :ref:`amdgpu-address-spaces`). 4950 OpenCL Memory Model [OpenCL]_ 4951 The OpenCL memory model which has separate happens-before relations for the 4952 global and local address spaces. Only a fence specifying both global and 4953 local address space, and seq_cst instructions join the relationships. Since 4954 the LLVM ``memfence`` instruction does not allow an address space to be 4955 specified the OpenCL fence has to conservatively assume both local and 4956 global address space was specified. However, optimizations can often be 4957 done to eliminate the additional ``s_waitcnt`` instructions when there are 4958 no intervening memory instructions which access the corresponding address 4959 space. The code sequences in the table indicate what can be omitted for the 4960 OpenCL memory. The target triple environment is used to determine if the 4961 source language is OpenCL (see :ref:`amdgpu-opencl`). 4962 4963``ds/flat_load/store/atomic`` instructions to local memory are termed LDS 4964operations. 4965 4966``buffer/global/flat_load/store/atomic`` instructions to global memory are 4967termed vector memory operations. 4968 4969Private address space uses ``buffer_load/store`` using the scratch V# 4970(GFX6-GFX8), or ``scratch_load/store`` (GFX9-GFX10). Since only a single thread 4971is accessing the memory, atomic memory orderings are not meaningful, and all 4972accesses are treated as non-atomic. 4973 4974Constant address space uses ``buffer/global_load`` instructions (or equivalent 4975scalar memory instructions). Since the constant address space contents do not 4976change during the execution of a kernel dispatch it is not legal to perform 4977stores, and atomic memory orderings are not meaningful, and all accesses are 4978treated as non-atomic. 4979 4980A memory synchronization scope wider than work-group is not meaningful for the 4981group (LDS) address space and is treated as work-group. 4982 4983The memory model does not support the region address space which is treated as 4984non-atomic. 4985 4986Acquire memory ordering is not meaningful on store atomic instructions and is 4987treated as non-atomic. 4988 4989Release memory ordering is not meaningful on load atomic instructions and is 4990treated a non-atomic. 4991 4992Acquire-release memory ordering is not meaningful on load or store atomic 4993instructions and is treated as acquire and release respectively. 4994 4995The memory order also adds the single thread optimization constraints defined in 4996table 4997:ref:`amdgpu-amdhsa-memory-model-single-thread-optimization-constraints-table`. 4998 4999 .. table:: AMDHSA Memory Model Single Thread Optimization Constraints 5000 :name: amdgpu-amdhsa-memory-model-single-thread-optimization-constraints-table 5001 5002 ============ ============================================================== 5003 LLVM Memory Optimization Constraints 5004 Ordering 5005 ============ ============================================================== 5006 unordered *none* 5007 monotonic *none* 5008 acquire - If a load atomic/atomicrmw then no following load/load 5009 atomic/store/store atomic/atomicrmw/fence instruction can be 5010 moved before the acquire. 5011 - If a fence then same as load atomic, plus no preceding 5012 associated fence-paired-atomic can be moved after the fence. 5013 release - If a store atomic/atomicrmw then no preceding load/load 5014 atomic/store/store atomic/atomicrmw/fence instruction can be 5015 moved after the release. 5016 - If a fence then same as store atomic, plus no following 5017 associated fence-paired-atomic can be moved before the 5018 fence. 5019 acq_rel Same constraints as both acquire and release. 5020 seq_cst - If a load atomic then same constraints as acquire, plus no 5021 preceding sequentially consistent load atomic/store 5022 atomic/atomicrmw/fence instruction can be moved after the 5023 seq_cst. 5024 - If a store atomic then the same constraints as release, plus 5025 no following sequentially consistent load atomic/store 5026 atomic/atomicrmw/fence instruction can be moved before the 5027 seq_cst. 5028 - If an atomicrmw/fence then same constraints as acq_rel. 5029 ============ ============================================================== 5030 5031The code sequences used to implement the memory model are defined in the 5032following sections: 5033 5034* :ref:`amdgpu-amdhsa-memory-model-gfx6-gfx9` 5035* :ref:`amdgpu-amdhsa-memory-model-gfx90a` 5036* :ref:`amdgpu-amdhsa-memory-model-gfx940` 5037* :ref:`amdgpu-amdhsa-memory-model-gfx10` 5038 5039.. _amdgpu-amdhsa-memory-model-gfx6-gfx9: 5040 5041Memory Model GFX6-GFX9 5042++++++++++++++++++++++ 5043 5044For GFX6-GFX9: 5045 5046* Each agent has multiple shader arrays (SA). 5047* Each SA has multiple compute units (CU). 5048* Each CU has multiple SIMDs that execute wavefronts. 5049* The wavefronts for a single work-group are executed in the same CU but may be 5050 executed by different SIMDs. 5051* Each CU has a single LDS memory shared by the wavefronts of the work-groups 5052 executing on it. 5053* All LDS operations of a CU are performed as wavefront wide operations in a 5054 global order and involve no caching. Completion is reported to a wavefront in 5055 execution order. 5056* The LDS memory has multiple request queues shared by the SIMDs of a 5057 CU. Therefore, the LDS operations performed by different wavefronts of a 5058 work-group can be reordered relative to each other, which can result in 5059 reordering the visibility of vector memory operations with respect to LDS 5060 operations of other wavefronts in the same work-group. A ``s_waitcnt 5061 lgkmcnt(0)`` is required to ensure synchronization between LDS operations and 5062 vector memory operations between wavefronts of a work-group, but not between 5063 operations performed by the same wavefront. 5064* The vector memory operations are performed as wavefront wide operations and 5065 completion is reported to a wavefront in execution order. The exception is 5066 that for GFX7-GFX9 ``flat_load/store/atomic`` instructions can report out of 5067 vector memory order if they access LDS memory, and out of LDS operation order 5068 if they access global memory. 5069* The vector memory operations access a single vector L1 cache shared by all 5070 SIMDs a CU. Therefore, no special action is required for coherence between the 5071 lanes of a single wavefront, or for coherence between wavefronts in the same 5072 work-group. A ``buffer_wbinvl1_vol`` is required for coherence between 5073 wavefronts executing in different work-groups as they may be executing on 5074 different CUs. 5075* The scalar memory operations access a scalar L1 cache shared by all wavefronts 5076 on a group of CUs. The scalar and vector L1 caches are not coherent. However, 5077 scalar operations are used in a restricted way so do not impact the memory 5078 model. See :ref:`amdgpu-amdhsa-memory-spaces`. 5079* The vector and scalar memory operations use an L2 cache shared by all CUs on 5080 the same agent. 5081* The L2 cache has independent channels to service disjoint ranges of virtual 5082 addresses. 5083* Each CU has a separate request queue per channel. Therefore, the vector and 5084 scalar memory operations performed by wavefronts executing in different 5085 work-groups (which may be executing on different CUs) of an agent can be 5086 reordered relative to each other. A ``s_waitcnt vmcnt(0)`` is required to 5087 ensure synchronization between vector memory operations of different CUs. It 5088 ensures a previous vector memory operation has completed before executing a 5089 subsequent vector memory or LDS operation and so can be used to meet the 5090 requirements of acquire and release. 5091* The L2 cache can be kept coherent with other agents on some targets, or ranges 5092 of virtual addresses can be set up to bypass it to ensure system coherence. 5093 5094Scalar memory operations are only used to access memory that is proven to not 5095change during the execution of the kernel dispatch. This includes constant 5096address space and global address space for program scope ``const`` variables. 5097Therefore, the kernel machine code does not have to maintain the scalar cache to 5098ensure it is coherent with the vector caches. The scalar and vector caches are 5099invalidated between kernel dispatches by CP since constant address space data 5100may change between kernel dispatch executions. See 5101:ref:`amdgpu-amdhsa-memory-spaces`. 5102 5103The one exception is if scalar writes are used to spill SGPR registers. In this 5104case the AMDGPU backend ensures the memory location used to spill is never 5105accessed by vector memory operations at the same time. If scalar writes are used 5106then a ``s_dcache_wb`` is inserted before the ``s_endpgm`` and before a function 5107return since the locations may be used for vector memory instructions by a 5108future wavefront that uses the same scratch area, or a function call that 5109creates a frame at the same address, respectively. There is no need for a 5110``s_dcache_inv`` as all scalar writes are write-before-read in the same thread. 5111 5112For kernarg backing memory: 5113 5114* CP invalidates the L1 cache at the start of each kernel dispatch. 5115* On dGPU the kernarg backing memory is allocated in host memory accessed as 5116 MTYPE UC (uncached) to avoid needing to invalidate the L2 cache. This also 5117 causes it to be treated as non-volatile and so is not invalidated by 5118 ``*_vol``. 5119* On APU the kernarg backing memory it is accessed as MTYPE CC (cache coherent) 5120 and so the L2 cache will be coherent with the CPU and other agents. 5121 5122Scratch backing memory (which is used for the private address space) is accessed 5123with MTYPE NC_NV (non-coherent non-volatile). Since the private address space is 5124only accessed by a single thread, and is always write-before-read, there is 5125never a need to invalidate these entries from the L1 cache. Hence all cache 5126invalidates are done as ``*_vol`` to only invalidate the volatile cache lines. 5127 5128The code sequences used to implement the memory model for GFX6-GFX9 are defined 5129in table :ref:`amdgpu-amdhsa-memory-model-code-sequences-gfx6-gfx9-table`. 5130 5131 .. table:: AMDHSA Memory Model Code Sequences GFX6-GFX9 5132 :name: amdgpu-amdhsa-memory-model-code-sequences-gfx6-gfx9-table 5133 5134 ============ ============ ============== ========== ================================ 5135 LLVM Instr LLVM Memory LLVM Memory AMDGPU AMDGPU Machine Code 5136 Ordering Sync Scope Address GFX6-GFX9 5137 Space 5138 ============ ============ ============== ========== ================================ 5139 **Non-Atomic** 5140 ------------------------------------------------------------------------------------ 5141 load *none* *none* - global - !volatile & !nontemporal 5142 - generic 5143 - private 1. buffer/global/flat_load 5144 - constant 5145 - !volatile & nontemporal 5146 5147 1. buffer/global/flat_load 5148 glc=1 slc=1 5149 5150 - volatile 5151 5152 1. buffer/global/flat_load 5153 glc=1 5154 2. s_waitcnt vmcnt(0) 5155 5156 - Must happen before 5157 any following volatile 5158 global/generic 5159 load/store. 5160 - Ensures that 5161 volatile 5162 operations to 5163 different 5164 addresses will not 5165 be reordered by 5166 hardware. 5167 5168 load *none* *none* - local 1. ds_load 5169 store *none* *none* - global - !volatile & !nontemporal 5170 - generic 5171 - private 1. buffer/global/flat_store 5172 - constant 5173 - !volatile & nontemporal 5174 5175 1. buffer/global/flat_store 5176 glc=1 slc=1 5177 5178 - volatile 5179 5180 1. buffer/global/flat_store 5181 2. s_waitcnt vmcnt(0) 5182 5183 - Must happen before 5184 any following volatile 5185 global/generic 5186 load/store. 5187 - Ensures that 5188 volatile 5189 operations to 5190 different 5191 addresses will not 5192 be reordered by 5193 hardware. 5194 5195 store *none* *none* - local 1. ds_store 5196 **Unordered Atomic** 5197 ------------------------------------------------------------------------------------ 5198 load atomic unordered *any* *any* *Same as non-atomic*. 5199 store atomic unordered *any* *any* *Same as non-atomic*. 5200 atomicrmw unordered *any* *any* *Same as monotonic atomic*. 5201 **Monotonic Atomic** 5202 ------------------------------------------------------------------------------------ 5203 load atomic monotonic - singlethread - global 1. buffer/global/ds/flat_load 5204 - wavefront - local 5205 - workgroup - generic 5206 load atomic monotonic - agent - global 1. buffer/global/flat_load 5207 - system - generic glc=1 5208 store atomic monotonic - singlethread - global 1. buffer/global/flat_store 5209 - wavefront - generic 5210 - workgroup 5211 - agent 5212 - system 5213 store atomic monotonic - singlethread - local 1. ds_store 5214 - wavefront 5215 - workgroup 5216 atomicrmw monotonic - singlethread - global 1. buffer/global/flat_atomic 5217 - wavefront - generic 5218 - workgroup 5219 - agent 5220 - system 5221 atomicrmw monotonic - singlethread - local 1. ds_atomic 5222 - wavefront 5223 - workgroup 5224 **Acquire Atomic** 5225 ------------------------------------------------------------------------------------ 5226 load atomic acquire - singlethread - global 1. buffer/global/ds/flat_load 5227 - wavefront - local 5228 - generic 5229 load atomic acquire - workgroup - global 1. buffer/global_load 5230 load atomic acquire - workgroup - local 1. ds/flat_load 5231 - generic 2. s_waitcnt lgkmcnt(0) 5232 5233 - If OpenCL, omit. 5234 - Must happen before 5235 any following 5236 global/generic 5237 load/load 5238 atomic/store/store 5239 atomic/atomicrmw. 5240 - Ensures any 5241 following global 5242 data read is no 5243 older than a local load 5244 atomic value being 5245 acquired. 5246 5247 load atomic acquire - agent - global 1. buffer/global_load 5248 - system glc=1 5249 2. s_waitcnt vmcnt(0) 5250 5251 - Must happen before 5252 following 5253 buffer_wbinvl1_vol. 5254 - Ensures the load 5255 has completed 5256 before invalidating 5257 the cache. 5258 5259 3. buffer_wbinvl1_vol 5260 5261 - Must happen before 5262 any following 5263 global/generic 5264 load/load 5265 atomic/atomicrmw. 5266 - Ensures that 5267 following 5268 loads will not see 5269 stale global data. 5270 5271 load atomic acquire - agent - generic 1. flat_load glc=1 5272 - system 2. s_waitcnt vmcnt(0) & 5273 lgkmcnt(0) 5274 5275 - If OpenCL omit 5276 lgkmcnt(0). 5277 - Must happen before 5278 following 5279 buffer_wbinvl1_vol. 5280 - Ensures the flat_load 5281 has completed 5282 before invalidating 5283 the cache. 5284 5285 3. buffer_wbinvl1_vol 5286 5287 - Must happen before 5288 any following 5289 global/generic 5290 load/load 5291 atomic/atomicrmw. 5292 - Ensures that 5293 following loads 5294 will not see stale 5295 global data. 5296 5297 atomicrmw acquire - singlethread - global 1. buffer/global/ds/flat_atomic 5298 - wavefront - local 5299 - generic 5300 atomicrmw acquire - workgroup - global 1. buffer/global_atomic 5301 atomicrmw acquire - workgroup - local 1. ds/flat_atomic 5302 - generic 2. s_waitcnt lgkmcnt(0) 5303 5304 - If OpenCL, omit. 5305 - Must happen before 5306 any following 5307 global/generic 5308 load/load 5309 atomic/store/store 5310 atomic/atomicrmw. 5311 - Ensures any 5312 following global 5313 data read is no 5314 older than a local 5315 atomicrmw value 5316 being acquired. 5317 5318 atomicrmw acquire - agent - global 1. buffer/global_atomic 5319 - system 2. s_waitcnt vmcnt(0) 5320 5321 - Must happen before 5322 following 5323 buffer_wbinvl1_vol. 5324 - Ensures the 5325 atomicrmw has 5326 completed before 5327 invalidating the 5328 cache. 5329 5330 3. buffer_wbinvl1_vol 5331 5332 - Must happen before 5333 any following 5334 global/generic 5335 load/load 5336 atomic/atomicrmw. 5337 - Ensures that 5338 following loads 5339 will not see stale 5340 global data. 5341 5342 atomicrmw acquire - agent - generic 1. flat_atomic 5343 - system 2. s_waitcnt vmcnt(0) & 5344 lgkmcnt(0) 5345 5346 - If OpenCL, omit 5347 lgkmcnt(0). 5348 - Must happen before 5349 following 5350 buffer_wbinvl1_vol. 5351 - Ensures the 5352 atomicrmw has 5353 completed before 5354 invalidating the 5355 cache. 5356 5357 3. buffer_wbinvl1_vol 5358 5359 - Must happen before 5360 any following 5361 global/generic 5362 load/load 5363 atomic/atomicrmw. 5364 - Ensures that 5365 following loads 5366 will not see stale 5367 global data. 5368 5369 fence acquire - singlethread *none* *none* 5370 - wavefront 5371 fence acquire - workgroup *none* 1. s_waitcnt lgkmcnt(0) 5372 5373 - If OpenCL and 5374 address space is 5375 not generic, omit. 5376 - However, since LLVM 5377 currently has no 5378 address space on 5379 the fence need to 5380 conservatively 5381 always generate. If 5382 fence had an 5383 address space then 5384 set to address 5385 space of OpenCL 5386 fence flag, or to 5387 generic if both 5388 local and global 5389 flags are 5390 specified. 5391 - Must happen after 5392 any preceding 5393 local/generic load 5394 atomic/atomicrmw 5395 with an equal or 5396 wider sync scope 5397 and memory ordering 5398 stronger than 5399 unordered (this is 5400 termed the 5401 fence-paired-atomic). 5402 - Must happen before 5403 any following 5404 global/generic 5405 load/load 5406 atomic/store/store 5407 atomic/atomicrmw. 5408 - Ensures any 5409 following global 5410 data read is no 5411 older than the 5412 value read by the 5413 fence-paired-atomic. 5414 5415 fence acquire - agent *none* 1. s_waitcnt lgkmcnt(0) & 5416 - system vmcnt(0) 5417 5418 - If OpenCL and 5419 address space is 5420 not generic, omit 5421 lgkmcnt(0). 5422 - However, since LLVM 5423 currently has no 5424 address space on 5425 the fence need to 5426 conservatively 5427 always generate 5428 (see comment for 5429 previous fence). 5430 - Could be split into 5431 separate s_waitcnt 5432 vmcnt(0) and 5433 s_waitcnt 5434 lgkmcnt(0) to allow 5435 them to be 5436 independently moved 5437 according to the 5438 following rules. 5439 - s_waitcnt vmcnt(0) 5440 must happen after 5441 any preceding 5442 global/generic load 5443 atomic/atomicrmw 5444 with an equal or 5445 wider sync scope 5446 and memory ordering 5447 stronger than 5448 unordered (this is 5449 termed the 5450 fence-paired-atomic). 5451 - s_waitcnt lgkmcnt(0) 5452 must happen after 5453 any preceding 5454 local/generic load 5455 atomic/atomicrmw 5456 with an equal or 5457 wider sync scope 5458 and memory ordering 5459 stronger than 5460 unordered (this is 5461 termed the 5462 fence-paired-atomic). 5463 - Must happen before 5464 the following 5465 buffer_wbinvl1_vol. 5466 - Ensures that the 5467 fence-paired atomic 5468 has completed 5469 before invalidating 5470 the 5471 cache. Therefore 5472 any following 5473 locations read must 5474 be no older than 5475 the value read by 5476 the 5477 fence-paired-atomic. 5478 5479 2. buffer_wbinvl1_vol 5480 5481 - Must happen before any 5482 following global/generic 5483 load/load 5484 atomic/store/store 5485 atomic/atomicrmw. 5486 - Ensures that 5487 following loads 5488 will not see stale 5489 global data. 5490 5491 **Release Atomic** 5492 ------------------------------------------------------------------------------------ 5493 store atomic release - singlethread - global 1. buffer/global/ds/flat_store 5494 - wavefront - local 5495 - generic 5496 store atomic release - workgroup - global 1. s_waitcnt lgkmcnt(0) 5497 - generic 5498 - If OpenCL, omit. 5499 - Must happen after 5500 any preceding 5501 local/generic 5502 load/store/load 5503 atomic/store 5504 atomic/atomicrmw. 5505 - Must happen before 5506 the following 5507 store. 5508 - Ensures that all 5509 memory operations 5510 to local have 5511 completed before 5512 performing the 5513 store that is being 5514 released. 5515 5516 2. buffer/global/flat_store 5517 store atomic release - workgroup - local 1. ds_store 5518 store atomic release - agent - global 1. s_waitcnt lgkmcnt(0) & 5519 - system - generic vmcnt(0) 5520 5521 - If OpenCL and 5522 address space is 5523 not generic, omit 5524 lgkmcnt(0). 5525 - Could be split into 5526 separate s_waitcnt 5527 vmcnt(0) and 5528 s_waitcnt 5529 lgkmcnt(0) to allow 5530 them to be 5531 independently moved 5532 according to the 5533 following rules. 5534 - s_waitcnt vmcnt(0) 5535 must happen after 5536 any preceding 5537 global/generic 5538 load/store/load 5539 atomic/store 5540 atomic/atomicrmw. 5541 - s_waitcnt lgkmcnt(0) 5542 must happen after 5543 any preceding 5544 local/generic 5545 load/store/load 5546 atomic/store 5547 atomic/atomicrmw. 5548 - Must happen before 5549 the following 5550 store. 5551 - Ensures that all 5552 memory operations 5553 to memory have 5554 completed before 5555 performing the 5556 store that is being 5557 released. 5558 5559 2. buffer/global/flat_store 5560 atomicrmw release - singlethread - global 1. buffer/global/ds/flat_atomic 5561 - wavefront - local 5562 - generic 5563 atomicrmw release - workgroup - global 1. s_waitcnt lgkmcnt(0) 5564 - generic 5565 - If OpenCL, omit. 5566 - Must happen after 5567 any preceding 5568 local/generic 5569 load/store/load 5570 atomic/store 5571 atomic/atomicrmw. 5572 - Must happen before 5573 the following 5574 atomicrmw. 5575 - Ensures that all 5576 memory operations 5577 to local have 5578 completed before 5579 performing the 5580 atomicrmw that is 5581 being released. 5582 5583 2. buffer/global/flat_atomic 5584 atomicrmw release - workgroup - local 1. ds_atomic 5585 atomicrmw release - agent - global 1. s_waitcnt lgkmcnt(0) & 5586 - system - generic vmcnt(0) 5587 5588 - If OpenCL, omit 5589 lgkmcnt(0). 5590 - Could be split into 5591 separate s_waitcnt 5592 vmcnt(0) and 5593 s_waitcnt 5594 lgkmcnt(0) to allow 5595 them to be 5596 independently moved 5597 according to the 5598 following rules. 5599 - s_waitcnt vmcnt(0) 5600 must happen after 5601 any preceding 5602 global/generic 5603 load/store/load 5604 atomic/store 5605 atomic/atomicrmw. 5606 - s_waitcnt lgkmcnt(0) 5607 must happen after 5608 any preceding 5609 local/generic 5610 load/store/load 5611 atomic/store 5612 atomic/atomicrmw. 5613 - Must happen before 5614 the following 5615 atomicrmw. 5616 - Ensures that all 5617 memory operations 5618 to global and local 5619 have completed 5620 before performing 5621 the atomicrmw that 5622 is being released. 5623 5624 2. buffer/global/flat_atomic 5625 fence release - singlethread *none* *none* 5626 - wavefront 5627 fence release - workgroup *none* 1. s_waitcnt lgkmcnt(0) 5628 5629 - If OpenCL and 5630 address space is 5631 not generic, omit. 5632 - However, since LLVM 5633 currently has no 5634 address space on 5635 the fence need to 5636 conservatively 5637 always generate. If 5638 fence had an 5639 address space then 5640 set to address 5641 space of OpenCL 5642 fence flag, or to 5643 generic if both 5644 local and global 5645 flags are 5646 specified. 5647 - Must happen after 5648 any preceding 5649 local/generic 5650 load/load 5651 atomic/store/store 5652 atomic/atomicrmw. 5653 - Must happen before 5654 any following store 5655 atomic/atomicrmw 5656 with an equal or 5657 wider sync scope 5658 and memory ordering 5659 stronger than 5660 unordered (this is 5661 termed the 5662 fence-paired-atomic). 5663 - Ensures that all 5664 memory operations 5665 to local have 5666 completed before 5667 performing the 5668 following 5669 fence-paired-atomic. 5670 5671 fence release - agent *none* 1. s_waitcnt lgkmcnt(0) & 5672 - system vmcnt(0) 5673 5674 - If OpenCL and 5675 address space is 5676 not generic, omit 5677 lgkmcnt(0). 5678 - If OpenCL and 5679 address space is 5680 local, omit 5681 vmcnt(0). 5682 - However, since LLVM 5683 currently has no 5684 address space on 5685 the fence need to 5686 conservatively 5687 always generate. If 5688 fence had an 5689 address space then 5690 set to address 5691 space of OpenCL 5692 fence flag, or to 5693 generic if both 5694 local and global 5695 flags are 5696 specified. 5697 - Could be split into 5698 separate s_waitcnt 5699 vmcnt(0) and 5700 s_waitcnt 5701 lgkmcnt(0) to allow 5702 them to be 5703 independently moved 5704 according to the 5705 following rules. 5706 - s_waitcnt vmcnt(0) 5707 must happen after 5708 any preceding 5709 global/generic 5710 load/store/load 5711 atomic/store 5712 atomic/atomicrmw. 5713 - s_waitcnt lgkmcnt(0) 5714 must happen after 5715 any preceding 5716 local/generic 5717 load/store/load 5718 atomic/store 5719 atomic/atomicrmw. 5720 - Must happen before 5721 any following store 5722 atomic/atomicrmw 5723 with an equal or 5724 wider sync scope 5725 and memory ordering 5726 stronger than 5727 unordered (this is 5728 termed the 5729 fence-paired-atomic). 5730 - Ensures that all 5731 memory operations 5732 have 5733 completed before 5734 performing the 5735 following 5736 fence-paired-atomic. 5737 5738 **Acquire-Release Atomic** 5739 ------------------------------------------------------------------------------------ 5740 atomicrmw acq_rel - singlethread - global 1. buffer/global/ds/flat_atomic 5741 - wavefront - local 5742 - generic 5743 atomicrmw acq_rel - workgroup - global 1. s_waitcnt lgkmcnt(0) 5744 5745 - If OpenCL, omit. 5746 - Must happen after 5747 any preceding 5748 local/generic 5749 load/store/load 5750 atomic/store 5751 atomic/atomicrmw. 5752 - Must happen before 5753 the following 5754 atomicrmw. 5755 - Ensures that all 5756 memory operations 5757 to local have 5758 completed before 5759 performing the 5760 atomicrmw that is 5761 being released. 5762 5763 2. buffer/global_atomic 5764 5765 atomicrmw acq_rel - workgroup - local 1. ds_atomic 5766 2. s_waitcnt lgkmcnt(0) 5767 5768 - If OpenCL, omit. 5769 - Must happen before 5770 any following 5771 global/generic 5772 load/load 5773 atomic/store/store 5774 atomic/atomicrmw. 5775 - Ensures any 5776 following global 5777 data read is no 5778 older than the local load 5779 atomic value being 5780 acquired. 5781 5782 atomicrmw acq_rel - workgroup - generic 1. s_waitcnt lgkmcnt(0) 5783 5784 - If OpenCL, omit. 5785 - Must happen after 5786 any preceding 5787 local/generic 5788 load/store/load 5789 atomic/store 5790 atomic/atomicrmw. 5791 - Must happen before 5792 the following 5793 atomicrmw. 5794 - Ensures that all 5795 memory operations 5796 to local have 5797 completed before 5798 performing the 5799 atomicrmw that is 5800 being released. 5801 5802 2. flat_atomic 5803 3. s_waitcnt lgkmcnt(0) 5804 5805 - If OpenCL, omit. 5806 - Must happen before 5807 any following 5808 global/generic 5809 load/load 5810 atomic/store/store 5811 atomic/atomicrmw. 5812 - Ensures any 5813 following global 5814 data read is no 5815 older than a local load 5816 atomic value being 5817 acquired. 5818 5819 atomicrmw acq_rel - agent - global 1. s_waitcnt lgkmcnt(0) & 5820 - system vmcnt(0) 5821 5822 - If OpenCL, omit 5823 lgkmcnt(0). 5824 - Could be split into 5825 separate s_waitcnt 5826 vmcnt(0) and 5827 s_waitcnt 5828 lgkmcnt(0) to allow 5829 them to be 5830 independently moved 5831 according to the 5832 following rules. 5833 - s_waitcnt vmcnt(0) 5834 must happen after 5835 any preceding 5836 global/generic 5837 load/store/load 5838 atomic/store 5839 atomic/atomicrmw. 5840 - s_waitcnt lgkmcnt(0) 5841 must happen after 5842 any preceding 5843 local/generic 5844 load/store/load 5845 atomic/store 5846 atomic/atomicrmw. 5847 - Must happen before 5848 the following 5849 atomicrmw. 5850 - Ensures that all 5851 memory operations 5852 to global have 5853 completed before 5854 performing the 5855 atomicrmw that is 5856 being released. 5857 5858 2. buffer/global_atomic 5859 3. s_waitcnt vmcnt(0) 5860 5861 - Must happen before 5862 following 5863 buffer_wbinvl1_vol. 5864 - Ensures the 5865 atomicrmw has 5866 completed before 5867 invalidating the 5868 cache. 5869 5870 4. buffer_wbinvl1_vol 5871 5872 - Must happen before 5873 any following 5874 global/generic 5875 load/load 5876 atomic/atomicrmw. 5877 - Ensures that 5878 following loads 5879 will not see stale 5880 global data. 5881 5882 atomicrmw acq_rel - agent - generic 1. s_waitcnt lgkmcnt(0) & 5883 - system vmcnt(0) 5884 5885 - If OpenCL, omit 5886 lgkmcnt(0). 5887 - Could be split into 5888 separate s_waitcnt 5889 vmcnt(0) and 5890 s_waitcnt 5891 lgkmcnt(0) to allow 5892 them to be 5893 independently moved 5894 according to the 5895 following rules. 5896 - s_waitcnt vmcnt(0) 5897 must happen after 5898 any preceding 5899 global/generic 5900 load/store/load 5901 atomic/store 5902 atomic/atomicrmw. 5903 - s_waitcnt lgkmcnt(0) 5904 must happen after 5905 any preceding 5906 local/generic 5907 load/store/load 5908 atomic/store 5909 atomic/atomicrmw. 5910 - Must happen before 5911 the following 5912 atomicrmw. 5913 - Ensures that all 5914 memory operations 5915 to global have 5916 completed before 5917 performing the 5918 atomicrmw that is 5919 being released. 5920 5921 2. flat_atomic 5922 3. s_waitcnt vmcnt(0) & 5923 lgkmcnt(0) 5924 5925 - If OpenCL, omit 5926 lgkmcnt(0). 5927 - Must happen before 5928 following 5929 buffer_wbinvl1_vol. 5930 - Ensures the 5931 atomicrmw has 5932 completed before 5933 invalidating the 5934 cache. 5935 5936 4. buffer_wbinvl1_vol 5937 5938 - Must happen before 5939 any following 5940 global/generic 5941 load/load 5942 atomic/atomicrmw. 5943 - Ensures that 5944 following loads 5945 will not see stale 5946 global data. 5947 5948 fence acq_rel - singlethread *none* *none* 5949 - wavefront 5950 fence acq_rel - workgroup *none* 1. s_waitcnt lgkmcnt(0) 5951 5952 - If OpenCL and 5953 address space is 5954 not generic, omit. 5955 - However, 5956 since LLVM 5957 currently has no 5958 address space on 5959 the fence need to 5960 conservatively 5961 always generate 5962 (see comment for 5963 previous fence). 5964 - Must happen after 5965 any preceding 5966 local/generic 5967 load/load 5968 atomic/store/store 5969 atomic/atomicrmw. 5970 - Must happen before 5971 any following 5972 global/generic 5973 load/load 5974 atomic/store/store 5975 atomic/atomicrmw. 5976 - Ensures that all 5977 memory operations 5978 to local have 5979 completed before 5980 performing any 5981 following global 5982 memory operations. 5983 - Ensures that the 5984 preceding 5985 local/generic load 5986 atomic/atomicrmw 5987 with an equal or 5988 wider sync scope 5989 and memory ordering 5990 stronger than 5991 unordered (this is 5992 termed the 5993 acquire-fence-paired-atomic) 5994 has completed 5995 before following 5996 global memory 5997 operations. This 5998 satisfies the 5999 requirements of 6000 acquire. 6001 - Ensures that all 6002 previous memory 6003 operations have 6004 completed before a 6005 following 6006 local/generic store 6007 atomic/atomicrmw 6008 with an equal or 6009 wider sync scope 6010 and memory ordering 6011 stronger than 6012 unordered (this is 6013 termed the 6014 release-fence-paired-atomic). 6015 This satisfies the 6016 requirements of 6017 release. 6018 6019 fence acq_rel - agent *none* 1. s_waitcnt lgkmcnt(0) & 6020 - system vmcnt(0) 6021 6022 - If OpenCL and 6023 address space is 6024 not generic, omit 6025 lgkmcnt(0). 6026 - However, since LLVM 6027 currently has no 6028 address space on 6029 the fence need to 6030 conservatively 6031 always generate 6032 (see comment for 6033 previous fence). 6034 - Could be split into 6035 separate s_waitcnt 6036 vmcnt(0) and 6037 s_waitcnt 6038 lgkmcnt(0) to allow 6039 them to be 6040 independently moved 6041 according to the 6042 following rules. 6043 - s_waitcnt vmcnt(0) 6044 must happen after 6045 any preceding 6046 global/generic 6047 load/store/load 6048 atomic/store 6049 atomic/atomicrmw. 6050 - s_waitcnt lgkmcnt(0) 6051 must happen after 6052 any preceding 6053 local/generic 6054 load/store/load 6055 atomic/store 6056 atomic/atomicrmw. 6057 - Must happen before 6058 the following 6059 buffer_wbinvl1_vol. 6060 - Ensures that the 6061 preceding 6062 global/local/generic 6063 load 6064 atomic/atomicrmw 6065 with an equal or 6066 wider sync scope 6067 and memory ordering 6068 stronger than 6069 unordered (this is 6070 termed the 6071 acquire-fence-paired-atomic) 6072 has completed 6073 before invalidating 6074 the cache. This 6075 satisfies the 6076 requirements of 6077 acquire. 6078 - Ensures that all 6079 previous memory 6080 operations have 6081 completed before a 6082 following 6083 global/local/generic 6084 store 6085 atomic/atomicrmw 6086 with an equal or 6087 wider sync scope 6088 and memory ordering 6089 stronger than 6090 unordered (this is 6091 termed the 6092 release-fence-paired-atomic). 6093 This satisfies the 6094 requirements of 6095 release. 6096 6097 2. buffer_wbinvl1_vol 6098 6099 - Must happen before 6100 any following 6101 global/generic 6102 load/load 6103 atomic/store/store 6104 atomic/atomicrmw. 6105 - Ensures that 6106 following loads 6107 will not see stale 6108 global data. This 6109 satisfies the 6110 requirements of 6111 acquire. 6112 6113 **Sequential Consistent Atomic** 6114 ------------------------------------------------------------------------------------ 6115 load atomic seq_cst - singlethread - global *Same as corresponding 6116 - wavefront - local load atomic acquire, 6117 - generic except must generate 6118 all instructions even 6119 for OpenCL.* 6120 load atomic seq_cst - workgroup - global 1. s_waitcnt lgkmcnt(0) 6121 - generic 6122 6123 - Must 6124 happen after 6125 preceding 6126 local/generic load 6127 atomic/store 6128 atomic/atomicrmw 6129 with memory 6130 ordering of seq_cst 6131 and with equal or 6132 wider sync scope. 6133 (Note that seq_cst 6134 fences have their 6135 own s_waitcnt 6136 lgkmcnt(0) and so do 6137 not need to be 6138 considered.) 6139 - Ensures any 6140 preceding 6141 sequential 6142 consistent local 6143 memory instructions 6144 have completed 6145 before executing 6146 this sequentially 6147 consistent 6148 instruction. This 6149 prevents reordering 6150 a seq_cst store 6151 followed by a 6152 seq_cst load. (Note 6153 that seq_cst is 6154 stronger than 6155 acquire/release as 6156 the reordering of 6157 load acquire 6158 followed by a store 6159 release is 6160 prevented by the 6161 s_waitcnt of 6162 the release, but 6163 there is nothing 6164 preventing a store 6165 release followed by 6166 load acquire from 6167 completing out of 6168 order. The s_waitcnt 6169 could be placed after 6170 seq_store or before 6171 the seq_load. We 6172 choose the load to 6173 make the s_waitcnt be 6174 as late as possible 6175 so that the store 6176 may have already 6177 completed.) 6178 6179 2. *Following 6180 instructions same as 6181 corresponding load 6182 atomic acquire, 6183 except must generate 6184 all instructions even 6185 for OpenCL.* 6186 load atomic seq_cst - workgroup - local *Same as corresponding 6187 load atomic acquire, 6188 except must generate 6189 all instructions even 6190 for OpenCL.* 6191 6192 load atomic seq_cst - agent - global 1. s_waitcnt lgkmcnt(0) & 6193 - system - generic vmcnt(0) 6194 6195 - Could be split into 6196 separate s_waitcnt 6197 vmcnt(0) 6198 and s_waitcnt 6199 lgkmcnt(0) to allow 6200 them to be 6201 independently moved 6202 according to the 6203 following rules. 6204 - s_waitcnt lgkmcnt(0) 6205 must happen after 6206 preceding 6207 global/generic load 6208 atomic/store 6209 atomic/atomicrmw 6210 with memory 6211 ordering of seq_cst 6212 and with equal or 6213 wider sync scope. 6214 (Note that seq_cst 6215 fences have their 6216 own s_waitcnt 6217 lgkmcnt(0) and so do 6218 not need to be 6219 considered.) 6220 - s_waitcnt vmcnt(0) 6221 must happen after 6222 preceding 6223 global/generic load 6224 atomic/store 6225 atomic/atomicrmw 6226 with memory 6227 ordering of seq_cst 6228 and with equal or 6229 wider sync scope. 6230 (Note that seq_cst 6231 fences have their 6232 own s_waitcnt 6233 vmcnt(0) and so do 6234 not need to be 6235 considered.) 6236 - Ensures any 6237 preceding 6238 sequential 6239 consistent global 6240 memory instructions 6241 have completed 6242 before executing 6243 this sequentially 6244 consistent 6245 instruction. This 6246 prevents reordering 6247 a seq_cst store 6248 followed by a 6249 seq_cst load. (Note 6250 that seq_cst is 6251 stronger than 6252 acquire/release as 6253 the reordering of 6254 load acquire 6255 followed by a store 6256 release is 6257 prevented by the 6258 s_waitcnt of 6259 the release, but 6260 there is nothing 6261 preventing a store 6262 release followed by 6263 load acquire from 6264 completing out of 6265 order. The s_waitcnt 6266 could be placed after 6267 seq_store or before 6268 the seq_load. We 6269 choose the load to 6270 make the s_waitcnt be 6271 as late as possible 6272 so that the store 6273 may have already 6274 completed.) 6275 6276 2. *Following 6277 instructions same as 6278 corresponding load 6279 atomic acquire, 6280 except must generate 6281 all instructions even 6282 for OpenCL.* 6283 store atomic seq_cst - singlethread - global *Same as corresponding 6284 - wavefront - local store atomic release, 6285 - workgroup - generic except must generate 6286 - agent all instructions even 6287 - system for OpenCL.* 6288 atomicrmw seq_cst - singlethread - global *Same as corresponding 6289 - wavefront - local atomicrmw acq_rel, 6290 - workgroup - generic except must generate 6291 - agent all instructions even 6292 - system for OpenCL.* 6293 fence seq_cst - singlethread *none* *Same as corresponding 6294 - wavefront fence acq_rel, 6295 - workgroup except must generate 6296 - agent all instructions even 6297 - system for OpenCL.* 6298 ============ ============ ============== ========== ================================ 6299 6300.. _amdgpu-amdhsa-memory-model-gfx90a: 6301 6302Memory Model GFX90A 6303+++++++++++++++++++ 6304 6305For GFX90A: 6306 6307* Each agent has multiple shader arrays (SA). 6308* Each SA has multiple compute units (CU). 6309* Each CU has multiple SIMDs that execute wavefronts. 6310* The wavefronts for a single work-group are executed in the same CU but may be 6311 executed by different SIMDs. The exception is when in tgsplit execution mode 6312 when the wavefronts may be executed by different SIMDs in different CUs. 6313* Each CU has a single LDS memory shared by the wavefronts of the work-groups 6314 executing on it. The exception is when in tgsplit execution mode when no LDS 6315 is allocated as wavefronts of the same work-group can be in different CUs. 6316* All LDS operations of a CU are performed as wavefront wide operations in a 6317 global order and involve no caching. Completion is reported to a wavefront in 6318 execution order. 6319* The LDS memory has multiple request queues shared by the SIMDs of a 6320 CU. Therefore, the LDS operations performed by different wavefronts of a 6321 work-group can be reordered relative to each other, which can result in 6322 reordering the visibility of vector memory operations with respect to LDS 6323 operations of other wavefronts in the same work-group. A ``s_waitcnt 6324 lgkmcnt(0)`` is required to ensure synchronization between LDS operations and 6325 vector memory operations between wavefronts of a work-group, but not between 6326 operations performed by the same wavefront. 6327* The vector memory operations are performed as wavefront wide operations and 6328 completion is reported to a wavefront in execution order. The exception is 6329 that ``flat_load/store/atomic`` instructions can report out of vector memory 6330 order if they access LDS memory, and out of LDS operation order if they access 6331 global memory. 6332* The vector memory operations access a single vector L1 cache shared by all 6333 SIMDs a CU. Therefore: 6334 6335 * No special action is required for coherence between the lanes of a single 6336 wavefront. 6337 6338 * No special action is required for coherence between wavefronts in the same 6339 work-group since they execute on the same CU. The exception is when in 6340 tgsplit execution mode as wavefronts of the same work-group can be in 6341 different CUs and so a ``buffer_wbinvl1_vol`` is required as described in 6342 the following item. 6343 6344 * A ``buffer_wbinvl1_vol`` is required for coherence between wavefronts 6345 executing in different work-groups as they may be executing on different 6346 CUs. 6347 6348* The scalar memory operations access a scalar L1 cache shared by all wavefronts 6349 on a group of CUs. The scalar and vector L1 caches are not coherent. However, 6350 scalar operations are used in a restricted way so do not impact the memory 6351 model. See :ref:`amdgpu-amdhsa-memory-spaces`. 6352* The vector and scalar memory operations use an L2 cache shared by all CUs on 6353 the same agent. 6354 6355 * The L2 cache has independent channels to service disjoint ranges of virtual 6356 addresses. 6357 * Each CU has a separate request queue per channel. Therefore, the vector and 6358 scalar memory operations performed by wavefronts executing in different 6359 work-groups (which may be executing on different CUs), or the same 6360 work-group if executing in tgsplit mode, of an agent can be reordered 6361 relative to each other. A ``s_waitcnt vmcnt(0)`` is required to ensure 6362 synchronization between vector memory operations of different CUs. It 6363 ensures a previous vector memory operation has completed before executing a 6364 subsequent vector memory or LDS operation and so can be used to meet the 6365 requirements of acquire and release. 6366 * The L2 cache of one agent can be kept coherent with other agents by: 6367 using the MTYPE RW (read-write) or MTYPE CC (cache-coherent) with the PTE 6368 C-bit for memory local to the L2; and using the MTYPE NC (non-coherent) with 6369 the PTE C-bit set or MTYPE UC (uncached) for memory not local to the L2. 6370 6371 * Any local memory cache lines will be automatically invalidated by writes 6372 from CUs associated with other L2 caches, or writes from the CPU, due to 6373 the cache probe caused by coherent requests. Coherent requests are caused 6374 by GPU accesses to pages with the PTE C-bit set, by CPU accesses over 6375 XGMI, and by PCIe requests that are configured to be coherent requests. 6376 * XGMI accesses from the CPU to local memory may be cached on the CPU. 6377 Subsequent access from the GPU will automatically invalidate or writeback 6378 the CPU cache due to the L2 probe filter and and the PTE C-bit being set. 6379 * Since all work-groups on the same agent share the same L2, no L2 6380 invalidation or writeback is required for coherence. 6381 * To ensure coherence of local and remote memory writes of work-groups in 6382 different agents a ``buffer_wbl2`` is required. It will writeback dirty L2 6383 cache lines of MTYPE RW (used for local coarse grain memory) and MTYPE NC 6384 ()used for remote coarse grain memory). Note that MTYPE CC (used for local 6385 fine grain memory) causes write through to DRAM, and MTYPE UC (used for 6386 remote fine grain memory) bypasses the L2, so both will never result in 6387 dirty L2 cache lines. 6388 * To ensure coherence of local and remote memory reads of work-groups in 6389 different agents a ``buffer_invl2`` is required. It will invalidate L2 6390 cache lines with MTYPE NC (used for remote coarse grain memory). Note that 6391 MTYPE CC (used for local fine grain memory) and MTYPE RW (used for local 6392 coarse memory) cause local reads to be invalidated by remote writes with 6393 with the PTE C-bit so these cache lines are not invalidated. Note that 6394 MTYPE UC (used for remote fine grain memory) bypasses the L2, so will 6395 never result in L2 cache lines that need to be invalidated. 6396 6397 * PCIe access from the GPU to the CPU memory is kept coherent by using the 6398 MTYPE UC (uncached) which bypasses the L2. 6399 6400Scalar memory operations are only used to access memory that is proven to not 6401change during the execution of the kernel dispatch. This includes constant 6402address space and global address space for program scope ``const`` variables. 6403Therefore, the kernel machine code does not have to maintain the scalar cache to 6404ensure it is coherent with the vector caches. The scalar and vector caches are 6405invalidated between kernel dispatches by CP since constant address space data 6406may change between kernel dispatch executions. See 6407:ref:`amdgpu-amdhsa-memory-spaces`. 6408 6409The one exception is if scalar writes are used to spill SGPR registers. In this 6410case the AMDGPU backend ensures the memory location used to spill is never 6411accessed by vector memory operations at the same time. If scalar writes are used 6412then a ``s_dcache_wb`` is inserted before the ``s_endpgm`` and before a function 6413return since the locations may be used for vector memory instructions by a 6414future wavefront that uses the same scratch area, or a function call that 6415creates a frame at the same address, respectively. There is no need for a 6416``s_dcache_inv`` as all scalar writes are write-before-read in the same thread. 6417 6418For kernarg backing memory: 6419 6420* CP invalidates the L1 cache at the start of each kernel dispatch. 6421* On dGPU over XGMI or PCIe the kernarg backing memory is allocated in host 6422 memory accessed as MTYPE UC (uncached) to avoid needing to invalidate the L2 6423 cache. This also causes it to be treated as non-volatile and so is not 6424 invalidated by ``*_vol``. 6425* On APU the kernarg backing memory is accessed as MTYPE CC (cache coherent) and 6426 so the L2 cache will be coherent with the CPU and other agents. 6427 6428Scratch backing memory (which is used for the private address space) is accessed 6429with MTYPE NC_NV (non-coherent non-volatile). Since the private address space is 6430only accessed by a single thread, and is always write-before-read, there is 6431never a need to invalidate these entries from the L1 cache. Hence all cache 6432invalidates are done as ``*_vol`` to only invalidate the volatile cache lines. 6433 6434The code sequences used to implement the memory model for GFX90A are defined 6435in table :ref:`amdgpu-amdhsa-memory-model-code-sequences-gfx90a-table`. 6436 6437 .. table:: AMDHSA Memory Model Code Sequences GFX90A 6438 :name: amdgpu-amdhsa-memory-model-code-sequences-gfx90a-table 6439 6440 ============ ============ ============== ========== ================================ 6441 LLVM Instr LLVM Memory LLVM Memory AMDGPU AMDGPU Machine Code 6442 Ordering Sync Scope Address GFX90A 6443 Space 6444 ============ ============ ============== ========== ================================ 6445 **Non-Atomic** 6446 ------------------------------------------------------------------------------------ 6447 load *none* *none* - global - !volatile & !nontemporal 6448 - generic 6449 - private 1. buffer/global/flat_load 6450 - constant 6451 - !volatile & nontemporal 6452 6453 1. buffer/global/flat_load 6454 glc=1 slc=1 6455 6456 - volatile 6457 6458 1. buffer/global/flat_load 6459 glc=1 6460 2. s_waitcnt vmcnt(0) 6461 6462 - Must happen before 6463 any following volatile 6464 global/generic 6465 load/store. 6466 - Ensures that 6467 volatile 6468 operations to 6469 different 6470 addresses will not 6471 be reordered by 6472 hardware. 6473 6474 load *none* *none* - local 1. ds_load 6475 store *none* *none* - global - !volatile & !nontemporal 6476 - generic 6477 - private 1. buffer/global/flat_store 6478 - constant 6479 - !volatile & nontemporal 6480 6481 1. buffer/global/flat_store 6482 glc=1 slc=1 6483 6484 - volatile 6485 6486 1. buffer/global/flat_store 6487 2. s_waitcnt vmcnt(0) 6488 6489 - Must happen before 6490 any following volatile 6491 global/generic 6492 load/store. 6493 - Ensures that 6494 volatile 6495 operations to 6496 different 6497 addresses will not 6498 be reordered by 6499 hardware. 6500 6501 store *none* *none* - local 1. ds_store 6502 **Unordered Atomic** 6503 ------------------------------------------------------------------------------------ 6504 load atomic unordered *any* *any* *Same as non-atomic*. 6505 store atomic unordered *any* *any* *Same as non-atomic*. 6506 atomicrmw unordered *any* *any* *Same as monotonic atomic*. 6507 **Monotonic Atomic** 6508 ------------------------------------------------------------------------------------ 6509 load atomic monotonic - singlethread - global 1. buffer/global/flat_load 6510 - wavefront - generic 6511 load atomic monotonic - workgroup - global 1. buffer/global/flat_load 6512 - generic glc=1 6513 6514 - If not TgSplit execution 6515 mode, omit glc=1. 6516 6517 load atomic monotonic - singlethread - local *If TgSplit execution mode, 6518 - wavefront local address space cannot 6519 - workgroup be used.* 6520 6521 1. ds_load 6522 load atomic monotonic - agent - global 1. buffer/global/flat_load 6523 - generic glc=1 6524 load atomic monotonic - system - global 1. buffer/global/flat_load 6525 - generic glc=1 6526 store atomic monotonic - singlethread - global 1. buffer/global/flat_store 6527 - wavefront - generic 6528 - workgroup 6529 - agent 6530 store atomic monotonic - system - global 1. buffer/global/flat_store 6531 - generic 6532 store atomic monotonic - singlethread - local *If TgSplit execution mode, 6533 - wavefront local address space cannot 6534 - workgroup be used.* 6535 6536 1. ds_store 6537 atomicrmw monotonic - singlethread - global 1. buffer/global/flat_atomic 6538 - wavefront - generic 6539 - workgroup 6540 - agent 6541 atomicrmw monotonic - system - global 1. buffer/global/flat_atomic 6542 - generic 6543 atomicrmw monotonic - singlethread - local *If TgSplit execution mode, 6544 - wavefront local address space cannot 6545 - workgroup be used.* 6546 6547 1. ds_atomic 6548 **Acquire Atomic** 6549 ------------------------------------------------------------------------------------ 6550 load atomic acquire - singlethread - global 1. buffer/global/ds/flat_load 6551 - wavefront - local 6552 - generic 6553 load atomic acquire - workgroup - global 1. buffer/global_load glc=1 6554 6555 - If not TgSplit execution 6556 mode, omit glc=1. 6557 6558 2. s_waitcnt vmcnt(0) 6559 6560 - If not TgSplit execution 6561 mode, omit. 6562 - Must happen before the 6563 following buffer_wbinvl1_vol. 6564 6565 3. buffer_wbinvl1_vol 6566 6567 - If not TgSplit execution 6568 mode, omit. 6569 - Must happen before 6570 any following 6571 global/generic 6572 load/load 6573 atomic/store/store 6574 atomic/atomicrmw. 6575 - Ensures that 6576 following 6577 loads will not see 6578 stale data. 6579 6580 load atomic acquire - workgroup - local *If TgSplit execution mode, 6581 local address space cannot 6582 be used.* 6583 6584 1. ds_load 6585 2. s_waitcnt lgkmcnt(0) 6586 6587 - If OpenCL, omit. 6588 - Must happen before 6589 any following 6590 global/generic 6591 load/load 6592 atomic/store/store 6593 atomic/atomicrmw. 6594 - Ensures any 6595 following global 6596 data read is no 6597 older than the local load 6598 atomic value being 6599 acquired. 6600 6601 load atomic acquire - workgroup - generic 1. flat_load glc=1 6602 6603 - If not TgSplit execution 6604 mode, omit glc=1. 6605 6606 2. s_waitcnt lgkm/vmcnt(0) 6607 6608 - Use lgkmcnt(0) if not 6609 TgSplit execution mode 6610 and vmcnt(0) if TgSplit 6611 execution mode. 6612 - If OpenCL, omit lgkmcnt(0). 6613 - Must happen before 6614 the following 6615 buffer_wbinvl1_vol and any 6616 following global/generic 6617 load/load 6618 atomic/store/store 6619 atomic/atomicrmw. 6620 - Ensures any 6621 following global 6622 data read is no 6623 older than a local load 6624 atomic value being 6625 acquired. 6626 6627 3. buffer_wbinvl1_vol 6628 6629 - If not TgSplit execution 6630 mode, omit. 6631 - Ensures that 6632 following 6633 loads will not see 6634 stale data. 6635 6636 load atomic acquire - agent - global 1. buffer/global_load 6637 glc=1 6638 2. s_waitcnt vmcnt(0) 6639 6640 - Must happen before 6641 following 6642 buffer_wbinvl1_vol. 6643 - Ensures the load 6644 has completed 6645 before invalidating 6646 the cache. 6647 6648 3. buffer_wbinvl1_vol 6649 6650 - Must happen before 6651 any following 6652 global/generic 6653 load/load 6654 atomic/atomicrmw. 6655 - Ensures that 6656 following 6657 loads will not see 6658 stale global data. 6659 6660 load atomic acquire - system - global 1. buffer/global/flat_load 6661 glc=1 6662 2. s_waitcnt vmcnt(0) 6663 6664 - Must happen before 6665 following buffer_invl2 and 6666 buffer_wbinvl1_vol. 6667 - Ensures the load 6668 has completed 6669 before invalidating 6670 the cache. 6671 6672 3. buffer_invl2; 6673 buffer_wbinvl1_vol 6674 6675 - Must happen before 6676 any following 6677 global/generic 6678 load/load 6679 atomic/atomicrmw. 6680 - Ensures that 6681 following 6682 loads will not see 6683 stale L1 global data, 6684 nor see stale L2 MTYPE 6685 NC global data. 6686 MTYPE RW and CC memory will 6687 never be stale in L2 due to 6688 the memory probes. 6689 6690 load atomic acquire - agent - generic 1. flat_load glc=1 6691 2. s_waitcnt vmcnt(0) & 6692 lgkmcnt(0) 6693 6694 - If TgSplit execution mode, 6695 omit lgkmcnt(0). 6696 - If OpenCL omit 6697 lgkmcnt(0). 6698 - Must happen before 6699 following 6700 buffer_wbinvl1_vol. 6701 - Ensures the flat_load 6702 has completed 6703 before invalidating 6704 the cache. 6705 6706 3. buffer_wbinvl1_vol 6707 6708 - Must happen before 6709 any following 6710 global/generic 6711 load/load 6712 atomic/atomicrmw. 6713 - Ensures that 6714 following loads 6715 will not see stale 6716 global data. 6717 6718 load atomic acquire - system - generic 1. flat_load glc=1 6719 2. s_waitcnt vmcnt(0) & 6720 lgkmcnt(0) 6721 6722 - If TgSplit execution mode, 6723 omit lgkmcnt(0). 6724 - If OpenCL omit 6725 lgkmcnt(0). 6726 - Must happen before 6727 following 6728 buffer_invl2 and 6729 buffer_wbinvl1_vol. 6730 - Ensures the flat_load 6731 has completed 6732 before invalidating 6733 the caches. 6734 6735 3. buffer_invl2; 6736 buffer_wbinvl1_vol 6737 6738 - Must happen before 6739 any following 6740 global/generic 6741 load/load 6742 atomic/atomicrmw. 6743 - Ensures that 6744 following 6745 loads will not see 6746 stale L1 global data, 6747 nor see stale L2 MTYPE 6748 NC global data. 6749 MTYPE RW and CC memory will 6750 never be stale in L2 due to 6751 the memory probes. 6752 6753 atomicrmw acquire - singlethread - global 1. buffer/global/flat_atomic 6754 - wavefront - generic 6755 atomicrmw acquire - singlethread - local *If TgSplit execution mode, 6756 - wavefront local address space cannot 6757 be used.* 6758 6759 1. ds_atomic 6760 atomicrmw acquire - workgroup - global 1. buffer/global_atomic 6761 2. s_waitcnt vmcnt(0) 6762 6763 - If not TgSplit execution 6764 mode, omit. 6765 - Must happen before the 6766 following buffer_wbinvl1_vol. 6767 - Ensures the atomicrmw 6768 has completed 6769 before invalidating 6770 the cache. 6771 6772 3. buffer_wbinvl1_vol 6773 6774 - If not TgSplit execution 6775 mode, omit. 6776 - Must happen before 6777 any following 6778 global/generic 6779 load/load 6780 atomic/atomicrmw. 6781 - Ensures that 6782 following loads 6783 will not see stale 6784 global data. 6785 6786 atomicrmw acquire - workgroup - local *If TgSplit execution mode, 6787 local address space cannot 6788 be used.* 6789 6790 1. ds_atomic 6791 2. s_waitcnt lgkmcnt(0) 6792 6793 - If OpenCL, omit. 6794 - Must happen before 6795 any following 6796 global/generic 6797 load/load 6798 atomic/store/store 6799 atomic/atomicrmw. 6800 - Ensures any 6801 following global 6802 data read is no 6803 older than the local 6804 atomicrmw value 6805 being acquired. 6806 6807 atomicrmw acquire - workgroup - generic 1. flat_atomic 6808 2. s_waitcnt lgkm/vmcnt(0) 6809 6810 - Use lgkmcnt(0) if not 6811 TgSplit execution mode 6812 and vmcnt(0) if TgSplit 6813 execution mode. 6814 - If OpenCL, omit lgkmcnt(0). 6815 - Must happen before 6816 the following 6817 buffer_wbinvl1_vol and 6818 any following 6819 global/generic 6820 load/load 6821 atomic/store/store 6822 atomic/atomicrmw. 6823 - Ensures any 6824 following global 6825 data read is no 6826 older than a local 6827 atomicrmw value 6828 being acquired. 6829 6830 3. buffer_wbinvl1_vol 6831 6832 - If not TgSplit execution 6833 mode, omit. 6834 - Ensures that 6835 following 6836 loads will not see 6837 stale data. 6838 6839 atomicrmw acquire - agent - global 1. buffer/global_atomic 6840 2. s_waitcnt vmcnt(0) 6841 6842 - Must happen before 6843 following 6844 buffer_wbinvl1_vol. 6845 - Ensures the 6846 atomicrmw has 6847 completed before 6848 invalidating the 6849 cache. 6850 6851 3. buffer_wbinvl1_vol 6852 6853 - Must happen before 6854 any following 6855 global/generic 6856 load/load 6857 atomic/atomicrmw. 6858 - Ensures that 6859 following loads 6860 will not see stale 6861 global data. 6862 6863 atomicrmw acquire - system - global 1. buffer/global_atomic 6864 2. s_waitcnt vmcnt(0) 6865 6866 - Must happen before 6867 following buffer_invl2 and 6868 buffer_wbinvl1_vol. 6869 - Ensures the 6870 atomicrmw has 6871 completed before 6872 invalidating the 6873 caches. 6874 6875 3. buffer_invl2; 6876 buffer_wbinvl1_vol 6877 6878 - Must happen before 6879 any following 6880 global/generic 6881 load/load 6882 atomic/atomicrmw. 6883 - Ensures that 6884 following 6885 loads will not see 6886 stale L1 global data, 6887 nor see stale L2 MTYPE 6888 NC global data. 6889 MTYPE RW and CC memory will 6890 never be stale in L2 due to 6891 the memory probes. 6892 6893 atomicrmw acquire - agent - generic 1. flat_atomic 6894 2. s_waitcnt vmcnt(0) & 6895 lgkmcnt(0) 6896 6897 - If TgSplit execution mode, 6898 omit lgkmcnt(0). 6899 - If OpenCL, omit 6900 lgkmcnt(0). 6901 - Must happen before 6902 following 6903 buffer_wbinvl1_vol. 6904 - Ensures the 6905 atomicrmw has 6906 completed before 6907 invalidating the 6908 cache. 6909 6910 3. buffer_wbinvl1_vol 6911 6912 - Must happen before 6913 any following 6914 global/generic 6915 load/load 6916 atomic/atomicrmw. 6917 - Ensures that 6918 following loads 6919 will not see stale 6920 global data. 6921 6922 atomicrmw acquire - system - generic 1. flat_atomic 6923 2. s_waitcnt vmcnt(0) & 6924 lgkmcnt(0) 6925 6926 - If TgSplit execution mode, 6927 omit lgkmcnt(0). 6928 - If OpenCL, omit 6929 lgkmcnt(0). 6930 - Must happen before 6931 following 6932 buffer_invl2 and 6933 buffer_wbinvl1_vol. 6934 - Ensures the 6935 atomicrmw has 6936 completed before 6937 invalidating the 6938 caches. 6939 6940 3. buffer_invl2; 6941 buffer_wbinvl1_vol 6942 6943 - Must happen before 6944 any following 6945 global/generic 6946 load/load 6947 atomic/atomicrmw. 6948 - Ensures that 6949 following 6950 loads will not see 6951 stale L1 global data, 6952 nor see stale L2 MTYPE 6953 NC global data. 6954 MTYPE RW and CC memory will 6955 never be stale in L2 due to 6956 the memory probes. 6957 6958 fence acquire - singlethread *none* *none* 6959 - wavefront 6960 fence acquire - workgroup *none* 1. s_waitcnt lgkm/vmcnt(0) 6961 6962 - Use lgkmcnt(0) if not 6963 TgSplit execution mode 6964 and vmcnt(0) if TgSplit 6965 execution mode. 6966 - If OpenCL and 6967 address space is 6968 not generic, omit 6969 lgkmcnt(0). 6970 - If OpenCL and 6971 address space is 6972 local, omit 6973 vmcnt(0). 6974 - However, since LLVM 6975 currently has no 6976 address space on 6977 the fence need to 6978 conservatively 6979 always generate. If 6980 fence had an 6981 address space then 6982 set to address 6983 space of OpenCL 6984 fence flag, or to 6985 generic if both 6986 local and global 6987 flags are 6988 specified. 6989 - s_waitcnt vmcnt(0) 6990 must happen after 6991 any preceding 6992 global/generic load 6993 atomic/ 6994 atomicrmw 6995 with an equal or 6996 wider sync scope 6997 and memory ordering 6998 stronger than 6999 unordered (this is 7000 termed the 7001 fence-paired-atomic). 7002 - s_waitcnt lgkmcnt(0) 7003 must happen after 7004 any preceding 7005 local/generic load 7006 atomic/atomicrmw 7007 with an equal or 7008 wider sync scope 7009 and memory ordering 7010 stronger than 7011 unordered (this is 7012 termed the 7013 fence-paired-atomic). 7014 - Must happen before 7015 the following 7016 buffer_wbinvl1_vol and 7017 any following 7018 global/generic 7019 load/load 7020 atomic/store/store 7021 atomic/atomicrmw. 7022 - Ensures any 7023 following global 7024 data read is no 7025 older than the 7026 value read by the 7027 fence-paired-atomic. 7028 7029 2. buffer_wbinvl1_vol 7030 7031 - If not TgSplit execution 7032 mode, omit. 7033 - Ensures that 7034 following 7035 loads will not see 7036 stale data. 7037 7038 fence acquire - agent *none* 1. s_waitcnt lgkmcnt(0) & 7039 vmcnt(0) 7040 7041 - If TgSplit execution mode, 7042 omit lgkmcnt(0). 7043 - If OpenCL and 7044 address space is 7045 not generic, omit 7046 lgkmcnt(0). 7047 - However, since LLVM 7048 currently has no 7049 address space on 7050 the fence need to 7051 conservatively 7052 always generate 7053 (see comment for 7054 previous fence). 7055 - Could be split into 7056 separate s_waitcnt 7057 vmcnt(0) and 7058 s_waitcnt 7059 lgkmcnt(0) to allow 7060 them to be 7061 independently moved 7062 according to the 7063 following rules. 7064 - s_waitcnt vmcnt(0) 7065 must happen after 7066 any preceding 7067 global/generic load 7068 atomic/atomicrmw 7069 with an equal or 7070 wider sync scope 7071 and memory ordering 7072 stronger than 7073 unordered (this is 7074 termed the 7075 fence-paired-atomic). 7076 - s_waitcnt lgkmcnt(0) 7077 must happen after 7078 any preceding 7079 local/generic load 7080 atomic/atomicrmw 7081 with an equal or 7082 wider sync scope 7083 and memory ordering 7084 stronger than 7085 unordered (this is 7086 termed the 7087 fence-paired-atomic). 7088 - Must happen before 7089 the following 7090 buffer_wbinvl1_vol. 7091 - Ensures that the 7092 fence-paired atomic 7093 has completed 7094 before invalidating 7095 the 7096 cache. Therefore 7097 any following 7098 locations read must 7099 be no older than 7100 the value read by 7101 the 7102 fence-paired-atomic. 7103 7104 2. buffer_wbinvl1_vol 7105 7106 - Must happen before any 7107 following global/generic 7108 load/load 7109 atomic/store/store 7110 atomic/atomicrmw. 7111 - Ensures that 7112 following loads 7113 will not see stale 7114 global data. 7115 7116 fence acquire - system *none* 1. s_waitcnt lgkmcnt(0) & 7117 vmcnt(0) 7118 7119 - If TgSplit execution mode, 7120 omit lgkmcnt(0). 7121 - If OpenCL and 7122 address space is 7123 not generic, omit 7124 lgkmcnt(0). 7125 - However, since LLVM 7126 currently has no 7127 address space on 7128 the fence need to 7129 conservatively 7130 always generate 7131 (see comment for 7132 previous fence). 7133 - Could be split into 7134 separate s_waitcnt 7135 vmcnt(0) and 7136 s_waitcnt 7137 lgkmcnt(0) to allow 7138 them to be 7139 independently moved 7140 according to the 7141 following rules. 7142 - s_waitcnt vmcnt(0) 7143 must happen after 7144 any preceding 7145 global/generic load 7146 atomic/atomicrmw 7147 with an equal or 7148 wider sync scope 7149 and memory ordering 7150 stronger than 7151 unordered (this is 7152 termed the 7153 fence-paired-atomic). 7154 - s_waitcnt lgkmcnt(0) 7155 must happen after 7156 any preceding 7157 local/generic load 7158 atomic/atomicrmw 7159 with an equal or 7160 wider sync scope 7161 and memory ordering 7162 stronger than 7163 unordered (this is 7164 termed the 7165 fence-paired-atomic). 7166 - Must happen before 7167 the following buffer_invl2 and 7168 buffer_wbinvl1_vol. 7169 - Ensures that the 7170 fence-paired atomic 7171 has completed 7172 before invalidating 7173 the 7174 cache. Therefore 7175 any following 7176 locations read must 7177 be no older than 7178 the value read by 7179 the 7180 fence-paired-atomic. 7181 7182 2. buffer_invl2; 7183 buffer_wbinvl1_vol 7184 7185 - Must happen before any 7186 following global/generic 7187 load/load 7188 atomic/store/store 7189 atomic/atomicrmw. 7190 - Ensures that 7191 following 7192 loads will not see 7193 stale L1 global data, 7194 nor see stale L2 MTYPE 7195 NC global data. 7196 MTYPE RW and CC memory will 7197 never be stale in L2 due to 7198 the memory probes. 7199 **Release Atomic** 7200 ------------------------------------------------------------------------------------ 7201 store atomic release - singlethread - global 1. buffer/global/flat_store 7202 - wavefront - generic 7203 store atomic release - singlethread - local *If TgSplit execution mode, 7204 - wavefront local address space cannot 7205 be used.* 7206 7207 1. ds_store 7208 store atomic release - workgroup - global 1. s_waitcnt lgkm/vmcnt(0) 7209 - generic 7210 - Use lgkmcnt(0) if not 7211 TgSplit execution mode 7212 and vmcnt(0) if TgSplit 7213 execution mode. 7214 - If OpenCL, omit lgkmcnt(0). 7215 - s_waitcnt vmcnt(0) 7216 must happen after 7217 any preceding 7218 global/generic load/store/ 7219 load atomic/store atomic/ 7220 atomicrmw. 7221 - s_waitcnt lgkmcnt(0) 7222 must happen after 7223 any preceding 7224 local/generic 7225 load/store/load 7226 atomic/store 7227 atomic/atomicrmw. 7228 - Must happen before 7229 the following 7230 store. 7231 - Ensures that all 7232 memory operations 7233 have 7234 completed before 7235 performing the 7236 store that is being 7237 released. 7238 7239 2. buffer/global/flat_store 7240 store atomic release - workgroup - local *If TgSplit execution mode, 7241 local address space cannot 7242 be used.* 7243 7244 1. ds_store 7245 store atomic release - agent - global 1. s_waitcnt lgkmcnt(0) & 7246 - generic vmcnt(0) 7247 7248 - If TgSplit execution mode, 7249 omit lgkmcnt(0). 7250 - If OpenCL and 7251 address space is 7252 not generic, omit 7253 lgkmcnt(0). 7254 - Could be split into 7255 separate s_waitcnt 7256 vmcnt(0) and 7257 s_waitcnt 7258 lgkmcnt(0) to allow 7259 them to be 7260 independently moved 7261 according to the 7262 following rules. 7263 - s_waitcnt vmcnt(0) 7264 must happen after 7265 any preceding 7266 global/generic 7267 load/store/load 7268 atomic/store 7269 atomic/atomicrmw. 7270 - s_waitcnt lgkmcnt(0) 7271 must happen after 7272 any preceding 7273 local/generic 7274 load/store/load 7275 atomic/store 7276 atomic/atomicrmw. 7277 - Must happen before 7278 the following 7279 store. 7280 - Ensures that all 7281 memory operations 7282 to memory have 7283 completed before 7284 performing the 7285 store that is being 7286 released. 7287 7288 2. buffer/global/flat_store 7289 store atomic release - system - global 1. buffer_wbl2 7290 - generic 7291 - Must happen before 7292 following s_waitcnt. 7293 - Performs L2 writeback to 7294 ensure previous 7295 global/generic 7296 store/atomicrmw are 7297 visible at system scope. 7298 7299 2. s_waitcnt lgkmcnt(0) & 7300 vmcnt(0) 7301 7302 - If TgSplit execution mode, 7303 omit lgkmcnt(0). 7304 - If OpenCL and 7305 address space is 7306 not generic, omit 7307 lgkmcnt(0). 7308 - Could be split into 7309 separate s_waitcnt 7310 vmcnt(0) and 7311 s_waitcnt 7312 lgkmcnt(0) to allow 7313 them to be 7314 independently moved 7315 according to the 7316 following rules. 7317 - s_waitcnt vmcnt(0) 7318 must happen after any 7319 preceding 7320 global/generic 7321 load/store/load 7322 atomic/store 7323 atomic/atomicrmw. 7324 - s_waitcnt lgkmcnt(0) 7325 must happen after any 7326 preceding 7327 local/generic 7328 load/store/load 7329 atomic/store 7330 atomic/atomicrmw. 7331 - Must happen before 7332 the following 7333 store. 7334 - Ensures that all 7335 memory operations 7336 to memory and the L2 7337 writeback have 7338 completed before 7339 performing the 7340 store that is being 7341 released. 7342 7343 3. buffer/global/flat_store 7344 atomicrmw release - singlethread - global 1. buffer/global/flat_atomic 7345 - wavefront - generic 7346 atomicrmw release - singlethread - local *If TgSplit execution mode, 7347 - wavefront local address space cannot 7348 be used.* 7349 7350 1. ds_atomic 7351 atomicrmw release - workgroup - global 1. s_waitcnt lgkm/vmcnt(0) 7352 - generic 7353 - Use lgkmcnt(0) if not 7354 TgSplit execution mode 7355 and vmcnt(0) if TgSplit 7356 execution mode. 7357 - If OpenCL, omit 7358 lgkmcnt(0). 7359 - s_waitcnt vmcnt(0) 7360 must happen after 7361 any preceding 7362 global/generic load/store/ 7363 load atomic/store atomic/ 7364 atomicrmw. 7365 - s_waitcnt lgkmcnt(0) 7366 must happen after 7367 any preceding 7368 local/generic 7369 load/store/load 7370 atomic/store 7371 atomic/atomicrmw. 7372 - Must happen before 7373 the following 7374 atomicrmw. 7375 - Ensures that all 7376 memory operations 7377 have 7378 completed before 7379 performing the 7380 atomicrmw that is 7381 being released. 7382 7383 2. buffer/global/flat_atomic 7384 atomicrmw release - workgroup - local *If TgSplit execution mode, 7385 local address space cannot 7386 be used.* 7387 7388 1. ds_atomic 7389 atomicrmw release - agent - global 1. s_waitcnt lgkmcnt(0) & 7390 - generic vmcnt(0) 7391 7392 - If TgSplit execution mode, 7393 omit lgkmcnt(0). 7394 - If OpenCL, omit 7395 lgkmcnt(0). 7396 - Could be split into 7397 separate s_waitcnt 7398 vmcnt(0) and 7399 s_waitcnt 7400 lgkmcnt(0) to allow 7401 them to be 7402 independently moved 7403 according to the 7404 following rules. 7405 - s_waitcnt vmcnt(0) 7406 must happen after 7407 any preceding 7408 global/generic 7409 load/store/load 7410 atomic/store 7411 atomic/atomicrmw. 7412 - s_waitcnt lgkmcnt(0) 7413 must happen after 7414 any preceding 7415 local/generic 7416 load/store/load 7417 atomic/store 7418 atomic/atomicrmw. 7419 - Must happen before 7420 the following 7421 atomicrmw. 7422 - Ensures that all 7423 memory operations 7424 to global and local 7425 have completed 7426 before performing 7427 the atomicrmw that 7428 is being released. 7429 7430 2. buffer/global/flat_atomic 7431 atomicrmw release - system - global 1. buffer_wbl2 7432 - generic 7433 - Must happen before 7434 following s_waitcnt. 7435 - Performs L2 writeback to 7436 ensure previous 7437 global/generic 7438 store/atomicrmw are 7439 visible at system scope. 7440 7441 2. s_waitcnt lgkmcnt(0) & 7442 vmcnt(0) 7443 7444 - If TgSplit execution mode, 7445 omit lgkmcnt(0). 7446 - If OpenCL, omit 7447 lgkmcnt(0). 7448 - Could be split into 7449 separate s_waitcnt 7450 vmcnt(0) and 7451 s_waitcnt 7452 lgkmcnt(0) to allow 7453 them to be 7454 independently moved 7455 according to the 7456 following rules. 7457 - s_waitcnt vmcnt(0) 7458 must happen after 7459 any preceding 7460 global/generic 7461 load/store/load 7462 atomic/store 7463 atomic/atomicrmw. 7464 - s_waitcnt lgkmcnt(0) 7465 must happen after 7466 any preceding 7467 local/generic 7468 load/store/load 7469 atomic/store 7470 atomic/atomicrmw. 7471 - Must happen before 7472 the following 7473 atomicrmw. 7474 - Ensures that all 7475 memory operations 7476 to memory and the L2 7477 writeback have 7478 completed before 7479 performing the 7480 store that is being 7481 released. 7482 7483 3. buffer/global/flat_atomic 7484 fence release - singlethread *none* *none* 7485 - wavefront 7486 fence release - workgroup *none* 1. s_waitcnt lgkm/vmcnt(0) 7487 7488 - Use lgkmcnt(0) if not 7489 TgSplit execution mode 7490 and vmcnt(0) if TgSplit 7491 execution mode. 7492 - If OpenCL and 7493 address space is 7494 not generic, omit 7495 lgkmcnt(0). 7496 - If OpenCL and 7497 address space is 7498 local, omit 7499 vmcnt(0). 7500 - However, since LLVM 7501 currently has no 7502 address space on 7503 the fence need to 7504 conservatively 7505 always generate. If 7506 fence had an 7507 address space then 7508 set to address 7509 space of OpenCL 7510 fence flag, or to 7511 generic if both 7512 local and global 7513 flags are 7514 specified. 7515 - s_waitcnt vmcnt(0) 7516 must happen after 7517 any preceding 7518 global/generic 7519 load/store/ 7520 load atomic/store atomic/ 7521 atomicrmw. 7522 - s_waitcnt lgkmcnt(0) 7523 must happen after 7524 any preceding 7525 local/generic 7526 load/load 7527 atomic/store/store 7528 atomic/atomicrmw. 7529 - Must happen before 7530 any following store 7531 atomic/atomicrmw 7532 with an equal or 7533 wider sync scope 7534 and memory ordering 7535 stronger than 7536 unordered (this is 7537 termed the 7538 fence-paired-atomic). 7539 - Ensures that all 7540 memory operations 7541 have 7542 completed before 7543 performing the 7544 following 7545 fence-paired-atomic. 7546 7547 fence release - agent *none* 1. s_waitcnt lgkmcnt(0) & 7548 vmcnt(0) 7549 7550 - If TgSplit execution mode, 7551 omit lgkmcnt(0). 7552 - If OpenCL and 7553 address space is 7554 not generic, omit 7555 lgkmcnt(0). 7556 - If OpenCL and 7557 address space is 7558 local, omit 7559 vmcnt(0). 7560 - However, since LLVM 7561 currently has no 7562 address space on 7563 the fence need to 7564 conservatively 7565 always generate. If 7566 fence had an 7567 address space then 7568 set to address 7569 space of OpenCL 7570 fence flag, or to 7571 generic if both 7572 local and global 7573 flags are 7574 specified. 7575 - Could be split into 7576 separate s_waitcnt 7577 vmcnt(0) and 7578 s_waitcnt 7579 lgkmcnt(0) to allow 7580 them to be 7581 independently moved 7582 according to the 7583 following rules. 7584 - s_waitcnt vmcnt(0) 7585 must happen after 7586 any preceding 7587 global/generic 7588 load/store/load 7589 atomic/store 7590 atomic/atomicrmw. 7591 - s_waitcnt lgkmcnt(0) 7592 must happen after 7593 any preceding 7594 local/generic 7595 load/store/load 7596 atomic/store 7597 atomic/atomicrmw. 7598 - Must happen before 7599 any following store 7600 atomic/atomicrmw 7601 with an equal or 7602 wider sync scope 7603 and memory ordering 7604 stronger than 7605 unordered (this is 7606 termed the 7607 fence-paired-atomic). 7608 - Ensures that all 7609 memory operations 7610 have 7611 completed before 7612 performing the 7613 following 7614 fence-paired-atomic. 7615 7616 fence release - system *none* 1. buffer_wbl2 7617 7618 - If OpenCL and 7619 address space is 7620 local, omit. 7621 - Must happen before 7622 following s_waitcnt. 7623 - Performs L2 writeback to 7624 ensure previous 7625 global/generic 7626 store/atomicrmw are 7627 visible at system scope. 7628 7629 2. s_waitcnt lgkmcnt(0) & 7630 vmcnt(0) 7631 7632 - If TgSplit execution mode, 7633 omit lgkmcnt(0). 7634 - If OpenCL and 7635 address space is 7636 not generic, omit 7637 lgkmcnt(0). 7638 - If OpenCL and 7639 address space is 7640 local, omit 7641 vmcnt(0). 7642 - However, since LLVM 7643 currently has no 7644 address space on 7645 the fence need to 7646 conservatively 7647 always generate. If 7648 fence had an 7649 address space then 7650 set to address 7651 space of OpenCL 7652 fence flag, or to 7653 generic if both 7654 local and global 7655 flags are 7656 specified. 7657 - Could be split into 7658 separate s_waitcnt 7659 vmcnt(0) and 7660 s_waitcnt 7661 lgkmcnt(0) to allow 7662 them to be 7663 independently moved 7664 according to the 7665 following rules. 7666 - s_waitcnt vmcnt(0) 7667 must happen after 7668 any preceding 7669 global/generic 7670 load/store/load 7671 atomic/store 7672 atomic/atomicrmw. 7673 - s_waitcnt lgkmcnt(0) 7674 must happen after 7675 any preceding 7676 local/generic 7677 load/store/load 7678 atomic/store 7679 atomic/atomicrmw. 7680 - Must happen before 7681 any following store 7682 atomic/atomicrmw 7683 with an equal or 7684 wider sync scope 7685 and memory ordering 7686 stronger than 7687 unordered (this is 7688 termed the 7689 fence-paired-atomic). 7690 - Ensures that all 7691 memory operations 7692 have 7693 completed before 7694 performing the 7695 following 7696 fence-paired-atomic. 7697 7698 **Acquire-Release Atomic** 7699 ------------------------------------------------------------------------------------ 7700 atomicrmw acq_rel - singlethread - global 1. buffer/global/flat_atomic 7701 - wavefront - generic 7702 atomicrmw acq_rel - singlethread - local *If TgSplit execution mode, 7703 - wavefront local address space cannot 7704 be used.* 7705 7706 1. ds_atomic 7707 atomicrmw acq_rel - workgroup - global 1. s_waitcnt lgkm/vmcnt(0) 7708 7709 - Use lgkmcnt(0) if not 7710 TgSplit execution mode 7711 and vmcnt(0) if TgSplit 7712 execution mode. 7713 - If OpenCL, omit 7714 lgkmcnt(0). 7715 - Must happen after 7716 any preceding 7717 local/generic 7718 load/store/load 7719 atomic/store 7720 atomic/atomicrmw. 7721 - s_waitcnt vmcnt(0) 7722 must happen after 7723 any preceding 7724 global/generic load/store/ 7725 load atomic/store atomic/ 7726 atomicrmw. 7727 - s_waitcnt lgkmcnt(0) 7728 must happen after 7729 any preceding 7730 local/generic 7731 load/store/load 7732 atomic/store 7733 atomic/atomicrmw. 7734 - Must happen before 7735 the following 7736 atomicrmw. 7737 - Ensures that all 7738 memory operations 7739 have 7740 completed before 7741 performing the 7742 atomicrmw that is 7743 being released. 7744 7745 2. buffer/global_atomic 7746 3. s_waitcnt vmcnt(0) 7747 7748 - If not TgSplit execution 7749 mode, omit. 7750 - Must happen before 7751 the following 7752 buffer_wbinvl1_vol. 7753 - Ensures any 7754 following global 7755 data read is no 7756 older than the 7757 atomicrmw value 7758 being acquired. 7759 7760 4. buffer_wbinvl1_vol 7761 7762 - If not TgSplit execution 7763 mode, omit. 7764 - Ensures that 7765 following 7766 loads will not see 7767 stale data. 7768 7769 atomicrmw acq_rel - workgroup - local *If TgSplit execution mode, 7770 local address space cannot 7771 be used.* 7772 7773 1. ds_atomic 7774 2. s_waitcnt lgkmcnt(0) 7775 7776 - If OpenCL, omit. 7777 - Must happen before 7778 any following 7779 global/generic 7780 load/load 7781 atomic/store/store 7782 atomic/atomicrmw. 7783 - Ensures any 7784 following global 7785 data read is no 7786 older than the local load 7787 atomic value being 7788 acquired. 7789 7790 atomicrmw acq_rel - workgroup - generic 1. s_waitcnt lgkm/vmcnt(0) 7791 7792 - Use lgkmcnt(0) if not 7793 TgSplit execution mode 7794 and vmcnt(0) if TgSplit 7795 execution mode. 7796 - If OpenCL, omit 7797 lgkmcnt(0). 7798 - s_waitcnt vmcnt(0) 7799 must happen after 7800 any preceding 7801 global/generic load/store/ 7802 load atomic/store atomic/ 7803 atomicrmw. 7804 - s_waitcnt lgkmcnt(0) 7805 must happen after 7806 any preceding 7807 local/generic 7808 load/store/load 7809 atomic/store 7810 atomic/atomicrmw. 7811 - Must happen before 7812 the following 7813 atomicrmw. 7814 - Ensures that all 7815 memory operations 7816 have 7817 completed before 7818 performing the 7819 atomicrmw that is 7820 being released. 7821 7822 2. flat_atomic 7823 3. s_waitcnt lgkmcnt(0) & 7824 vmcnt(0) 7825 7826 - If not TgSplit execution 7827 mode, omit vmcnt(0). 7828 - If OpenCL, omit 7829 lgkmcnt(0). 7830 - Must happen before 7831 the following 7832 buffer_wbinvl1_vol and 7833 any following 7834 global/generic 7835 load/load 7836 atomic/store/store 7837 atomic/atomicrmw. 7838 - Ensures any 7839 following global 7840 data read is no 7841 older than a local load 7842 atomic value being 7843 acquired. 7844 7845 3. buffer_wbinvl1_vol 7846 7847 - If not TgSplit execution 7848 mode, omit. 7849 - Ensures that 7850 following 7851 loads will not see 7852 stale data. 7853 7854 atomicrmw acq_rel - agent - global 1. s_waitcnt lgkmcnt(0) & 7855 vmcnt(0) 7856 7857 - If TgSplit execution mode, 7858 omit lgkmcnt(0). 7859 - If OpenCL, omit 7860 lgkmcnt(0). 7861 - Could be split into 7862 separate s_waitcnt 7863 vmcnt(0) and 7864 s_waitcnt 7865 lgkmcnt(0) to allow 7866 them to be 7867 independently moved 7868 according to the 7869 following rules. 7870 - s_waitcnt vmcnt(0) 7871 must happen after 7872 any preceding 7873 global/generic 7874 load/store/load 7875 atomic/store 7876 atomic/atomicrmw. 7877 - s_waitcnt lgkmcnt(0) 7878 must happen after 7879 any preceding 7880 local/generic 7881 load/store/load 7882 atomic/store 7883 atomic/atomicrmw. 7884 - Must happen before 7885 the following 7886 atomicrmw. 7887 - Ensures that all 7888 memory operations 7889 to global have 7890 completed before 7891 performing the 7892 atomicrmw that is 7893 being released. 7894 7895 2. buffer/global_atomic 7896 3. s_waitcnt vmcnt(0) 7897 7898 - Must happen before 7899 following 7900 buffer_wbinvl1_vol. 7901 - Ensures the 7902 atomicrmw has 7903 completed before 7904 invalidating the 7905 cache. 7906 7907 4. buffer_wbinvl1_vol 7908 7909 - Must happen before 7910 any following 7911 global/generic 7912 load/load 7913 atomic/atomicrmw. 7914 - Ensures that 7915 following loads 7916 will not see stale 7917 global data. 7918 7919 atomicrmw acq_rel - system - global 1. buffer_wbl2 7920 7921 - Must happen before 7922 following s_waitcnt. 7923 - Performs L2 writeback to 7924 ensure previous 7925 global/generic 7926 store/atomicrmw are 7927 visible at system scope. 7928 7929 2. s_waitcnt lgkmcnt(0) & 7930 vmcnt(0) 7931 7932 - If TgSplit execution mode, 7933 omit lgkmcnt(0). 7934 - If OpenCL, omit 7935 lgkmcnt(0). 7936 - Could be split into 7937 separate s_waitcnt 7938 vmcnt(0) and 7939 s_waitcnt 7940 lgkmcnt(0) to allow 7941 them to be 7942 independently moved 7943 according to the 7944 following rules. 7945 - s_waitcnt vmcnt(0) 7946 must happen after 7947 any preceding 7948 global/generic 7949 load/store/load 7950 atomic/store 7951 atomic/atomicrmw. 7952 - s_waitcnt lgkmcnt(0) 7953 must happen after 7954 any preceding 7955 local/generic 7956 load/store/load 7957 atomic/store 7958 atomic/atomicrmw. 7959 - Must happen before 7960 the following 7961 atomicrmw. 7962 - Ensures that all 7963 memory operations 7964 to global and L2 writeback 7965 have completed before 7966 performing the 7967 atomicrmw that is 7968 being released. 7969 7970 3. buffer/global_atomic 7971 4. s_waitcnt vmcnt(0) 7972 7973 - Must happen before 7974 following buffer_invl2 and 7975 buffer_wbinvl1_vol. 7976 - Ensures the 7977 atomicrmw has 7978 completed before 7979 invalidating the 7980 caches. 7981 7982 5. buffer_invl2; 7983 buffer_wbinvl1_vol 7984 7985 - Must happen before 7986 any following 7987 global/generic 7988 load/load 7989 atomic/atomicrmw. 7990 - Ensures that 7991 following 7992 loads will not see 7993 stale L1 global data, 7994 nor see stale L2 MTYPE 7995 NC global data. 7996 MTYPE RW and CC memory will 7997 never be stale in L2 due to 7998 the memory probes. 7999 8000 atomicrmw acq_rel - agent - generic 1. s_waitcnt lgkmcnt(0) & 8001 vmcnt(0) 8002 8003 - If TgSplit execution mode, 8004 omit lgkmcnt(0). 8005 - If OpenCL, omit 8006 lgkmcnt(0). 8007 - Could be split into 8008 separate s_waitcnt 8009 vmcnt(0) and 8010 s_waitcnt 8011 lgkmcnt(0) to allow 8012 them to be 8013 independently moved 8014 according to the 8015 following rules. 8016 - s_waitcnt vmcnt(0) 8017 must happen after 8018 any preceding 8019 global/generic 8020 load/store/load 8021 atomic/store 8022 atomic/atomicrmw. 8023 - s_waitcnt lgkmcnt(0) 8024 must happen after 8025 any preceding 8026 local/generic 8027 load/store/load 8028 atomic/store 8029 atomic/atomicrmw. 8030 - Must happen before 8031 the following 8032 atomicrmw. 8033 - Ensures that all 8034 memory operations 8035 to global have 8036 completed before 8037 performing the 8038 atomicrmw that is 8039 being released. 8040 8041 2. flat_atomic 8042 3. s_waitcnt vmcnt(0) & 8043 lgkmcnt(0) 8044 8045 - If TgSplit execution mode, 8046 omit lgkmcnt(0). 8047 - If OpenCL, omit 8048 lgkmcnt(0). 8049 - Must happen before 8050 following 8051 buffer_wbinvl1_vol. 8052 - Ensures the 8053 atomicrmw has 8054 completed before 8055 invalidating the 8056 cache. 8057 8058 4. buffer_wbinvl1_vol 8059 8060 - Must happen before 8061 any following 8062 global/generic 8063 load/load 8064 atomic/atomicrmw. 8065 - Ensures that 8066 following loads 8067 will not see stale 8068 global data. 8069 8070 atomicrmw acq_rel - system - generic 1. buffer_wbl2 8071 8072 - Must happen before 8073 following s_waitcnt. 8074 - Performs L2 writeback to 8075 ensure previous 8076 global/generic 8077 store/atomicrmw are 8078 visible at system scope. 8079 8080 2. s_waitcnt lgkmcnt(0) & 8081 vmcnt(0) 8082 8083 - If TgSplit execution mode, 8084 omit lgkmcnt(0). 8085 - If OpenCL, omit 8086 lgkmcnt(0). 8087 - Could be split into 8088 separate s_waitcnt 8089 vmcnt(0) and 8090 s_waitcnt 8091 lgkmcnt(0) to allow 8092 them to be 8093 independently moved 8094 according to the 8095 following rules. 8096 - s_waitcnt vmcnt(0) 8097 must happen after 8098 any preceding 8099 global/generic 8100 load/store/load 8101 atomic/store 8102 atomic/atomicrmw. 8103 - s_waitcnt lgkmcnt(0) 8104 must happen after 8105 any preceding 8106 local/generic 8107 load/store/load 8108 atomic/store 8109 atomic/atomicrmw. 8110 - Must happen before 8111 the following 8112 atomicrmw. 8113 - Ensures that all 8114 memory operations 8115 to global and L2 writeback 8116 have completed before 8117 performing the 8118 atomicrmw that is 8119 being released. 8120 8121 3. flat_atomic 8122 4. s_waitcnt vmcnt(0) & 8123 lgkmcnt(0) 8124 8125 - If TgSplit execution mode, 8126 omit lgkmcnt(0). 8127 - If OpenCL, omit 8128 lgkmcnt(0). 8129 - Must happen before 8130 following buffer_invl2 and 8131 buffer_wbinvl1_vol. 8132 - Ensures the 8133 atomicrmw has 8134 completed before 8135 invalidating the 8136 caches. 8137 8138 5. buffer_invl2; 8139 buffer_wbinvl1_vol 8140 8141 - Must happen before 8142 any following 8143 global/generic 8144 load/load 8145 atomic/atomicrmw. 8146 - Ensures that 8147 following 8148 loads will not see 8149 stale L1 global data, 8150 nor see stale L2 MTYPE 8151 NC global data. 8152 MTYPE RW and CC memory will 8153 never be stale in L2 due to 8154 the memory probes. 8155 8156 fence acq_rel - singlethread *none* *none* 8157 - wavefront 8158 fence acq_rel - workgroup *none* 1. s_waitcnt lgkm/vmcnt(0) 8159 8160 - Use lgkmcnt(0) if not 8161 TgSplit execution mode 8162 and vmcnt(0) if TgSplit 8163 execution mode. 8164 - If OpenCL and 8165 address space is 8166 not generic, omit 8167 lgkmcnt(0). 8168 - If OpenCL and 8169 address space is 8170 local, omit 8171 vmcnt(0). 8172 - However, 8173 since LLVM 8174 currently has no 8175 address space on 8176 the fence need to 8177 conservatively 8178 always generate 8179 (see comment for 8180 previous fence). 8181 - s_waitcnt vmcnt(0) 8182 must happen after 8183 any preceding 8184 global/generic 8185 load/store/ 8186 load atomic/store atomic/ 8187 atomicrmw. 8188 - s_waitcnt lgkmcnt(0) 8189 must happen after 8190 any preceding 8191 local/generic 8192 load/load 8193 atomic/store/store 8194 atomic/atomicrmw. 8195 - Must happen before 8196 any following 8197 global/generic 8198 load/load 8199 atomic/store/store 8200 atomic/atomicrmw. 8201 - Ensures that all 8202 memory operations 8203 have 8204 completed before 8205 performing any 8206 following global 8207 memory operations. 8208 - Ensures that the 8209 preceding 8210 local/generic load 8211 atomic/atomicrmw 8212 with an equal or 8213 wider sync scope 8214 and memory ordering 8215 stronger than 8216 unordered (this is 8217 termed the 8218 acquire-fence-paired-atomic) 8219 has completed 8220 before following 8221 global memory 8222 operations. This 8223 satisfies the 8224 requirements of 8225 acquire. 8226 - Ensures that all 8227 previous memory 8228 operations have 8229 completed before a 8230 following 8231 local/generic store 8232 atomic/atomicrmw 8233 with an equal or 8234 wider sync scope 8235 and memory ordering 8236 stronger than 8237 unordered (this is 8238 termed the 8239 release-fence-paired-atomic). 8240 This satisfies the 8241 requirements of 8242 release. 8243 - Must happen before 8244 the following 8245 buffer_wbinvl1_vol. 8246 - Ensures that the 8247 acquire-fence-paired 8248 atomic has completed 8249 before invalidating 8250 the 8251 cache. Therefore 8252 any following 8253 locations read must 8254 be no older than 8255 the value read by 8256 the 8257 acquire-fence-paired-atomic. 8258 8259 2. buffer_wbinvl1_vol 8260 8261 - If not TgSplit execution 8262 mode, omit. 8263 - Ensures that 8264 following 8265 loads will not see 8266 stale data. 8267 8268 fence acq_rel - agent *none* 1. s_waitcnt lgkmcnt(0) & 8269 vmcnt(0) 8270 8271 - If TgSplit execution mode, 8272 omit lgkmcnt(0). 8273 - If OpenCL and 8274 address space is 8275 not generic, omit 8276 lgkmcnt(0). 8277 - However, since LLVM 8278 currently has no 8279 address space on 8280 the fence need to 8281 conservatively 8282 always generate 8283 (see comment for 8284 previous fence). 8285 - Could be split into 8286 separate s_waitcnt 8287 vmcnt(0) and 8288 s_waitcnt 8289 lgkmcnt(0) to allow 8290 them to be 8291 independently moved 8292 according to the 8293 following rules. 8294 - s_waitcnt vmcnt(0) 8295 must happen after 8296 any preceding 8297 global/generic 8298 load/store/load 8299 atomic/store 8300 atomic/atomicrmw. 8301 - s_waitcnt lgkmcnt(0) 8302 must happen after 8303 any preceding 8304 local/generic 8305 load/store/load 8306 atomic/store 8307 atomic/atomicrmw. 8308 - Must happen before 8309 the following 8310 buffer_wbinvl1_vol. 8311 - Ensures that the 8312 preceding 8313 global/local/generic 8314 load 8315 atomic/atomicrmw 8316 with an equal or 8317 wider sync scope 8318 and memory ordering 8319 stronger than 8320 unordered (this is 8321 termed the 8322 acquire-fence-paired-atomic) 8323 has completed 8324 before invalidating 8325 the cache. This 8326 satisfies the 8327 requirements of 8328 acquire. 8329 - Ensures that all 8330 previous memory 8331 operations have 8332 completed before a 8333 following 8334 global/local/generic 8335 store 8336 atomic/atomicrmw 8337 with an equal or 8338 wider sync scope 8339 and memory ordering 8340 stronger than 8341 unordered (this is 8342 termed the 8343 release-fence-paired-atomic). 8344 This satisfies the 8345 requirements of 8346 release. 8347 8348 2. buffer_wbinvl1_vol 8349 8350 - Must happen before 8351 any following 8352 global/generic 8353 load/load 8354 atomic/store/store 8355 atomic/atomicrmw. 8356 - Ensures that 8357 following loads 8358 will not see stale 8359 global data. This 8360 satisfies the 8361 requirements of 8362 acquire. 8363 8364 fence acq_rel - system *none* 1. buffer_wbl2 8365 8366 - If OpenCL and 8367 address space is 8368 local, omit. 8369 - Must happen before 8370 following s_waitcnt. 8371 - Performs L2 writeback to 8372 ensure previous 8373 global/generic 8374 store/atomicrmw are 8375 visible at system scope. 8376 8377 2. s_waitcnt lgkmcnt(0) & 8378 vmcnt(0) 8379 8380 - If TgSplit execution mode, 8381 omit lgkmcnt(0). 8382 - If OpenCL and 8383 address space is 8384 not generic, omit 8385 lgkmcnt(0). 8386 - However, since LLVM 8387 currently has no 8388 address space on 8389 the fence need to 8390 conservatively 8391 always generate 8392 (see comment for 8393 previous fence). 8394 - Could be split into 8395 separate s_waitcnt 8396 vmcnt(0) and 8397 s_waitcnt 8398 lgkmcnt(0) to allow 8399 them to be 8400 independently moved 8401 according to the 8402 following rules. 8403 - s_waitcnt vmcnt(0) 8404 must happen after 8405 any preceding 8406 global/generic 8407 load/store/load 8408 atomic/store 8409 atomic/atomicrmw. 8410 - s_waitcnt lgkmcnt(0) 8411 must happen after 8412 any preceding 8413 local/generic 8414 load/store/load 8415 atomic/store 8416 atomic/atomicrmw. 8417 - Must happen before 8418 the following buffer_invl2 and 8419 buffer_wbinvl1_vol. 8420 - Ensures that the 8421 preceding 8422 global/local/generic 8423 load 8424 atomic/atomicrmw 8425 with an equal or 8426 wider sync scope 8427 and memory ordering 8428 stronger than 8429 unordered (this is 8430 termed the 8431 acquire-fence-paired-atomic) 8432 has completed 8433 before invalidating 8434 the cache. This 8435 satisfies the 8436 requirements of 8437 acquire. 8438 - Ensures that all 8439 previous memory 8440 operations have 8441 completed before a 8442 following 8443 global/local/generic 8444 store 8445 atomic/atomicrmw 8446 with an equal or 8447 wider sync scope 8448 and memory ordering 8449 stronger than 8450 unordered (this is 8451 termed the 8452 release-fence-paired-atomic). 8453 This satisfies the 8454 requirements of 8455 release. 8456 8457 3. buffer_invl2; 8458 buffer_wbinvl1_vol 8459 8460 - Must happen before 8461 any following 8462 global/generic 8463 load/load 8464 atomic/store/store 8465 atomic/atomicrmw. 8466 - Ensures that 8467 following 8468 loads will not see 8469 stale L1 global data, 8470 nor see stale L2 MTYPE 8471 NC global data. 8472 MTYPE RW and CC memory will 8473 never be stale in L2 due to 8474 the memory probes. 8475 8476 **Sequential Consistent Atomic** 8477 ------------------------------------------------------------------------------------ 8478 load atomic seq_cst - singlethread - global *Same as corresponding 8479 - wavefront - local load atomic acquire, 8480 - generic except must generate 8481 all instructions even 8482 for OpenCL.* 8483 load atomic seq_cst - workgroup - global 1. s_waitcnt lgkm/vmcnt(0) 8484 - generic 8485 - Use lgkmcnt(0) if not 8486 TgSplit execution mode 8487 and vmcnt(0) if TgSplit 8488 execution mode. 8489 - s_waitcnt lgkmcnt(0) must 8490 happen after 8491 preceding 8492 local/generic load 8493 atomic/store 8494 atomic/atomicrmw 8495 with memory 8496 ordering of seq_cst 8497 and with equal or 8498 wider sync scope. 8499 (Note that seq_cst 8500 fences have their 8501 own s_waitcnt 8502 lgkmcnt(0) and so do 8503 not need to be 8504 considered.) 8505 - s_waitcnt vmcnt(0) 8506 must happen after 8507 preceding 8508 global/generic load 8509 atomic/store 8510 atomic/atomicrmw 8511 with memory 8512 ordering of seq_cst 8513 and with equal or 8514 wider sync scope. 8515 (Note that seq_cst 8516 fences have their 8517 own s_waitcnt 8518 vmcnt(0) and so do 8519 not need to be 8520 considered.) 8521 - Ensures any 8522 preceding 8523 sequential 8524 consistent global/local 8525 memory instructions 8526 have completed 8527 before executing 8528 this sequentially 8529 consistent 8530 instruction. This 8531 prevents reordering 8532 a seq_cst store 8533 followed by a 8534 seq_cst load. (Note 8535 that seq_cst is 8536 stronger than 8537 acquire/release as 8538 the reordering of 8539 load acquire 8540 followed by a store 8541 release is 8542 prevented by the 8543 s_waitcnt of 8544 the release, but 8545 there is nothing 8546 preventing a store 8547 release followed by 8548 load acquire from 8549 completing out of 8550 order. The s_waitcnt 8551 could be placed after 8552 seq_store or before 8553 the seq_load. We 8554 choose the load to 8555 make the s_waitcnt be 8556 as late as possible 8557 so that the store 8558 may have already 8559 completed.) 8560 8561 2. *Following 8562 instructions same as 8563 corresponding load 8564 atomic acquire, 8565 except must generate 8566 all instructions even 8567 for OpenCL.* 8568 load atomic seq_cst - workgroup - local *If TgSplit execution mode, 8569 local address space cannot 8570 be used.* 8571 8572 *Same as corresponding 8573 load atomic acquire, 8574 except must generate 8575 all instructions even 8576 for OpenCL.* 8577 8578 load atomic seq_cst - agent - global 1. s_waitcnt lgkmcnt(0) & 8579 - system - generic vmcnt(0) 8580 8581 - If TgSplit execution mode, 8582 omit lgkmcnt(0). 8583 - Could be split into 8584 separate s_waitcnt 8585 vmcnt(0) 8586 and s_waitcnt 8587 lgkmcnt(0) to allow 8588 them to be 8589 independently moved 8590 according to the 8591 following rules. 8592 - s_waitcnt lgkmcnt(0) 8593 must happen after 8594 preceding 8595 global/generic load 8596 atomic/store 8597 atomic/atomicrmw 8598 with memory 8599 ordering of seq_cst 8600 and with equal or 8601 wider sync scope. 8602 (Note that seq_cst 8603 fences have their 8604 own s_waitcnt 8605 lgkmcnt(0) and so do 8606 not need to be 8607 considered.) 8608 - s_waitcnt vmcnt(0) 8609 must happen after 8610 preceding 8611 global/generic load 8612 atomic/store 8613 atomic/atomicrmw 8614 with memory 8615 ordering of seq_cst 8616 and with equal or 8617 wider sync scope. 8618 (Note that seq_cst 8619 fences have their 8620 own s_waitcnt 8621 vmcnt(0) and so do 8622 not need to be 8623 considered.) 8624 - Ensures any 8625 preceding 8626 sequential 8627 consistent global 8628 memory instructions 8629 have completed 8630 before executing 8631 this sequentially 8632 consistent 8633 instruction. This 8634 prevents reordering 8635 a seq_cst store 8636 followed by a 8637 seq_cst load. (Note 8638 that seq_cst is 8639 stronger than 8640 acquire/release as 8641 the reordering of 8642 load acquire 8643 followed by a store 8644 release is 8645 prevented by the 8646 s_waitcnt of 8647 the release, but 8648 there is nothing 8649 preventing a store 8650 release followed by 8651 load acquire from 8652 completing out of 8653 order. The s_waitcnt 8654 could be placed after 8655 seq_store or before 8656 the seq_load. We 8657 choose the load to 8658 make the s_waitcnt be 8659 as late as possible 8660 so that the store 8661 may have already 8662 completed.) 8663 8664 2. *Following 8665 instructions same as 8666 corresponding load 8667 atomic acquire, 8668 except must generate 8669 all instructions even 8670 for OpenCL.* 8671 store atomic seq_cst - singlethread - global *Same as corresponding 8672 - wavefront - local store atomic release, 8673 - workgroup - generic except must generate 8674 - agent all instructions even 8675 - system for OpenCL.* 8676 atomicrmw seq_cst - singlethread - global *Same as corresponding 8677 - wavefront - local atomicrmw acq_rel, 8678 - workgroup - generic except must generate 8679 - agent all instructions even 8680 - system for OpenCL.* 8681 fence seq_cst - singlethread *none* *Same as corresponding 8682 - wavefront fence acq_rel, 8683 - workgroup except must generate 8684 - agent all instructions even 8685 - system for OpenCL.* 8686 ============ ============ ============== ========== ================================ 8687 8688.. _amdgpu-amdhsa-memory-model-gfx940: 8689 8690Memory Model GFX940 8691+++++++++++++++++++ 8692 8693For GFX940: 8694 8695* Each agent has multiple shader arrays (SA). 8696* Each SA has multiple compute units (CU). 8697* Each CU has multiple SIMDs that execute wavefronts. 8698* The wavefronts for a single work-group are executed in the same CU but may be 8699 executed by different SIMDs. The exception is when in tgsplit execution mode 8700 when the wavefronts may be executed by different SIMDs in different CUs. 8701* Each CU has a single LDS memory shared by the wavefronts of the work-groups 8702 executing on it. The exception is when in tgsplit execution mode when no LDS 8703 is allocated as wavefronts of the same work-group can be in different CUs. 8704* All LDS operations of a CU are performed as wavefront wide operations in a 8705 global order and involve no caching. Completion is reported to a wavefront in 8706 execution order. 8707* The LDS memory has multiple request queues shared by the SIMDs of a 8708 CU. Therefore, the LDS operations performed by different wavefronts of a 8709 work-group can be reordered relative to each other, which can result in 8710 reordering the visibility of vector memory operations with respect to LDS 8711 operations of other wavefronts in the same work-group. A ``s_waitcnt 8712 lgkmcnt(0)`` is required to ensure synchronization between LDS operations and 8713 vector memory operations between wavefronts of a work-group, but not between 8714 operations performed by the same wavefront. 8715* The vector memory operations are performed as wavefront wide operations and 8716 completion is reported to a wavefront in execution order. The exception is 8717 that ``flat_load/store/atomic`` instructions can report out of vector memory 8718 order if they access LDS memory, and out of LDS operation order if they access 8719 global memory. 8720* The vector memory operations access a single vector L1 cache shared by all 8721 SIMDs a CU. Therefore: 8722 8723 * No special action is required for coherence between the lanes of a single 8724 wavefront. 8725 8726 * No special action is required for coherence between wavefronts in the same 8727 work-group since they execute on the same CU. The exception is when in 8728 tgsplit execution mode as wavefronts of the same work-group can be in 8729 different CUs and so a ``buffer_inv sc0`` is required which will invalidate 8730 the L1 cache. 8731 8732 * A ``buffer_inv sc0`` is required to invalidate the L1 cache for coherence 8733 between wavefronts executing in different work-groups as they may be 8734 executing on different CUs. 8735 8736 * Atomic read-modify-write instructions implicitly bypass the L1 cache. 8737 Therefore, they do not use the sc0 bit for coherence and instead use it to 8738 indicate if the instruction returns the original value being updated. They 8739 do use sc1 to indicate system or agent scope coherence. 8740 8741* The scalar memory operations access a scalar L1 cache shared by all wavefronts 8742 on a group of CUs. The scalar and vector L1 caches are not coherent. However, 8743 scalar operations are used in a restricted way so do not impact the memory 8744 model. See :ref:`amdgpu-amdhsa-memory-spaces`. 8745* The vector and scalar memory operations use an L2 cache. 8746 8747 * The gfx940 can be configured as a number of smaller agents with each having 8748 a single L2 shared by all CUs on the same agent, or as fewer (possibly one) 8749 larger agents with groups of CUs on each agent each sharing separate L2 8750 caches. 8751 * The L2 cache has independent channels to service disjoint ranges of virtual 8752 addresses. 8753 * Each CU has a separate request queue per channel for its associated L2. 8754 Therefore, the vector and scalar memory operations performed by wavefronts 8755 executing with different L1 caches and the same L2 cache can be reordered 8756 relative to each other. 8757 * A ``s_waitcnt vmcnt(0)`` is required to ensure synchronization between 8758 vector memory operations of different CUs. It ensures a previous vector 8759 memory operation has completed before executing a subsequent vector memory 8760 or LDS operation and so can be used to meet the requirements of acquire and 8761 release. 8762 * An L2 cache can be kept coherent with other L2 caches by using the MTYPE RW 8763 (read-write) for memory local to the L2, and MTYPE NC (non-coherent) with 8764 the PTE C-bit set for memory not local to the L2. 8765 8766 * Any local memory cache lines will be automatically invalidated by writes 8767 from CUs associated with other L2 caches, or writes from the CPU, due to 8768 the cache probe caused by the PTE C-bit. 8769 * XGMI accesses from the CPU to local memory may be cached on the CPU. 8770 Subsequent access from the GPU will automatically invalidate or writeback 8771 the CPU cache due to the L2 probe filter. 8772 * To ensure coherence of local memory writes of CUs with different L1 caches 8773 in the same agent a ``buffer_wbl2`` is required. It does nothing if the 8774 agent is configured to have a single L2, or will writeback dirty L2 cache 8775 lines if configured to have multiple L2 caches. 8776 * To ensure coherence of local memory writes of CUs in different agents a 8777 ``buffer_wbl2 sc1`` is required. It will writeback dirty L2 cache lines. 8778 * To ensure coherence of local memory reads of CUs with different L1 caches 8779 in the same agent a ``buffer_inv sc1`` is required. It does nothing if the 8780 agent is configured to have a single L2, or will invalidate non-local L2 8781 cache lines if configured to have multiple L2 caches. 8782 * To ensure coherence of local memory reads of CUs in different agents a 8783 ``buffer_inv sc0 sc1`` is required. It will invalidate non-local L2 cache 8784 lines if configured to have multiple L2 caches. 8785 8786 * PCIe access from the GPU to the CPU can be kept coherent by using the MTYPE 8787 UC (uncached) which bypasses the L2. 8788 8789Scalar memory operations are only used to access memory that is proven to not 8790change during the execution of the kernel dispatch. This includes constant 8791address space and global address space for program scope ``const`` variables. 8792Therefore, the kernel machine code does not have to maintain the scalar cache to 8793ensure it is coherent with the vector caches. The scalar and vector caches are 8794invalidated between kernel dispatches by CP since constant address space data 8795may change between kernel dispatch executions. See 8796:ref:`amdgpu-amdhsa-memory-spaces`. 8797 8798The one exception is if scalar writes are used to spill SGPR registers. In this 8799case the AMDGPU backend ensures the memory location used to spill is never 8800accessed by vector memory operations at the same time. If scalar writes are used 8801then a ``s_dcache_wb`` is inserted before the ``s_endpgm`` and before a function 8802return since the locations may be used for vector memory instructions by a 8803future wavefront that uses the same scratch area, or a function call that 8804creates a frame at the same address, respectively. There is no need for a 8805``s_dcache_inv`` as all scalar writes are write-before-read in the same thread. 8806 8807For kernarg backing memory: 8808 8809* CP invalidates the L1 cache at the start of each kernel dispatch. 8810* On dGPU over XGMI or PCIe the kernarg backing memory is allocated in host 8811 memory accessed as MTYPE UC (uncached) to avoid needing to invalidate the L2 8812 cache. This also causes it to be treated as non-volatile and so is not 8813 invalidated by ``*_vol``. 8814* On APU the kernarg backing memory is accessed as MTYPE CC (cache coherent) and 8815 so the L2 cache will be coherent with the CPU and other agents. 8816 8817Scratch backing memory (which is used for the private address space) is accessed 8818with MTYPE NC_NV (non-coherent non-volatile). Since the private address space is 8819only accessed by a single thread, and is always write-before-read, there is 8820never a need to invalidate these entries from the L1 cache. Hence all cache 8821invalidates are done as ``*_vol`` to only invalidate the volatile cache lines. 8822 8823The code sequences used to implement the memory model for GFX940 are defined 8824in table :ref:`amdgpu-amdhsa-memory-model-code-sequences-gfx940-table`. 8825 8826 .. table:: AMDHSA Memory Model Code Sequences GFX940 8827 :name: amdgpu-amdhsa-memory-model-code-sequences-gfx940-table 8828 8829 ============ ============ ============== ========== ================================ 8830 LLVM Instr LLVM Memory LLVM Memory AMDGPU AMDGPU Machine Code 8831 Ordering Sync Scope Address GFX940 8832 Space 8833 ============ ============ ============== ========== ================================ 8834 **Non-Atomic** 8835 ------------------------------------------------------------------------------------ 8836 load *none* *none* - global - !volatile & !nontemporal 8837 - generic 8838 - private 1. buffer/global/flat_load 8839 - constant 8840 - !volatile & nontemporal 8841 8842 1. buffer/global/flat_load 8843 nt=1 8844 8845 - volatile 8846 8847 1. buffer/global/flat_load 8848 sc0=1 sc1=1 8849 2. s_waitcnt vmcnt(0) 8850 8851 - Must happen before 8852 any following volatile 8853 global/generic 8854 load/store. 8855 - Ensures that 8856 volatile 8857 operations to 8858 different 8859 addresses will not 8860 be reordered by 8861 hardware. 8862 8863 load *none* *none* - local 1. ds_load 8864 store *none* *none* - global - !volatile & !nontemporal 8865 - generic 8866 - private 1. buffer/global/flat_store 8867 - constant 8868 - !volatile & nontemporal 8869 8870 1. buffer/global/flat_store 8871 nt=1 8872 8873 - volatile 8874 8875 1. buffer/global/flat_store 8876 sc0=1 sc1=1 8877 2. s_waitcnt vmcnt(0) 8878 8879 - Must happen before 8880 any following volatile 8881 global/generic 8882 load/store. 8883 - Ensures that 8884 volatile 8885 operations to 8886 different 8887 addresses will not 8888 be reordered by 8889 hardware. 8890 8891 store *none* *none* - local 1. ds_store 8892 **Unordered Atomic** 8893 ------------------------------------------------------------------------------------ 8894 load atomic unordered *any* *any* *Same as non-atomic*. 8895 store atomic unordered *any* *any* *Same as non-atomic*. 8896 atomicrmw unordered *any* *any* *Same as monotonic atomic*. 8897 **Monotonic Atomic** 8898 ------------------------------------------------------------------------------------ 8899 load atomic monotonic - singlethread - global 1. buffer/global/flat_load 8900 - wavefront - generic 8901 load atomic monotonic - workgroup - global 1. buffer/global/flat_load 8902 - generic sc0=1 8903 load atomic monotonic - singlethread - local *If TgSplit execution mode, 8904 - wavefront local address space cannot 8905 - workgroup be used.* 8906 8907 1. ds_load 8908 load atomic monotonic - agent - global 1. buffer/global/flat_load 8909 - generic sc1=1 8910 load atomic monotonic - system - global 1. buffer/global/flat_load 8911 - generic sc0=1 sc1=1 8912 store atomic monotonic - singlethread - global 1. buffer/global/flat_store 8913 - wavefront - generic 8914 store atomic monotonic - workgroup - global 1. buffer/global/flat_store 8915 - generic sc0=1 8916 store atomic monotonic - agent - global 1. buffer/global/flat_store 8917 - generic sc1=1 8918 store atomic monotonic - system - global 1. buffer/global/flat_store 8919 - generic sc0=1 sc1=1 8920 store atomic monotonic - singlethread - local *If TgSplit execution mode, 8921 - wavefront local address space cannot 8922 - workgroup be used.* 8923 8924 1. ds_store 8925 atomicrmw monotonic - singlethread - global 1. buffer/global/flat_atomic 8926 - wavefront - generic 8927 - workgroup 8928 - agent 8929 atomicrmw monotonic - system - global 1. buffer/global/flat_atomic 8930 - generic sc1=1 8931 atomicrmw monotonic - singlethread - local *If TgSplit execution mode, 8932 - wavefront local address space cannot 8933 - workgroup be used.* 8934 8935 1. ds_atomic 8936 **Acquire Atomic** 8937 ------------------------------------------------------------------------------------ 8938 load atomic acquire - singlethread - global 1. buffer/global/ds/flat_load 8939 - wavefront - local 8940 - generic 8941 load atomic acquire - workgroup - global 1. buffer/global_load sc0=1 8942 2. s_waitcnt vmcnt(0) 8943 8944 - If not TgSplit execution 8945 mode, omit. 8946 - Must happen before the 8947 following buffer_inv. 8948 8949 3. buffer_inv sc0=1 8950 8951 - If not TgSplit execution 8952 mode, omit. 8953 - Must happen before 8954 any following 8955 global/generic 8956 load/load 8957 atomic/store/store 8958 atomic/atomicrmw. 8959 - Ensures that 8960 following 8961 loads will not see 8962 stale data. 8963 8964 load atomic acquire - workgroup - local *If TgSplit execution mode, 8965 local address space cannot 8966 be used.* 8967 8968 1. ds_load 8969 2. s_waitcnt lgkmcnt(0) 8970 8971 - If OpenCL, omit. 8972 - Must happen before 8973 any following 8974 global/generic 8975 load/load 8976 atomic/store/store 8977 atomic/atomicrmw. 8978 - Ensures any 8979 following global 8980 data read is no 8981 older than the local load 8982 atomic value being 8983 acquired. 8984 8985 load atomic acquire - workgroup - generic 1. flat_load sc0=1 8986 2. s_waitcnt lgkm/vmcnt(0) 8987 8988 - Use lgkmcnt(0) if not 8989 TgSplit execution mode 8990 and vmcnt(0) if TgSplit 8991 execution mode. 8992 - If OpenCL, omit lgkmcnt(0). 8993 - Must happen before 8994 the following 8995 buffer_inv and any 8996 following global/generic 8997 load/load 8998 atomic/store/store 8999 atomic/atomicrmw. 9000 - Ensures any 9001 following global 9002 data read is no 9003 older than a local load 9004 atomic value being 9005 acquired. 9006 9007 3. buffer_inv sc0=1 9008 9009 - If not TgSplit execution 9010 mode, omit. 9011 - Ensures that 9012 following 9013 loads will not see 9014 stale data. 9015 9016 load atomic acquire - agent - global 1. buffer/global_load 9017 sc1=1 9018 2. s_waitcnt vmcnt(0) 9019 9020 - Must happen before 9021 following 9022 buffer_inv. 9023 - Ensures the load 9024 has completed 9025 before invalidating 9026 the cache. 9027 9028 3. buffer_inv sc1=1 9029 9030 - Must happen before 9031 any following 9032 global/generic 9033 load/load 9034 atomic/atomicrmw. 9035 - Ensures that 9036 following 9037 loads will not see 9038 stale global data. 9039 9040 load atomic acquire - system - global 1. buffer/global/flat_load 9041 sc0=1 sc1=1 9042 2. s_waitcnt vmcnt(0) 9043 9044 - Must happen before 9045 following 9046 buffer_inv. 9047 - Ensures the load 9048 has completed 9049 before invalidating 9050 the cache. 9051 9052 3. buffer_inv sc0=1 sc1=1 9053 9054 - Must happen before 9055 any following 9056 global/generic 9057 load/load 9058 atomic/atomicrmw. 9059 - Ensures that 9060 following 9061 loads will not see 9062 stale MTYPE NC global data. 9063 MTYPE RW and CC memory will 9064 never be stale due to the 9065 memory probes. 9066 9067 load atomic acquire - agent - generic 1. flat_load sc1=1 9068 2. s_waitcnt vmcnt(0) & 9069 lgkmcnt(0) 9070 9071 - If TgSplit execution mode, 9072 omit lgkmcnt(0). 9073 - If OpenCL omit 9074 lgkmcnt(0). 9075 - Must happen before 9076 following 9077 buffer_inv. 9078 - Ensures the flat_load 9079 has completed 9080 before invalidating 9081 the cache. 9082 9083 3. buffer_inv sc1=1 9084 9085 - Must happen before 9086 any following 9087 global/generic 9088 load/load 9089 atomic/atomicrmw. 9090 - Ensures that 9091 following loads 9092 will not see stale 9093 global data. 9094 9095 load atomic acquire - system - generic 1. flat_load sc0=1 sc1=1 9096 2. s_waitcnt vmcnt(0) & 9097 lgkmcnt(0) 9098 9099 - If TgSplit execution mode, 9100 omit lgkmcnt(0). 9101 - If OpenCL omit 9102 lgkmcnt(0). 9103 - Must happen before 9104 the following 9105 buffer_inv. 9106 - Ensures the flat_load 9107 has completed 9108 before invalidating 9109 the caches. 9110 9111 3. buffer_inv sc0=1 sc1=1 9112 9113 - Must happen before 9114 any following 9115 global/generic 9116 load/load 9117 atomic/atomicrmw. 9118 - Ensures that 9119 following 9120 loads will not see 9121 stale MTYPE NC global data. 9122 MTYPE RW and CC memory will 9123 never be stale due to the 9124 memory probes. 9125 9126 atomicrmw acquire - singlethread - global 1. buffer/global/flat_atomic 9127 - wavefront - generic 9128 atomicrmw acquire - singlethread - local *If TgSplit execution mode, 9129 - wavefront local address space cannot 9130 be used.* 9131 9132 1. ds_atomic 9133 atomicrmw acquire - workgroup - global 1. buffer/global_atomic 9134 2. s_waitcnt vmcnt(0) 9135 9136 - If not TgSplit execution 9137 mode, omit. 9138 - Must happen before the 9139 following buffer_inv. 9140 - Ensures the atomicrmw 9141 has completed 9142 before invalidating 9143 the cache. 9144 9145 3. buffer_inv sc0=1 9146 9147 - If not TgSplit execution 9148 mode, omit. 9149 - Must happen before 9150 any following 9151 global/generic 9152 load/load 9153 atomic/atomicrmw. 9154 - Ensures that 9155 following loads 9156 will not see stale 9157 global data. 9158 9159 atomicrmw acquire - workgroup - local *If TgSplit execution mode, 9160 local address space cannot 9161 be used.* 9162 9163 1. ds_atomic 9164 2. s_waitcnt lgkmcnt(0) 9165 9166 - If OpenCL, omit. 9167 - Must happen before 9168 any following 9169 global/generic 9170 load/load 9171 atomic/store/store 9172 atomic/atomicrmw. 9173 - Ensures any 9174 following global 9175 data read is no 9176 older than the local 9177 atomicrmw value 9178 being acquired. 9179 9180 atomicrmw acquire - workgroup - generic 1. flat_atomic 9181 2. s_waitcnt lgkm/vmcnt(0) 9182 9183 - Use lgkmcnt(0) if not 9184 TgSplit execution mode 9185 and vmcnt(0) if TgSplit 9186 execution mode. 9187 - If OpenCL, omit lgkmcnt(0). 9188 - Must happen before 9189 the following 9190 buffer_inv and 9191 any following 9192 global/generic 9193 load/load 9194 atomic/store/store 9195 atomic/atomicrmw. 9196 - Ensures any 9197 following global 9198 data read is no 9199 older than a local 9200 atomicrmw value 9201 being acquired. 9202 9203 3. buffer_inv sc0=1 9204 9205 - If not TgSplit execution 9206 mode, omit. 9207 - Ensures that 9208 following 9209 loads will not see 9210 stale data. 9211 9212 atomicrmw acquire - agent - global 1. buffer/global_atomic 9213 2. s_waitcnt vmcnt(0) 9214 9215 - Must happen before 9216 following 9217 buffer_inv. 9218 - Ensures the 9219 atomicrmw has 9220 completed before 9221 invalidating the 9222 cache. 9223 9224 3. buffer_inv sc1=1 9225 9226 - Must happen before 9227 any following 9228 global/generic 9229 load/load 9230 atomic/atomicrmw. 9231 - Ensures that 9232 following loads 9233 will not see stale 9234 global data. 9235 9236 atomicrmw acquire - system - global 1. buffer/global_atomic 9237 sc1=1 9238 2. s_waitcnt vmcnt(0) 9239 9240 - Must happen before 9241 following 9242 buffer_inv. 9243 - Ensures the 9244 atomicrmw has 9245 completed before 9246 invalidating the 9247 caches. 9248 9249 3. buffer_inv sc0=1 sc1=1 9250 9251 - Must happen before 9252 any following 9253 global/generic 9254 load/load 9255 atomic/atomicrmw. 9256 - Ensures that 9257 following 9258 loads will not see 9259 stale MTYPE NC global data. 9260 MTYPE RW and CC memory will 9261 never be stale due to the 9262 memory probes. 9263 9264 atomicrmw acquire - agent - generic 1. flat_atomic 9265 2. s_waitcnt vmcnt(0) & 9266 lgkmcnt(0) 9267 9268 - If TgSplit execution mode, 9269 omit lgkmcnt(0). 9270 - If OpenCL, omit 9271 lgkmcnt(0). 9272 - Must happen before 9273 following 9274 buffer_inv. 9275 - Ensures the 9276 atomicrmw has 9277 completed before 9278 invalidating the 9279 cache. 9280 9281 3. buffer_inv sc1=1 9282 9283 - Must happen before 9284 any following 9285 global/generic 9286 load/load 9287 atomic/atomicrmw. 9288 - Ensures that 9289 following loads 9290 will not see stale 9291 global data. 9292 9293 atomicrmw acquire - system - generic 1. flat_atomic sc1=1 9294 2. s_waitcnt vmcnt(0) & 9295 lgkmcnt(0) 9296 9297 - If TgSplit execution mode, 9298 omit lgkmcnt(0). 9299 - If OpenCL, omit 9300 lgkmcnt(0). 9301 - Must happen before 9302 following 9303 buffer_inv. 9304 - Ensures the 9305 atomicrmw has 9306 completed before 9307 invalidating the 9308 caches. 9309 9310 3. buffer_inv sc0=1 sc1=1 9311 9312 - Must happen before 9313 any following 9314 global/generic 9315 load/load 9316 atomic/atomicrmw. 9317 - Ensures that 9318 following 9319 loads will not see 9320 stale MTYPE NC global data. 9321 MTYPE RW and CC memory will 9322 never be stale due to the 9323 memory probes. 9324 9325 fence acquire - singlethread *none* *none* 9326 - wavefront 9327 fence acquire - workgroup *none* 1. s_waitcnt lgkm/vmcnt(0) 9328 9329 - Use lgkmcnt(0) if not 9330 TgSplit execution mode 9331 and vmcnt(0) if TgSplit 9332 execution mode. 9333 - If OpenCL and 9334 address space is 9335 not generic, omit 9336 lgkmcnt(0). 9337 - If OpenCL and 9338 address space is 9339 local, omit 9340 vmcnt(0). 9341 - However, since LLVM 9342 currently has no 9343 address space on 9344 the fence need to 9345 conservatively 9346 always generate. If 9347 fence had an 9348 address space then 9349 set to address 9350 space of OpenCL 9351 fence flag, or to 9352 generic if both 9353 local and global 9354 flags are 9355 specified. 9356 - s_waitcnt vmcnt(0) 9357 must happen after 9358 any preceding 9359 global/generic load 9360 atomic/ 9361 atomicrmw 9362 with an equal or 9363 wider sync scope 9364 and memory ordering 9365 stronger than 9366 unordered (this is 9367 termed the 9368 fence-paired-atomic). 9369 - s_waitcnt lgkmcnt(0) 9370 must happen after 9371 any preceding 9372 local/generic load 9373 atomic/atomicrmw 9374 with an equal or 9375 wider sync scope 9376 and memory ordering 9377 stronger than 9378 unordered (this is 9379 termed the 9380 fence-paired-atomic). 9381 - Must happen before 9382 the following 9383 buffer_inv and 9384 any following 9385 global/generic 9386 load/load 9387 atomic/store/store 9388 atomic/atomicrmw. 9389 - Ensures any 9390 following global 9391 data read is no 9392 older than the 9393 value read by the 9394 fence-paired-atomic. 9395 9396 3. buffer_inv sc0=1 9397 9398 - If not TgSplit execution 9399 mode, omit. 9400 - Ensures that 9401 following 9402 loads will not see 9403 stale data. 9404 9405 fence acquire - agent *none* 1. s_waitcnt lgkmcnt(0) & 9406 vmcnt(0) 9407 9408 - If TgSplit execution mode, 9409 omit lgkmcnt(0). 9410 - If OpenCL and 9411 address space is 9412 not generic, omit 9413 lgkmcnt(0). 9414 - However, since LLVM 9415 currently has no 9416 address space on 9417 the fence need to 9418 conservatively 9419 always generate 9420 (see comment for 9421 previous fence). 9422 - Could be split into 9423 separate s_waitcnt 9424 vmcnt(0) and 9425 s_waitcnt 9426 lgkmcnt(0) to allow 9427 them to be 9428 independently moved 9429 according to the 9430 following rules. 9431 - s_waitcnt vmcnt(0) 9432 must happen after 9433 any preceding 9434 global/generic load 9435 atomic/atomicrmw 9436 with an equal or 9437 wider sync scope 9438 and memory ordering 9439 stronger than 9440 unordered (this is 9441 termed the 9442 fence-paired-atomic). 9443 - s_waitcnt lgkmcnt(0) 9444 must happen after 9445 any preceding 9446 local/generic load 9447 atomic/atomicrmw 9448 with an equal or 9449 wider sync scope 9450 and memory ordering 9451 stronger than 9452 unordered (this is 9453 termed the 9454 fence-paired-atomic). 9455 - Must happen before 9456 the following 9457 buffer_inv. 9458 - Ensures that the 9459 fence-paired atomic 9460 has completed 9461 before invalidating 9462 the 9463 cache. Therefore 9464 any following 9465 locations read must 9466 be no older than 9467 the value read by 9468 the 9469 fence-paired-atomic. 9470 9471 2. buffer_inv sc1=1 9472 9473 - Must happen before any 9474 following global/generic 9475 load/load 9476 atomic/store/store 9477 atomic/atomicrmw. 9478 - Ensures that 9479 following loads 9480 will not see stale 9481 global data. 9482 9483 fence acquire - system *none* 1. s_waitcnt lgkmcnt(0) & 9484 vmcnt(0) 9485 9486 - If TgSplit execution mode, 9487 omit lgkmcnt(0). 9488 - If OpenCL and 9489 address space is 9490 not generic, omit 9491 lgkmcnt(0). 9492 - However, since LLVM 9493 currently has no 9494 address space on 9495 the fence need to 9496 conservatively 9497 always generate 9498 (see comment for 9499 previous fence). 9500 - Could be split into 9501 separate s_waitcnt 9502 vmcnt(0) and 9503 s_waitcnt 9504 lgkmcnt(0) to allow 9505 them to be 9506 independently moved 9507 according to the 9508 following rules. 9509 - s_waitcnt vmcnt(0) 9510 must happen after 9511 any preceding 9512 global/generic load 9513 atomic/atomicrmw 9514 with an equal or 9515 wider sync scope 9516 and memory ordering 9517 stronger than 9518 unordered (this is 9519 termed the 9520 fence-paired-atomic). 9521 - s_waitcnt lgkmcnt(0) 9522 must happen after 9523 any preceding 9524 local/generic load 9525 atomic/atomicrmw 9526 with an equal or 9527 wider sync scope 9528 and memory ordering 9529 stronger than 9530 unordered (this is 9531 termed the 9532 fence-paired-atomic). 9533 - Must happen before 9534 the following 9535 buffer_inv. 9536 - Ensures that the 9537 fence-paired atomic 9538 has completed 9539 before invalidating 9540 the 9541 cache. Therefore 9542 any following 9543 locations read must 9544 be no older than 9545 the value read by 9546 the 9547 fence-paired-atomic. 9548 9549 2. buffer_inv sc0=1 sc1=1 9550 9551 - Must happen before any 9552 following global/generic 9553 load/load 9554 atomic/store/store 9555 atomic/atomicrmw. 9556 - Ensures that 9557 following loads 9558 will not see stale 9559 global data. 9560 9561 **Release Atomic** 9562 ------------------------------------------------------------------------------------ 9563 store atomic release - singlethread - global 1. buffer/global/flat_store 9564 - wavefront - generic 9565 store atomic release - singlethread - local *If TgSplit execution mode, 9566 - wavefront local address space cannot 9567 be used.* 9568 9569 1. ds_store 9570 store atomic release - workgroup - global 1. s_waitcnt lgkm/vmcnt(0) 9571 - generic 9572 - Use lgkmcnt(0) if not 9573 TgSplit execution mode 9574 and vmcnt(0) if TgSplit 9575 execution mode. 9576 - If OpenCL, omit lgkmcnt(0). 9577 - s_waitcnt vmcnt(0) 9578 must happen after 9579 any preceding 9580 global/generic load/store/ 9581 load atomic/store atomic/ 9582 atomicrmw. 9583 - s_waitcnt lgkmcnt(0) 9584 must happen after 9585 any preceding 9586 local/generic 9587 load/store/load 9588 atomic/store 9589 atomic/atomicrmw. 9590 - Must happen before 9591 the following 9592 store. 9593 - Ensures that all 9594 memory operations 9595 have 9596 completed before 9597 performing the 9598 store that is being 9599 released. 9600 9601 2. buffer/global/flat_store sc0=1 9602 store atomic release - workgroup - local *If TgSplit execution mode, 9603 local address space cannot 9604 be used.* 9605 9606 1. ds_store 9607 store atomic release - agent - global 1. buffer_wbl2 sc1=1 9608 - generic 9609 - Must happen before 9610 following s_waitcnt. 9611 - Performs L2 writeback to 9612 ensure previous 9613 global/generic 9614 store/atomicrmw are 9615 visible at agent scope. 9616 9617 2. s_waitcnt lgkmcnt(0) & 9618 vmcnt(0) 9619 9620 - If TgSplit execution mode, 9621 omit lgkmcnt(0). 9622 - If OpenCL and 9623 address space is 9624 not generic, omit 9625 lgkmcnt(0). 9626 - Could be split into 9627 separate s_waitcnt 9628 vmcnt(0) and 9629 s_waitcnt 9630 lgkmcnt(0) to allow 9631 them to be 9632 independently moved 9633 according to the 9634 following rules. 9635 - s_waitcnt vmcnt(0) 9636 must happen after 9637 any preceding 9638 global/generic 9639 load/store/load 9640 atomic/store 9641 atomic/atomicrmw. 9642 - s_waitcnt lgkmcnt(0) 9643 must happen after 9644 any preceding 9645 local/generic 9646 load/store/load 9647 atomic/store 9648 atomic/atomicrmw. 9649 - Must happen before 9650 the following 9651 store. 9652 - Ensures that all 9653 memory operations 9654 to memory have 9655 completed before 9656 performing the 9657 store that is being 9658 released. 9659 9660 3. buffer/global/flat_store sc1=1 9661 store atomic release - system - global 1. buffer_wbl2 sc0=1 sc1=1 9662 - generic 9663 - Must happen before 9664 following s_waitcnt. 9665 - Performs L2 writeback to 9666 ensure previous 9667 global/generic 9668 store/atomicrmw are 9669 visible at system scope. 9670 9671 2. s_waitcnt lgkmcnt(0) & 9672 vmcnt(0) 9673 9674 - If TgSplit execution mode, 9675 omit lgkmcnt(0). 9676 - If OpenCL and 9677 address space is 9678 not generic, omit 9679 lgkmcnt(0). 9680 - Could be split into 9681 separate s_waitcnt 9682 vmcnt(0) and 9683 s_waitcnt 9684 lgkmcnt(0) to allow 9685 them to be 9686 independently moved 9687 according to the 9688 following rules. 9689 - s_waitcnt vmcnt(0) 9690 must happen after any 9691 preceding 9692 global/generic 9693 load/store/load 9694 atomic/store 9695 atomic/atomicrmw. 9696 - s_waitcnt lgkmcnt(0) 9697 must happen after any 9698 preceding 9699 local/generic 9700 load/store/load 9701 atomic/store 9702 atomic/atomicrmw. 9703 - Must happen before 9704 the following 9705 store. 9706 - Ensures that all 9707 memory operations 9708 to memory and the L2 9709 writeback have 9710 completed before 9711 performing the 9712 store that is being 9713 released. 9714 9715 3. buffer/global/flat_store 9716 sc0=1 sc1=1 9717 atomicrmw release - singlethread - global 1. buffer/global/flat_atomic 9718 - wavefront - generic 9719 atomicrmw release - singlethread - local *If TgSplit execution mode, 9720 - wavefront local address space cannot 9721 be used.* 9722 9723 1. ds_atomic 9724 atomicrmw release - workgroup - global 1. s_waitcnt lgkm/vmcnt(0) 9725 - generic 9726 - Use lgkmcnt(0) if not 9727 TgSplit execution mode 9728 and vmcnt(0) if TgSplit 9729 execution mode. 9730 - If OpenCL, omit 9731 lgkmcnt(0). 9732 - s_waitcnt vmcnt(0) 9733 must happen after 9734 any preceding 9735 global/generic load/store/ 9736 load atomic/store atomic/ 9737 atomicrmw. 9738 - s_waitcnt lgkmcnt(0) 9739 must happen after 9740 any preceding 9741 local/generic 9742 load/store/load 9743 atomic/store 9744 atomic/atomicrmw. 9745 - Must happen before 9746 the following 9747 atomicrmw. 9748 - Ensures that all 9749 memory operations 9750 have 9751 completed before 9752 performing the 9753 atomicrmw that is 9754 being released. 9755 9756 2. buffer/global/flat_atomic sc0=1 9757 atomicrmw release - workgroup - local *If TgSplit execution mode, 9758 local address space cannot 9759 be used.* 9760 9761 1. ds_atomic 9762 atomicrmw release - agent - global 1. buffer_wbl2 sc1=1 9763 - generic 9764 - Must happen before 9765 following s_waitcnt. 9766 - Performs L2 writeback to 9767 ensure previous 9768 global/generic 9769 store/atomicrmw are 9770 visible at agent scope. 9771 9772 2. s_waitcnt lgkmcnt(0) & 9773 vmcnt(0) 9774 9775 - If TgSplit execution mode, 9776 omit lgkmcnt(0). 9777 - If OpenCL, omit 9778 lgkmcnt(0). 9779 - Could be split into 9780 separate s_waitcnt 9781 vmcnt(0) and 9782 s_waitcnt 9783 lgkmcnt(0) to allow 9784 them to be 9785 independently moved 9786 according to the 9787 following rules. 9788 - s_waitcnt vmcnt(0) 9789 must happen after 9790 any preceding 9791 global/generic 9792 load/store/load 9793 atomic/store 9794 atomic/atomicrmw. 9795 - s_waitcnt lgkmcnt(0) 9796 must happen after 9797 any preceding 9798 local/generic 9799 load/store/load 9800 atomic/store 9801 atomic/atomicrmw. 9802 - Must happen before 9803 the following 9804 atomicrmw. 9805 - Ensures that all 9806 memory operations 9807 to global and local 9808 have completed 9809 before performing 9810 the atomicrmw that 9811 is being released. 9812 9813 3. buffer/global/flat_atomic sc1=1 9814 atomicrmw release - system - global 1. buffer_wbl2 sc0=1 sc1=1 9815 - generic 9816 - Must happen before 9817 following s_waitcnt. 9818 - Performs L2 writeback to 9819 ensure previous 9820 global/generic 9821 store/atomicrmw are 9822 visible at system scope. 9823 9824 2. s_waitcnt lgkmcnt(0) & 9825 vmcnt(0) 9826 9827 - If TgSplit execution mode, 9828 omit lgkmcnt(0). 9829 - If OpenCL, omit 9830 lgkmcnt(0). 9831 - Could be split into 9832 separate s_waitcnt 9833 vmcnt(0) and 9834 s_waitcnt 9835 lgkmcnt(0) to allow 9836 them to be 9837 independently moved 9838 according to the 9839 following rules. 9840 - s_waitcnt vmcnt(0) 9841 must happen after 9842 any preceding 9843 global/generic 9844 load/store/load 9845 atomic/store 9846 atomic/atomicrmw. 9847 - s_waitcnt lgkmcnt(0) 9848 must happen after 9849 any preceding 9850 local/generic 9851 load/store/load 9852 atomic/store 9853 atomic/atomicrmw. 9854 - Must happen before 9855 the following 9856 atomicrmw. 9857 - Ensures that all 9858 memory operations 9859 to memory and the L2 9860 writeback have 9861 completed before 9862 performing the 9863 store that is being 9864 released. 9865 9866 3. buffer/global/flat_atomic 9867 sc0=1 sc1=1 9868 fence release - singlethread *none* *none* 9869 - wavefront 9870 fence release - workgroup *none* 1. s_waitcnt lgkm/vmcnt(0) 9871 9872 - Use lgkmcnt(0) if not 9873 TgSplit execution mode 9874 and vmcnt(0) if TgSplit 9875 execution mode. 9876 - If OpenCL and 9877 address space is 9878 not generic, omit 9879 lgkmcnt(0). 9880 - If OpenCL and 9881 address space is 9882 local, omit 9883 vmcnt(0). 9884 - However, since LLVM 9885 currently has no 9886 address space on 9887 the fence need to 9888 conservatively 9889 always generate. If 9890 fence had an 9891 address space then 9892 set to address 9893 space of OpenCL 9894 fence flag, or to 9895 generic if both 9896 local and global 9897 flags are 9898 specified. 9899 - s_waitcnt vmcnt(0) 9900 must happen after 9901 any preceding 9902 global/generic 9903 load/store/ 9904 load atomic/store atomic/ 9905 atomicrmw. 9906 - s_waitcnt lgkmcnt(0) 9907 must happen after 9908 any preceding 9909 local/generic 9910 load/load 9911 atomic/store/store 9912 atomic/atomicrmw. 9913 - Must happen before 9914 any following store 9915 atomic/atomicrmw 9916 with an equal or 9917 wider sync scope 9918 and memory ordering 9919 stronger than 9920 unordered (this is 9921 termed the 9922 fence-paired-atomic). 9923 - Ensures that all 9924 memory operations 9925 have 9926 completed before 9927 performing the 9928 following 9929 fence-paired-atomic. 9930 9931 fence release - agent *none* 1. buffer_wbl2 sc1=1 9932 9933 - If OpenCL and 9934 address space is 9935 local, omit. 9936 - Must happen before 9937 following s_waitcnt. 9938 - Performs L2 writeback to 9939 ensure previous 9940 global/generic 9941 store/atomicrmw are 9942 visible at agent scope. 9943 9944 2. s_waitcnt lgkmcnt(0) & 9945 vmcnt(0) 9946 9947 - If TgSplit execution mode, 9948 omit lgkmcnt(0). 9949 - If OpenCL and 9950 address space is 9951 not generic, omit 9952 lgkmcnt(0). 9953 - If OpenCL and 9954 address space is 9955 local, omit 9956 vmcnt(0). 9957 - However, since LLVM 9958 currently has no 9959 address space on 9960 the fence need to 9961 conservatively 9962 always generate. If 9963 fence had an 9964 address space then 9965 set to address 9966 space of OpenCL 9967 fence flag, or to 9968 generic if both 9969 local and global 9970 flags are 9971 specified. 9972 - Could be split into 9973 separate s_waitcnt 9974 vmcnt(0) and 9975 s_waitcnt 9976 lgkmcnt(0) to allow 9977 them to be 9978 independently moved 9979 according to the 9980 following rules. 9981 - s_waitcnt vmcnt(0) 9982 must happen after 9983 any preceding 9984 global/generic 9985 load/store/load 9986 atomic/store 9987 atomic/atomicrmw. 9988 - s_waitcnt lgkmcnt(0) 9989 must happen after 9990 any preceding 9991 local/generic 9992 load/store/load 9993 atomic/store 9994 atomic/atomicrmw. 9995 - Must happen before 9996 any following store 9997 atomic/atomicrmw 9998 with an equal or 9999 wider sync scope 10000 and memory ordering 10001 stronger than 10002 unordered (this is 10003 termed the 10004 fence-paired-atomic). 10005 - Ensures that all 10006 memory operations 10007 have 10008 completed before 10009 performing the 10010 following 10011 fence-paired-atomic. 10012 10013 fence release - system *none* 1. buffer_wbl2 sc0=1 sc1=1 10014 10015 - Must happen before 10016 following s_waitcnt. 10017 - Performs L2 writeback to 10018 ensure previous 10019 global/generic 10020 store/atomicrmw are 10021 visible at system scope. 10022 10023 2. s_waitcnt lgkmcnt(0) & 10024 vmcnt(0) 10025 10026 - If TgSplit execution mode, 10027 omit lgkmcnt(0). 10028 - If OpenCL and 10029 address space is 10030 not generic, omit 10031 lgkmcnt(0). 10032 - If OpenCL and 10033 address space is 10034 local, omit 10035 vmcnt(0). 10036 - However, since LLVM 10037 currently has no 10038 address space on 10039 the fence need to 10040 conservatively 10041 always generate. If 10042 fence had an 10043 address space then 10044 set to address 10045 space of OpenCL 10046 fence flag, or to 10047 generic if both 10048 local and global 10049 flags are 10050 specified. 10051 - Could be split into 10052 separate s_waitcnt 10053 vmcnt(0) and 10054 s_waitcnt 10055 lgkmcnt(0) to allow 10056 them to be 10057 independently moved 10058 according to the 10059 following rules. 10060 - s_waitcnt vmcnt(0) 10061 must happen after 10062 any preceding 10063 global/generic 10064 load/store/load 10065 atomic/store 10066 atomic/atomicrmw. 10067 - s_waitcnt lgkmcnt(0) 10068 must happen after 10069 any preceding 10070 local/generic 10071 load/store/load 10072 atomic/store 10073 atomic/atomicrmw. 10074 - Must happen before 10075 any following store 10076 atomic/atomicrmw 10077 with an equal or 10078 wider sync scope 10079 and memory ordering 10080 stronger than 10081 unordered (this is 10082 termed the 10083 fence-paired-atomic). 10084 - Ensures that all 10085 memory operations 10086 have 10087 completed before 10088 performing the 10089 following 10090 fence-paired-atomic. 10091 10092 **Acquire-Release Atomic** 10093 ------------------------------------------------------------------------------------ 10094 atomicrmw acq_rel - singlethread - global 1. buffer/global/flat_atomic 10095 - wavefront - generic 10096 atomicrmw acq_rel - singlethread - local *If TgSplit execution mode, 10097 - wavefront local address space cannot 10098 be used.* 10099 10100 1. ds_atomic 10101 atomicrmw acq_rel - workgroup - global 1. s_waitcnt lgkm/vmcnt(0) 10102 10103 - Use lgkmcnt(0) if not 10104 TgSplit execution mode 10105 and vmcnt(0) if TgSplit 10106 execution mode. 10107 - If OpenCL, omit 10108 lgkmcnt(0). 10109 - Must happen after 10110 any preceding 10111 local/generic 10112 load/store/load 10113 atomic/store 10114 atomic/atomicrmw. 10115 - s_waitcnt vmcnt(0) 10116 must happen after 10117 any preceding 10118 global/generic load/store/ 10119 load atomic/store atomic/ 10120 atomicrmw. 10121 - s_waitcnt lgkmcnt(0) 10122 must happen after 10123 any preceding 10124 local/generic 10125 load/store/load 10126 atomic/store 10127 atomic/atomicrmw. 10128 - Must happen before 10129 the following 10130 atomicrmw. 10131 - Ensures that all 10132 memory operations 10133 have 10134 completed before 10135 performing the 10136 atomicrmw that is 10137 being released. 10138 10139 2. buffer/global_atomic 10140 3. s_waitcnt vmcnt(0) 10141 10142 - If not TgSplit execution 10143 mode, omit. 10144 - Must happen before 10145 the following 10146 buffer_inv. 10147 - Ensures any 10148 following global 10149 data read is no 10150 older than the 10151 atomicrmw value 10152 being acquired. 10153 10154 4. buffer_inv sc0=1 10155 10156 - If not TgSplit execution 10157 mode, omit. 10158 - Ensures that 10159 following 10160 loads will not see 10161 stale data. 10162 10163 atomicrmw acq_rel - workgroup - local *If TgSplit execution mode, 10164 local address space cannot 10165 be used.* 10166 10167 1. ds_atomic 10168 2. s_waitcnt lgkmcnt(0) 10169 10170 - If OpenCL, omit. 10171 - Must happen before 10172 any following 10173 global/generic 10174 load/load 10175 atomic/store/store 10176 atomic/atomicrmw. 10177 - Ensures any 10178 following global 10179 data read is no 10180 older than the local load 10181 atomic value being 10182 acquired. 10183 10184 atomicrmw acq_rel - workgroup - generic 1. s_waitcnt lgkm/vmcnt(0) 10185 10186 - Use lgkmcnt(0) if not 10187 TgSplit execution mode 10188 and vmcnt(0) if TgSplit 10189 execution mode. 10190 - If OpenCL, omit 10191 lgkmcnt(0). 10192 - s_waitcnt vmcnt(0) 10193 must happen after 10194 any preceding 10195 global/generic load/store/ 10196 load atomic/store atomic/ 10197 atomicrmw. 10198 - s_waitcnt lgkmcnt(0) 10199 must happen after 10200 any preceding 10201 local/generic 10202 load/store/load 10203 atomic/store 10204 atomic/atomicrmw. 10205 - Must happen before 10206 the following 10207 atomicrmw. 10208 - Ensures that all 10209 memory operations 10210 have 10211 completed before 10212 performing the 10213 atomicrmw that is 10214 being released. 10215 10216 2. flat_atomic 10217 3. s_waitcnt lgkmcnt(0) & 10218 vmcnt(0) 10219 10220 - If not TgSplit execution 10221 mode, omit vmcnt(0). 10222 - If OpenCL, omit 10223 lgkmcnt(0). 10224 - Must happen before 10225 the following 10226 buffer_inv and 10227 any following 10228 global/generic 10229 load/load 10230 atomic/store/store 10231 atomic/atomicrmw. 10232 - Ensures any 10233 following global 10234 data read is no 10235 older than a local load 10236 atomic value being 10237 acquired. 10238 10239 3. buffer_inv sc0=1 10240 10241 - If not TgSplit execution 10242 mode, omit. 10243 - Ensures that 10244 following 10245 loads will not see 10246 stale data. 10247 10248 atomicrmw acq_rel - agent - global 1. buffer_wbl2 sc1=1 10249 10250 - Must happen before 10251 following s_waitcnt. 10252 - Performs L2 writeback to 10253 ensure previous 10254 global/generic 10255 store/atomicrmw are 10256 visible at agent scope. 10257 10258 2. s_waitcnt lgkmcnt(0) & 10259 vmcnt(0) 10260 10261 - If TgSplit execution mode, 10262 omit lgkmcnt(0). 10263 - If OpenCL, omit 10264 lgkmcnt(0). 10265 - Could be split into 10266 separate s_waitcnt 10267 vmcnt(0) and 10268 s_waitcnt 10269 lgkmcnt(0) to allow 10270 them to be 10271 independently moved 10272 according to the 10273 following rules. 10274 - s_waitcnt vmcnt(0) 10275 must happen after 10276 any preceding 10277 global/generic 10278 load/store/load 10279 atomic/store 10280 atomic/atomicrmw. 10281 - s_waitcnt lgkmcnt(0) 10282 must happen after 10283 any preceding 10284 local/generic 10285 load/store/load 10286 atomic/store 10287 atomic/atomicrmw. 10288 - Must happen before 10289 the following 10290 atomicrmw. 10291 - Ensures that all 10292 memory operations 10293 to global have 10294 completed before 10295 performing the 10296 atomicrmw that is 10297 being released. 10298 10299 3. buffer/global_atomic 10300 4. s_waitcnt vmcnt(0) 10301 10302 - Must happen before 10303 following 10304 buffer_inv. 10305 - Ensures the 10306 atomicrmw has 10307 completed before 10308 invalidating the 10309 cache. 10310 10311 5. buffer_inv sc1=1 10312 10313 - Must happen before 10314 any following 10315 global/generic 10316 load/load 10317 atomic/atomicrmw. 10318 - Ensures that 10319 following loads 10320 will not see stale 10321 global data. 10322 10323 atomicrmw acq_rel - system - global 1. buffer_wbl2 sc0=1 sc1=1 10324 10325 - Must happen before 10326 following s_waitcnt. 10327 - Performs L2 writeback to 10328 ensure previous 10329 global/generic 10330 store/atomicrmw are 10331 visible at system scope. 10332 10333 2. s_waitcnt lgkmcnt(0) & 10334 vmcnt(0) 10335 10336 - If TgSplit execution mode, 10337 omit lgkmcnt(0). 10338 - If OpenCL, omit 10339 lgkmcnt(0). 10340 - Could be split into 10341 separate s_waitcnt 10342 vmcnt(0) and 10343 s_waitcnt 10344 lgkmcnt(0) to allow 10345 them to be 10346 independently moved 10347 according to the 10348 following rules. 10349 - s_waitcnt vmcnt(0) 10350 must happen after 10351 any preceding 10352 global/generic 10353 load/store/load 10354 atomic/store 10355 atomic/atomicrmw. 10356 - s_waitcnt lgkmcnt(0) 10357 must happen after 10358 any preceding 10359 local/generic 10360 load/store/load 10361 atomic/store 10362 atomic/atomicrmw. 10363 - Must happen before 10364 the following 10365 atomicrmw. 10366 - Ensures that all 10367 memory operations 10368 to global and L2 writeback 10369 have completed before 10370 performing the 10371 atomicrmw that is 10372 being released. 10373 10374 3. buffer/global_atomic 10375 sc1=1 10376 4. s_waitcnt vmcnt(0) 10377 10378 - Must happen before 10379 following 10380 buffer_inv. 10381 - Ensures the 10382 atomicrmw has 10383 completed before 10384 invalidating the 10385 caches. 10386 10387 5. buffer_inv sc0=1 sc1=1 10388 10389 - Must happen before 10390 any following 10391 global/generic 10392 load/load 10393 atomic/atomicrmw. 10394 - Ensures that 10395 following loads 10396 will not see stale 10397 MTYPE NC global data. 10398 MTYPE RW and CC memory will 10399 never be stale due to the 10400 memory probes. 10401 10402 atomicrmw acq_rel - agent - generic 1. buffer_wbl2 sc1=1 10403 10404 - Must happen before 10405 following s_waitcnt. 10406 - Performs L2 writeback to 10407 ensure previous 10408 global/generic 10409 store/atomicrmw are 10410 visible at agent scope. 10411 10412 2. s_waitcnt lgkmcnt(0) & 10413 vmcnt(0) 10414 10415 - If TgSplit execution mode, 10416 omit lgkmcnt(0). 10417 - If OpenCL, omit 10418 lgkmcnt(0). 10419 - Could be split into 10420 separate s_waitcnt 10421 vmcnt(0) and 10422 s_waitcnt 10423 lgkmcnt(0) to allow 10424 them to be 10425 independently moved 10426 according to the 10427 following rules. 10428 - s_waitcnt vmcnt(0) 10429 must happen after 10430 any preceding 10431 global/generic 10432 load/store/load 10433 atomic/store 10434 atomic/atomicrmw. 10435 - s_waitcnt lgkmcnt(0) 10436 must happen after 10437 any preceding 10438 local/generic 10439 load/store/load 10440 atomic/store 10441 atomic/atomicrmw. 10442 - Must happen before 10443 the following 10444 atomicrmw. 10445 - Ensures that all 10446 memory operations 10447 to global have 10448 completed before 10449 performing the 10450 atomicrmw that is 10451 being released. 10452 10453 3. flat_atomic 10454 4. s_waitcnt vmcnt(0) & 10455 lgkmcnt(0) 10456 10457 - If TgSplit execution mode, 10458 omit lgkmcnt(0). 10459 - If OpenCL, omit 10460 lgkmcnt(0). 10461 - Must happen before 10462 following 10463 buffer_inv. 10464 - Ensures the 10465 atomicrmw has 10466 completed before 10467 invalidating the 10468 cache. 10469 10470 5. buffer_inv sc1=1 10471 10472 - Must happen before 10473 any following 10474 global/generic 10475 load/load 10476 atomic/atomicrmw. 10477 - Ensures that 10478 following loads 10479 will not see stale 10480 global data. 10481 10482 atomicrmw acq_rel - system - generic 1. buffer_wbl2 sc0=1 sc1=1 10483 10484 - Must happen before 10485 following s_waitcnt. 10486 - Performs L2 writeback to 10487 ensure previous 10488 global/generic 10489 store/atomicrmw are 10490 visible at system scope. 10491 10492 2. s_waitcnt lgkmcnt(0) & 10493 vmcnt(0) 10494 10495 - If TgSplit execution mode, 10496 omit lgkmcnt(0). 10497 - If OpenCL, omit 10498 lgkmcnt(0). 10499 - Could be split into 10500 separate s_waitcnt 10501 vmcnt(0) and 10502 s_waitcnt 10503 lgkmcnt(0) to allow 10504 them to be 10505 independently moved 10506 according to the 10507 following rules. 10508 - s_waitcnt vmcnt(0) 10509 must happen after 10510 any preceding 10511 global/generic 10512 load/store/load 10513 atomic/store 10514 atomic/atomicrmw. 10515 - s_waitcnt lgkmcnt(0) 10516 must happen after 10517 any preceding 10518 local/generic 10519 load/store/load 10520 atomic/store 10521 atomic/atomicrmw. 10522 - Must happen before 10523 the following 10524 atomicrmw. 10525 - Ensures that all 10526 memory operations 10527 to global and L2 writeback 10528 have completed before 10529 performing the 10530 atomicrmw that is 10531 being released. 10532 10533 3. flat_atomic sc1=1 10534 4. s_waitcnt vmcnt(0) & 10535 lgkmcnt(0) 10536 10537 - If TgSplit execution mode, 10538 omit lgkmcnt(0). 10539 - If OpenCL, omit 10540 lgkmcnt(0). 10541 - Must happen before 10542 following 10543 buffer_inv. 10544 - Ensures the 10545 atomicrmw has 10546 completed before 10547 invalidating the 10548 caches. 10549 10550 5. buffer_inv sc0=1 sc1=1 10551 10552 - Must happen before 10553 any following 10554 global/generic 10555 load/load 10556 atomic/atomicrmw. 10557 - Ensures that 10558 following loads 10559 will not see stale 10560 MTYPE NC global data. 10561 MTYPE RW and CC memory will 10562 never be stale due to the 10563 memory probes. 10564 10565 fence acq_rel - singlethread *none* *none* 10566 - wavefront 10567 fence acq_rel - workgroup *none* 1. s_waitcnt lgkm/vmcnt(0) 10568 10569 - Use lgkmcnt(0) if not 10570 TgSplit execution mode 10571 and vmcnt(0) if TgSplit 10572 execution mode. 10573 - If OpenCL and 10574 address space is 10575 not generic, omit 10576 lgkmcnt(0). 10577 - If OpenCL and 10578 address space is 10579 local, omit 10580 vmcnt(0). 10581 - However, 10582 since LLVM 10583 currently has no 10584 address space on 10585 the fence need to 10586 conservatively 10587 always generate 10588 (see comment for 10589 previous fence). 10590 - s_waitcnt vmcnt(0) 10591 must happen after 10592 any preceding 10593 global/generic 10594 load/store/ 10595 load atomic/store atomic/ 10596 atomicrmw. 10597 - s_waitcnt lgkmcnt(0) 10598 must happen after 10599 any preceding 10600 local/generic 10601 load/load 10602 atomic/store/store 10603 atomic/atomicrmw. 10604 - Must happen before 10605 any following 10606 global/generic 10607 load/load 10608 atomic/store/store 10609 atomic/atomicrmw. 10610 - Ensures that all 10611 memory operations 10612 have 10613 completed before 10614 performing any 10615 following global 10616 memory operations. 10617 - Ensures that the 10618 preceding 10619 local/generic load 10620 atomic/atomicrmw 10621 with an equal or 10622 wider sync scope 10623 and memory ordering 10624 stronger than 10625 unordered (this is 10626 termed the 10627 acquire-fence-paired-atomic) 10628 has completed 10629 before following 10630 global memory 10631 operations. This 10632 satisfies the 10633 requirements of 10634 acquire. 10635 - Ensures that all 10636 previous memory 10637 operations have 10638 completed before a 10639 following 10640 local/generic store 10641 atomic/atomicrmw 10642 with an equal or 10643 wider sync scope 10644 and memory ordering 10645 stronger than 10646 unordered (this is 10647 termed the 10648 release-fence-paired-atomic). 10649 This satisfies the 10650 requirements of 10651 release. 10652 - Must happen before 10653 the following 10654 buffer_inv. 10655 - Ensures that the 10656 acquire-fence-paired 10657 atomic has completed 10658 before invalidating 10659 the 10660 cache. Therefore 10661 any following 10662 locations read must 10663 be no older than 10664 the value read by 10665 the 10666 acquire-fence-paired-atomic. 10667 10668 3. buffer_inv sc0=1 10669 10670 - If not TgSplit execution 10671 mode, omit. 10672 - Ensures that 10673 following 10674 loads will not see 10675 stale data. 10676 10677 fence acq_rel - agent *none* 1. buffer_wbl2 sc1=1 10678 10679 - If OpenCL and 10680 address space is 10681 local, omit. 10682 - Must happen before 10683 following s_waitcnt. 10684 - Performs L2 writeback to 10685 ensure previous 10686 global/generic 10687 store/atomicrmw are 10688 visible at agent scope. 10689 10690 2. s_waitcnt lgkmcnt(0) & 10691 vmcnt(0) 10692 10693 - If TgSplit execution mode, 10694 omit lgkmcnt(0). 10695 - If OpenCL and 10696 address space is 10697 not generic, omit 10698 lgkmcnt(0). 10699 - However, since LLVM 10700 currently has no 10701 address space on 10702 the fence need to 10703 conservatively 10704 always generate 10705 (see comment for 10706 previous fence). 10707 - Could be split into 10708 separate s_waitcnt 10709 vmcnt(0) and 10710 s_waitcnt 10711 lgkmcnt(0) to allow 10712 them to be 10713 independently moved 10714 according to the 10715 following rules. 10716 - s_waitcnt vmcnt(0) 10717 must happen after 10718 any preceding 10719 global/generic 10720 load/store/load 10721 atomic/store 10722 atomic/atomicrmw. 10723 - s_waitcnt lgkmcnt(0) 10724 must happen after 10725 any preceding 10726 local/generic 10727 load/store/load 10728 atomic/store 10729 atomic/atomicrmw. 10730 - Must happen before 10731 the following 10732 buffer_inv. 10733 - Ensures that the 10734 preceding 10735 global/local/generic 10736 load 10737 atomic/atomicrmw 10738 with an equal or 10739 wider sync scope 10740 and memory ordering 10741 stronger than 10742 unordered (this is 10743 termed the 10744 acquire-fence-paired-atomic) 10745 has completed 10746 before invalidating 10747 the cache. This 10748 satisfies the 10749 requirements of 10750 acquire. 10751 - Ensures that all 10752 previous memory 10753 operations have 10754 completed before a 10755 following 10756 global/local/generic 10757 store 10758 atomic/atomicrmw 10759 with an equal or 10760 wider sync scope 10761 and memory ordering 10762 stronger than 10763 unordered (this is 10764 termed the 10765 release-fence-paired-atomic). 10766 This satisfies the 10767 requirements of 10768 release. 10769 10770 3. buffer_inv sc1=1 10771 10772 - Must happen before 10773 any following 10774 global/generic 10775 load/load 10776 atomic/store/store 10777 atomic/atomicrmw. 10778 - Ensures that 10779 following loads 10780 will not see stale 10781 global data. This 10782 satisfies the 10783 requirements of 10784 acquire. 10785 10786 fence acq_rel - system *none* 1. buffer_wbl2 sc0=1 sc1=1 10787 10788 - If OpenCL and 10789 address space is 10790 local, omit. 10791 - Must happen before 10792 following s_waitcnt. 10793 - Performs L2 writeback to 10794 ensure previous 10795 global/generic 10796 store/atomicrmw are 10797 visible at system scope. 10798 10799 1. s_waitcnt lgkmcnt(0) & 10800 vmcnt(0) 10801 10802 - If TgSplit execution mode, 10803 omit lgkmcnt(0). 10804 - If OpenCL and 10805 address space is 10806 not generic, omit 10807 lgkmcnt(0). 10808 - However, since LLVM 10809 currently has no 10810 address space on 10811 the fence need to 10812 conservatively 10813 always generate 10814 (see comment for 10815 previous fence). 10816 - Could be split into 10817 separate s_waitcnt 10818 vmcnt(0) and 10819 s_waitcnt 10820 lgkmcnt(0) to allow 10821 them to be 10822 independently moved 10823 according to the 10824 following rules. 10825 - s_waitcnt vmcnt(0) 10826 must happen after 10827 any preceding 10828 global/generic 10829 load/store/load 10830 atomic/store 10831 atomic/atomicrmw. 10832 - s_waitcnt lgkmcnt(0) 10833 must happen after 10834 any preceding 10835 local/generic 10836 load/store/load 10837 atomic/store 10838 atomic/atomicrmw. 10839 - Must happen before 10840 the following 10841 buffer_inv. 10842 - Ensures that the 10843 preceding 10844 global/local/generic 10845 load 10846 atomic/atomicrmw 10847 with an equal or 10848 wider sync scope 10849 and memory ordering 10850 stronger than 10851 unordered (this is 10852 termed the 10853 acquire-fence-paired-atomic) 10854 has completed 10855 before invalidating 10856 the cache. This 10857 satisfies the 10858 requirements of 10859 acquire. 10860 - Ensures that all 10861 previous memory 10862 operations have 10863 completed before a 10864 following 10865 global/local/generic 10866 store 10867 atomic/atomicrmw 10868 with an equal or 10869 wider sync scope 10870 and memory ordering 10871 stronger than 10872 unordered (this is 10873 termed the 10874 release-fence-paired-atomic). 10875 This satisfies the 10876 requirements of 10877 release. 10878 10879 2. buffer_inv sc0=1 sc1=1 10880 10881 - Must happen before 10882 any following 10883 global/generic 10884 load/load 10885 atomic/store/store 10886 atomic/atomicrmw. 10887 - Ensures that 10888 following loads 10889 will not see stale 10890 MTYPE NC global data. 10891 MTYPE RW and CC memory will 10892 never be stale due to the 10893 memory probes. 10894 10895 **Sequential Consistent Atomic** 10896 ------------------------------------------------------------------------------------ 10897 load atomic seq_cst - singlethread - global *Same as corresponding 10898 - wavefront - local load atomic acquire, 10899 - generic except must generate 10900 all instructions even 10901 for OpenCL.* 10902 load atomic seq_cst - workgroup - global 1. s_waitcnt lgkm/vmcnt(0) 10903 - generic 10904 - Use lgkmcnt(0) if not 10905 TgSplit execution mode 10906 and vmcnt(0) if TgSplit 10907 execution mode. 10908 - s_waitcnt lgkmcnt(0) must 10909 happen after 10910 preceding 10911 local/generic load 10912 atomic/store 10913 atomic/atomicrmw 10914 with memory 10915 ordering of seq_cst 10916 and with equal or 10917 wider sync scope. 10918 (Note that seq_cst 10919 fences have their 10920 own s_waitcnt 10921 lgkmcnt(0) and so do 10922 not need to be 10923 considered.) 10924 - s_waitcnt vmcnt(0) 10925 must happen after 10926 preceding 10927 global/generic load 10928 atomic/store 10929 atomic/atomicrmw 10930 with memory 10931 ordering of seq_cst 10932 and with equal or 10933 wider sync scope. 10934 (Note that seq_cst 10935 fences have their 10936 own s_waitcnt 10937 vmcnt(0) and so do 10938 not need to be 10939 considered.) 10940 - Ensures any 10941 preceding 10942 sequential 10943 consistent global/local 10944 memory instructions 10945 have completed 10946 before executing 10947 this sequentially 10948 consistent 10949 instruction. This 10950 prevents reordering 10951 a seq_cst store 10952 followed by a 10953 seq_cst load. (Note 10954 that seq_cst is 10955 stronger than 10956 acquire/release as 10957 the reordering of 10958 load acquire 10959 followed by a store 10960 release is 10961 prevented by the 10962 s_waitcnt of 10963 the release, but 10964 there is nothing 10965 preventing a store 10966 release followed by 10967 load acquire from 10968 completing out of 10969 order. The s_waitcnt 10970 could be placed after 10971 seq_store or before 10972 the seq_load. We 10973 choose the load to 10974 make the s_waitcnt be 10975 as late as possible 10976 so that the store 10977 may have already 10978 completed.) 10979 10980 2. *Following 10981 instructions same as 10982 corresponding load 10983 atomic acquire, 10984 except must generate 10985 all instructions even 10986 for OpenCL.* 10987 load atomic seq_cst - workgroup - local *If TgSplit execution mode, 10988 local address space cannot 10989 be used.* 10990 10991 *Same as corresponding 10992 load atomic acquire, 10993 except must generate 10994 all instructions even 10995 for OpenCL.* 10996 10997 load atomic seq_cst - agent - global 1. s_waitcnt lgkmcnt(0) & 10998 - system - generic vmcnt(0) 10999 11000 - If TgSplit execution mode, 11001 omit lgkmcnt(0). 11002 - Could be split into 11003 separate s_waitcnt 11004 vmcnt(0) 11005 and s_waitcnt 11006 lgkmcnt(0) to allow 11007 them to be 11008 independently moved 11009 according to the 11010 following rules. 11011 - s_waitcnt lgkmcnt(0) 11012 must happen after 11013 preceding 11014 global/generic load 11015 atomic/store 11016 atomic/atomicrmw 11017 with memory 11018 ordering of seq_cst 11019 and with equal or 11020 wider sync scope. 11021 (Note that seq_cst 11022 fences have their 11023 own s_waitcnt 11024 lgkmcnt(0) and so do 11025 not need to be 11026 considered.) 11027 - s_waitcnt vmcnt(0) 11028 must happen after 11029 preceding 11030 global/generic load 11031 atomic/store 11032 atomic/atomicrmw 11033 with memory 11034 ordering of seq_cst 11035 and with equal or 11036 wider sync scope. 11037 (Note that seq_cst 11038 fences have their 11039 own s_waitcnt 11040 vmcnt(0) and so do 11041 not need to be 11042 considered.) 11043 - Ensures any 11044 preceding 11045 sequential 11046 consistent global 11047 memory instructions 11048 have completed 11049 before executing 11050 this sequentially 11051 consistent 11052 instruction. This 11053 prevents reordering 11054 a seq_cst store 11055 followed by a 11056 seq_cst load. (Note 11057 that seq_cst is 11058 stronger than 11059 acquire/release as 11060 the reordering of 11061 load acquire 11062 followed by a store 11063 release is 11064 prevented by the 11065 s_waitcnt of 11066 the release, but 11067 there is nothing 11068 preventing a store 11069 release followed by 11070 load acquire from 11071 completing out of 11072 order. The s_waitcnt 11073 could be placed after 11074 seq_store or before 11075 the seq_load. We 11076 choose the load to 11077 make the s_waitcnt be 11078 as late as possible 11079 so that the store 11080 may have already 11081 completed.) 11082 11083 2. *Following 11084 instructions same as 11085 corresponding load 11086 atomic acquire, 11087 except must generate 11088 all instructions even 11089 for OpenCL.* 11090 store atomic seq_cst - singlethread - global *Same as corresponding 11091 - wavefront - local store atomic release, 11092 - workgroup - generic except must generate 11093 - agent all instructions even 11094 - system for OpenCL.* 11095 atomicrmw seq_cst - singlethread - global *Same as corresponding 11096 - wavefront - local atomicrmw acq_rel, 11097 - workgroup - generic except must generate 11098 - agent all instructions even 11099 - system for OpenCL.* 11100 fence seq_cst - singlethread *none* *Same as corresponding 11101 - wavefront fence acq_rel, 11102 - workgroup except must generate 11103 - agent all instructions even 11104 - system for OpenCL.* 11105 ============ ============ ============== ========== ================================ 11106 11107.. _amdgpu-amdhsa-memory-model-gfx10: 11108 11109Memory Model GFX10 11110++++++++++++++++++ 11111 11112For GFX10: 11113 11114* Each agent has multiple shader arrays (SA). 11115* Each SA has multiple work-group processors (WGP). 11116* Each WGP has multiple compute units (CU). 11117* Each CU has multiple SIMDs that execute wavefronts. 11118* The wavefronts for a single work-group are executed in the same 11119 WGP. In CU wavefront execution mode the wavefronts may be executed by 11120 different SIMDs in the same CU. In WGP wavefront execution mode the 11121 wavefronts may be executed by different SIMDs in different CUs in the same 11122 WGP. 11123* Each WGP has a single LDS memory shared by the wavefronts of the work-groups 11124 executing on it. 11125* All LDS operations of a WGP are performed as wavefront wide operations in a 11126 global order and involve no caching. Completion is reported to a wavefront in 11127 execution order. 11128* The LDS memory has multiple request queues shared by the SIMDs of a 11129 WGP. Therefore, the LDS operations performed by different wavefronts of a 11130 work-group can be reordered relative to each other, which can result in 11131 reordering the visibility of vector memory operations with respect to LDS 11132 operations of other wavefronts in the same work-group. A ``s_waitcnt 11133 lgkmcnt(0)`` is required to ensure synchronization between LDS operations and 11134 vector memory operations between wavefronts of a work-group, but not between 11135 operations performed by the same wavefront. 11136* The vector memory operations are performed as wavefront wide operations. 11137 Completion of load/store/sample operations are reported to a wavefront in 11138 execution order of other load/store/sample operations performed by that 11139 wavefront. 11140* The vector memory operations access a vector L0 cache. There is a single L0 11141 cache per CU. Each SIMD of a CU accesses the same L0 cache. Therefore, no 11142 special action is required for coherence between the lanes of a single 11143 wavefront. However, a ``buffer_gl0_inv`` is required for coherence between 11144 wavefronts executing in the same work-group as they may be executing on SIMDs 11145 of different CUs that access different L0s. A ``buffer_gl0_inv`` is also 11146 required for coherence between wavefronts executing in different work-groups 11147 as they may be executing on different WGPs. 11148* The scalar memory operations access a scalar L0 cache shared by all wavefronts 11149 on a WGP. The scalar and vector L0 caches are not coherent. However, scalar 11150 operations are used in a restricted way so do not impact the memory model. See 11151 :ref:`amdgpu-amdhsa-memory-spaces`. 11152* The vector and scalar memory L0 caches use an L1 cache shared by all WGPs on 11153 the same SA. Therefore, no special action is required for coherence between 11154 the wavefronts of a single work-group. However, a ``buffer_gl1_inv`` is 11155 required for coherence between wavefronts executing in different work-groups 11156 as they may be executing on different SAs that access different L1s. 11157* The L1 caches have independent quadrants to service disjoint ranges of virtual 11158 addresses. 11159* Each L0 cache has a separate request queue per L1 quadrant. Therefore, the 11160 vector and scalar memory operations performed by different wavefronts, whether 11161 executing in the same or different work-groups (which may be executing on 11162 different CUs accessing different L0s), can be reordered relative to each 11163 other. A ``s_waitcnt vmcnt(0) & vscnt(0)`` is required to ensure 11164 synchronization between vector memory operations of different wavefronts. It 11165 ensures a previous vector memory operation has completed before executing a 11166 subsequent vector memory or LDS operation and so can be used to meet the 11167 requirements of acquire, release and sequential consistency. 11168* The L1 caches use an L2 cache shared by all SAs on the same agent. 11169* The L2 cache has independent channels to service disjoint ranges of virtual 11170 addresses. 11171* Each L1 quadrant of a single SA accesses a different L2 channel. Each L1 11172 quadrant has a separate request queue per L2 channel. Therefore, the vector 11173 and scalar memory operations performed by wavefronts executing in different 11174 work-groups (which may be executing on different SAs) of an agent can be 11175 reordered relative to each other. A ``s_waitcnt vmcnt(0) & vscnt(0)`` is 11176 required to ensure synchronization between vector memory operations of 11177 different SAs. It ensures a previous vector memory operation has completed 11178 before executing a subsequent vector memory and so can be used to meet the 11179 requirements of acquire, release and sequential consistency. 11180* The L2 cache can be kept coherent with other agents on some targets, or ranges 11181 of virtual addresses can be set up to bypass it to ensure system coherence. 11182* On GFX10.3 a memory attached last level (MALL) cache exists for GPU memory. 11183 The MALL cache is fully coherent with GPU memory and has no impact on system 11184 coherence. All agents (GPU and CPU) access GPU memory through the MALL cache. 11185 11186Scalar memory operations are only used to access memory that is proven to not 11187change during the execution of the kernel dispatch. This includes constant 11188address space and global address space for program scope ``const`` variables. 11189Therefore, the kernel machine code does not have to maintain the scalar cache to 11190ensure it is coherent with the vector caches. The scalar and vector caches are 11191invalidated between kernel dispatches by CP since constant address space data 11192may change between kernel dispatch executions. See 11193:ref:`amdgpu-amdhsa-memory-spaces`. 11194 11195The one exception is if scalar writes are used to spill SGPR registers. In this 11196case the AMDGPU backend ensures the memory location used to spill is never 11197accessed by vector memory operations at the same time. If scalar writes are used 11198then a ``s_dcache_wb`` is inserted before the ``s_endpgm`` and before a function 11199return since the locations may be used for vector memory instructions by a 11200future wavefront that uses the same scratch area, or a function call that 11201creates a frame at the same address, respectively. There is no need for a 11202``s_dcache_inv`` as all scalar writes are write-before-read in the same thread. 11203 11204For kernarg backing memory: 11205 11206* CP invalidates the L0 and L1 caches at the start of each kernel dispatch. 11207* On dGPU the kernarg backing memory is accessed as MTYPE UC (uncached) to avoid 11208 needing to invalidate the L2 cache. 11209* On APU the kernarg backing memory is accessed as MTYPE CC (cache coherent) and 11210 so the L2 cache will be coherent with the CPU and other agents. 11211 11212Scratch backing memory (which is used for the private address space) is accessed 11213with MTYPE NC (non-coherent). Since the private address space is only accessed 11214by a single thread, and is always write-before-read, there is never a need to 11215invalidate these entries from the L0 or L1 caches. 11216 11217Wavefronts are executed in native mode with in-order reporting of loads and 11218sample instructions. In this mode vmcnt reports completion of load, atomic with 11219return and sample instructions in order, and the vscnt reports the completion of 11220store and atomic without return in order. See ``MEM_ORDERED`` field in 11221:ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 11222 11223Wavefronts can be executed in WGP or CU wavefront execution mode: 11224 11225* In WGP wavefront execution mode the wavefronts of a work-group are executed 11226 on the SIMDs of both CUs of the WGP. Therefore, explicit management of the per 11227 CU L0 caches is required for work-group synchronization. Also accesses to L1 11228 at work-group scope need to be explicitly ordered as the accesses from 11229 different CUs are not ordered. 11230* In CU wavefront execution mode the wavefronts of a work-group are executed on 11231 the SIMDs of a single CU of the WGP. Therefore, all global memory access by 11232 the work-group access the same L0 which in turn ensures L1 accesses are 11233 ordered and so do not require explicit management of the caches for 11234 work-group synchronization. 11235 11236See ``WGP_MODE`` field in 11237:ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table` and 11238:ref:`amdgpu-target-features`. 11239 11240The code sequences used to implement the memory model for GFX10 are defined in 11241table :ref:`amdgpu-amdhsa-memory-model-code-sequences-gfx10-table`. 11242 11243 .. table:: AMDHSA Memory Model Code Sequences GFX10 11244 :name: amdgpu-amdhsa-memory-model-code-sequences-gfx10-table 11245 11246 ============ ============ ============== ========== ================================ 11247 LLVM Instr LLVM Memory LLVM Memory AMDGPU AMDGPU Machine Code 11248 Ordering Sync Scope Address GFX10 11249 Space 11250 ============ ============ ============== ========== ================================ 11251 **Non-Atomic** 11252 ------------------------------------------------------------------------------------ 11253 load *none* *none* - global - !volatile & !nontemporal 11254 - generic 11255 - private 1. buffer/global/flat_load 11256 - constant 11257 - !volatile & nontemporal 11258 11259 1. buffer/global/flat_load 11260 slc=1 11261 11262 - volatile 11263 11264 1. buffer/global/flat_load 11265 glc=1 dlc=1 11266 2. s_waitcnt vmcnt(0) 11267 11268 - Must happen before 11269 any following volatile 11270 global/generic 11271 load/store. 11272 - Ensures that 11273 volatile 11274 operations to 11275 different 11276 addresses will not 11277 be reordered by 11278 hardware. 11279 11280 load *none* *none* - local 1. ds_load 11281 store *none* *none* - global - !volatile & !nontemporal 11282 - generic 11283 - private 1. buffer/global/flat_store 11284 - constant 11285 - !volatile & nontemporal 11286 11287 1. buffer/global/flat_store 11288 glc=1 slc=1 11289 11290 - volatile 11291 11292 1. buffer/global/flat_store 11293 2. s_waitcnt vscnt(0) 11294 11295 - Must happen before 11296 any following volatile 11297 global/generic 11298 load/store. 11299 - Ensures that 11300 volatile 11301 operations to 11302 different 11303 addresses will not 11304 be reordered by 11305 hardware. 11306 11307 store *none* *none* - local 1. ds_store 11308 **Unordered Atomic** 11309 ------------------------------------------------------------------------------------ 11310 load atomic unordered *any* *any* *Same as non-atomic*. 11311 store atomic unordered *any* *any* *Same as non-atomic*. 11312 atomicrmw unordered *any* *any* *Same as monotonic atomic*. 11313 **Monotonic Atomic** 11314 ------------------------------------------------------------------------------------ 11315 load atomic monotonic - singlethread - global 1. buffer/global/flat_load 11316 - wavefront - generic 11317 load atomic monotonic - workgroup - global 1. buffer/global/flat_load 11318 - generic glc=1 11319 11320 - If CU wavefront execution 11321 mode, omit glc=1. 11322 11323 load atomic monotonic - singlethread - local 1. ds_load 11324 - wavefront 11325 - workgroup 11326 load atomic monotonic - agent - global 1. buffer/global/flat_load 11327 - system - generic glc=1 dlc=1 11328 store atomic monotonic - singlethread - global 1. buffer/global/flat_store 11329 - wavefront - generic 11330 - workgroup 11331 - agent 11332 - system 11333 store atomic monotonic - singlethread - local 1. ds_store 11334 - wavefront 11335 - workgroup 11336 atomicrmw monotonic - singlethread - global 1. buffer/global/flat_atomic 11337 - wavefront - generic 11338 - workgroup 11339 - agent 11340 - system 11341 atomicrmw monotonic - singlethread - local 1. ds_atomic 11342 - wavefront 11343 - workgroup 11344 **Acquire Atomic** 11345 ------------------------------------------------------------------------------------ 11346 load atomic acquire - singlethread - global 1. buffer/global/ds/flat_load 11347 - wavefront - local 11348 - generic 11349 load atomic acquire - workgroup - global 1. buffer/global_load glc=1 11350 11351 - If CU wavefront execution 11352 mode, omit glc=1. 11353 11354 2. s_waitcnt vmcnt(0) 11355 11356 - If CU wavefront execution 11357 mode, omit. 11358 - Must happen before 11359 the following buffer_gl0_inv 11360 and before any following 11361 global/generic 11362 load/load 11363 atomic/store/store 11364 atomic/atomicrmw. 11365 11366 3. buffer_gl0_inv 11367 11368 - If CU wavefront execution 11369 mode, omit. 11370 - Ensures that 11371 following 11372 loads will not see 11373 stale data. 11374 11375 load atomic acquire - workgroup - local 1. ds_load 11376 2. s_waitcnt lgkmcnt(0) 11377 11378 - If OpenCL, omit. 11379 - Must happen before 11380 the following buffer_gl0_inv 11381 and before any following 11382 global/generic load/load 11383 atomic/store/store 11384 atomic/atomicrmw. 11385 - Ensures any 11386 following global 11387 data read is no 11388 older than the local load 11389 atomic value being 11390 acquired. 11391 11392 3. buffer_gl0_inv 11393 11394 - If CU wavefront execution 11395 mode, omit. 11396 - If OpenCL, omit. 11397 - Ensures that 11398 following 11399 loads will not see 11400 stale data. 11401 11402 load atomic acquire - workgroup - generic 1. flat_load glc=1 11403 11404 - If CU wavefront execution 11405 mode, omit glc=1. 11406 11407 2. s_waitcnt lgkmcnt(0) & 11408 vmcnt(0) 11409 11410 - If CU wavefront execution 11411 mode, omit vmcnt(0). 11412 - If OpenCL, omit 11413 lgkmcnt(0). 11414 - Must happen before 11415 the following 11416 buffer_gl0_inv and any 11417 following global/generic 11418 load/load 11419 atomic/store/store 11420 atomic/atomicrmw. 11421 - Ensures any 11422 following global 11423 data read is no 11424 older than a local load 11425 atomic value being 11426 acquired. 11427 11428 3. buffer_gl0_inv 11429 11430 - If CU wavefront execution 11431 mode, omit. 11432 - Ensures that 11433 following 11434 loads will not see 11435 stale data. 11436 11437 load atomic acquire - agent - global 1. buffer/global_load 11438 - system glc=1 dlc=1 11439 2. s_waitcnt vmcnt(0) 11440 11441 - Must happen before 11442 following 11443 buffer_gl*_inv. 11444 - Ensures the load 11445 has completed 11446 before invalidating 11447 the caches. 11448 11449 3. buffer_gl0_inv; 11450 buffer_gl1_inv 11451 11452 - Must happen before 11453 any following 11454 global/generic 11455 load/load 11456 atomic/atomicrmw. 11457 - Ensures that 11458 following 11459 loads will not see 11460 stale global data. 11461 11462 load atomic acquire - agent - generic 1. flat_load glc=1 dlc=1 11463 - system 2. s_waitcnt vmcnt(0) & 11464 lgkmcnt(0) 11465 11466 - If OpenCL omit 11467 lgkmcnt(0). 11468 - Must happen before 11469 following 11470 buffer_gl*_invl. 11471 - Ensures the flat_load 11472 has completed 11473 before invalidating 11474 the caches. 11475 11476 3. buffer_gl0_inv; 11477 buffer_gl1_inv 11478 11479 - Must happen before 11480 any following 11481 global/generic 11482 load/load 11483 atomic/atomicrmw. 11484 - Ensures that 11485 following loads 11486 will not see stale 11487 global data. 11488 11489 atomicrmw acquire - singlethread - global 1. buffer/global/ds/flat_atomic 11490 - wavefront - local 11491 - generic 11492 atomicrmw acquire - workgroup - global 1. buffer/global_atomic 11493 2. s_waitcnt vm/vscnt(0) 11494 11495 - If CU wavefront execution 11496 mode, omit. 11497 - Use vmcnt(0) if atomic with 11498 return and vscnt(0) if 11499 atomic with no-return. 11500 - Must happen before 11501 the following buffer_gl0_inv 11502 and before any following 11503 global/generic 11504 load/load 11505 atomic/store/store 11506 atomic/atomicrmw. 11507 11508 3. buffer_gl0_inv 11509 11510 - If CU wavefront execution 11511 mode, omit. 11512 - Ensures that 11513 following 11514 loads will not see 11515 stale data. 11516 11517 atomicrmw acquire - workgroup - local 1. ds_atomic 11518 2. s_waitcnt lgkmcnt(0) 11519 11520 - If OpenCL, omit. 11521 - Must happen before 11522 the following 11523 buffer_gl0_inv. 11524 - Ensures any 11525 following global 11526 data read is no 11527 older than the local 11528 atomicrmw value 11529 being acquired. 11530 11531 3. buffer_gl0_inv 11532 11533 - If OpenCL omit. 11534 - Ensures that 11535 following 11536 loads will not see 11537 stale data. 11538 11539 atomicrmw acquire - workgroup - generic 1. flat_atomic 11540 2. s_waitcnt lgkmcnt(0) & 11541 vm/vscnt(0) 11542 11543 - If CU wavefront execution 11544 mode, omit vm/vscnt(0). 11545 - If OpenCL, omit lgkmcnt(0). 11546 - Use vmcnt(0) if atomic with 11547 return and vscnt(0) if 11548 atomic with no-return. 11549 - Must happen before 11550 the following 11551 buffer_gl0_inv. 11552 - Ensures any 11553 following global 11554 data read is no 11555 older than a local 11556 atomicrmw value 11557 being acquired. 11558 11559 3. buffer_gl0_inv 11560 11561 - If CU wavefront execution 11562 mode, omit. 11563 - Ensures that 11564 following 11565 loads will not see 11566 stale data. 11567 11568 atomicrmw acquire - agent - global 1. buffer/global_atomic 11569 - system 2. s_waitcnt vm/vscnt(0) 11570 11571 - Use vmcnt(0) if atomic with 11572 return and vscnt(0) if 11573 atomic with no-return. 11574 - Must happen before 11575 following 11576 buffer_gl*_inv. 11577 - Ensures the 11578 atomicrmw has 11579 completed before 11580 invalidating the 11581 caches. 11582 11583 3. buffer_gl0_inv; 11584 buffer_gl1_inv 11585 11586 - Must happen before 11587 any following 11588 global/generic 11589 load/load 11590 atomic/atomicrmw. 11591 - Ensures that 11592 following loads 11593 will not see stale 11594 global data. 11595 11596 atomicrmw acquire - agent - generic 1. flat_atomic 11597 - system 2. s_waitcnt vm/vscnt(0) & 11598 lgkmcnt(0) 11599 11600 - If OpenCL, omit 11601 lgkmcnt(0). 11602 - Use vmcnt(0) if atomic with 11603 return and vscnt(0) if 11604 atomic with no-return. 11605 - Must happen before 11606 following 11607 buffer_gl*_inv. 11608 - Ensures the 11609 atomicrmw has 11610 completed before 11611 invalidating the 11612 caches. 11613 11614 3. buffer_gl0_inv; 11615 buffer_gl1_inv 11616 11617 - Must happen before 11618 any following 11619 global/generic 11620 load/load 11621 atomic/atomicrmw. 11622 - Ensures that 11623 following loads 11624 will not see stale 11625 global data. 11626 11627 fence acquire - singlethread *none* *none* 11628 - wavefront 11629 fence acquire - workgroup *none* 1. s_waitcnt lgkmcnt(0) & 11630 vmcnt(0) & vscnt(0) 11631 11632 - If CU wavefront execution 11633 mode, omit vmcnt(0) and 11634 vscnt(0). 11635 - If OpenCL and 11636 address space is 11637 not generic, omit 11638 lgkmcnt(0). 11639 - If OpenCL and 11640 address space is 11641 local, omit 11642 vmcnt(0) and vscnt(0). 11643 - However, since LLVM 11644 currently has no 11645 address space on 11646 the fence need to 11647 conservatively 11648 always generate. If 11649 fence had an 11650 address space then 11651 set to address 11652 space of OpenCL 11653 fence flag, or to 11654 generic if both 11655 local and global 11656 flags are 11657 specified. 11658 - Could be split into 11659 separate s_waitcnt 11660 vmcnt(0), s_waitcnt 11661 vscnt(0) and s_waitcnt 11662 lgkmcnt(0) to allow 11663 them to be 11664 independently moved 11665 according to the 11666 following rules. 11667 - s_waitcnt vmcnt(0) 11668 must happen after 11669 any preceding 11670 global/generic load 11671 atomic/ 11672 atomicrmw-with-return-value 11673 with an equal or 11674 wider sync scope 11675 and memory ordering 11676 stronger than 11677 unordered (this is 11678 termed the 11679 fence-paired-atomic). 11680 - s_waitcnt vscnt(0) 11681 must happen after 11682 any preceding 11683 global/generic 11684 atomicrmw-no-return-value 11685 with an equal or 11686 wider sync scope 11687 and memory ordering 11688 stronger than 11689 unordered (this is 11690 termed the 11691 fence-paired-atomic). 11692 - s_waitcnt lgkmcnt(0) 11693 must happen after 11694 any preceding 11695 local/generic load 11696 atomic/atomicrmw 11697 with an equal or 11698 wider sync scope 11699 and memory ordering 11700 stronger than 11701 unordered (this is 11702 termed the 11703 fence-paired-atomic). 11704 - Must happen before 11705 the following 11706 buffer_gl0_inv. 11707 - Ensures that the 11708 fence-paired atomic 11709 has completed 11710 before invalidating 11711 the 11712 cache. Therefore 11713 any following 11714 locations read must 11715 be no older than 11716 the value read by 11717 the 11718 fence-paired-atomic. 11719 11720 3. buffer_gl0_inv 11721 11722 - If CU wavefront execution 11723 mode, omit. 11724 - Ensures that 11725 following 11726 loads will not see 11727 stale data. 11728 11729 fence acquire - agent *none* 1. s_waitcnt lgkmcnt(0) & 11730 - system vmcnt(0) & vscnt(0) 11731 11732 - If OpenCL and 11733 address space is 11734 not generic, omit 11735 lgkmcnt(0). 11736 - If OpenCL and 11737 address space is 11738 local, omit 11739 vmcnt(0) and vscnt(0). 11740 - However, since LLVM 11741 currently has no 11742 address space on 11743 the fence need to 11744 conservatively 11745 always generate 11746 (see comment for 11747 previous fence). 11748 - Could be split into 11749 separate s_waitcnt 11750 vmcnt(0), s_waitcnt 11751 vscnt(0) and s_waitcnt 11752 lgkmcnt(0) to allow 11753 them to be 11754 independently moved 11755 according to the 11756 following rules. 11757 - s_waitcnt vmcnt(0) 11758 must happen after 11759 any preceding 11760 global/generic load 11761 atomic/ 11762 atomicrmw-with-return-value 11763 with an equal or 11764 wider sync scope 11765 and memory ordering 11766 stronger than 11767 unordered (this is 11768 termed the 11769 fence-paired-atomic). 11770 - s_waitcnt vscnt(0) 11771 must happen after 11772 any preceding 11773 global/generic 11774 atomicrmw-no-return-value 11775 with an equal or 11776 wider sync scope 11777 and memory ordering 11778 stronger than 11779 unordered (this is 11780 termed the 11781 fence-paired-atomic). 11782 - s_waitcnt lgkmcnt(0) 11783 must happen after 11784 any preceding 11785 local/generic load 11786 atomic/atomicrmw 11787 with an equal or 11788 wider sync scope 11789 and memory ordering 11790 stronger than 11791 unordered (this is 11792 termed the 11793 fence-paired-atomic). 11794 - Must happen before 11795 the following 11796 buffer_gl*_inv. 11797 - Ensures that the 11798 fence-paired atomic 11799 has completed 11800 before invalidating 11801 the 11802 caches. Therefore 11803 any following 11804 locations read must 11805 be no older than 11806 the value read by 11807 the 11808 fence-paired-atomic. 11809 11810 2. buffer_gl0_inv; 11811 buffer_gl1_inv 11812 11813 - Must happen before any 11814 following global/generic 11815 load/load 11816 atomic/store/store 11817 atomic/atomicrmw. 11818 - Ensures that 11819 following loads 11820 will not see stale 11821 global data. 11822 11823 **Release Atomic** 11824 ------------------------------------------------------------------------------------ 11825 store atomic release - singlethread - global 1. buffer/global/ds/flat_store 11826 - wavefront - local 11827 - generic 11828 store atomic release - workgroup - global 1. s_waitcnt lgkmcnt(0) & 11829 - generic vmcnt(0) & vscnt(0) 11830 11831 - If CU wavefront execution 11832 mode, omit vmcnt(0) and 11833 vscnt(0). 11834 - If OpenCL, omit 11835 lgkmcnt(0). 11836 - Could be split into 11837 separate s_waitcnt 11838 vmcnt(0), s_waitcnt 11839 vscnt(0) and s_waitcnt 11840 lgkmcnt(0) to allow 11841 them to be 11842 independently moved 11843 according to the 11844 following rules. 11845 - s_waitcnt vmcnt(0) 11846 must happen after 11847 any preceding 11848 global/generic load/load 11849 atomic/ 11850 atomicrmw-with-return-value. 11851 - s_waitcnt vscnt(0) 11852 must happen after 11853 any preceding 11854 global/generic 11855 store/store 11856 atomic/ 11857 atomicrmw-no-return-value. 11858 - s_waitcnt lgkmcnt(0) 11859 must happen after 11860 any preceding 11861 local/generic 11862 load/store/load 11863 atomic/store 11864 atomic/atomicrmw. 11865 - Must happen before 11866 the following 11867 store. 11868 - Ensures that all 11869 memory operations 11870 have 11871 completed before 11872 performing the 11873 store that is being 11874 released. 11875 11876 2. buffer/global/flat_store 11877 store atomic release - workgroup - local 1. s_waitcnt vmcnt(0) & vscnt(0) 11878 11879 - If CU wavefront execution 11880 mode, omit. 11881 - If OpenCL, omit. 11882 - Could be split into 11883 separate s_waitcnt 11884 vmcnt(0) and s_waitcnt 11885 vscnt(0) to allow 11886 them to be 11887 independently moved 11888 according to the 11889 following rules. 11890 - s_waitcnt vmcnt(0) 11891 must happen after 11892 any preceding 11893 global/generic load/load 11894 atomic/ 11895 atomicrmw-with-return-value. 11896 - s_waitcnt vscnt(0) 11897 must happen after 11898 any preceding 11899 global/generic 11900 store/store atomic/ 11901 atomicrmw-no-return-value. 11902 - Must happen before 11903 the following 11904 store. 11905 - Ensures that all 11906 global memory 11907 operations have 11908 completed before 11909 performing the 11910 store that is being 11911 released. 11912 11913 2. ds_store 11914 store atomic release - agent - global 1. s_waitcnt lgkmcnt(0) & 11915 - system - generic vmcnt(0) & vscnt(0) 11916 11917 - If OpenCL and 11918 address space is 11919 not generic, omit 11920 lgkmcnt(0). 11921 - Could be split into 11922 separate s_waitcnt 11923 vmcnt(0), s_waitcnt vscnt(0) 11924 and s_waitcnt 11925 lgkmcnt(0) to allow 11926 them to be 11927 independently moved 11928 according to the 11929 following rules. 11930 - s_waitcnt vmcnt(0) 11931 must happen after 11932 any preceding 11933 global/generic 11934 load/load 11935 atomic/ 11936 atomicrmw-with-return-value. 11937 - s_waitcnt vscnt(0) 11938 must happen after 11939 any preceding 11940 global/generic 11941 store/store atomic/ 11942 atomicrmw-no-return-value. 11943 - s_waitcnt lgkmcnt(0) 11944 must happen after 11945 any preceding 11946 local/generic 11947 load/store/load 11948 atomic/store 11949 atomic/atomicrmw. 11950 - Must happen before 11951 the following 11952 store. 11953 - Ensures that all 11954 memory operations 11955 have 11956 completed before 11957 performing the 11958 store that is being 11959 released. 11960 11961 2. buffer/global/flat_store 11962 atomicrmw release - singlethread - global 1. buffer/global/ds/flat_atomic 11963 - wavefront - local 11964 - generic 11965 atomicrmw release - workgroup - global 1. s_waitcnt lgkmcnt(0) & 11966 - generic vmcnt(0) & vscnt(0) 11967 11968 - If CU wavefront execution 11969 mode, omit vmcnt(0) and 11970 vscnt(0). 11971 - If OpenCL, omit lgkmcnt(0). 11972 - Could be split into 11973 separate s_waitcnt 11974 vmcnt(0), s_waitcnt 11975 vscnt(0) and s_waitcnt 11976 lgkmcnt(0) to allow 11977 them to be 11978 independently moved 11979 according to the 11980 following rules. 11981 - s_waitcnt vmcnt(0) 11982 must happen after 11983 any preceding 11984 global/generic load/load 11985 atomic/ 11986 atomicrmw-with-return-value. 11987 - s_waitcnt vscnt(0) 11988 must happen after 11989 any preceding 11990 global/generic 11991 store/store 11992 atomic/ 11993 atomicrmw-no-return-value. 11994 - s_waitcnt lgkmcnt(0) 11995 must happen after 11996 any preceding 11997 local/generic 11998 load/store/load 11999 atomic/store 12000 atomic/atomicrmw. 12001 - Must happen before 12002 the following 12003 atomicrmw. 12004 - Ensures that all 12005 memory operations 12006 have 12007 completed before 12008 performing the 12009 atomicrmw that is 12010 being released. 12011 12012 2. buffer/global/flat_atomic 12013 atomicrmw release - workgroup - local 1. s_waitcnt vmcnt(0) & vscnt(0) 12014 12015 - If CU wavefront execution 12016 mode, omit. 12017 - If OpenCL, omit. 12018 - Could be split into 12019 separate s_waitcnt 12020 vmcnt(0) and s_waitcnt 12021 vscnt(0) to allow 12022 them to be 12023 independently moved 12024 according to the 12025 following rules. 12026 - s_waitcnt vmcnt(0) 12027 must happen after 12028 any preceding 12029 global/generic load/load 12030 atomic/ 12031 atomicrmw-with-return-value. 12032 - s_waitcnt vscnt(0) 12033 must happen after 12034 any preceding 12035 global/generic 12036 store/store atomic/ 12037 atomicrmw-no-return-value. 12038 - Must happen before 12039 the following 12040 store. 12041 - Ensures that all 12042 global memory 12043 operations have 12044 completed before 12045 performing the 12046 store that is being 12047 released. 12048 12049 2. ds_atomic 12050 atomicrmw release - agent - global 1. s_waitcnt lgkmcnt(0) & 12051 - system - generic vmcnt(0) & vscnt(0) 12052 12053 - If OpenCL, omit 12054 lgkmcnt(0). 12055 - Could be split into 12056 separate s_waitcnt 12057 vmcnt(0), s_waitcnt 12058 vscnt(0) and s_waitcnt 12059 lgkmcnt(0) to allow 12060 them to be 12061 independently moved 12062 according to the 12063 following rules. 12064 - s_waitcnt vmcnt(0) 12065 must happen after 12066 any preceding 12067 global/generic 12068 load/load atomic/ 12069 atomicrmw-with-return-value. 12070 - s_waitcnt vscnt(0) 12071 must happen after 12072 any preceding 12073 global/generic 12074 store/store atomic/ 12075 atomicrmw-no-return-value. 12076 - s_waitcnt lgkmcnt(0) 12077 must happen after 12078 any preceding 12079 local/generic 12080 load/store/load 12081 atomic/store 12082 atomic/atomicrmw. 12083 - Must happen before 12084 the following 12085 atomicrmw. 12086 - Ensures that all 12087 memory operations 12088 to global and local 12089 have completed 12090 before performing 12091 the atomicrmw that 12092 is being released. 12093 12094 2. buffer/global/flat_atomic 12095 fence release - singlethread *none* *none* 12096 - wavefront 12097 fence release - workgroup *none* 1. s_waitcnt lgkmcnt(0) & 12098 vmcnt(0) & vscnt(0) 12099 12100 - If CU wavefront execution 12101 mode, omit vmcnt(0) and 12102 vscnt(0). 12103 - If OpenCL and 12104 address space is 12105 not generic, omit 12106 lgkmcnt(0). 12107 - If OpenCL and 12108 address space is 12109 local, omit 12110 vmcnt(0) and vscnt(0). 12111 - However, since LLVM 12112 currently has no 12113 address space on 12114 the fence need to 12115 conservatively 12116 always generate. If 12117 fence had an 12118 address space then 12119 set to address 12120 space of OpenCL 12121 fence flag, or to 12122 generic if both 12123 local and global 12124 flags are 12125 specified. 12126 - Could be split into 12127 separate s_waitcnt 12128 vmcnt(0), s_waitcnt 12129 vscnt(0) and s_waitcnt 12130 lgkmcnt(0) to allow 12131 them to be 12132 independently moved 12133 according to the 12134 following rules. 12135 - s_waitcnt vmcnt(0) 12136 must happen after 12137 any preceding 12138 global/generic 12139 load/load 12140 atomic/ 12141 atomicrmw-with-return-value. 12142 - s_waitcnt vscnt(0) 12143 must happen after 12144 any preceding 12145 global/generic 12146 store/store atomic/ 12147 atomicrmw-no-return-value. 12148 - s_waitcnt lgkmcnt(0) 12149 must happen after 12150 any preceding 12151 local/generic 12152 load/store/load 12153 atomic/store atomic/ 12154 atomicrmw. 12155 - Must happen before 12156 any following store 12157 atomic/atomicrmw 12158 with an equal or 12159 wider sync scope 12160 and memory ordering 12161 stronger than 12162 unordered (this is 12163 termed the 12164 fence-paired-atomic). 12165 - Ensures that all 12166 memory operations 12167 have 12168 completed before 12169 performing the 12170 following 12171 fence-paired-atomic. 12172 12173 fence release - agent *none* 1. s_waitcnt lgkmcnt(0) & 12174 - system vmcnt(0) & vscnt(0) 12175 12176 - If OpenCL and 12177 address space is 12178 not generic, omit 12179 lgkmcnt(0). 12180 - If OpenCL and 12181 address space is 12182 local, omit 12183 vmcnt(0) and vscnt(0). 12184 - However, since LLVM 12185 currently has no 12186 address space on 12187 the fence need to 12188 conservatively 12189 always generate. If 12190 fence had an 12191 address space then 12192 set to address 12193 space of OpenCL 12194 fence flag, or to 12195 generic if both 12196 local and global 12197 flags are 12198 specified. 12199 - Could be split into 12200 separate s_waitcnt 12201 vmcnt(0), s_waitcnt 12202 vscnt(0) and s_waitcnt 12203 lgkmcnt(0) to allow 12204 them to be 12205 independently moved 12206 according to the 12207 following rules. 12208 - s_waitcnt vmcnt(0) 12209 must happen after 12210 any preceding 12211 global/generic 12212 load/load atomic/ 12213 atomicrmw-with-return-value. 12214 - s_waitcnt vscnt(0) 12215 must happen after 12216 any preceding 12217 global/generic 12218 store/store atomic/ 12219 atomicrmw-no-return-value. 12220 - s_waitcnt lgkmcnt(0) 12221 must happen after 12222 any preceding 12223 local/generic 12224 load/store/load 12225 atomic/store 12226 atomic/atomicrmw. 12227 - Must happen before 12228 any following store 12229 atomic/atomicrmw 12230 with an equal or 12231 wider sync scope 12232 and memory ordering 12233 stronger than 12234 unordered (this is 12235 termed the 12236 fence-paired-atomic). 12237 - Ensures that all 12238 memory operations 12239 have 12240 completed before 12241 performing the 12242 following 12243 fence-paired-atomic. 12244 12245 **Acquire-Release Atomic** 12246 ------------------------------------------------------------------------------------ 12247 atomicrmw acq_rel - singlethread - global 1. buffer/global/ds/flat_atomic 12248 - wavefront - local 12249 - generic 12250 atomicrmw acq_rel - workgroup - global 1. s_waitcnt lgkmcnt(0) & 12251 vmcnt(0) & vscnt(0) 12252 12253 - If CU wavefront execution 12254 mode, omit vmcnt(0) and 12255 vscnt(0). 12256 - If OpenCL, omit 12257 lgkmcnt(0). 12258 - Must happen after 12259 any preceding 12260 local/generic 12261 load/store/load 12262 atomic/store 12263 atomic/atomicrmw. 12264 - Could be split into 12265 separate s_waitcnt 12266 vmcnt(0), s_waitcnt 12267 vscnt(0), and s_waitcnt 12268 lgkmcnt(0) to allow 12269 them to be 12270 independently moved 12271 according to the 12272 following rules. 12273 - s_waitcnt vmcnt(0) 12274 must happen after 12275 any preceding 12276 global/generic load/load 12277 atomic/ 12278 atomicrmw-with-return-value. 12279 - s_waitcnt vscnt(0) 12280 must happen after 12281 any preceding 12282 global/generic 12283 store/store 12284 atomic/ 12285 atomicrmw-no-return-value. 12286 - s_waitcnt lgkmcnt(0) 12287 must happen after 12288 any preceding 12289 local/generic 12290 load/store/load 12291 atomic/store 12292 atomic/atomicrmw. 12293 - Must happen before 12294 the following 12295 atomicrmw. 12296 - Ensures that all 12297 memory operations 12298 have 12299 completed before 12300 performing the 12301 atomicrmw that is 12302 being released. 12303 12304 2. buffer/global_atomic 12305 3. s_waitcnt vm/vscnt(0) 12306 12307 - If CU wavefront execution 12308 mode, omit. 12309 - Use vmcnt(0) if atomic with 12310 return and vscnt(0) if 12311 atomic with no-return. 12312 - Must happen before 12313 the following 12314 buffer_gl0_inv. 12315 - Ensures any 12316 following global 12317 data read is no 12318 older than the 12319 atomicrmw value 12320 being acquired. 12321 12322 4. buffer_gl0_inv 12323 12324 - If CU wavefront execution 12325 mode, omit. 12326 - Ensures that 12327 following 12328 loads will not see 12329 stale data. 12330 12331 atomicrmw acq_rel - workgroup - local 1. s_waitcnt vmcnt(0) & vscnt(0) 12332 12333 - If CU wavefront execution 12334 mode, omit. 12335 - If OpenCL, omit. 12336 - Could be split into 12337 separate s_waitcnt 12338 vmcnt(0) and s_waitcnt 12339 vscnt(0) to allow 12340 them to be 12341 independently moved 12342 according to the 12343 following rules. 12344 - s_waitcnt vmcnt(0) 12345 must happen after 12346 any preceding 12347 global/generic load/load 12348 atomic/ 12349 atomicrmw-with-return-value. 12350 - s_waitcnt vscnt(0) 12351 must happen after 12352 any preceding 12353 global/generic 12354 store/store atomic/ 12355 atomicrmw-no-return-value. 12356 - Must happen before 12357 the following 12358 store. 12359 - Ensures that all 12360 global memory 12361 operations have 12362 completed before 12363 performing the 12364 store that is being 12365 released. 12366 12367 2. ds_atomic 12368 3. s_waitcnt lgkmcnt(0) 12369 12370 - If OpenCL, omit. 12371 - Must happen before 12372 the following 12373 buffer_gl0_inv. 12374 - Ensures any 12375 following global 12376 data read is no 12377 older than the local load 12378 atomic value being 12379 acquired. 12380 12381 4. buffer_gl0_inv 12382 12383 - If CU wavefront execution 12384 mode, omit. 12385 - If OpenCL omit. 12386 - Ensures that 12387 following 12388 loads will not see 12389 stale data. 12390 12391 atomicrmw acq_rel - workgroup - generic 1. s_waitcnt lgkmcnt(0) & 12392 vmcnt(0) & vscnt(0) 12393 12394 - If CU wavefront execution 12395 mode, omit vmcnt(0) and 12396 vscnt(0). 12397 - If OpenCL, omit lgkmcnt(0). 12398 - Could be split into 12399 separate s_waitcnt 12400 vmcnt(0), s_waitcnt 12401 vscnt(0) and s_waitcnt 12402 lgkmcnt(0) to allow 12403 them to be 12404 independently moved 12405 according to the 12406 following rules. 12407 - s_waitcnt vmcnt(0) 12408 must happen after 12409 any preceding 12410 global/generic load/load 12411 atomic/ 12412 atomicrmw-with-return-value. 12413 - s_waitcnt vscnt(0) 12414 must happen after 12415 any preceding 12416 global/generic 12417 store/store 12418 atomic/ 12419 atomicrmw-no-return-value. 12420 - s_waitcnt lgkmcnt(0) 12421 must happen after 12422 any preceding 12423 local/generic 12424 load/store/load 12425 atomic/store 12426 atomic/atomicrmw. 12427 - Must happen before 12428 the following 12429 atomicrmw. 12430 - Ensures that all 12431 memory operations 12432 have 12433 completed before 12434 performing the 12435 atomicrmw that is 12436 being released. 12437 12438 2. flat_atomic 12439 3. s_waitcnt lgkmcnt(0) & 12440 vmcnt(0) & vscnt(0) 12441 12442 - If CU wavefront execution 12443 mode, omit vmcnt(0) and 12444 vscnt(0). 12445 - If OpenCL, omit lgkmcnt(0). 12446 - Must happen before 12447 the following 12448 buffer_gl0_inv. 12449 - Ensures any 12450 following global 12451 data read is no 12452 older than the load 12453 atomic value being 12454 acquired. 12455 12456 3. buffer_gl0_inv 12457 12458 - If CU wavefront execution 12459 mode, omit. 12460 - Ensures that 12461 following 12462 loads will not see 12463 stale data. 12464 12465 atomicrmw acq_rel - agent - global 1. s_waitcnt lgkmcnt(0) & 12466 - system vmcnt(0) & vscnt(0) 12467 12468 - If OpenCL, omit 12469 lgkmcnt(0). 12470 - Could be split into 12471 separate s_waitcnt 12472 vmcnt(0), s_waitcnt 12473 vscnt(0) and s_waitcnt 12474 lgkmcnt(0) to allow 12475 them to be 12476 independently moved 12477 according to the 12478 following rules. 12479 - s_waitcnt vmcnt(0) 12480 must happen after 12481 any preceding 12482 global/generic 12483 load/load atomic/ 12484 atomicrmw-with-return-value. 12485 - s_waitcnt vscnt(0) 12486 must happen after 12487 any preceding 12488 global/generic 12489 store/store atomic/ 12490 atomicrmw-no-return-value. 12491 - s_waitcnt lgkmcnt(0) 12492 must happen after 12493 any preceding 12494 local/generic 12495 load/store/load 12496 atomic/store 12497 atomic/atomicrmw. 12498 - Must happen before 12499 the following 12500 atomicrmw. 12501 - Ensures that all 12502 memory operations 12503 to global have 12504 completed before 12505 performing the 12506 atomicrmw that is 12507 being released. 12508 12509 2. buffer/global_atomic 12510 3. s_waitcnt vm/vscnt(0) 12511 12512 - Use vmcnt(0) if atomic with 12513 return and vscnt(0) if 12514 atomic with no-return. 12515 - Must happen before 12516 following 12517 buffer_gl*_inv. 12518 - Ensures the 12519 atomicrmw has 12520 completed before 12521 invalidating the 12522 caches. 12523 12524 4. buffer_gl0_inv; 12525 buffer_gl1_inv 12526 12527 - Must happen before 12528 any following 12529 global/generic 12530 load/load 12531 atomic/atomicrmw. 12532 - Ensures that 12533 following loads 12534 will not see stale 12535 global data. 12536 12537 atomicrmw acq_rel - agent - generic 1. s_waitcnt lgkmcnt(0) & 12538 - system vmcnt(0) & vscnt(0) 12539 12540 - If OpenCL, omit 12541 lgkmcnt(0). 12542 - Could be split into 12543 separate s_waitcnt 12544 vmcnt(0), s_waitcnt 12545 vscnt(0), and s_waitcnt 12546 lgkmcnt(0) to allow 12547 them to be 12548 independently moved 12549 according to the 12550 following rules. 12551 - s_waitcnt vmcnt(0) 12552 must happen after 12553 any preceding 12554 global/generic 12555 load/load atomic 12556 atomicrmw-with-return-value. 12557 - s_waitcnt vscnt(0) 12558 must happen after 12559 any preceding 12560 global/generic 12561 store/store atomic/ 12562 atomicrmw-no-return-value. 12563 - s_waitcnt lgkmcnt(0) 12564 must happen after 12565 any preceding 12566 local/generic 12567 load/store/load 12568 atomic/store 12569 atomic/atomicrmw. 12570 - Must happen before 12571 the following 12572 atomicrmw. 12573 - Ensures that all 12574 memory operations 12575 have 12576 completed before 12577 performing the 12578 atomicrmw that is 12579 being released. 12580 12581 2. flat_atomic 12582 3. s_waitcnt vm/vscnt(0) & 12583 lgkmcnt(0) 12584 12585 - If OpenCL, omit 12586 lgkmcnt(0). 12587 - Use vmcnt(0) if atomic with 12588 return and vscnt(0) if 12589 atomic with no-return. 12590 - Must happen before 12591 following 12592 buffer_gl*_inv. 12593 - Ensures the 12594 atomicrmw has 12595 completed before 12596 invalidating the 12597 caches. 12598 12599 4. buffer_gl0_inv; 12600 buffer_gl1_inv 12601 12602 - Must happen before 12603 any following 12604 global/generic 12605 load/load 12606 atomic/atomicrmw. 12607 - Ensures that 12608 following loads 12609 will not see stale 12610 global data. 12611 12612 fence acq_rel - singlethread *none* *none* 12613 - wavefront 12614 fence acq_rel - workgroup *none* 1. s_waitcnt lgkmcnt(0) & 12615 vmcnt(0) & vscnt(0) 12616 12617 - If CU wavefront execution 12618 mode, omit vmcnt(0) and 12619 vscnt(0). 12620 - If OpenCL and 12621 address space is 12622 not generic, omit 12623 lgkmcnt(0). 12624 - If OpenCL and 12625 address space is 12626 local, omit 12627 vmcnt(0) and vscnt(0). 12628 - However, 12629 since LLVM 12630 currently has no 12631 address space on 12632 the fence need to 12633 conservatively 12634 always generate 12635 (see comment for 12636 previous fence). 12637 - Could be split into 12638 separate s_waitcnt 12639 vmcnt(0), s_waitcnt 12640 vscnt(0) and s_waitcnt 12641 lgkmcnt(0) to allow 12642 them to be 12643 independently moved 12644 according to the 12645 following rules. 12646 - s_waitcnt vmcnt(0) 12647 must happen after 12648 any preceding 12649 global/generic 12650 load/load 12651 atomic/ 12652 atomicrmw-with-return-value. 12653 - s_waitcnt vscnt(0) 12654 must happen after 12655 any preceding 12656 global/generic 12657 store/store atomic/ 12658 atomicrmw-no-return-value. 12659 - s_waitcnt lgkmcnt(0) 12660 must happen after 12661 any preceding 12662 local/generic 12663 load/store/load 12664 atomic/store atomic/ 12665 atomicrmw. 12666 - Must happen before 12667 any following 12668 global/generic 12669 load/load 12670 atomic/store/store 12671 atomic/atomicrmw. 12672 - Ensures that all 12673 memory operations 12674 have 12675 completed before 12676 performing any 12677 following global 12678 memory operations. 12679 - Ensures that the 12680 preceding 12681 local/generic load 12682 atomic/atomicrmw 12683 with an equal or 12684 wider sync scope 12685 and memory ordering 12686 stronger than 12687 unordered (this is 12688 termed the 12689 acquire-fence-paired-atomic) 12690 has completed 12691 before following 12692 global memory 12693 operations. This 12694 satisfies the 12695 requirements of 12696 acquire. 12697 - Ensures that all 12698 previous memory 12699 operations have 12700 completed before a 12701 following 12702 local/generic store 12703 atomic/atomicrmw 12704 with an equal or 12705 wider sync scope 12706 and memory ordering 12707 stronger than 12708 unordered (this is 12709 termed the 12710 release-fence-paired-atomic). 12711 This satisfies the 12712 requirements of 12713 release. 12714 - Must happen before 12715 the following 12716 buffer_gl0_inv. 12717 - Ensures that the 12718 acquire-fence-paired 12719 atomic has completed 12720 before invalidating 12721 the 12722 cache. Therefore 12723 any following 12724 locations read must 12725 be no older than 12726 the value read by 12727 the 12728 acquire-fence-paired-atomic. 12729 12730 3. buffer_gl0_inv 12731 12732 - If CU wavefront execution 12733 mode, omit. 12734 - Ensures that 12735 following 12736 loads will not see 12737 stale data. 12738 12739 fence acq_rel - agent *none* 1. s_waitcnt lgkmcnt(0) & 12740 - system vmcnt(0) & vscnt(0) 12741 12742 - If OpenCL and 12743 address space is 12744 not generic, omit 12745 lgkmcnt(0). 12746 - If OpenCL and 12747 address space is 12748 local, omit 12749 vmcnt(0) and vscnt(0). 12750 - However, since LLVM 12751 currently has no 12752 address space on 12753 the fence need to 12754 conservatively 12755 always generate 12756 (see comment for 12757 previous fence). 12758 - Could be split into 12759 separate s_waitcnt 12760 vmcnt(0), s_waitcnt 12761 vscnt(0) and s_waitcnt 12762 lgkmcnt(0) to allow 12763 them to be 12764 independently moved 12765 according to the 12766 following rules. 12767 - s_waitcnt vmcnt(0) 12768 must happen after 12769 any preceding 12770 global/generic 12771 load/load 12772 atomic/ 12773 atomicrmw-with-return-value. 12774 - s_waitcnt vscnt(0) 12775 must happen after 12776 any preceding 12777 global/generic 12778 store/store atomic/ 12779 atomicrmw-no-return-value. 12780 - s_waitcnt lgkmcnt(0) 12781 must happen after 12782 any preceding 12783 local/generic 12784 load/store/load 12785 atomic/store 12786 atomic/atomicrmw. 12787 - Must happen before 12788 the following 12789 buffer_gl*_inv. 12790 - Ensures that the 12791 preceding 12792 global/local/generic 12793 load 12794 atomic/atomicrmw 12795 with an equal or 12796 wider sync scope 12797 and memory ordering 12798 stronger than 12799 unordered (this is 12800 termed the 12801 acquire-fence-paired-atomic) 12802 has completed 12803 before invalidating 12804 the caches. This 12805 satisfies the 12806 requirements of 12807 acquire. 12808 - Ensures that all 12809 previous memory 12810 operations have 12811 completed before a 12812 following 12813 global/local/generic 12814 store 12815 atomic/atomicrmw 12816 with an equal or 12817 wider sync scope 12818 and memory ordering 12819 stronger than 12820 unordered (this is 12821 termed the 12822 release-fence-paired-atomic). 12823 This satisfies the 12824 requirements of 12825 release. 12826 12827 2. buffer_gl0_inv; 12828 buffer_gl1_inv 12829 12830 - Must happen before 12831 any following 12832 global/generic 12833 load/load 12834 atomic/store/store 12835 atomic/atomicrmw. 12836 - Ensures that 12837 following loads 12838 will not see stale 12839 global data. This 12840 satisfies the 12841 requirements of 12842 acquire. 12843 12844 **Sequential Consistent Atomic** 12845 ------------------------------------------------------------------------------------ 12846 load atomic seq_cst - singlethread - global *Same as corresponding 12847 - wavefront - local load atomic acquire, 12848 - generic except must generate 12849 all instructions even 12850 for OpenCL.* 12851 load atomic seq_cst - workgroup - global 1. s_waitcnt lgkmcnt(0) & 12852 - generic vmcnt(0) & vscnt(0) 12853 12854 - If CU wavefront execution 12855 mode, omit vmcnt(0) and 12856 vscnt(0). 12857 - Could be split into 12858 separate s_waitcnt 12859 vmcnt(0), s_waitcnt 12860 vscnt(0), and s_waitcnt 12861 lgkmcnt(0) to allow 12862 them to be 12863 independently moved 12864 according to the 12865 following rules. 12866 - s_waitcnt lgkmcnt(0) must 12867 happen after 12868 preceding 12869 local/generic load 12870 atomic/store 12871 atomic/atomicrmw 12872 with memory 12873 ordering of seq_cst 12874 and with equal or 12875 wider sync scope. 12876 (Note that seq_cst 12877 fences have their 12878 own s_waitcnt 12879 lgkmcnt(0) and so do 12880 not need to be 12881 considered.) 12882 - s_waitcnt vmcnt(0) 12883 must happen after 12884 preceding 12885 global/generic load 12886 atomic/ 12887 atomicrmw-with-return-value 12888 with memory 12889 ordering of seq_cst 12890 and with equal or 12891 wider sync scope. 12892 (Note that seq_cst 12893 fences have their 12894 own s_waitcnt 12895 vmcnt(0) and so do 12896 not need to be 12897 considered.) 12898 - s_waitcnt vscnt(0) 12899 Must happen after 12900 preceding 12901 global/generic store 12902 atomic/ 12903 atomicrmw-no-return-value 12904 with memory 12905 ordering of seq_cst 12906 and with equal or 12907 wider sync scope. 12908 (Note that seq_cst 12909 fences have their 12910 own s_waitcnt 12911 vscnt(0) and so do 12912 not need to be 12913 considered.) 12914 - Ensures any 12915 preceding 12916 sequential 12917 consistent global/local 12918 memory instructions 12919 have completed 12920 before executing 12921 this sequentially 12922 consistent 12923 instruction. This 12924 prevents reordering 12925 a seq_cst store 12926 followed by a 12927 seq_cst load. (Note 12928 that seq_cst is 12929 stronger than 12930 acquire/release as 12931 the reordering of 12932 load acquire 12933 followed by a store 12934 release is 12935 prevented by the 12936 s_waitcnt of 12937 the release, but 12938 there is nothing 12939 preventing a store 12940 release followed by 12941 load acquire from 12942 completing out of 12943 order. The s_waitcnt 12944 could be placed after 12945 seq_store or before 12946 the seq_load. We 12947 choose the load to 12948 make the s_waitcnt be 12949 as late as possible 12950 so that the store 12951 may have already 12952 completed.) 12953 12954 2. *Following 12955 instructions same as 12956 corresponding load 12957 atomic acquire, 12958 except must generate 12959 all instructions even 12960 for OpenCL.* 12961 load atomic seq_cst - workgroup - local 12962 12963 1. s_waitcnt vmcnt(0) & vscnt(0) 12964 12965 - If CU wavefront execution 12966 mode, omit. 12967 - Could be split into 12968 separate s_waitcnt 12969 vmcnt(0) and s_waitcnt 12970 vscnt(0) to allow 12971 them to be 12972 independently moved 12973 according to the 12974 following rules. 12975 - s_waitcnt vmcnt(0) 12976 Must happen after 12977 preceding 12978 global/generic load 12979 atomic/ 12980 atomicrmw-with-return-value 12981 with memory 12982 ordering of seq_cst 12983 and with equal or 12984 wider sync scope. 12985 (Note that seq_cst 12986 fences have their 12987 own s_waitcnt 12988 vmcnt(0) and so do 12989 not need to be 12990 considered.) 12991 - s_waitcnt vscnt(0) 12992 Must happen after 12993 preceding 12994 global/generic store 12995 atomic/ 12996 atomicrmw-no-return-value 12997 with memory 12998 ordering of seq_cst 12999 and with equal or 13000 wider sync scope. 13001 (Note that seq_cst 13002 fences have their 13003 own s_waitcnt 13004 vscnt(0) and so do 13005 not need to be 13006 considered.) 13007 - Ensures any 13008 preceding 13009 sequential 13010 consistent global 13011 memory instructions 13012 have completed 13013 before executing 13014 this sequentially 13015 consistent 13016 instruction. This 13017 prevents reordering 13018 a seq_cst store 13019 followed by a 13020 seq_cst load. (Note 13021 that seq_cst is 13022 stronger than 13023 acquire/release as 13024 the reordering of 13025 load acquire 13026 followed by a store 13027 release is 13028 prevented by the 13029 s_waitcnt of 13030 the release, but 13031 there is nothing 13032 preventing a store 13033 release followed by 13034 load acquire from 13035 completing out of 13036 order. The s_waitcnt 13037 could be placed after 13038 seq_store or before 13039 the seq_load. We 13040 choose the load to 13041 make the s_waitcnt be 13042 as late as possible 13043 so that the store 13044 may have already 13045 completed.) 13046 13047 2. *Following 13048 instructions same as 13049 corresponding load 13050 atomic acquire, 13051 except must generate 13052 all instructions even 13053 for OpenCL.* 13054 13055 load atomic seq_cst - agent - global 1. s_waitcnt lgkmcnt(0) & 13056 - system - generic vmcnt(0) & vscnt(0) 13057 13058 - Could be split into 13059 separate s_waitcnt 13060 vmcnt(0), s_waitcnt 13061 vscnt(0) and s_waitcnt 13062 lgkmcnt(0) to allow 13063 them to be 13064 independently moved 13065 according to the 13066 following rules. 13067 - s_waitcnt lgkmcnt(0) 13068 must happen after 13069 preceding 13070 local load 13071 atomic/store 13072 atomic/atomicrmw 13073 with memory 13074 ordering of seq_cst 13075 and with equal or 13076 wider sync scope. 13077 (Note that seq_cst 13078 fences have their 13079 own s_waitcnt 13080 lgkmcnt(0) and so do 13081 not need to be 13082 considered.) 13083 - s_waitcnt vmcnt(0) 13084 must happen after 13085 preceding 13086 global/generic load 13087 atomic/ 13088 atomicrmw-with-return-value 13089 with memory 13090 ordering of seq_cst 13091 and with equal or 13092 wider sync scope. 13093 (Note that seq_cst 13094 fences have their 13095 own s_waitcnt 13096 vmcnt(0) and so do 13097 not need to be 13098 considered.) 13099 - s_waitcnt vscnt(0) 13100 Must happen after 13101 preceding 13102 global/generic store 13103 atomic/ 13104 atomicrmw-no-return-value 13105 with memory 13106 ordering of seq_cst 13107 and with equal or 13108 wider sync scope. 13109 (Note that seq_cst 13110 fences have their 13111 own s_waitcnt 13112 vscnt(0) and so do 13113 not need to be 13114 considered.) 13115 - Ensures any 13116 preceding 13117 sequential 13118 consistent global 13119 memory instructions 13120 have completed 13121 before executing 13122 this sequentially 13123 consistent 13124 instruction. This 13125 prevents reordering 13126 a seq_cst store 13127 followed by a 13128 seq_cst load. (Note 13129 that seq_cst is 13130 stronger than 13131 acquire/release as 13132 the reordering of 13133 load acquire 13134 followed by a store 13135 release is 13136 prevented by the 13137 s_waitcnt of 13138 the release, but 13139 there is nothing 13140 preventing a store 13141 release followed by 13142 load acquire from 13143 completing out of 13144 order. The s_waitcnt 13145 could be placed after 13146 seq_store or before 13147 the seq_load. We 13148 choose the load to 13149 make the s_waitcnt be 13150 as late as possible 13151 so that the store 13152 may have already 13153 completed.) 13154 13155 2. *Following 13156 instructions same as 13157 corresponding load 13158 atomic acquire, 13159 except must generate 13160 all instructions even 13161 for OpenCL.* 13162 store atomic seq_cst - singlethread - global *Same as corresponding 13163 - wavefront - local store atomic release, 13164 - workgroup - generic except must generate 13165 - agent all instructions even 13166 - system for OpenCL.* 13167 atomicrmw seq_cst - singlethread - global *Same as corresponding 13168 - wavefront - local atomicrmw acq_rel, 13169 - workgroup - generic except must generate 13170 - agent all instructions even 13171 - system for OpenCL.* 13172 fence seq_cst - singlethread *none* *Same as corresponding 13173 - wavefront fence acq_rel, 13174 - workgroup except must generate 13175 - agent all instructions even 13176 - system for OpenCL.* 13177 ============ ============ ============== ========== ================================ 13178 13179.. _amdgpu-amdhsa-trap-handler-abi: 13180 13181Trap Handler ABI 13182~~~~~~~~~~~~~~~~ 13183 13184For code objects generated by the AMDGPU backend for HSA [HSA]_ compatible 13185runtimes (see :ref:`amdgpu-os`), the runtime installs a trap handler that 13186supports the ``s_trap`` instruction. For usage see: 13187 13188- :ref:`amdgpu-trap-handler-for-amdhsa-os-v2-table` 13189- :ref:`amdgpu-trap-handler-for-amdhsa-os-v3-table` 13190- :ref:`amdgpu-trap-handler-for-amdhsa-os-v4-onwards-table` 13191 13192 .. table:: AMDGPU Trap Handler for AMDHSA OS Code Object V2 13193 :name: amdgpu-trap-handler-for-amdhsa-os-v2-table 13194 13195 =================== =============== =============== ======================================= 13196 Usage Code Sequence Trap Handler Description 13197 Inputs 13198 =================== =============== =============== ======================================= 13199 reserved ``s_trap 0x00`` Reserved by hardware. 13200 ``debugtrap(arg)`` ``s_trap 0x01`` ``SGPR0-1``: Reserved for Finalizer HSA ``debugtrap`` 13201 ``queue_ptr`` intrinsic (not implemented). 13202 ``VGPR0``: 13203 ``arg`` 13204 ``llvm.trap`` ``s_trap 0x02`` ``SGPR0-1``: Causes wave to be halted with the PC at 13205 ``queue_ptr`` the trap instruction. The associated 13206 queue is signalled to put it into the 13207 error state. When the queue is put in 13208 the error state, the waves executing 13209 dispatches on the queue will be 13210 terminated. 13211 ``llvm.debugtrap`` ``s_trap 0x03`` *none* - If debugger not enabled then behaves 13212 as a no-operation. The trap handler 13213 is entered and immediately returns to 13214 continue execution of the wavefront. 13215 - If the debugger is enabled, causes 13216 the debug trap to be reported by the 13217 debugger and the wavefront is put in 13218 the halt state with the PC at the 13219 instruction. The debugger must 13220 increment the PC and resume the wave. 13221 reserved ``s_trap 0x04`` Reserved. 13222 reserved ``s_trap 0x05`` Reserved. 13223 reserved ``s_trap 0x06`` Reserved. 13224 reserved ``s_trap 0x07`` Reserved. 13225 reserved ``s_trap 0x08`` Reserved. 13226 reserved ``s_trap 0xfe`` Reserved. 13227 reserved ``s_trap 0xff`` Reserved. 13228 =================== =============== =============== ======================================= 13229 13230.. 13231 13232 .. table:: AMDGPU Trap Handler for AMDHSA OS Code Object V3 13233 :name: amdgpu-trap-handler-for-amdhsa-os-v3-table 13234 13235 =================== =============== =============== ======================================= 13236 Usage Code Sequence Trap Handler Description 13237 Inputs 13238 =================== =============== =============== ======================================= 13239 reserved ``s_trap 0x00`` Reserved by hardware. 13240 debugger breakpoint ``s_trap 0x01`` *none* Reserved for debugger to use for 13241 breakpoints. Causes wave to be halted 13242 with the PC at the trap instruction. 13243 The debugger is responsible to resume 13244 the wave, including the instruction 13245 that the breakpoint overwrote. 13246 ``llvm.trap`` ``s_trap 0x02`` ``SGPR0-1``: Causes wave to be halted with the PC at 13247 ``queue_ptr`` the trap instruction. The associated 13248 queue is signalled to put it into the 13249 error state. When the queue is put in 13250 the error state, the waves executing 13251 dispatches on the queue will be 13252 terminated. 13253 ``llvm.debugtrap`` ``s_trap 0x03`` *none* - If debugger not enabled then behaves 13254 as a no-operation. The trap handler 13255 is entered and immediately returns to 13256 continue execution of the wavefront. 13257 - If the debugger is enabled, causes 13258 the debug trap to be reported by the 13259 debugger and the wavefront is put in 13260 the halt state with the PC at the 13261 instruction. The debugger must 13262 increment the PC and resume the wave. 13263 reserved ``s_trap 0x04`` Reserved. 13264 reserved ``s_trap 0x05`` Reserved. 13265 reserved ``s_trap 0x06`` Reserved. 13266 reserved ``s_trap 0x07`` Reserved. 13267 reserved ``s_trap 0x08`` Reserved. 13268 reserved ``s_trap 0xfe`` Reserved. 13269 reserved ``s_trap 0xff`` Reserved. 13270 =================== =============== =============== ======================================= 13271 13272.. 13273 13274 .. table:: AMDGPU Trap Handler for AMDHSA OS Code Object V4 and Above 13275 :name: amdgpu-trap-handler-for-amdhsa-os-v4-onwards-table 13276 13277 =================== =============== ================ ================= ======================================= 13278 Usage Code Sequence GFX6-GFX8 Inputs GFX9-GFX10 Inputs Description 13279 =================== =============== ================ ================= ======================================= 13280 reserved ``s_trap 0x00`` Reserved by hardware. 13281 debugger breakpoint ``s_trap 0x01`` *none* *none* Reserved for debugger to use for 13282 breakpoints. Causes wave to be halted 13283 with the PC at the trap instruction. 13284 The debugger is responsible to resume 13285 the wave, including the instruction 13286 that the breakpoint overwrote. 13287 ``llvm.trap`` ``s_trap 0x02`` ``SGPR0-1``: *none* Causes wave to be halted with the PC at 13288 ``queue_ptr`` the trap instruction. The associated 13289 queue is signalled to put it into the 13290 error state. When the queue is put in 13291 the error state, the waves executing 13292 dispatches on the queue will be 13293 terminated. 13294 ``llvm.debugtrap`` ``s_trap 0x03`` *none* *none* - If debugger not enabled then behaves 13295 as a no-operation. The trap handler 13296 is entered and immediately returns to 13297 continue execution of the wavefront. 13298 - If the debugger is enabled, causes 13299 the debug trap to be reported by the 13300 debugger and the wavefront is put in 13301 the halt state with the PC at the 13302 instruction. The debugger must 13303 increment the PC and resume the wave. 13304 reserved ``s_trap 0x04`` Reserved. 13305 reserved ``s_trap 0x05`` Reserved. 13306 reserved ``s_trap 0x06`` Reserved. 13307 reserved ``s_trap 0x07`` Reserved. 13308 reserved ``s_trap 0x08`` Reserved. 13309 reserved ``s_trap 0xfe`` Reserved. 13310 reserved ``s_trap 0xff`` Reserved. 13311 =================== =============== ================ ================= ======================================= 13312 13313.. _amdgpu-amdhsa-function-call-convention: 13314 13315Call Convention 13316~~~~~~~~~~~~~~~ 13317 13318.. note:: 13319 13320 This section is currently incomplete and has inaccuracies. It is WIP that will 13321 be updated as information is determined. 13322 13323See :ref:`amdgpu-dwarf-address-space-identifier` for information on swizzled 13324addresses. Unswizzled addresses are normal linear addresses. 13325 13326.. _amdgpu-amdhsa-function-call-convention-kernel-functions: 13327 13328Kernel Functions 13329++++++++++++++++ 13330 13331This section describes the call convention ABI for the outer kernel function. 13332 13333See :ref:`amdgpu-amdhsa-initial-kernel-execution-state` for the kernel call 13334convention. 13335 13336The following is not part of the AMDGPU kernel calling convention but describes 13337how the AMDGPU implements function calls: 13338 133391. Clang decides the kernarg layout to match the *HSA Programmer's Language 13340 Reference* [HSA]_. 13341 13342 - All structs are passed directly. 13343 - Lambda values are passed *TBA*. 13344 13345 .. TODO:: 13346 13347 - Does this really follow HSA rules? Or are structs >16 bytes passed 13348 by-value struct? 13349 - What is ABI for lambda values? 13350 133514. The kernel performs certain setup in its prolog, as described in 13352 :ref:`amdgpu-amdhsa-kernel-prolog`. 13353 13354.. _amdgpu-amdhsa-function-call-convention-non-kernel-functions: 13355 13356Non-Kernel Functions 13357++++++++++++++++++++ 13358 13359This section describes the call convention ABI for functions other than the 13360outer kernel function. 13361 13362If a kernel has function calls then scratch is always allocated and used for 13363the call stack which grows from low address to high address using the swizzled 13364scratch address space. 13365 13366On entry to a function: 13367 133681. SGPR0-3 contain a V# with the following properties (see 13369 :ref:`amdgpu-amdhsa-kernel-prolog-private-segment-buffer`): 13370 13371 * Base address pointing to the beginning of the wavefront scratch backing 13372 memory. 13373 * Swizzled with dword element size and stride of wavefront size elements. 13374 133752. The FLAT_SCRATCH register pair is setup. See 13376 :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`. 133773. GFX6-GFX8: M0 register set to the size of LDS in bytes. See 13378 :ref:`amdgpu-amdhsa-kernel-prolog-m0`. 133794. The EXEC register is set to the lanes active on entry to the function. 133805. MODE register: *TBD* 133816. VGPR0-31 and SGPR4-29 are used to pass function input arguments as described 13382 below. 133837. SGPR30-31 return address (RA). The code address that the function must 13384 return to when it completes. The value is undefined if the function is *no 13385 return*. 133868. SGPR32 is used for the stack pointer (SP). It is an unswizzled scratch 13387 offset relative to the beginning of the wavefront scratch backing memory. 13388 13389 The unswizzled SP can be used with buffer instructions as an unswizzled SGPR 13390 offset with the scratch V# in SGPR0-3 to access the stack in a swizzled 13391 manner. 13392 13393 The unswizzled SP value can be converted into the swizzled SP value by: 13394 13395 | swizzled SP = unswizzled SP / wavefront size 13396 13397 This may be used to obtain the private address space address of stack 13398 objects and to convert this address to a flat address by adding the flat 13399 scratch aperture base address. 13400 13401 The swizzled SP value is always 4 bytes aligned for the ``r600`` 13402 architecture and 16 byte aligned for the ``amdgcn`` architecture. 13403 13404 .. note:: 13405 13406 The ``amdgcn`` value is selected to avoid dynamic stack alignment for the 13407 OpenCL language which has the largest base type defined as 16 bytes. 13408 13409 On entry, the swizzled SP value is the address of the first function 13410 argument passed on the stack. Other stack passed arguments are positive 13411 offsets from the entry swizzled SP value. 13412 13413 The function may use positive offsets beyond the last stack passed argument 13414 for stack allocated local variables and register spill slots. If necessary, 13415 the function may align these to greater alignment than 16 bytes. After these 13416 the function may dynamically allocate space for such things as runtime sized 13417 ``alloca`` local allocations. 13418 13419 If the function calls another function, it will place any stack allocated 13420 arguments after the last local allocation and adjust SGPR32 to the address 13421 after the last local allocation. 13422 134239. All other registers are unspecified. 1342410. Any necessary ``s_waitcnt`` has been performed to ensure memory is available 13425 to the function. 13426 13427On exit from a function: 13428 134291. VGPR0-31 and SGPR4-29 are used to pass function result arguments as 13430 described below. Any registers used are considered clobbered registers. 134312. The following registers are preserved and have the same value as on entry: 13432 13433 * FLAT_SCRATCH 13434 * EXEC 13435 * GFX6-GFX8: M0 13436 * All SGPR registers except the clobbered registers of SGPR4-31. 13437 * VGPR40-47 13438 * VGPR56-63 13439 * VGPR72-79 13440 * VGPR88-95 13441 * VGPR104-111 13442 * VGPR120-127 13443 * VGPR136-143 13444 * VGPR152-159 13445 * VGPR168-175 13446 * VGPR184-191 13447 * VGPR200-207 13448 * VGPR216-223 13449 * VGPR232-239 13450 * VGPR248-255 13451 13452 .. note:: 13453 13454 Except the argument registers, the VGPRs clobbered and the preserved 13455 registers are intermixed at regular intervals in order to keep a 13456 similar ratio independent of the number of allocated VGPRs. 13457 13458 * GFX90A: All AGPR registers except the clobbered registers AGPR0-31. 13459 * Lanes of all VGPRs that are inactive at the call site. 13460 13461 For the AMDGPU backend, an inter-procedural register allocation (IPRA) 13462 optimization may mark some of clobbered SGPR and VGPR registers as 13463 preserved if it can be determined that the called function does not change 13464 their value. 13465 134662. The PC is set to the RA provided on entry. 134673. MODE register: *TBD*. 134684. All other registers are clobbered. 134695. Any necessary ``s_waitcnt`` has been performed to ensure memory accessed by 13470 function is available to the caller. 13471 13472.. TODO:: 13473 13474 - How are function results returned? The address of structured types is passed 13475 by reference, but what about other types? 13476 13477The function input arguments are made up of the formal arguments explicitly 13478declared by the source language function plus the implicit input arguments used 13479by the implementation. 13480 13481The source language input arguments are: 13482 134831. Any source language implicit ``this`` or ``self`` argument comes first as a 13484 pointer type. 134852. Followed by the function formal arguments in left to right source order. 13486 13487The source language result arguments are: 13488 134891. The function result argument. 13490 13491The source language input or result struct type arguments that are less than or 13492equal to 16 bytes, are decomposed recursively into their base type fields, and 13493each field is passed as if a separate argument. For input arguments, if the 13494called function requires the struct to be in memory, for example because its 13495address is taken, then the function body is responsible for allocating a stack 13496location and copying the field arguments into it. Clang terms this *direct 13497struct*. 13498 13499The source language input struct type arguments that are greater than 16 bytes, 13500are passed by reference. The caller is responsible for allocating a stack 13501location to make a copy of the struct value and pass the address as the input 13502argument. The called function is responsible to perform the dereference when 13503accessing the input argument. Clang terms this *by-value struct*. 13504 13505A source language result struct type argument that is greater than 16 bytes, is 13506returned by reference. The caller is responsible for allocating a stack location 13507to hold the result value and passes the address as the last input argument 13508(before the implicit input arguments). In this case there are no result 13509arguments. The called function is responsible to perform the dereference when 13510storing the result value. Clang terms this *structured return (sret)*. 13511 13512*TODO: correct the ``sret`` definition.* 13513 13514.. TODO:: 13515 13516 Is this definition correct? Or is ``sret`` only used if passing in registers, and 13517 pass as non-decomposed struct as stack argument? Or something else? Is the 13518 memory location in the caller stack frame, or a stack memory argument and so 13519 no address is passed as the caller can directly write to the argument stack 13520 location? But then the stack location is still live after return. If an 13521 argument stack location is it the first stack argument or the last one? 13522 13523Lambda argument types are treated as struct types with an implementation defined 13524set of fields. 13525 13526.. TODO:: 13527 13528 Need to specify the ABI for lambda types for AMDGPU. 13529 13530For AMDGPU backend all source language arguments (including the decomposed 13531struct type arguments) are passed in VGPRs unless marked ``inreg`` in which case 13532they are passed in SGPRs. 13533 13534The AMDGPU backend walks the function call graph from the leaves to determine 13535which implicit input arguments are used, propagating to each caller of the 13536function. The used implicit arguments are appended to the function arguments 13537after the source language arguments in the following order: 13538 13539.. TODO:: 13540 13541 Is recursion or external functions supported? 13542 135431. Work-Item ID (1 VGPR) 13544 13545 The X, Y and Z work-item ID are packed into a single VGRP with the following 13546 layout. Only fields actually used by the function are set. The other bits 13547 are undefined. 13548 13549 The values come from the initial kernel execution state. See 13550 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`. 13551 13552 .. table:: Work-item implicit argument layout 13553 :name: amdgpu-amdhsa-workitem-implicit-argument-layout-table 13554 13555 ======= ======= ============== 13556 Bits Size Field Name 13557 ======= ======= ============== 13558 9:0 10 bits X Work-Item ID 13559 19:10 10 bits Y Work-Item ID 13560 29:20 10 bits Z Work-Item ID 13561 31:30 2 bits Unused 13562 ======= ======= ============== 13563 135642. Dispatch Ptr (2 SGPRs) 13565 13566 The value comes from the initial kernel execution state. See 13567 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 13568 135693. Queue Ptr (2 SGPRs) 13570 13571 The value comes from the initial kernel execution state. See 13572 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 13573 135744. Kernarg Segment Ptr (2 SGPRs) 13575 13576 The value comes from the initial kernel execution state. See 13577 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 13578 135795. Dispatch id (2 SGPRs) 13580 13581 The value comes from the initial kernel execution state. See 13582 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 13583 135846. Work-Group ID X (1 SGPR) 13585 13586 The value comes from the initial kernel execution state. See 13587 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 13588 135897. Work-Group ID Y (1 SGPR) 13590 13591 The value comes from the initial kernel execution state. See 13592 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 13593 135948. Work-Group ID Z (1 SGPR) 13595 13596 The value comes from the initial kernel execution state. See 13597 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 13598 135999. Implicit Argument Ptr (2 SGPRs) 13600 13601 The value is computed by adding an offset to Kernarg Segment Ptr to get the 13602 global address space pointer to the first kernarg implicit argument. 13603 13604The input and result arguments are assigned in order in the following manner: 13605 13606.. note:: 13607 13608 There are likely some errors and omissions in the following description that 13609 need correction. 13610 13611 .. TODO:: 13612 13613 Check the Clang source code to decipher how function arguments and return 13614 results are handled. Also see the AMDGPU specific values used. 13615 13616* VGPR arguments are assigned to consecutive VGPRs starting at VGPR0 up to 13617 VGPR31. 13618 13619 If there are more arguments than will fit in these registers, the remaining 13620 arguments are allocated on the stack in order on naturally aligned 13621 addresses. 13622 13623 .. TODO:: 13624 13625 How are overly aligned structures allocated on the stack? 13626 13627* SGPR arguments are assigned to consecutive SGPRs starting at SGPR0 up to 13628 SGPR29. 13629 13630 If there are more arguments than will fit in these registers, the remaining 13631 arguments are allocated on the stack in order on naturally aligned 13632 addresses. 13633 13634Note that decomposed struct type arguments may have some fields passed in 13635registers and some in memory. 13636 13637.. TODO:: 13638 13639 So, a struct which can pass some fields as decomposed register arguments, will 13640 pass the rest as decomposed stack elements? But an argument that will not start 13641 in registers will not be decomposed and will be passed as a non-decomposed 13642 stack value? 13643 13644The following is not part of the AMDGPU function calling convention but 13645describes how the AMDGPU implements function calls: 13646 136471. SGPR33 is used as a frame pointer (FP) if necessary. Like the SP it is an 13648 unswizzled scratch address. It is only needed if runtime sized ``alloca`` 13649 are used, or for the reasons defined in ``SIFrameLowering``. 136502. Runtime stack alignment is supported. SGPR34 is used as a base pointer (BP) 13651 to access the incoming stack arguments in the function. The BP is needed 13652 only when the function requires the runtime stack alignment. 13653 136543. Allocating SGPR arguments on the stack are not supported. 13655 136564. No CFI is currently generated. See 13657 :ref:`amdgpu-dwarf-call-frame-information`. 13658 13659 .. note:: 13660 13661 CFI will be generated that defines the CFA as the unswizzled address 13662 relative to the wave scratch base in the unswizzled private address space 13663 of the lowest address stack allocated local variable. 13664 13665 ``DW_AT_frame_base`` will be defined as the swizzled address in the 13666 swizzled private address space by dividing the CFA by the wavefront size 13667 (since CFA is always at least dword aligned which matches the scratch 13668 swizzle element size). 13669 13670 If no dynamic stack alignment was performed, the stack allocated arguments 13671 are accessed as negative offsets relative to ``DW_AT_frame_base``, and the 13672 local variables and register spill slots are accessed as positive offsets 13673 relative to ``DW_AT_frame_base``. 13674 136755. Function argument passing is implemented by copying the input physical 13676 registers to virtual registers on entry. The register allocator can spill if 13677 necessary. These are copied back to physical registers at call sites. The 13678 net effect is that each function call can have these values in entirely 13679 distinct locations. The IPRA can help avoid shuffling argument registers. 136806. Call sites are implemented by setting up the arguments at positive offsets 13681 from SP. Then SP is incremented to account for the known frame size before 13682 the call and decremented after the call. 13683 13684 .. note:: 13685 13686 The CFI will reflect the changed calculation needed to compute the CFA 13687 from SP. 13688 136897. 4 byte spill slots are used in the stack frame. One slot is allocated for an 13690 emergency spill slot. Buffer instructions are used for stack accesses and 13691 not the ``flat_scratch`` instruction. 13692 13693 .. TODO:: 13694 13695 Explain when the emergency spill slot is used. 13696 13697.. TODO:: 13698 13699 Possible broken issues: 13700 13701 - Stack arguments must be aligned to required alignment. 13702 - Stack is aligned to max(16, max formal argument alignment) 13703 - Direct argument < 64 bits should check register budget. 13704 - Register budget calculation should respect ``inreg`` for SGPR. 13705 - SGPR overflow is not handled. 13706 - struct with 1 member unpeeling is not checking size of member. 13707 - ``sret`` is after ``this`` pointer. 13708 - Caller is not implementing stack realignment: need an extra pointer. 13709 - Should say AMDGPU passes FP rather than SP. 13710 - Should CFI define CFA as address of locals or arguments. Difference is 13711 apparent when have implemented dynamic alignment. 13712 - If ``SCRATCH`` instruction could allow negative offsets, then can make FP be 13713 highest address of stack frame and use negative offset for locals. Would 13714 allow SP to be the same as FP and could support signal-handler-like as now 13715 have a real SP for the top of the stack. 13716 - How is ``sret`` passed on the stack? In argument stack area? Can it overlay 13717 arguments? 13718 13719AMDPAL 13720------ 13721 13722This section provides code conventions used when the target triple OS is 13723``amdpal`` (see :ref:`amdgpu-target-triples`). 13724 13725.. _amdgpu-amdpal-code-object-metadata-section: 13726 13727Code Object Metadata 13728~~~~~~~~~~~~~~~~~~~~ 13729 13730.. note:: 13731 13732 The metadata is currently in development and is subject to major 13733 changes. Only the current version is supported. *When this document 13734 was generated the version was 2.6.* 13735 13736Code object metadata is specified by the ``NT_AMDGPU_METADATA`` note 13737record (see :ref:`amdgpu-note-records-v3-onwards`). 13738 13739The metadata is represented as Message Pack formatted binary data (see 13740[MsgPack]_). The top level is a Message Pack map that includes the keys 13741defined in table :ref:`amdgpu-amdpal-code-object-metadata-map-table` 13742and referenced tables. 13743 13744Additional information can be added to the maps. To avoid conflicts, any 13745key names should be prefixed by "*vendor-name*." where ``vendor-name`` 13746can be the name of the vendor and specific vendor tool that generates the 13747information. The prefix is abbreviated to simply "." when it appears 13748within a map that has been added by the same *vendor-name*. 13749 13750 .. table:: AMDPAL Code Object Metadata Map 13751 :name: amdgpu-amdpal-code-object-metadata-map-table 13752 13753 =================== ============== ========= ====================================================================== 13754 String Key Value Type Required? Description 13755 =================== ============== ========= ====================================================================== 13756 "amdpal.version" sequence of Required PAL code object metadata (major, minor) version. The current values 13757 2 integers are defined by *Util::Abi::PipelineMetadata(Major|Minor)Version*. 13758 "amdpal.pipelines" sequence of Required Per-pipeline metadata. See 13759 map :ref:`amdgpu-amdpal-code-object-pipeline-metadata-map-table` for the 13760 definition of the keys included in that map. 13761 =================== ============== ========= ====================================================================== 13762 13763.. 13764 13765 .. table:: AMDPAL Code Object Pipeline Metadata Map 13766 :name: amdgpu-amdpal-code-object-pipeline-metadata-map-table 13767 13768 ====================================== ============== ========= =================================================== 13769 String Key Value Type Required? Description 13770 ====================================== ============== ========= =================================================== 13771 ".name" string Source name of the pipeline. 13772 ".type" string Pipeline type, e.g. VsPs. Values include: 13773 13774 - "VsPs" 13775 - "Gs" 13776 - "Cs" 13777 - "Ngg" 13778 - "Tess" 13779 - "GsTess" 13780 - "NggTess" 13781 13782 ".internal_pipeline_hash" sequence of Required Internal compiler hash for this pipeline. Lower 13783 2 integers 64 bits is the "stable" portion of the hash, used 13784 for e.g. shader replacement lookup. Upper 64 bits 13785 is the "unique" portion of the hash, used for 13786 e.g. pipeline cache lookup. The value is 13787 implementation defined, and can not be relied on 13788 between different builds of the compiler. 13789 ".shaders" map Per-API shader metadata. See 13790 :ref:`amdgpu-amdpal-code-object-shader-map-table` 13791 for the definition of the keys included in that 13792 map. 13793 ".hardware_stages" map Per-hardware stage metadata. See 13794 :ref:`amdgpu-amdpal-code-object-hardware-stage-map-table` 13795 for the definition of the keys included in that 13796 map. 13797 ".shader_functions" map Per-shader function metadata. See 13798 :ref:`amdgpu-amdpal-code-object-shader-function-map-table` 13799 for the definition of the keys included in that 13800 map. 13801 ".registers" map Required Hardware register configuration. See 13802 :ref:`amdgpu-amdpal-code-object-register-map-table` 13803 for the definition of the keys included in that 13804 map. 13805 ".user_data_limit" integer Number of user data entries accessed by this 13806 pipeline. 13807 ".spill_threshold" integer The user data spill threshold. 0xFFFF for 13808 NoUserDataSpilling. 13809 ".uses_viewport_array_index" boolean Indicates whether or not the pipeline uses the 13810 viewport array index feature. Pipelines which use 13811 this feature can render into all 16 viewports, 13812 whereas pipelines which do not use it are 13813 restricted to viewport #0. 13814 ".es_gs_lds_size" integer Size in bytes of LDS space used internally for 13815 handling data-passing between the ES and GS 13816 shader stages. This can be zero if the data is 13817 passed using off-chip buffers. This value should 13818 be used to program all user-SGPRs which have been 13819 marked with "UserDataMapping::EsGsLdsSize" 13820 (typically only the GS and VS HW stages will ever 13821 have a user-SGPR so marked). 13822 ".nggSubgroupSize" integer Explicit maximum subgroup size for NGG shaders 13823 (maximum number of threads in a subgroup). 13824 ".num_interpolants" integer Graphics only. Number of PS interpolants. 13825 ".mesh_scratch_memory_size" integer Max mesh shader scratch memory used. 13826 ".api" string Name of the client graphics API. 13827 ".api_create_info" binary Graphics API shader create info binary blob. Can 13828 be defined by the driver using the compiler if 13829 they want to be able to correlate API-specific 13830 information used during creation at a later time. 13831 ====================================== ============== ========= =================================================== 13832 13833.. 13834 13835 .. table:: AMDPAL Code Object Shader Map 13836 :name: amdgpu-amdpal-code-object-shader-map-table 13837 13838 13839 +-------------+--------------+-------------------------------------------------------------------+ 13840 |String Key |Value Type |Description | 13841 +=============+==============+===================================================================+ 13842 |- ".compute" |map |See :ref:`amdgpu-amdpal-code-object-api-shader-metadata-map-table` | 13843 |- ".vertex" | |for the definition of the keys included in that map. | 13844 |- ".hull" | | | 13845 |- ".domain" | | | 13846 |- ".geometry"| | | 13847 |- ".pixel" | | | 13848 +-------------+--------------+-------------------------------------------------------------------+ 13849 13850.. 13851 13852 .. table:: AMDPAL Code Object API Shader Metadata Map 13853 :name: amdgpu-amdpal-code-object-api-shader-metadata-map-table 13854 13855 ==================== ============== ========= ===================================================================== 13856 String Key Value Type Required? Description 13857 ==================== ============== ========= ===================================================================== 13858 ".api_shader_hash" sequence of Required Input shader hash, typically passed in from the client. The value 13859 2 integers is implementation defined, and can not be relied on between 13860 different builds of the compiler. 13861 ".hardware_mapping" sequence of Required Flags indicating the HW stages this API shader maps to. Values 13862 string include: 13863 13864 - ".ls" 13865 - ".hs" 13866 - ".es" 13867 - ".gs" 13868 - ".vs" 13869 - ".ps" 13870 - ".cs" 13871 13872 ==================== ============== ========= ===================================================================== 13873 13874.. 13875 13876 .. table:: AMDPAL Code Object Hardware Stage Map 13877 :name: amdgpu-amdpal-code-object-hardware-stage-map-table 13878 13879 +-------------+--------------+-----------------------------------------------------------------------+ 13880 |String Key |Value Type |Description | 13881 +=============+==============+=======================================================================+ 13882 |- ".ls" |map |See :ref:`amdgpu-amdpal-code-object-hardware-stage-metadata-map-table` | 13883 |- ".hs" | |for the definition of the keys included in that map. | 13884 |- ".es" | | | 13885 |- ".gs" | | | 13886 |- ".vs" | | | 13887 |- ".ps" | | | 13888 |- ".cs" | | | 13889 +-------------+--------------+-----------------------------------------------------------------------+ 13890 13891.. 13892 13893 .. table:: AMDPAL Code Object Hardware Stage Metadata Map 13894 :name: amdgpu-amdpal-code-object-hardware-stage-metadata-map-table 13895 13896 ========================== ============== ========= =============================================================== 13897 String Key Value Type Required? Description 13898 ========================== ============== ========= =============================================================== 13899 ".entry_point" string The ELF symbol pointing to this pipeline's stage entry point. 13900 ".scratch_memory_size" integer Scratch memory size in bytes. 13901 ".lds_size" integer Local Data Share size in bytes. 13902 ".perf_data_buffer_size" integer Performance data buffer size in bytes. 13903 ".vgpr_count" integer Number of VGPRs used. 13904 ".agpr_count" integer Number of AGPRs used. 13905 ".sgpr_count" integer Number of SGPRs used. 13906 ".vgpr_limit" integer If non-zero, indicates the shader was compiled with a 13907 directive to instruct the compiler to limit the VGPR usage to 13908 be less than or equal to the specified value (only set if 13909 different from HW default). 13910 ".sgpr_limit" integer SGPR count upper limit (only set if different from HW 13911 default). 13912 ".threadgroup_dimensions" sequence of Thread-group X/Y/Z dimensions (Compute only). 13913 3 integers 13914 ".wavefront_size" integer Wavefront size (only set if different from HW default). 13915 ".uses_uavs" boolean The shader reads or writes UAVs. 13916 ".uses_rovs" boolean The shader reads or writes ROVs. 13917 ".writes_uavs" boolean The shader writes to one or more UAVs. 13918 ".writes_depth" boolean The shader writes out a depth value. 13919 ".uses_append_consume" boolean The shader uses append and/or consume operations, either 13920 memory or GDS. 13921 ".uses_prim_id" boolean The shader uses PrimID. 13922 ========================== ============== ========= =============================================================== 13923 13924.. 13925 13926 .. table:: AMDPAL Code Object Shader Function Map 13927 :name: amdgpu-amdpal-code-object-shader-function-map-table 13928 13929 =============== ============== ==================================================================== 13930 String Key Value Type Description 13931 =============== ============== ==================================================================== 13932 *symbol name* map *symbol name* is the ELF symbol name of the shader function code 13933 entry address. The value is the function's metadata. See 13934 :ref:`amdgpu-amdpal-code-object-shader-function-metadata-map-table`. 13935 =============== ============== ==================================================================== 13936 13937.. 13938 13939 .. table:: AMDPAL Code Object Shader Function Metadata Map 13940 :name: amdgpu-amdpal-code-object-shader-function-metadata-map-table 13941 13942 ============================= ============== ================================================================= 13943 String Key Value Type Description 13944 ============================= ============== ================================================================= 13945 ".api_shader_hash" sequence of Input shader hash, typically passed in from the client. The value 13946 2 integers is implementation defined, and can not be relied on between 13947 different builds of the compiler. 13948 ".scratch_memory_size" integer Size in bytes of scratch memory used by the shader. 13949 ".lds_size" integer Size in bytes of LDS memory. 13950 ".vgpr_count" integer Number of VGPRs used by the shader. 13951 ".sgpr_count" integer Number of SGPRs used by the shader. 13952 ".stack_frame_size_in_bytes" integer Amount of stack size used by the shader. 13953 ".shader_subtype" string Shader subtype/kind. Values include: 13954 13955 - "Unknown" 13956 13957 ============================= ============== ================================================================= 13958 13959.. 13960 13961 .. table:: AMDPAL Code Object Register Map 13962 :name: amdgpu-amdpal-code-object-register-map-table 13963 13964 ========================== ============== ==================================================================== 13965 32-bit Integer Key Value Type Description 13966 ========================== ============== ==================================================================== 13967 ``reg offset`` 32-bit integer ``reg offset`` is the dword offset into the GFXIP register space of 13968 a GRBM register (i.e., driver accessible GPU register number, not 13969 shader GPR register number). The driver is required to program each 13970 specified register to the corresponding specified value when 13971 executing this pipeline. Typically, the ``reg offsets`` are the 13972 ``uint16_t`` offsets to each register as defined by the hardware 13973 chip headers. The register is set to the provided value. However, a 13974 ``reg offset`` that specifies a user data register (e.g., 13975 COMPUTE_USER_DATA_0) needs special treatment. See 13976 :ref:`amdgpu-amdpal-code-object-user-data-section` section for more 13977 information. 13978 ========================== ============== ==================================================================== 13979 13980.. _amdgpu-amdpal-code-object-user-data-section: 13981 13982User Data 13983+++++++++ 13984 13985Each hardware stage has a set of 32-bit physical SPI *user data registers* 13986(either 16 or 32 based on graphics IP and the stage) which can be 13987written from a command buffer and then loaded into SGPRs when waves are 13988launched via a subsequent dispatch or draw operation. This is the way 13989most arguments are passed from the application/runtime to a hardware 13990shader. 13991 13992PAL abstracts this functionality by exposing a set of 128 *user data 13993entries* per pipeline a client can use to pass arguments from a command 13994buffer to one or more shaders in that pipeline. The ELF code object must 13995specify a mapping from virtualized *user data entries* to physical *user 13996data registers*, and PAL is responsible for implementing that mapping, 13997including spilling overflow *user data entries* to memory if needed. 13998 13999Since the *user data registers* are GRBM-accessible SPI registers, this 14000mapping is actually embedded in the ``.registers`` metadata entry. For 14001most registers, the value in that map is a literal 32-bit value that 14002should be written to the register by the driver. However, when the 14003register is a *user data register* (any USER_DATA register e.g., 14004SPI_SHADER_USER_DATA_PS_5), the value is instead an encoding that tells 14005the driver to write either a *user data entry* value or one of several 14006driver-internal values to the register. This encoding is described in 14007the following table: 14008 14009.. note:: 14010 14011 Currently, *user data registers* 0 and 1 (e.g., SPI_SHADER_USER_DATA_PS_0, 14012 and SPI_SHADER_USER_DATA_PS_1) are reserved. *User data register* 0 must 14013 always be programmed to the address of the GlobalTable, and *user data 14014 register* 1 must always be programmed to the address of the PerShaderTable. 14015 14016.. 14017 14018 .. table:: AMDPAL User Data Mapping 14019 :name: amdgpu-amdpal-code-object-metadata-user-data-mapping-table 14020 14021 ========== ================= =============================================================================== 14022 Value Name Description 14023 ========== ================= =============================================================================== 14024 0..127 *User Data Entry* 32-bit value of user_data_entry[N] as specified via *CmdSetUserData()* 14025 0x10000000 GlobalTable 32-bit pointer to GPU memory containing the global internal table (should 14026 always point to *user data register* 0). 14027 0x10000001 PerShaderTable 32-bit pointer to GPU memory containing the per-shader internal table. See 14028 :ref:`amdgpu-amdpal-code-object-metadata-user-data-per-shader-table-section` 14029 for more detail (should always point to *user data register* 1). 14030 0x10000002 SpillTable 32-bit pointer to GPU memory containing the user data spill table. See 14031 :ref:`amdgpu-amdpal-code-object-metadata-user-data-spill-table-section` for 14032 more detail. 14033 0x10000003 BaseVertex Vertex offset (32-bit unsigned integer). Not needed if the pipeline doesn't 14034 reference the draw index in the vertex shader. Only supported by the first 14035 stage in a graphics pipeline. 14036 0x10000004 BaseInstance Instance offset (32-bit unsigned integer). Only supported by the first stage in 14037 a graphics pipeline. 14038 0x10000005 DrawIndex Draw index (32-bit unsigned integer). Only supported by the first stage in a 14039 graphics pipeline. 14040 0x10000006 Workgroup Thread group count (32-bit unsigned integer). Low half of a 64-bit address of 14041 a buffer containing the grid dimensions for a Compute dispatch operation. The 14042 high half of the address is stored in the next sequential user-SGPR. Only 14043 supported by compute pipelines. 14044 0x1000000A EsGsLdsSize Indicates that PAL will program this user-SGPR to contain the amount of LDS 14045 space used for the ES/GS pseudo-ring-buffer for passing data between shader 14046 stages. 14047 0x1000000B ViewId View id (32-bit unsigned integer) identifies a view of graphic 14048 pipeline instancing. 14049 0x1000000C StreamOutTable 32-bit pointer to GPU memory containing the stream out target SRD table. This 14050 can only appear for one shader stage per pipeline. 14051 0x1000000D PerShaderPerfData 32-bit pointer to GPU memory containing the per-shader performance data buffer. 14052 0x1000000F VertexBufferTable 32-bit pointer to GPU memory containing the vertex buffer SRD table. This can 14053 only appear for one shader stage per pipeline. 14054 0x10000010 UavExportTable 32-bit pointer to GPU memory containing the UAV export SRD table. This can 14055 only appear for one shader stage per pipeline (PS). These replace color targets 14056 and are completely separate from any UAVs used by the shader. This is optional, 14057 and only used by the PS when UAV exports are used to replace color-target 14058 exports to optimize specific shaders. 14059 0x10000011 NggCullingData 64-bit pointer to GPU memory containing the hardware register data needed by 14060 some NGG pipelines to perform culling. This value contains the address of the 14061 first of two consecutive registers which provide the full GPU address. 14062 0x10000015 FetchShaderPtr 64-bit pointer to GPU memory containing the fetch shader subroutine. 14063 ========== ================= =============================================================================== 14064 14065.. _amdgpu-amdpal-code-object-metadata-user-data-per-shader-table-section: 14066 14067Per-Shader Table 14068################ 14069 14070Low 32 bits of the GPU address for an optional buffer in the ``.data`` 14071section of the ELF. The high 32 bits of the address match the high 32 bits 14072of the shader's program counter. 14073 14074The buffer can be anything the shader compiler needs it for, and 14075allows each shader to have its own region of the ``.data`` section. 14076Typically, this could be a table of buffer SRD's and the data pointed to 14077by the buffer SRD's, but it could be a flat-address region of memory as 14078well. Its layout and usage are defined by the shader compiler. 14079 14080Each shader's table in the ``.data`` section is referenced by the symbol 14081``_amdgpu_``\ *xs*\ ``_shdr_intrl_data`` where *xs* corresponds with the 14082hardware shader stage the data is for. E.g., 14083``_amdgpu_cs_shdr_intrl_data`` for the compute shader hardware stage. 14084 14085.. _amdgpu-amdpal-code-object-metadata-user-data-spill-table-section: 14086 14087Spill Table 14088########### 14089 14090It is possible for a hardware shader to need access to more *user data 14091entries* than there are slots available in user data registers for one 14092or more hardware shader stages. In that case, the PAL runtime expects 14093the necessary *user data entries* to be spilled to GPU memory and use 14094one user data register to point to the spilled user data memory. The 14095value of the *user data entry* must then represent the location where 14096a shader expects to read the low 32-bits of the table's GPU virtual 14097address. The *spill table* itself represents a set of 32-bit values 14098managed by the PAL runtime in GPU-accessible memory that can be made 14099indirectly accessible to a hardware shader. 14100 14101Unspecified OS 14102-------------- 14103 14104This section provides code conventions used when the target triple OS is 14105empty (see :ref:`amdgpu-target-triples`). 14106 14107Trap Handler ABI 14108~~~~~~~~~~~~~~~~ 14109 14110For code objects generated by AMDGPU backend for non-amdhsa OS, the runtime does 14111not install a trap handler. The ``llvm.trap`` and ``llvm.debugtrap`` 14112instructions are handled as follows: 14113 14114 .. table:: AMDGPU Trap Handler for Non-AMDHSA OS 14115 :name: amdgpu-trap-handler-for-non-amdhsa-os-table 14116 14117 =============== =============== =========================================== 14118 Usage Code Sequence Description 14119 =============== =============== =========================================== 14120 llvm.trap s_endpgm Causes wavefront to be terminated. 14121 llvm.debugtrap *none* Compiler warning given that there is no 14122 trap handler installed. 14123 =============== =============== =========================================== 14124 14125Source Languages 14126================ 14127 14128.. _amdgpu-opencl: 14129 14130OpenCL 14131------ 14132 14133When the language is OpenCL the following differences occur: 14134 141351. The OpenCL memory model is used (see :ref:`amdgpu-amdhsa-memory-model`). 141362. The AMDGPU backend appends additional arguments to the kernel's explicit 14137 arguments for the AMDHSA OS (see 14138 :ref:`opencl-kernel-implicit-arguments-appended-for-amdhsa-os-table`). 141393. Additional metadata is generated 14140 (see :ref:`amdgpu-amdhsa-code-object-metadata`). 14141 14142 .. table:: OpenCL kernel implicit arguments appended for AMDHSA OS 14143 :name: opencl-kernel-implicit-arguments-appended-for-amdhsa-os-table 14144 14145 ======== ==== ========= =========================================== 14146 Position Byte Byte Description 14147 Size Alignment 14148 ======== ==== ========= =========================================== 14149 1 8 8 OpenCL Global Offset X 14150 2 8 8 OpenCL Global Offset Y 14151 3 8 8 OpenCL Global Offset Z 14152 4 8 8 OpenCL address of printf buffer 14153 5 8 8 OpenCL address of virtual queue used by 14154 enqueue_kernel. 14155 6 8 8 OpenCL address of AqlWrap struct used by 14156 enqueue_kernel. 14157 7 8 8 Pointer argument used for Multi-gird 14158 synchronization. 14159 ======== ==== ========= =========================================== 14160 14161.. _amdgpu-hcc: 14162 14163HCC 14164--- 14165 14166When the language is HCC the following differences occur: 14167 141681. The HSA memory model is used (see :ref:`amdgpu-amdhsa-memory-model`). 14169 14170.. _amdgpu-assembler: 14171 14172Assembler 14173--------- 14174 14175AMDGPU backend has LLVM-MC based assembler which is currently in development. 14176It supports AMDGCN GFX6-GFX10. 14177 14178This section describes general syntax for instructions and operands. 14179 14180Instructions 14181~~~~~~~~~~~~ 14182 14183An instruction has the following :doc:`syntax<AMDGPUInstructionSyntax>`: 14184 14185 | ``<``\ *opcode*\ ``> <``\ *operand0*\ ``>, <``\ *operand1*\ ``>,... 14186 <``\ *modifier0*\ ``> <``\ *modifier1*\ ``>...`` 14187 14188:doc:`Operands<AMDGPUOperandSyntax>` are comma-separated while 14189:doc:`modifiers<AMDGPUModifierSyntax>` are space-separated. 14190 14191The order of operands and modifiers is fixed. 14192Most modifiers are optional and may be omitted. 14193 14194Links to detailed instruction syntax description may be found in the following 14195table. Note that features under development are not included 14196in this description. 14197 14198 ============= ============================================= ======================================= 14199 Architecture Core ISA ISA Variants and Extensions 14200 ============= ============================================= ======================================= 14201 GCN 2 :doc:`GFX7<AMDGPU/AMDGPUAsmGFX7>` \- 14202 GCN 3, GCN 4 :doc:`GFX8<AMDGPU/AMDGPUAsmGFX8>` \- 14203 GCN 5 :doc:`GFX9<AMDGPU/AMDGPUAsmGFX9>` :doc:`gfx900<AMDGPU/AMDGPUAsmGFX900>` 14204 14205 :doc:`gfx902<AMDGPU/AMDGPUAsmGFX900>` 14206 14207 :doc:`gfx904<AMDGPU/AMDGPUAsmGFX904>` 14208 14209 :doc:`gfx906<AMDGPU/AMDGPUAsmGFX906>` 14210 14211 :doc:`gfx909<AMDGPU/AMDGPUAsmGFX900>` 14212 14213 CDNA 1 :doc:`GFX9<AMDGPU/AMDGPUAsmGFX9>` :doc:`gfx908<AMDGPU/AMDGPUAsmGFX908>` 14214 CDNA 2 :doc:`GFX9<AMDGPU/AMDGPUAsmGFX9>` :doc:`gfx90a<AMDGPU/AMDGPUAsmGFX90a>` 14215 RDNA 1 :doc:`GFX10 RDNA1<AMDGPU/AMDGPUAsmGFX10>` :doc:`gfx1010<AMDGPU/AMDGPUAsmGFX10>` 14216 14217 :doc:`gfx1011<AMDGPU/AMDGPUAsmGFX1011>` 14218 14219 :doc:`gfx1012<AMDGPU/AMDGPUAsmGFX1011>` 14220 14221 :doc:`gfx1013<AMDGPU/AMDGPUAsmGFX1013>` 14222 14223 RDNA 2 :doc:`GFX10 RDNA2<AMDGPU/AMDGPUAsmGFX1030>` :doc:`gfx1030<AMDGPU/AMDGPUAsmGFX1030>` 14224 14225 :doc:`gfx1031<AMDGPU/AMDGPUAsmGFX1030>` 14226 14227 :doc:`gfx1032<AMDGPU/AMDGPUAsmGFX1030>` 14228 14229 :doc:`gfx1033<AMDGPU/AMDGPUAsmGFX1030>` 14230 14231 :doc:`gfx1034<AMDGPU/AMDGPUAsmGFX1030>` 14232 14233 :doc:`gfx1035<AMDGPU/AMDGPUAsmGFX1030>` 14234 14235 :doc:`gfx1036<AMDGPU/AMDGPUAsmGFX1030>` 14236 ============= ============================================= ======================================= 14237 14238For more information about instructions, their semantics and supported 14239combinations of operands, refer to one of instruction set architecture manuals 14240[AMD-GCN-GFX6]_, [AMD-GCN-GFX7]_, [AMD-GCN-GFX8]_, 14241[AMD-GCN-GFX900-GFX904-VEGA]_, [AMD-GCN-GFX906-VEGA7NM]_, 14242[AMD-GCN-GFX908-CDNA1]_, [AMD-GCN-GFX90A-CDNA2]_, [AMD-GCN-GFX10-RDNA1]_ and 14243[AMD-GCN-GFX10-RDNA2]_. 14244 14245Operands 14246~~~~~~~~ 14247 14248Detailed description of operands may be found :doc:`here<AMDGPUOperandSyntax>`. 14249 14250Modifiers 14251~~~~~~~~~ 14252 14253Detailed description of modifiers may be found 14254:doc:`here<AMDGPUModifierSyntax>`. 14255 14256Instruction Examples 14257~~~~~~~~~~~~~~~~~~~~ 14258 14259DS 14260++ 14261 14262.. code-block:: nasm 14263 14264 ds_add_u32 v2, v4 offset:16 14265 ds_write_src2_b64 v2 offset0:4 offset1:8 14266 ds_cmpst_f32 v2, v4, v6 14267 ds_min_rtn_f64 v[8:9], v2, v[4:5] 14268 14269For full list of supported instructions, refer to "LDS/GDS instructions" in ISA 14270Manual. 14271 14272FLAT 14273++++ 14274 14275.. code-block:: nasm 14276 14277 flat_load_dword v1, v[3:4] 14278 flat_store_dwordx3 v[3:4], v[5:7] 14279 flat_atomic_swap v1, v[3:4], v5 glc 14280 flat_atomic_cmpswap v1, v[3:4], v[5:6] glc slc 14281 flat_atomic_fmax_x2 v[1:2], v[3:4], v[5:6] glc 14282 14283For full list of supported instructions, refer to "FLAT instructions" in ISA 14284Manual. 14285 14286MUBUF 14287+++++ 14288 14289.. code-block:: nasm 14290 14291 buffer_load_dword v1, off, s[4:7], s1 14292 buffer_store_dwordx4 v[1:4], v2, ttmp[4:7], s1 offen offset:4 glc tfe 14293 buffer_store_format_xy v[1:2], off, s[4:7], s1 14294 buffer_wbinvl1 14295 buffer_atomic_inc v1, v2, s[8:11], s4 idxen offset:4 slc 14296 14297For full list of supported instructions, refer to "MUBUF Instructions" in ISA 14298Manual. 14299 14300SMRD/SMEM 14301+++++++++ 14302 14303.. code-block:: nasm 14304 14305 s_load_dword s1, s[2:3], 0xfc 14306 s_load_dwordx8 s[8:15], s[2:3], s4 14307 s_load_dwordx16 s[88:103], s[2:3], s4 14308 s_dcache_inv_vol 14309 s_memtime s[4:5] 14310 14311For full list of supported instructions, refer to "Scalar Memory Operations" in 14312ISA Manual. 14313 14314SOP1 14315++++ 14316 14317.. code-block:: nasm 14318 14319 s_mov_b32 s1, s2 14320 s_mov_b64 s[0:1], 0x80000000 14321 s_cmov_b32 s1, 200 14322 s_wqm_b64 s[2:3], s[4:5] 14323 s_bcnt0_i32_b64 s1, s[2:3] 14324 s_swappc_b64 s[2:3], s[4:5] 14325 s_cbranch_join s[4:5] 14326 14327For full list of supported instructions, refer to "SOP1 Instructions" in ISA 14328Manual. 14329 14330SOP2 14331++++ 14332 14333.. code-block:: nasm 14334 14335 s_add_u32 s1, s2, s3 14336 s_and_b64 s[2:3], s[4:5], s[6:7] 14337 s_cselect_b32 s1, s2, s3 14338 s_andn2_b32 s2, s4, s6 14339 s_lshr_b64 s[2:3], s[4:5], s6 14340 s_ashr_i32 s2, s4, s6 14341 s_bfm_b64 s[2:3], s4, s6 14342 s_bfe_i64 s[2:3], s[4:5], s6 14343 s_cbranch_g_fork s[4:5], s[6:7] 14344 14345For full list of supported instructions, refer to "SOP2 Instructions" in ISA 14346Manual. 14347 14348SOPC 14349++++ 14350 14351.. code-block:: nasm 14352 14353 s_cmp_eq_i32 s1, s2 14354 s_bitcmp1_b32 s1, s2 14355 s_bitcmp0_b64 s[2:3], s4 14356 s_setvskip s3, s5 14357 14358For full list of supported instructions, refer to "SOPC Instructions" in ISA 14359Manual. 14360 14361SOPP 14362++++ 14363 14364.. code-block:: nasm 14365 14366 s_barrier 14367 s_nop 2 14368 s_endpgm 14369 s_waitcnt 0 ; Wait for all counters to be 0 14370 s_waitcnt vmcnt(0) & expcnt(0) & lgkmcnt(0) ; Equivalent to above 14371 s_waitcnt vmcnt(1) ; Wait for vmcnt counter to be 1. 14372 s_sethalt 9 14373 s_sleep 10 14374 s_sendmsg 0x1 14375 s_sendmsg sendmsg(MSG_INTERRUPT) 14376 s_trap 1 14377 14378For full list of supported instructions, refer to "SOPP Instructions" in ISA 14379Manual. 14380 14381Unless otherwise mentioned, little verification is performed on the operands 14382of SOPP Instructions, so it is up to the programmer to be familiar with the 14383range or acceptable values. 14384 14385VALU 14386++++ 14387 14388For vector ALU instruction opcodes (VOP1, VOP2, VOP3, VOPC, VOP_DPP, VOP_SDWA), 14389the assembler will automatically use optimal encoding based on its operands. To 14390force specific encoding, one can add a suffix to the opcode of the instruction: 14391 14392* _e32 for 32-bit VOP1/VOP2/VOPC 14393* _e64 for 64-bit VOP3 14394* _dpp for VOP_DPP 14395* _sdwa for VOP_SDWA 14396 14397VOP1/VOP2/VOP3/VOPC examples: 14398 14399.. code-block:: nasm 14400 14401 v_mov_b32 v1, v2 14402 v_mov_b32_e32 v1, v2 14403 v_nop 14404 v_cvt_f64_i32_e32 v[1:2], v2 14405 v_floor_f32_e32 v1, v2 14406 v_bfrev_b32_e32 v1, v2 14407 v_add_f32_e32 v1, v2, v3 14408 v_mul_i32_i24_e64 v1, v2, 3 14409 v_mul_i32_i24_e32 v1, -3, v3 14410 v_mul_i32_i24_e32 v1, -100, v3 14411 v_addc_u32 v1, s[0:1], v2, v3, s[2:3] 14412 v_max_f16_e32 v1, v2, v3 14413 14414VOP_DPP examples: 14415 14416.. code-block:: nasm 14417 14418 v_mov_b32 v0, v0 quad_perm:[0,2,1,1] 14419 v_sin_f32 v0, v0 row_shl:1 row_mask:0xa bank_mask:0x1 bound_ctrl:0 14420 v_mov_b32 v0, v0 wave_shl:1 14421 v_mov_b32 v0, v0 row_mirror 14422 v_mov_b32 v0, v0 row_bcast:31 14423 v_mov_b32 v0, v0 quad_perm:[1,3,0,1] row_mask:0xa bank_mask:0x1 bound_ctrl:0 14424 v_add_f32 v0, v0, |v0| row_shl:1 row_mask:0xa bank_mask:0x1 bound_ctrl:0 14425 v_max_f16 v1, v2, v3 row_shl:1 row_mask:0xa bank_mask:0x1 bound_ctrl:0 14426 14427VOP_SDWA examples: 14428 14429.. code-block:: nasm 14430 14431 v_mov_b32 v1, v2 dst_sel:BYTE_0 dst_unused:UNUSED_PRESERVE src0_sel:DWORD 14432 v_min_u32 v200, v200, v1 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:BYTE_1 src1_sel:DWORD 14433 v_sin_f32 v0, v0 dst_unused:UNUSED_PAD src0_sel:WORD_1 14434 v_fract_f32 v0, |v0| dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 14435 v_cmpx_le_u32 vcc, v1, v2 src0_sel:BYTE_2 src1_sel:WORD_0 14436 14437For full list of supported instructions, refer to "Vector ALU instructions". 14438 14439.. _amdgpu-amdhsa-assembler-predefined-symbols-v2: 14440 14441Code Object V2 Predefined Symbols 14442~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 14443 14444.. warning:: 14445 Code object V2 is not the default code object version emitted by 14446 this version of LLVM. 14447 14448The AMDGPU assembler defines and updates some symbols automatically. These 14449symbols do not affect code generation. 14450 14451.option.machine_version_major 14452+++++++++++++++++++++++++++++ 14453 14454Set to the GFX major generation number of the target being assembled for. For 14455example, when assembling for a "GFX9" target this will be set to the integer 14456value "9". The possible GFX major generation numbers are presented in 14457:ref:`amdgpu-processors`. 14458 14459.option.machine_version_minor 14460+++++++++++++++++++++++++++++ 14461 14462Set to the GFX minor generation number of the target being assembled for. For 14463example, when assembling for a "GFX810" target this will be set to the integer 14464value "1". The possible GFX minor generation numbers are presented in 14465:ref:`amdgpu-processors`. 14466 14467.option.machine_version_stepping 14468++++++++++++++++++++++++++++++++ 14469 14470Set to the GFX stepping generation number of the target being assembled for. 14471For example, when assembling for a "GFX704" target this will be set to the 14472integer value "4". The possible GFX stepping generation numbers are presented 14473in :ref:`amdgpu-processors`. 14474 14475.kernel.vgpr_count 14476++++++++++++++++++ 14477 14478Set to zero each time a 14479:ref:`amdgpu-amdhsa-assembler-directive-amdgpu_hsa_kernel` directive is 14480encountered. At each instruction, if the current value of this symbol is less 14481than or equal to the maximum VGPR number explicitly referenced within that 14482instruction then the symbol value is updated to equal that VGPR number plus 14483one. 14484 14485.kernel.sgpr_count 14486++++++++++++++++++ 14487 14488Set to zero each time a 14489:ref:`amdgpu-amdhsa-assembler-directive-amdgpu_hsa_kernel` directive is 14490encountered. At each instruction, if the current value of this symbol is less 14491than or equal to the maximum VGPR number explicitly referenced within that 14492instruction then the symbol value is updated to equal that SGPR number plus 14493one. 14494 14495.. _amdgpu-amdhsa-assembler-directives-v2: 14496 14497Code Object V2 Directives 14498~~~~~~~~~~~~~~~~~~~~~~~~~ 14499 14500.. warning:: 14501 Code object V2 is not the default code object version emitted by 14502 this version of LLVM. 14503 14504AMDGPU ABI defines auxiliary data in output code object. In assembly source, 14505one can specify them with assembler directives. 14506 14507.hsa_code_object_version major, minor 14508+++++++++++++++++++++++++++++++++++++ 14509 14510*major* and *minor* are integers that specify the version of the HSA code 14511object that will be generated by the assembler. 14512 14513.hsa_code_object_isa [major, minor, stepping, vendor, arch] 14514+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 14515 14516 14517*major*, *minor*, and *stepping* are all integers that describe the instruction 14518set architecture (ISA) version of the assembly program. 14519 14520*vendor* and *arch* are quoted strings. *vendor* should always be equal to 14521"AMD" and *arch* should always be equal to "AMDGPU". 14522 14523By default, the assembler will derive the ISA version, *vendor*, and *arch* 14524from the value of the -mcpu option that is passed to the assembler. 14525 14526.. _amdgpu-amdhsa-assembler-directive-amdgpu_hsa_kernel: 14527 14528.amdgpu_hsa_kernel (name) 14529+++++++++++++++++++++++++ 14530 14531This directives specifies that the symbol with given name is a kernel entry 14532point (label) and the object should contain corresponding symbol of type 14533STT_AMDGPU_HSA_KERNEL. 14534 14535.amd_kernel_code_t 14536++++++++++++++++++ 14537 14538This directive marks the beginning of a list of key / value pairs that are used 14539to specify the amd_kernel_code_t object that will be emitted by the assembler. 14540The list must be terminated by the *.end_amd_kernel_code_t* directive. For any 14541amd_kernel_code_t values that are unspecified a default value will be used. The 14542default value for all keys is 0, with the following exceptions: 14543 14544- *amd_code_version_major* defaults to 1. 14545- *amd_kernel_code_version_minor* defaults to 2. 14546- *amd_machine_kind* defaults to 1. 14547- *amd_machine_version_major*, *machine_version_minor*, and 14548 *amd_machine_version_stepping* are derived from the value of the -mcpu option 14549 that is passed to the assembler. 14550- *kernel_code_entry_byte_offset* defaults to 256. 14551- *wavefront_size* defaults 6 for all targets before GFX10. For GFX10 onwards 14552 defaults to 6 if target feature ``wavefrontsize64`` is enabled, otherwise 5. 14553 Note that wavefront size is specified as a power of two, so a value of **n** 14554 means a size of 2^ **n**. 14555- *call_convention* defaults to -1. 14556- *kernarg_segment_alignment*, *group_segment_alignment*, and 14557 *private_segment_alignment* default to 4. Note that alignments are specified 14558 as a power of 2, so a value of **n** means an alignment of 2^ **n**. 14559- *enable_tg_split* defaults to 1 if target feature ``tgsplit`` is enabled for 14560 GFX90A onwards. 14561- *enable_wgp_mode* defaults to 1 if target feature ``cumode`` is disabled for 14562 GFX10 onwards. 14563- *enable_mem_ordered* defaults to 1 for GFX10 onwards. 14564 14565The *.amd_kernel_code_t* directive must be placed immediately after the 14566function label and before any instructions. 14567 14568For a full list of amd_kernel_code_t keys, refer to AMDGPU ABI document, 14569comments in lib/Target/AMDGPU/AmdKernelCodeT.h and test/CodeGen/AMDGPU/hsa.s. 14570 14571.. _amdgpu-amdhsa-assembler-example-v2: 14572 14573Code Object V2 Example Source Code 14574~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 14575 14576.. warning:: 14577 Code Object V2 is not the default code object version emitted by 14578 this version of LLVM. 14579 14580Here is an example of a minimal assembly source file, defining one HSA kernel: 14581 14582.. code:: 14583 :number-lines: 14584 14585 .hsa_code_object_version 1,0 14586 .hsa_code_object_isa 14587 14588 .hsatext 14589 .globl hello_world 14590 .p2align 8 14591 .amdgpu_hsa_kernel hello_world 14592 14593 hello_world: 14594 14595 .amd_kernel_code_t 14596 enable_sgpr_kernarg_segment_ptr = 1 14597 is_ptr64 = 1 14598 compute_pgm_rsrc1_vgprs = 0 14599 compute_pgm_rsrc1_sgprs = 0 14600 compute_pgm_rsrc2_user_sgpr = 2 14601 compute_pgm_rsrc1_wgp_mode = 0 14602 compute_pgm_rsrc1_mem_ordered = 0 14603 compute_pgm_rsrc1_fwd_progress = 1 14604 .end_amd_kernel_code_t 14605 14606 s_load_dwordx2 s[0:1], s[0:1] 0x0 14607 v_mov_b32 v0, 3.14159 14608 s_waitcnt lgkmcnt(0) 14609 v_mov_b32 v1, s0 14610 v_mov_b32 v2, s1 14611 flat_store_dword v[1:2], v0 14612 s_endpgm 14613 .Lfunc_end0: 14614 .size hello_world, .Lfunc_end0-hello_world 14615 14616.. _amdgpu-amdhsa-assembler-predefined-symbols-v3-onwards: 14617 14618Code Object V3 and Above Predefined Symbols 14619~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 14620 14621The AMDGPU assembler defines and updates some symbols automatically. These 14622symbols do not affect code generation. 14623 14624.amdgcn.gfx_generation_number 14625+++++++++++++++++++++++++++++ 14626 14627Set to the GFX major generation number of the target being assembled for. For 14628example, when assembling for a "GFX9" target this will be set to the integer 14629value "9". The possible GFX major generation numbers are presented in 14630:ref:`amdgpu-processors`. 14631 14632.amdgcn.gfx_generation_minor 14633++++++++++++++++++++++++++++ 14634 14635Set to the GFX minor generation number of the target being assembled for. For 14636example, when assembling for a "GFX810" target this will be set to the integer 14637value "1". The possible GFX minor generation numbers are presented in 14638:ref:`amdgpu-processors`. 14639 14640.amdgcn.gfx_generation_stepping 14641+++++++++++++++++++++++++++++++ 14642 14643Set to the GFX stepping generation number of the target being assembled for. 14644For example, when assembling for a "GFX704" target this will be set to the 14645integer value "4". The possible GFX stepping generation numbers are presented 14646in :ref:`amdgpu-processors`. 14647 14648.. _amdgpu-amdhsa-assembler-symbol-next_free_vgpr: 14649 14650.amdgcn.next_free_vgpr 14651++++++++++++++++++++++ 14652 14653Set to zero before assembly begins. At each instruction, if the current value 14654of this symbol is less than or equal to the maximum VGPR number explicitly 14655referenced within that instruction then the symbol value is updated to equal 14656that VGPR number plus one. 14657 14658May be used to set the `.amdhsa_next_free_vgpr` directive in 14659:ref:`amdhsa-kernel-directives-table`. 14660 14661May be set at any time, e.g. manually set to zero at the start of each kernel. 14662 14663.. _amdgpu-amdhsa-assembler-symbol-next_free_sgpr: 14664 14665.amdgcn.next_free_sgpr 14666++++++++++++++++++++++ 14667 14668Set to zero before assembly begins. At each instruction, if the current value 14669of this symbol is less than or equal the maximum SGPR number explicitly 14670referenced within that instruction then the symbol value is updated to equal 14671that SGPR number plus one. 14672 14673May be used to set the `.amdhsa_next_free_spgr` directive in 14674:ref:`amdhsa-kernel-directives-table`. 14675 14676May be set at any time, e.g. manually set to zero at the start of each kernel. 14677 14678.. _amdgpu-amdhsa-assembler-directives-v3-onwards: 14679 14680Code Object V3 and Above Directives 14681~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 14682 14683Directives which begin with ``.amdgcn`` are valid for all ``amdgcn`` 14684architecture processors, and are not OS-specific. Directives which begin with 14685``.amdhsa`` are specific to ``amdgcn`` architecture processors when the 14686``amdhsa`` OS is specified. See :ref:`amdgpu-target-triples` and 14687:ref:`amdgpu-processors`. 14688 14689.. _amdgpu-assembler-directive-amdgcn-target: 14690 14691.amdgcn_target <target-triple> "-" <target-id> 14692++++++++++++++++++++++++++++++++++++++++++++++ 14693 14694Optional directive which declares the ``<target-triple>-<target-id>`` supported 14695by the containing assembler source file. Used by the assembler to validate 14696command-line options such as ``-triple``, ``-mcpu``, and 14697``--offload-arch=<target-id>``. A non-canonical target ID is allowed. See 14698:ref:`amdgpu-target-triples` and :ref:`amdgpu-target-id`. 14699 14700.. note:: 14701 14702 The target ID syntax used for code object V2 to V3 for this directive differs 14703 from that used elsewhere. See :ref:`amdgpu-target-id-v2-v3`. 14704 14705.amdhsa_kernel <name> 14706+++++++++++++++++++++ 14707 14708Creates a correctly aligned AMDHSA kernel descriptor and a symbol, 14709``<name>.kd``, in the current location of the current section. Only valid when 14710the OS is ``amdhsa``. ``<name>`` must be a symbol that labels the first 14711instruction to execute, and does not need to be previously defined. 14712 14713Marks the beginning of a list of directives used to generate the bytes of a 14714kernel descriptor, as described in :ref:`amdgpu-amdhsa-kernel-descriptor`. 14715Directives which may appear in this list are described in 14716:ref:`amdhsa-kernel-directives-table`. Directives may appear in any order, must 14717be valid for the target being assembled for, and cannot be repeated. Directives 14718support the range of values specified by the field they reference in 14719:ref:`amdgpu-amdhsa-kernel-descriptor`. If a directive is not specified, it is 14720assumed to have its default value, unless it is marked as "Required", in which 14721case it is an error to omit the directive. This list of directives is 14722terminated by an ``.end_amdhsa_kernel`` directive. 14723 14724 .. table:: AMDHSA Kernel Assembler Directives 14725 :name: amdhsa-kernel-directives-table 14726 14727 ======================================================== =================== ============ =================== 14728 Directive Default Supported On Description 14729 ======================================================== =================== ============ =================== 14730 ``.amdhsa_group_segment_fixed_size`` 0 GFX6-GFX10 Controls GROUP_SEGMENT_FIXED_SIZE in 14731 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 14732 ``.amdhsa_private_segment_fixed_size`` 0 GFX6-GFX10 Controls PRIVATE_SEGMENT_FIXED_SIZE in 14733 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 14734 ``.amdhsa_kernarg_size`` 0 GFX6-GFX10 Controls KERNARG_SIZE in 14735 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 14736 ``.amdhsa_user_sgpr_count`` 0 GFX6-GFX10 Controls USER_SGPR_COUNT in COMPUTE_PGM_RSRC2 14737 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table` 14738 ``.amdhsa_user_sgpr_private_segment_buffer`` 0 GFX6-GFX10 Controls ENABLE_SGPR_PRIVATE_SEGMENT_BUFFER in 14739 (except :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 14740 GFX940) 14741 ``.amdhsa_user_sgpr_dispatch_ptr`` 0 GFX6-GFX10 Controls ENABLE_SGPR_DISPATCH_PTR in 14742 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 14743 ``.amdhsa_user_sgpr_queue_ptr`` 0 GFX6-GFX10 Controls ENABLE_SGPR_QUEUE_PTR in 14744 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 14745 ``.amdhsa_user_sgpr_kernarg_segment_ptr`` 0 GFX6-GFX10 Controls ENABLE_SGPR_KERNARG_SEGMENT_PTR in 14746 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 14747 ``.amdhsa_user_sgpr_dispatch_id`` 0 GFX6-GFX10 Controls ENABLE_SGPR_DISPATCH_ID in 14748 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 14749 ``.amdhsa_user_sgpr_flat_scratch_init`` 0 GFX6-GFX10 Controls ENABLE_SGPR_FLAT_SCRATCH_INIT in 14750 (except :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 14751 GFX940) 14752 ``.amdhsa_user_sgpr_private_segment_size`` 0 GFX6-GFX10 Controls ENABLE_SGPR_PRIVATE_SEGMENT_SIZE in 14753 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 14754 ``.amdhsa_wavefront_size32`` Target GFX10 Controls ENABLE_WAVEFRONT_SIZE32 in 14755 Feature :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 14756 Specific 14757 (wavefrontsize64) 14758 ``.amdhsa_system_sgpr_private_segment_wavefront_offset`` 0 GFX6-GFX10 Controls ENABLE_PRIVATE_SEGMENT in 14759 (except :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 14760 GFX940) 14761 ``.amdhsa_enable_private_segment`` 0 GFX940 Controls ENABLE_PRIVATE_SEGMENT in 14762 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 14763 ``.amdhsa_system_sgpr_workgroup_id_x`` 1 GFX6-GFX10 Controls ENABLE_SGPR_WORKGROUP_ID_X in 14764 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 14765 ``.amdhsa_system_sgpr_workgroup_id_y`` 0 GFX6-GFX10 Controls ENABLE_SGPR_WORKGROUP_ID_Y in 14766 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 14767 ``.amdhsa_system_sgpr_workgroup_id_z`` 0 GFX6-GFX10 Controls ENABLE_SGPR_WORKGROUP_ID_Z in 14768 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 14769 ``.amdhsa_system_sgpr_workgroup_info`` 0 GFX6-GFX10 Controls ENABLE_SGPR_WORKGROUP_INFO in 14770 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 14771 ``.amdhsa_system_vgpr_workitem_id`` 0 GFX6-GFX10 Controls ENABLE_VGPR_WORKITEM_ID in 14772 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 14773 Possible values are defined in 14774 :ref:`amdgpu-amdhsa-system-vgpr-work-item-id-enumeration-values-table`. 14775 ``.amdhsa_next_free_vgpr`` Required GFX6-GFX10 Maximum VGPR number explicitly referenced, plus one. 14776 Used to calculate GRANULATED_WORKITEM_VGPR_COUNT in 14777 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 14778 ``.amdhsa_next_free_sgpr`` Required GFX6-GFX10 Maximum SGPR number explicitly referenced, plus one. 14779 Used to calculate GRANULATED_WAVEFRONT_SGPR_COUNT in 14780 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 14781 ``.amdhsa_accum_offset`` Required GFX90A, Offset of a first AccVGPR in the unified register file. 14782 GFX940 Used to calculate ACCUM_OFFSET in 14783 :ref:`amdgpu-amdhsa-compute_pgm_rsrc3-gfx90a-table`. 14784 ``.amdhsa_reserve_vcc`` 1 GFX6-GFX10 Whether the kernel may use the special VCC SGPR. 14785 Used to calculate GRANULATED_WAVEFRONT_SGPR_COUNT in 14786 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 14787 ``.amdhsa_reserve_flat_scratch`` 1 GFX7-GFX10 Whether the kernel may use flat instructions to access 14788 (except scratch memory. Used to calculate 14789 GFX940) GRANULATED_WAVEFRONT_SGPR_COUNT in 14790 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 14791 ``.amdhsa_reserve_xnack_mask`` Target GFX8-GFX10 Whether the kernel may trigger XNACK replay. 14792 Feature Used to calculate GRANULATED_WAVEFRONT_SGPR_COUNT in 14793 Specific :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 14794 (xnack) 14795 ``.amdhsa_float_round_mode_32`` 0 GFX6-GFX10 Controls FLOAT_ROUND_MODE_32 in 14796 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 14797 Possible values are defined in 14798 :ref:`amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table`. 14799 ``.amdhsa_float_round_mode_16_64`` 0 GFX6-GFX10 Controls FLOAT_ROUND_MODE_16_64 in 14800 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 14801 Possible values are defined in 14802 :ref:`amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table`. 14803 ``.amdhsa_float_denorm_mode_32`` 0 GFX6-GFX10 Controls FLOAT_DENORM_MODE_32 in 14804 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 14805 Possible values are defined in 14806 :ref:`amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table`. 14807 ``.amdhsa_float_denorm_mode_16_64`` 3 GFX6-GFX10 Controls FLOAT_DENORM_MODE_16_64 in 14808 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 14809 Possible values are defined in 14810 :ref:`amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table`. 14811 ``.amdhsa_dx10_clamp`` 1 GFX6-GFX10 Controls ENABLE_DX10_CLAMP in 14812 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 14813 ``.amdhsa_ieee_mode`` 1 GFX6-GFX10 Controls ENABLE_IEEE_MODE in 14814 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 14815 ``.amdhsa_fp16_overflow`` 0 GFX9-GFX10 Controls FP16_OVFL in 14816 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 14817 ``.amdhsa_tg_split`` Target GFX90A, Controls TG_SPLIT in 14818 Feature GFX940 :ref:`amdgpu-amdhsa-compute_pgm_rsrc3-gfx90a-table`. 14819 Specific 14820 (tgsplit) 14821 ``.amdhsa_workgroup_processor_mode`` Target GFX10 Controls ENABLE_WGP_MODE in 14822 Feature :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 14823 Specific 14824 (cumode) 14825 ``.amdhsa_memory_ordered`` 1 GFX10 Controls MEM_ORDERED in 14826 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 14827 ``.amdhsa_forward_progress`` 0 GFX10 Controls FWD_PROGRESS in 14828 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 14829 ``.amdhsa_shared_vgpr_count`` 0 GFX10 Controls SHARED_VGPR_COUNT in 14830 :ref:`amdgpu-amdhsa-compute_pgm_rsrc3-gfx10-table`. 14831 ``.amdhsa_exception_fp_ieee_invalid_op`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_INVALID_OPERATION in 14832 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 14833 ``.amdhsa_exception_fp_denorm_src`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_FP_DENORMAL_SOURCE in 14834 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 14835 ``.amdhsa_exception_fp_ieee_div_zero`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_DIVISION_BY_ZERO in 14836 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 14837 ``.amdhsa_exception_fp_ieee_overflow`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_OVERFLOW in 14838 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 14839 ``.amdhsa_exception_fp_ieee_underflow`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_UNDERFLOW in 14840 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 14841 ``.amdhsa_exception_fp_ieee_inexact`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_INEXACT in 14842 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 14843 ``.amdhsa_exception_int_div_zero`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_INT_DIVIDE_BY_ZERO in 14844 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 14845 ======================================================== =================== ============ =================== 14846 14847.amdgpu_metadata 14848++++++++++++++++ 14849 14850Optional directive which declares the contents of the ``NT_AMDGPU_METADATA`` 14851note record (see :ref:`amdgpu-elf-note-records-table-v3-onwards`). 14852 14853The contents must be in the [YAML]_ markup format, with the same structure and 14854semantics described in :ref:`amdgpu-amdhsa-code-object-metadata-v3`, 14855:ref:`amdgpu-amdhsa-code-object-metadata-v4` or 14856:ref:`amdgpu-amdhsa-code-object-metadata-v5`. 14857 14858This directive is terminated by an ``.end_amdgpu_metadata`` directive. 14859 14860.. _amdgpu-amdhsa-assembler-example-v3-onwards: 14861 14862Code Object V3 and Above Example Source Code 14863~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 14864 14865Here is an example of a minimal assembly source file, defining one HSA kernel: 14866 14867.. code:: 14868 :number-lines: 14869 14870 .amdgcn_target "amdgcn-amd-amdhsa--gfx900+xnack" // optional 14871 14872 .text 14873 .globl hello_world 14874 .p2align 8 14875 .type hello_world,@function 14876 hello_world: 14877 s_load_dwordx2 s[0:1], s[0:1] 0x0 14878 v_mov_b32 v0, 3.14159 14879 s_waitcnt lgkmcnt(0) 14880 v_mov_b32 v1, s0 14881 v_mov_b32 v2, s1 14882 flat_store_dword v[1:2], v0 14883 s_endpgm 14884 .Lfunc_end0: 14885 .size hello_world, .Lfunc_end0-hello_world 14886 14887 .rodata 14888 .p2align 6 14889 .amdhsa_kernel hello_world 14890 .amdhsa_user_sgpr_kernarg_segment_ptr 1 14891 .amdhsa_next_free_vgpr .amdgcn.next_free_vgpr 14892 .amdhsa_next_free_sgpr .amdgcn.next_free_sgpr 14893 .end_amdhsa_kernel 14894 14895 .amdgpu_metadata 14896 --- 14897 amdhsa.version: 14898 - 1 14899 - 0 14900 amdhsa.kernels: 14901 - .name: hello_world 14902 .symbol: hello_world.kd 14903 .kernarg_segment_size: 48 14904 .group_segment_fixed_size: 0 14905 .private_segment_fixed_size: 0 14906 .kernarg_segment_align: 4 14907 .wavefront_size: 64 14908 .sgpr_count: 2 14909 .vgpr_count: 3 14910 .max_flat_workgroup_size: 256 14911 .args: 14912 - .size: 8 14913 .offset: 0 14914 .value_kind: global_buffer 14915 .address_space: global 14916 .actual_access: write_only 14917 //... 14918 .end_amdgpu_metadata 14919 14920This kernel is equivalent to the following HIP program: 14921 14922.. code:: 14923 :number-lines: 14924 14925 __global__ void hello_world(float *p) { 14926 *p = 3.14159f; 14927 } 14928 14929If an assembly source file contains multiple kernels and/or functions, the 14930:ref:`amdgpu-amdhsa-assembler-symbol-next_free_vgpr` and 14931:ref:`amdgpu-amdhsa-assembler-symbol-next_free_sgpr` symbols may be reset using 14932the ``.set <symbol>, <expression>`` directive. For example, in the case of two 14933kernels, where ``function1`` is only called from ``kernel1`` it is sufficient 14934to group the function with the kernel that calls it and reset the symbols 14935between the two connected components: 14936 14937.. code:: 14938 :number-lines: 14939 14940 .amdgcn_target "amdgcn-amd-amdhsa--gfx900+xnack" // optional 14941 14942 // gpr tracking symbols are implicitly set to zero 14943 14944 .text 14945 .globl kern0 14946 .p2align 8 14947 .type kern0,@function 14948 kern0: 14949 // ... 14950 s_endpgm 14951 .Lkern0_end: 14952 .size kern0, .Lkern0_end-kern0 14953 14954 .rodata 14955 .p2align 6 14956 .amdhsa_kernel kern0 14957 // ... 14958 .amdhsa_next_free_vgpr .amdgcn.next_free_vgpr 14959 .amdhsa_next_free_sgpr .amdgcn.next_free_sgpr 14960 .end_amdhsa_kernel 14961 14962 // reset symbols to begin tracking usage in func1 and kern1 14963 .set .amdgcn.next_free_vgpr, 0 14964 .set .amdgcn.next_free_sgpr, 0 14965 14966 .text 14967 .hidden func1 14968 .global func1 14969 .p2align 2 14970 .type func1,@function 14971 func1: 14972 // ... 14973 s_setpc_b64 s[30:31] 14974 .Lfunc1_end: 14975 .size func1, .Lfunc1_end-func1 14976 14977 .globl kern1 14978 .p2align 8 14979 .type kern1,@function 14980 kern1: 14981 // ... 14982 s_getpc_b64 s[4:5] 14983 s_add_u32 s4, s4, func1@rel32@lo+4 14984 s_addc_u32 s5, s5, func1@rel32@lo+4 14985 s_swappc_b64 s[30:31], s[4:5] 14986 // ... 14987 s_endpgm 14988 .Lkern1_end: 14989 .size kern1, .Lkern1_end-kern1 14990 14991 .rodata 14992 .p2align 6 14993 .amdhsa_kernel kern1 14994 // ... 14995 .amdhsa_next_free_vgpr .amdgcn.next_free_vgpr 14996 .amdhsa_next_free_sgpr .amdgcn.next_free_sgpr 14997 .end_amdhsa_kernel 14998 14999These symbols cannot identify connected components in order to automatically 15000track the usage for each kernel. However, in some cases careful organization of 15001the kernels and functions in the source file means there is minimal additional 15002effort required to accurately calculate GPR usage. 15003 15004Additional Documentation 15005======================== 15006 15007.. [AMD-GCN-GFX6] `AMD Southern Islands Series ISA <http://developer.amd.com/wordpress/media/2012/12/AMD_Southern_Islands_Instruction_Set_Architecture.pdf>`__ 15008.. [AMD-GCN-GFX7] `AMD Sea Islands Series ISA <http://developer.amd.com/wordpress/media/2013/07/AMD_Sea_Islands_Instruction_Set_Architecture.pdf>`_ 15009.. [AMD-GCN-GFX8] `AMD GCN3 Instruction Set Architecture <http://amd-dev.wpengine.netdna-cdn.com/wordpress/media/2013/12/AMD_GCN3_Instruction_Set_Architecture_rev1.1.pdf>`__ 15010.. [AMD-GCN-GFX900-GFX904-VEGA] `AMD Vega Instruction Set Architecture <http://developer.amd.com/wordpress/media/2013/12/Vega_Shader_ISA_28July2017.pdf>`__ 15011.. [AMD-GCN-GFX906-VEGA7NM] `AMD Vega 7nm Instruction Set Architecture <https://gpuopen.com/wp-content/uploads/2019/11/Vega_7nm_Shader_ISA_26November2019.pdf>`__ 15012.. [AMD-GCN-GFX908-CDNA1] `AMD Instinct MI100 Instruction Set Architecture <https://developer.amd.com/wp-content/resources/CDNA1_Shader_ISA_14December2020.pdf>`__ 15013.. [AMD-GCN-GFX90A-CDNA2] `AMD Instinct MI200 Instruction Set Architecture <https://developer.amd.com/wp-content/resources/CDNA2_Shader_ISA_4February2022.pdf>`__ 15014.. [AMD-GCN-GFX10-RDNA1] `AMD RDNA 1.0 Instruction Set Architecture <https://gpuopen.com/wp-content/uploads/2019/08/RDNA_Shader_ISA_5August2019.pdf>`__ 15015.. [AMD-GCN-GFX10-RDNA2] `AMD RDNA 2 Instruction Set Architecture <https://developer.amd.com/wp-content/resources/RDNA2_Shader_ISA_November2020.pdf>`__ 15016.. [AMD-RADEON-HD-2000-3000] `AMD R6xx shader ISA <http://developer.amd.com/wordpress/media/2012/10/R600_Instruction_Set_Architecture.pdf>`__ 15017.. [AMD-RADEON-HD-4000] `AMD R7xx shader ISA <http://developer.amd.com/wordpress/media/2012/10/R700-Family_Instruction_Set_Architecture.pdf>`__ 15018.. [AMD-RADEON-HD-5000] `AMD Evergreen shader ISA <http://developer.amd.com/wordpress/media/2012/10/AMD_Evergreen-Family_Instruction_Set_Architecture.pdf>`__ 15019.. [AMD-RADEON-HD-6000] `AMD Cayman/Trinity shader ISA <http://developer.amd.com/wordpress/media/2012/10/AMD_HD_6900_Series_Instruction_Set_Architecture.pdf>`__ 15020.. [AMD-ROCm] `AMD ROCm™ Platform <https://rocmdocs.amd.com/>`__ 15021.. [AMD-ROCm-github] `AMD ROCm™ github <http://github.com/RadeonOpenCompute>`__ 15022.. [AMD-ROCm-Release-Notes] `AMD ROCm Release Notes <https://github.com/RadeonOpenCompute/ROCm>`__ 15023.. [CLANG-ATTR] `Attributes in Clang <https://clang.llvm.org/docs/AttributeReference.html>`__ 15024.. [DWARF] `DWARF Debugging Information Format <http://dwarfstd.org/>`__ 15025.. [ELF] `Executable and Linkable Format (ELF) <http://www.sco.com/developers/gabi/>`__ 15026.. [HRF] `Heterogeneous-race-free Memory Models <http://benedictgaster.org/wp-content/uploads/2014/01/asplos269-FINAL.pdf>`__ 15027.. [HSA] `Heterogeneous System Architecture (HSA) Foundation <http://www.hsafoundation.com/>`__ 15028.. [MsgPack] `Message Pack <http://www.msgpack.org/>`__ 15029.. [OpenCL] `The OpenCL Specification Version 2.0 <http://www.khronos.org/registry/cl/specs/opencl-2.0.pdf>`__ 15030.. [SEMVER] `Semantic Versioning <https://semver.org/>`__ 15031.. [YAML] `YAML Ain't Markup Language (YAML™) Version 1.2 <http://www.yaml.org/spec/1.2/spec.html>`__ 15032