1============================= 2User Guide for AMDGPU Backend 3============================= 4 5.. contents:: 6 :local: 7 8.. toctree:: 9 :hidden: 10 11 AMDGPU/AMDGPUAsmGFX7 12 AMDGPU/AMDGPUAsmGFX8 13 AMDGPU/AMDGPUAsmGFX9 14 AMDGPU/AMDGPUAsmGFX900 15 AMDGPU/AMDGPUAsmGFX904 16 AMDGPU/AMDGPUAsmGFX906 17 AMDGPU/AMDGPUAsmGFX908 18 AMDGPU/AMDGPUAsmGFX90a 19 AMDGPU/AMDGPUAsmGFX10 20 AMDGPU/AMDGPUAsmGFX1011 21 AMDGPU/AMDGPUAsmGFX1030 22 AMDGPUModifierSyntax 23 AMDGPUOperandSyntax 24 AMDGPUInstructionSyntax 25 AMDGPUInstructionNotation 26 AMDGPUDwarfExtensionsForHeterogeneousDebugging 27 AMDGPUDwarfExtensionAllowLocationDescriptionOnTheDwarfExpressionStack/AMDGPUDwarfExtensionAllowLocationDescriptionOnTheDwarfExpressionStack 28 29Introduction 30============ 31 32The AMDGPU backend provides ISA code generation for AMD GPUs, starting with the 33R600 family up until the current GCN families. It lives in the 34``llvm/lib/Target/AMDGPU`` directory. 35 36LLVM 37==== 38 39.. _amdgpu-target-triples: 40 41Target Triples 42-------------- 43 44Use the Clang option ``-target <Architecture>-<Vendor>-<OS>-<Environment>`` 45to specify the target triple: 46 47 .. table:: AMDGPU Architectures 48 :name: amdgpu-architecture-table 49 50 ============ ============================================================== 51 Architecture Description 52 ============ ============================================================== 53 ``r600`` AMD GPUs HD2XXX-HD6XXX for graphics and compute shaders. 54 ``amdgcn`` AMD GPUs GCN GFX6 onwards for graphics and compute shaders. 55 ============ ============================================================== 56 57 .. table:: AMDGPU Vendors 58 :name: amdgpu-vendor-table 59 60 ============ ============================================================== 61 Vendor Description 62 ============ ============================================================== 63 ``amd`` Can be used for all AMD GPU usage. 64 ``mesa3d`` Can be used if the OS is ``mesa3d``. 65 ============ ============================================================== 66 67 .. table:: AMDGPU Operating Systems 68 :name: amdgpu-os 69 70 ============== ============================================================ 71 OS Description 72 ============== ============================================================ 73 *<empty>* Defaults to the *unknown* OS. 74 ``amdhsa`` Compute kernels executed on HSA [HSA]_ compatible runtimes 75 such as: 76 77 - AMD's ROCm™ runtime [AMD-ROCm]_ using the *rocm-amdhsa* 78 loader on Linux. See *AMD ROCm Platform Release Notes* 79 [AMD-ROCm-Release-Notes]_ for supported hardware and 80 software. 81 - AMD's PAL runtime using the *pal-amdhsa* loader on 82 Windows. 83 84 ``amdpal`` Graphic shaders and compute kernels executed on AMD's PAL 85 runtime using the *pal-amdpal* loader on Windows and Linux 86 Pro. 87 ``mesa3d`` Graphic shaders and compute kernels executed on AMD's Mesa 88 3D runtime using the *mesa-mesa3d* loader on Linux. 89 ============== ============================================================ 90 91 .. table:: AMDGPU Environments 92 :name: amdgpu-environment-table 93 94 ============ ============================================================== 95 Environment Description 96 ============ ============================================================== 97 *<empty>* Default. 98 ============ ============================================================== 99 100.. _amdgpu-processors: 101 102Processors 103---------- 104 105Use the Clang options ``-mcpu=<target-id>`` or ``--offload-arch=<target-id>`` to 106specify the AMDGPU processor together with optional target features. See 107:ref:`amdgpu-target-id` and :ref:`amdgpu-target-features` for AMD GPU target 108specific information. 109 110Every processor supports every OS ABI (see :ref:`amdgpu-os`) with the following exceptions: 111 112* ``amdhsa`` is not supported in ``r600`` architecture (see :ref:`amdgpu-architecture-table`). 113 114 115 .. table:: AMDGPU Processors 116 :name: amdgpu-processor-table 117 118 =========== =============== ============ ===== ================= =============== =============== ====================== 119 Processor Alternative Target dGPU/ Target Target OS Support Example 120 Processor Triple APU Features Properties *(see* Products 121 Architecture Supported `amdgpu-os`_ 122 *and 123 corresponding 124 runtime release 125 notes for 126 current 127 information and 128 level of 129 support)* 130 =========== =============== ============ ===== ================= =============== =============== ====================== 131 **Radeon HD 2000/3000 Series (R600)** [AMD-RADEON-HD-2000-3000]_ 132 ----------------------------------------------------------------------------------------------------------------------- 133 ``r600`` ``r600`` dGPU - Does not 134 support 135 generic 136 address 137 space 138 ``r630`` ``r600`` dGPU - Does not 139 support 140 generic 141 address 142 space 143 ``rs880`` ``r600`` dGPU - Does not 144 support 145 generic 146 address 147 space 148 ``rv670`` ``r600`` dGPU - Does not 149 support 150 generic 151 address 152 space 153 **Radeon HD 4000 Series (R700)** [AMD-RADEON-HD-4000]_ 154 ----------------------------------------------------------------------------------------------------------------------- 155 ``rv710`` ``r600`` dGPU - Does not 156 support 157 generic 158 address 159 space 160 ``rv730`` ``r600`` dGPU - Does not 161 support 162 generic 163 address 164 space 165 ``rv770`` ``r600`` dGPU - Does not 166 support 167 generic 168 address 169 space 170 **Radeon HD 5000 Series (Evergreen)** [AMD-RADEON-HD-5000]_ 171 ----------------------------------------------------------------------------------------------------------------------- 172 ``cedar`` ``r600`` dGPU - Does not 173 support 174 generic 175 address 176 space 177 ``cypress`` ``r600`` dGPU - Does not 178 support 179 generic 180 address 181 space 182 ``juniper`` ``r600`` dGPU - Does not 183 support 184 generic 185 address 186 space 187 ``redwood`` ``r600`` dGPU - Does not 188 support 189 generic 190 address 191 space 192 ``sumo`` ``r600`` dGPU - Does not 193 support 194 generic 195 address 196 space 197 **Radeon HD 6000 Series (Northern Islands)** [AMD-RADEON-HD-6000]_ 198 ----------------------------------------------------------------------------------------------------------------------- 199 ``barts`` ``r600`` dGPU - Does not 200 support 201 generic 202 address 203 space 204 ``caicos`` ``r600`` dGPU - Does not 205 support 206 generic 207 address 208 space 209 ``cayman`` ``r600`` dGPU - Does not 210 support 211 generic 212 address 213 space 214 ``turks`` ``r600`` dGPU - Does not 215 support 216 generic 217 address 218 space 219 **GCN GFX6 (Southern Islands (SI))** [AMD-GCN-GFX6]_ 220 ----------------------------------------------------------------------------------------------------------------------- 221 ``gfx600`` - ``tahiti`` ``amdgcn`` dGPU - Does not - *pal-amdpal* 222 support 223 generic 224 address 225 space 226 ``gfx601`` - ``pitcairn`` ``amdgcn`` dGPU - Does not - *pal-amdpal* 227 - ``verde`` support 228 generic 229 address 230 space 231 ``gfx602`` - ``hainan`` ``amdgcn`` dGPU - Does not - *pal-amdpal* 232 - ``oland`` support 233 generic 234 address 235 space 236 **GCN GFX7 (Sea Islands (CI))** [AMD-GCN-GFX7]_ 237 ----------------------------------------------------------------------------------------------------------------------- 238 ``gfx700`` - ``kaveri`` ``amdgcn`` APU - Offset - *rocm-amdhsa* - A6-7000 239 flat - *pal-amdhsa* - A6 Pro-7050B 240 scratch - *pal-amdpal* - A8-7100 241 - A8 Pro-7150B 242 - A10-7300 243 - A10 Pro-7350B 244 - FX-7500 245 - A8-7200P 246 - A10-7400P 247 - FX-7600P 248 ``gfx701`` - ``hawaii`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - FirePro W8100 249 flat - *pal-amdhsa* - FirePro W9100 250 scratch - *pal-amdpal* - FirePro S9150 251 - FirePro S9170 252 ``gfx702`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - Radeon R9 290 253 flat - *pal-amdhsa* - Radeon R9 290x 254 scratch - *pal-amdpal* - Radeon R390 255 - Radeon R390x 256 ``gfx703`` - ``kabini`` ``amdgcn`` APU - Offset - *pal-amdhsa* - E1-2100 257 - ``mullins`` flat - *pal-amdpal* - E1-2200 258 scratch - E1-2500 259 - E2-3000 260 - E2-3800 261 - A4-5000 262 - A4-5100 263 - A6-5200 264 - A4 Pro-3340B 265 ``gfx704`` - ``bonaire`` ``amdgcn`` dGPU - Offset - *pal-amdhsa* - Radeon HD 7790 266 flat - *pal-amdpal* - Radeon HD 8770 267 scratch - R7 260 268 - R7 260X 269 ``gfx705`` ``amdgcn`` APU - Offset - *pal-amdhsa* *TBA* 270 flat - *pal-amdpal* 271 scratch .. TODO:: 272 273 Add product 274 names. 275 276 **GCN GFX8 (Volcanic Islands (VI))** [AMD-GCN-GFX8]_ 277 ----------------------------------------------------------------------------------------------------------------------- 278 ``gfx801`` - ``carrizo`` ``amdgcn`` APU - xnack - Offset - *rocm-amdhsa* - A6-8500P 279 flat - *pal-amdhsa* - Pro A6-8500B 280 scratch - *pal-amdpal* - A8-8600P 281 - Pro A8-8600B 282 - FX-8800P 283 - Pro A12-8800B 284 - A10-8700P 285 - Pro A10-8700B 286 - A10-8780P 287 - A10-9600P 288 - A10-9630P 289 - A12-9700P 290 - A12-9730P 291 - FX-9800P 292 - FX-9830P 293 - E2-9010 294 - A6-9210 295 - A9-9410 296 ``gfx802`` - ``iceland`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - Radeon R9 285 297 - ``tonga`` flat - *pal-amdhsa* - Radeon R9 380 298 scratch - *pal-amdpal* - Radeon R9 385 299 ``gfx803`` - ``fiji`` ``amdgcn`` dGPU - *rocm-amdhsa* - Radeon R9 Nano 300 - *pal-amdhsa* - Radeon R9 Fury 301 - *pal-amdpal* - Radeon R9 FuryX 302 - Radeon Pro Duo 303 - FirePro S9300x2 304 - Radeon Instinct MI8 305 \ - ``polaris10`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - Radeon RX 470 306 flat - *pal-amdhsa* - Radeon RX 480 307 scratch - *pal-amdpal* - Radeon Instinct MI6 308 \ - ``polaris11`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - Radeon RX 460 309 flat - *pal-amdhsa* 310 scratch - *pal-amdpal* 311 ``gfx805`` - ``tongapro`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - FirePro S7150 312 flat - *pal-amdhsa* - FirePro S7100 313 scratch - *pal-amdpal* - FirePro W7100 314 - Mobile FirePro 315 M7170 316 ``gfx810`` - ``stoney`` ``amdgcn`` APU - xnack - Offset - *rocm-amdhsa* *TBA* 317 flat - *pal-amdhsa* 318 scratch - *pal-amdpal* .. TODO:: 319 320 Add product 321 names. 322 323 **GCN GFX9 (Vega)** [AMD-GCN-GFX900-GFX904-VEGA]_ [AMD-GCN-GFX906-VEGA7NM]_ [AMD-GCN-GFX908-CDNA1]_ [AMD-GCN-GFX90A-CDNA2]_ 324 ----------------------------------------------------------------------------------------------------------------------- 325 ``gfx900`` ``amdgcn`` dGPU - xnack - Absolute - *rocm-amdhsa* - Radeon Vega 326 flat - *pal-amdhsa* Frontier Edition 327 scratch - *pal-amdpal* - Radeon RX Vega 56 328 - Radeon RX Vega 64 329 - Radeon RX Vega 64 330 Liquid 331 - Radeon Instinct MI25 332 ``gfx902`` ``amdgcn`` APU - xnack - Absolute - *rocm-amdhsa* - Ryzen 3 2200G 333 flat - *pal-amdhsa* - Ryzen 5 2400G 334 scratch - *pal-amdpal* 335 ``gfx904`` ``amdgcn`` dGPU - xnack - *rocm-amdhsa* *TBA* 336 - *pal-amdhsa* 337 - *pal-amdpal* .. TODO:: 338 339 Add product 340 names. 341 342 ``gfx906`` ``amdgcn`` dGPU - sramecc - Absolute - *rocm-amdhsa* - Radeon Instinct MI50 343 - xnack flat - *pal-amdhsa* - Radeon Instinct MI60 344 scratch - *pal-amdpal* - Radeon VII 345 - Radeon Pro VII 346 ``gfx908`` ``amdgcn`` dGPU - sramecc - *rocm-amdhsa* - AMD Instinct MI100 Accelerator 347 - xnack - Absolute 348 flat 349 scratch 350 ``gfx909`` ``amdgcn`` APU - xnack - Absolute - *pal-amdpal* *TBA* 351 flat 352 scratch .. TODO:: 353 354 Add product 355 names. 356 357 ``gfx90a`` ``amdgcn`` dGPU - sramecc - Absolute - *rocm-amdhsa* *TBA* 358 - tgsplit flat 359 - xnack scratch .. TODO:: 360 - Packed 361 work-item Add product 362 IDs names. 363 364 ``gfx90c`` ``amdgcn`` APU - xnack - Absolute - *pal-amdpal* - Ryzen 7 4700G 365 flat - Ryzen 7 4700GE 366 scratch - Ryzen 5 4600G 367 - Ryzen 5 4600GE 368 - Ryzen 3 4300G 369 - Ryzen 3 4300GE 370 - Ryzen Pro 4000G 371 - Ryzen 7 Pro 4700G 372 - Ryzen 7 Pro 4750GE 373 - Ryzen 5 Pro 4650G 374 - Ryzen 5 Pro 4650GE 375 - Ryzen 3 Pro 4350G 376 - Ryzen 3 Pro 4350GE 377 378 ``gfx940`` ``amdgcn`` dGPU - sramecc - Architected *TBA* 379 - tgsplit flat 380 - xnack scratch .. TODO:: 381 - Packed 382 work-item Add product 383 IDs names. 384 385 **GCN GFX10.1 (RDNA 1)** [AMD-GCN-GFX10-RDNA1]_ 386 ----------------------------------------------------------------------------------------------------------------------- 387 ``gfx1010`` ``amdgcn`` dGPU - cumode - Absolute - *rocm-amdhsa* - Radeon RX 5700 388 - wavefrontsize64 flat - *pal-amdhsa* - Radeon RX 5700 XT 389 - xnack scratch - *pal-amdpal* - Radeon Pro 5600 XT 390 - Radeon Pro 5600M 391 ``gfx1011`` ``amdgcn`` dGPU - cumode - *rocm-amdhsa* - Radeon Pro V520 392 - wavefrontsize64 - Absolute - *pal-amdhsa* 393 - xnack flat - *pal-amdpal* 394 scratch 395 ``gfx1012`` ``amdgcn`` dGPU - cumode - Absolute - *rocm-amdhsa* - Radeon RX 5500 396 - wavefrontsize64 flat - *pal-amdhsa* - Radeon RX 5500 XT 397 - xnack scratch - *pal-amdpal* 398 ``gfx1013`` ``amdgcn`` APU - cumode - Absolute - *rocm-amdhsa* *TBA* 399 - wavefrontsize64 flat - *pal-amdhsa* 400 - xnack scratch - *pal-amdpal* .. TODO:: 401 402 Add product 403 names. 404 405 **GCN GFX10.3 (RDNA 2)** [AMD-GCN-GFX10-RDNA2]_ 406 ----------------------------------------------------------------------------------------------------------------------- 407 ``gfx1030`` ``amdgcn`` dGPU - cumode - Absolute - *rocm-amdhsa* - Radeon RX 6800 408 - wavefrontsize64 flat - *pal-amdhsa* - Radeon RX 6800 XT 409 scratch - *pal-amdpal* - Radeon RX 6900 XT 410 ``gfx1031`` ``amdgcn`` dGPU - cumode - Absolute - *rocm-amdhsa* - Radeon RX 6700 XT 411 - wavefrontsize64 flat - *pal-amdhsa* 412 scratch - *pal-amdpal* 413 ``gfx1032`` ``amdgcn`` dGPU - cumode - Absolute - *rocm-amdhsa* *TBA* 414 - wavefrontsize64 flat - *pal-amdhsa* 415 scratch - *pal-amdpal* .. TODO:: 416 417 Add product 418 names. 419 420 ``gfx1033`` ``amdgcn`` APU - cumode - Absolute - *pal-amdpal* *TBA* 421 - wavefrontsize64 flat 422 scratch .. TODO:: 423 424 Add product 425 names. 426 ``gfx1034`` ``amdgcn`` dGPU - cumode - Absolute - *pal-amdpal* *TBA* 427 - wavefrontsize64 flat 428 scratch .. TODO:: 429 430 Add product 431 names. 432 433 ``gfx1035`` ``amdgcn`` APU - cumode - Absolute - *pal-amdpal* *TBA* 434 - wavefrontsize64 flat 435 scratch .. TODO:: 436 Add product 437 names. 438 439 ``gfx1036`` ``amdgcn`` APU - cumode - Absolute - *pal-amdpal* *TBA* 440 - wavefrontsize64 flat 441 scratch .. TODO:: 442 443 Add product 444 names. 445 446 =========== =============== ============ ===== ================= =============== =============== ====================== 447 448.. _amdgpu-target-features: 449 450Target Features 451--------------- 452 453Target features control how code is generated to support certain 454processor specific features. Not all target features are supported by 455all processors. The runtime must ensure that the features supported by 456the device used to execute the code match the features enabled when 457generating the code. A mismatch of features may result in incorrect 458execution, or a reduction in performance. 459 460The target features supported by each processor is listed in 461:ref:`amdgpu-processor-table`. 462 463Target features are controlled by exactly one of the following Clang 464options: 465 466``-mcpu=<target-id>`` or ``--offload-arch=<target-id>`` 467 468 The ``-mcpu`` and ``--offload-arch`` can specify the target feature as 469 optional components of the target ID. If omitted, the target feature has the 470 ``any`` value. See :ref:`amdgpu-target-id`. 471 472``-m[no-]<target-feature>`` 473 474 Target features not specified by the target ID are specified using a 475 separate option. These target features can have an ``on`` or ``off`` 476 value. ``on`` is specified by omitting the ``no-`` prefix, and 477 ``off`` is specified by including the ``no-`` prefix. The default 478 if not specified is ``off``. 479 480For example: 481 482``-mcpu=gfx908:xnack+`` 483 Enable the ``xnack`` feature. 484``-mcpu=gfx908:xnack-`` 485 Disable the ``xnack`` feature. 486``-mcumode`` 487 Enable the ``cumode`` feature. 488``-mno-cumode`` 489 Disable the ``cumode`` feature. 490 491 .. table:: AMDGPU Target Features 492 :name: amdgpu-target-features-table 493 494 =============== ============================ ================================================== 495 Target Feature Clang Option to Control Description 496 Name 497 =============== ============================ ================================================== 498 cumode - ``-m[no-]cumode`` Control the wavefront execution mode used 499 when generating code for kernels. When disabled 500 native WGP wavefront execution mode is used, 501 when enabled CU wavefront execution mode is used 502 (see :ref:`amdgpu-amdhsa-memory-model`). 503 504 sramecc - ``-mcpu`` If specified, generate code that can only be 505 - ``--offload-arch`` loaded and executed in a process that has a 506 matching setting for SRAMECC. 507 508 If not specified for code object V2 to V3, generate 509 code that can be loaded and executed in a process 510 with SRAMECC enabled. 511 512 If not specified for code object V4 or above, generate 513 code that can be loaded and executed in a process 514 with either setting of SRAMECC. 515 516 tgsplit ``-m[no-]tgsplit`` Enable/disable generating code that assumes 517 work-groups are launched in threadgroup split mode. 518 When enabled the waves of a work-group may be 519 launched in different CUs. 520 521 wavefrontsize64 - ``-m[no-]wavefrontsize64`` Control the wavefront size used when 522 generating code for kernels. When disabled 523 native wavefront size 32 is used, when enabled 524 wavefront size 64 is used. 525 526 xnack - ``-mcpu`` If specified, generate code that can only be 527 - ``--offload-arch`` loaded and executed in a process that has a 528 matching setting for XNACK replay. 529 530 If not specified for code object V2 to V3, generate 531 code that can be loaded and executed in a process 532 with XNACK replay enabled. 533 534 If not specified for code object V4 or above, generate 535 code that can be loaded and executed in a process 536 with either setting of XNACK replay. 537 538 XNACK replay can be used for demand paging and 539 page migration. If enabled in the device, then if 540 a page fault occurs the code may execute 541 incorrectly unless generated with XNACK replay 542 enabled, or generated for code object V4 or above without 543 specifying XNACK replay. Executing code that was 544 generated with XNACK replay enabled, or generated 545 for code object V4 or above without specifying XNACK replay, 546 on a device that does not have XNACK replay 547 enabled will execute correctly but may be less 548 performant than code generated for XNACK replay 549 disabled. 550 =============== ============================ ================================================== 551 552.. _amdgpu-target-id: 553 554Target ID 555--------- 556 557AMDGPU supports target IDs. See `Clang Offload Bundler 558<https://clang.llvm.org/docs/ClangOffloadBundler.html>`_ for a general 559description. The AMDGPU target specific information is: 560 561**processor** 562 Is an AMDGPU processor or alternative processor name specified in 563 :ref:`amdgpu-processor-table`. The non-canonical form target ID allows both 564 the primary processor and alternative processor names. The canonical form 565 target ID only allow the primary processor name. 566 567**target-feature** 568 Is a target feature name specified in :ref:`amdgpu-target-features-table` that 569 is supported by the processor. The target features supported by each processor 570 is specified in :ref:`amdgpu-processor-table`. Those that can be specified in 571 a target ID are marked as being controlled by ``-mcpu`` and 572 ``--offload-arch``. Each target feature must appear at most once in a target 573 ID. The non-canonical form target ID allows the target features to be 574 specified in any order. The canonical form target ID requires the target 575 features to be specified in alphabetic order. 576 577.. _amdgpu-target-id-v2-v3: 578 579Code Object V2 to V3 Target ID 580~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 581 582The target ID syntax for code object V2 to V3 is the same as defined in `Clang 583Offload Bundler <https://clang.llvm.org/docs/ClangOffloadBundler.html>`_ except 584when used in the :ref:`amdgpu-assembler-directive-amdgcn-target` assembler 585directive and the bundle entry ID. In those cases it has the following BNF 586syntax: 587 588.. code:: 589 590 <target-id> ::== <processor> ( "+" <target-feature> )* 591 592Where a target feature is omitted if *Off* and present if *On* or *Any*. 593 594.. note:: 595 596 The code object V2 to V3 cannot represent *Any* and treats it the same as 597 *On*. 598 599.. _amdgpu-embedding-bundled-objects: 600 601Embedding Bundled Code Objects 602------------------------------ 603 604AMDGPU supports the HIP and OpenMP languages that perform code object embedding 605as described in `Clang Offload Bundler 606<https://clang.llvm.org/docs/ClangOffloadBundler.html>`_. 607 608.. note:: 609 610 The target ID syntax used for code object V2 to V3 for a bundle entry ID 611 differs from that used elsewhere. See :ref:`amdgpu-target-id-v2-v3`. 612 613.. _amdgpu-address-spaces: 614 615Address Spaces 616-------------- 617 618The AMDGPU architecture supports a number of memory address spaces. The address 619space names use the OpenCL standard names, with some additions. 620 621The AMDGPU address spaces correspond to target architecture specific LLVM 622address space numbers used in LLVM IR. 623 624The AMDGPU address spaces are described in 625:ref:`amdgpu-address-spaces-table`. Only 64-bit process address spaces are 626supported for the ``amdgcn`` target. 627 628 .. table:: AMDGPU Address Spaces 629 :name: amdgpu-address-spaces-table 630 631 ================================= =============== =========== ================ ======= ============================ 632 .. 64-Bit Process Address Space 633 --------------------------------- --------------- ----------- ---------------- ------------------------------------ 634 Address Space Name LLVM IR Address HSA Segment Hardware Address NULL Value 635 Space Number Name Name Size 636 ================================= =============== =========== ================ ======= ============================ 637 Generic 0 flat flat 64 0x0000000000000000 638 Global 1 global global 64 0x0000000000000000 639 Region 2 N/A GDS 32 *not implemented for AMDHSA* 640 Local 3 group LDS 32 0xFFFFFFFF 641 Constant 4 constant *same as global* 64 0x0000000000000000 642 Private 5 private scratch 32 0xFFFFFFFF 643 Constant 32-bit 6 *TODO* 0x00000000 644 Buffer Fat Pointer (experimental) 7 *TODO* 645 ================================= =============== =========== ================ ======= ============================ 646 647**Generic** 648 The generic address space is supported unless the *Target Properties* column 649 of :ref:`amdgpu-processor-table` specifies *Does not support generic address 650 space*. 651 652 The generic address space uses the hardware flat address support for two fixed 653 ranges of virtual addresses (the private and local apertures), that are 654 outside the range of addressable global memory, to map from a flat address to 655 a private or local address. This uses FLAT instructions that can take a flat 656 address and access global, private (scratch), and group (LDS) memory depending 657 on if the address is within one of the aperture ranges. 658 659 Flat access to scratch requires hardware aperture setup and setup in the 660 kernel prologue (see :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`). Flat 661 access to LDS requires hardware aperture setup and M0 (GFX7-GFX8) register 662 setup (see :ref:`amdgpu-amdhsa-kernel-prolog-m0`). 663 664 To convert between a private or group address space address (termed a segment 665 address) and a flat address the base address of the corresponding aperture 666 can be used. For GFX7-GFX8 these are available in the 667 :ref:`amdgpu-amdhsa-hsa-aql-queue` the address of which can be obtained with 668 Queue Ptr SGPR (see :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). For 669 GFX9-GFX10 the aperture base addresses are directly available as inline 670 constant registers ``SRC_SHARED_BASE/LIMIT`` and ``SRC_PRIVATE_BASE/LIMIT``. 671 In 64-bit address mode the aperture sizes are 2^32 bytes and the base is 672 aligned to 2^32 which makes it easier to convert from flat to segment or 673 segment to flat. 674 675 A global address space address has the same value when used as a flat address 676 so no conversion is needed. 677 678**Global and Constant** 679 The global and constant address spaces both use global virtual addresses, 680 which are the same virtual address space used by the CPU. However, some 681 virtual addresses may only be accessible to the CPU, some only accessible 682 by the GPU, and some by both. 683 684 Using the constant address space indicates that the data will not change 685 during the execution of the kernel. This allows scalar read instructions to 686 be used. As the constant address space could only be modified on the host 687 side, a generic pointer loaded from the constant address space is safe to be 688 assumed as a global pointer since only the device global memory is visible 689 and managed on the host side. The vector and scalar L1 caches are invalidated 690 of volatile data before each kernel dispatch execution to allow constant 691 memory to change values between kernel dispatches. 692 693**Region** 694 The region address space uses the hardware Global Data Store (GDS). All 695 wavefronts executing on the same device will access the same memory for any 696 given region address. However, the same region address accessed by wavefronts 697 executing on different devices will access different memory. It is higher 698 performance than global memory. It is allocated by the runtime. The data 699 store (DS) instructions can be used to access it. 700 701**Local** 702 The local address space uses the hardware Local Data Store (LDS) which is 703 automatically allocated when the hardware creates the wavefronts of a 704 work-group, and freed when all the wavefronts of a work-group have 705 terminated. All wavefronts belonging to the same work-group will access the 706 same memory for any given local address. However, the same local address 707 accessed by wavefronts belonging to different work-groups will access 708 different memory. It is higher performance than global memory. The data store 709 (DS) instructions can be used to access it. 710 711**Private** 712 The private address space uses the hardware scratch memory support which 713 automatically allocates memory when it creates a wavefront and frees it when 714 a wavefronts terminates. The memory accessed by a lane of a wavefront for any 715 given private address will be different to the memory accessed by another lane 716 of the same or different wavefront for the same private address. 717 718 If a kernel dispatch uses scratch, then the hardware allocates memory from a 719 pool of backing memory allocated by the runtime for each wavefront. The lanes 720 of the wavefront access this using dword (4 byte) interleaving. The mapping 721 used from private address to backing memory address is: 722 723 ``wavefront-scratch-base + 724 ((private-address / 4) * wavefront-size * 4) + 725 (wavefront-lane-id * 4) + (private-address % 4)`` 726 727 If each lane of a wavefront accesses the same private address, the 728 interleaving results in adjacent dwords being accessed and hence requires 729 fewer cache lines to be fetched. 730 731 There are different ways that the wavefront scratch base address is 732 determined by a wavefront (see 733 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 734 735 Scratch memory can be accessed in an interleaved manner using buffer 736 instructions with the scratch buffer descriptor and per wavefront scratch 737 offset, by the scratch instructions, or by flat instructions. Multi-dword 738 access is not supported except by flat and scratch instructions in 739 GFX9-GFX10. 740 741**Constant 32-bit** 742 *TODO* 743 744**Buffer Fat Pointer** 745 The buffer fat pointer is an experimental address space that is currently 746 unsupported in the backend. It exposes a non-integral pointer that is in 747 the future intended to support the modelling of 128-bit buffer descriptors 748 plus a 32-bit offset into the buffer (in total encapsulating a 160-bit 749 *pointer*), allowing normal LLVM load/store/atomic operations to be used to 750 model the buffer descriptors used heavily in graphics workloads targeting 751 the backend. 752 753.. _amdgpu-memory-scopes: 754 755Memory Scopes 756------------- 757 758This section provides LLVM memory synchronization scopes supported by the AMDGPU 759backend memory model when the target triple OS is ``amdhsa`` (see 760:ref:`amdgpu-amdhsa-memory-model` and :ref:`amdgpu-target-triples`). 761 762The memory model supported is based on the HSA memory model [HSA]_ which is 763based in turn on HRF-indirect with scope inclusion [HRF]_. The happens-before 764relation is transitive over the synchronizes-with relation independent of scope 765and synchronizes-with allows the memory scope instances to be inclusive (see 766table :ref:`amdgpu-amdhsa-llvm-sync-scopes-table`). 767 768This is different to the OpenCL [OpenCL]_ memory model which does not have scope 769inclusion and requires the memory scopes to exactly match. However, this 770is conservatively correct for OpenCL. 771 772 .. table:: AMDHSA LLVM Sync Scopes 773 :name: amdgpu-amdhsa-llvm-sync-scopes-table 774 775 ======================= =================================================== 776 LLVM Sync Scope Description 777 ======================= =================================================== 778 *none* The default: ``system``. 779 780 Synchronizes with, and participates in modification 781 and seq_cst total orderings with, other operations 782 (except image operations) for all address spaces 783 (except private, or generic that accesses private) 784 provided the other operation's sync scope is: 785 786 - ``system``. 787 - ``agent`` and executed by a thread on the same 788 agent. 789 - ``workgroup`` and executed by a thread in the 790 same work-group. 791 - ``wavefront`` and executed by a thread in the 792 same wavefront. 793 794 ``agent`` Synchronizes with, and participates in modification 795 and seq_cst total orderings with, other operations 796 (except image operations) for all address spaces 797 (except private, or generic that accesses private) 798 provided the other operation's sync scope is: 799 800 - ``system`` or ``agent`` and executed by a thread 801 on the same agent. 802 - ``workgroup`` and executed by a thread in the 803 same work-group. 804 - ``wavefront`` and executed by a thread in the 805 same wavefront. 806 807 ``workgroup`` Synchronizes with, and participates in modification 808 and seq_cst total orderings with, other operations 809 (except image operations) for all address spaces 810 (except private, or generic that accesses private) 811 provided the other operation's sync scope is: 812 813 - ``system``, ``agent`` or ``workgroup`` and 814 executed by a thread in the same work-group. 815 - ``wavefront`` and executed by a thread in the 816 same wavefront. 817 818 ``wavefront`` Synchronizes with, and participates in modification 819 and seq_cst total orderings with, other operations 820 (except image operations) for all address spaces 821 (except private, or generic that accesses private) 822 provided the other operation's sync scope is: 823 824 - ``system``, ``agent``, ``workgroup`` or 825 ``wavefront`` and executed by a thread in the 826 same wavefront. 827 828 ``singlethread`` Only synchronizes with and participates in 829 modification and seq_cst total orderings with, 830 other operations (except image operations) running 831 in the same thread for all address spaces (for 832 example, in signal handlers). 833 834 ``one-as`` Same as ``system`` but only synchronizes with other 835 operations within the same address space. 836 837 ``agent-one-as`` Same as ``agent`` but only synchronizes with other 838 operations within the same address space. 839 840 ``workgroup-one-as`` Same as ``workgroup`` but only synchronizes with 841 other operations within the same address space. 842 843 ``wavefront-one-as`` Same as ``wavefront`` but only synchronizes with 844 other operations within the same address space. 845 846 ``singlethread-one-as`` Same as ``singlethread`` but only synchronizes with 847 other operations within the same address space. 848 ======================= =================================================== 849 850LLVM IR Intrinsics 851------------------ 852 853The AMDGPU backend implements the following LLVM IR intrinsics. 854 855*This section is WIP.* 856 857.. TODO:: 858 859 List AMDGPU intrinsics. 860 861LLVM IR Attributes 862------------------ 863 864The AMDGPU backend supports the following LLVM IR attributes. 865 866 .. table:: AMDGPU LLVM IR Attributes 867 :name: amdgpu-llvm-ir-attributes-table 868 869 ======================================= ========================================================== 870 LLVM Attribute Description 871 ======================================= ========================================================== 872 "amdgpu-flat-work-group-size"="min,max" Specify the minimum and maximum flat work group sizes that 873 will be specified when the kernel is dispatched. Generated 874 by the ``amdgpu_flat_work_group_size`` CLANG attribute [CLANG-ATTR]_. 875 The implied default value is 1,1024. 876 877 "amdgpu-implicitarg-num-bytes"="n" Number of kernel argument bytes to add to the kernel 878 argument block size for the implicit arguments. This 879 varies by OS and language (for OpenCL see 880 :ref:`opencl-kernel-implicit-arguments-appended-for-amdhsa-os-table`). 881 "amdgpu-num-sgpr"="n" Specifies the number of SGPRs to use. Generated by 882 the ``amdgpu_num_sgpr`` CLANG attribute [CLANG-ATTR]_. 883 "amdgpu-num-vgpr"="n" Specifies the number of VGPRs to use. Generated by the 884 ``amdgpu_num_vgpr`` CLANG attribute [CLANG-ATTR]_. 885 "amdgpu-waves-per-eu"="m,n" Specify the minimum and maximum number of waves per 886 execution unit. Generated by the ``amdgpu_waves_per_eu`` 887 CLANG attribute [CLANG-ATTR]_. This is an optimization hint, 888 and the backend may not be able to satisfy the request. If 889 the specified range is incompatible with the function's 890 "amdgpu-flat-work-group-size" value, the implied occupancy 891 bounds by the workgroup size takes precedence. 892 893 "amdgpu-ieee" true/false. Specify whether the function expects the IEEE field of the 894 mode register to be set on entry. Overrides the default for 895 the calling convention. 896 "amdgpu-dx10-clamp" true/false. Specify whether the function expects the DX10_CLAMP field of 897 the mode register to be set on entry. Overrides the default 898 for the calling convention. 899 900 "amdgpu-no-workitem-id-x" Indicates the function does not depend on the value of the 901 llvm.amdgcn.workitem.id.x intrinsic. If a function is marked with this 902 attribute, or reached through a call site marked with this attribute, 903 the value returned by the intrinsic is undefined. The backend can 904 generally infer this during code generation, so typically there is no 905 benefit to frontends marking functions with this. 906 907 "amdgpu-no-workitem-id-y" The same as amdgpu-no-workitem-id-x, except for the 908 llvm.amdgcn.workitem.id.y intrinsic. 909 910 "amdgpu-no-workitem-id-z" The same as amdgpu-no-workitem-id-x, except for the 911 llvm.amdgcn.workitem.id.z intrinsic. 912 913 "amdgpu-no-workgroup-id-x" The same as amdgpu-no-workitem-id-x, except for the 914 llvm.amdgcn.workgroup.id.x intrinsic. 915 916 "amdgpu-no-workgroup-id-y" The same as amdgpu-no-workitem-id-x, except for the 917 llvm.amdgcn.workgroup.id.y intrinsic. 918 919 "amdgpu-no-workgroup-id-z" The same as amdgpu-no-workitem-id-x, except for the 920 llvm.amdgcn.workgroup.id.z intrinsic. 921 922 "amdgpu-no-dispatch-ptr" The same as amdgpu-no-workitem-id-x, except for the 923 llvm.amdgcn.dispatch.ptr intrinsic. 924 925 "amdgpu-no-implicitarg-ptr" The same as amdgpu-no-workitem-id-x, except for the 926 llvm.amdgcn.implicitarg.ptr intrinsic. 927 928 "amdgpu-no-dispatch-id" The same as amdgpu-no-workitem-id-x, except for the 929 llvm.amdgcn.dispatch.id intrinsic. 930 931 "amdgpu-no-queue-ptr" Similar to amdgpu-no-workitem-id-x, except for the 932 llvm.amdgcn.queue.ptr intrinsic. Note that unlike the other ABI hint 933 attributes, the queue pointer may be required in situations where the 934 intrinsic call does not directly appear in the program. Some subtargets 935 require the queue pointer for to handle some addrspacecasts, as well 936 as the llvm.amdgcn.is.shared, llvm.amdgcn.is.private, llvm.trap, and 937 llvm.debug intrinsics. 938 939 "amdgpu-no-hostcall-ptr" Similar to amdgpu-no-implicitarg-ptr, except specific to the implicit 940 kernel argument that holds the pointer to the hostcall buffer. If this 941 attribute is absent, then the amdgpu-no-implicitarg-ptr is also removed. 942 943 "amdgpu-no-heap-ptr" Similar to amdgpu-no-implicitarg-ptr, except specific to the implicit 944 kernel argument that holds the pointer to an initialized memory buffer 945 that conforms to the requirements of the malloc/free device library V1 946 version implementation. If this attribute is absent, then the 947 amdgpu-no-implicitarg-ptr is also removed. 948 949 ======================================= ========================================================== 950 951.. _amdgpu-elf-code-object: 952 953ELF Code Object 954=============== 955 956The AMDGPU backend generates a standard ELF [ELF]_ relocatable code object that 957can be linked by ``lld`` to produce a standard ELF shared code object which can 958be loaded and executed on an AMDGPU target. 959 960.. _amdgpu-elf-header: 961 962Header 963------ 964 965The AMDGPU backend uses the following ELF header: 966 967 .. table:: AMDGPU ELF Header 968 :name: amdgpu-elf-header-table 969 970 ========================== =============================== 971 Field Value 972 ========================== =============================== 973 ``e_ident[EI_CLASS]`` ``ELFCLASS64`` 974 ``e_ident[EI_DATA]`` ``ELFDATA2LSB`` 975 ``e_ident[EI_OSABI]`` - ``ELFOSABI_NONE`` 976 - ``ELFOSABI_AMDGPU_HSA`` 977 - ``ELFOSABI_AMDGPU_PAL`` 978 - ``ELFOSABI_AMDGPU_MESA3D`` 979 ``e_ident[EI_ABIVERSION]`` - ``ELFABIVERSION_AMDGPU_HSA_V2`` 980 - ``ELFABIVERSION_AMDGPU_HSA_V3`` 981 - ``ELFABIVERSION_AMDGPU_HSA_V4`` 982 - ``ELFABIVERSION_AMDGPU_HSA_V5`` 983 - ``ELFABIVERSION_AMDGPU_PAL`` 984 - ``ELFABIVERSION_AMDGPU_MESA3D`` 985 ``e_type`` - ``ET_REL`` 986 - ``ET_DYN`` 987 ``e_machine`` ``EM_AMDGPU`` 988 ``e_entry`` 0 989 ``e_flags`` See :ref:`amdgpu-elf-header-e_flags-v2-table`, 990 :ref:`amdgpu-elf-header-e_flags-table-v3`, 991 and :ref:`amdgpu-elf-header-e_flags-table-v4-onwards` 992 ========================== =============================== 993 994.. 995 996 .. table:: AMDGPU ELF Header Enumeration Values 997 :name: amdgpu-elf-header-enumeration-values-table 998 999 =============================== ===== 1000 Name Value 1001 =============================== ===== 1002 ``EM_AMDGPU`` 224 1003 ``ELFOSABI_NONE`` 0 1004 ``ELFOSABI_AMDGPU_HSA`` 64 1005 ``ELFOSABI_AMDGPU_PAL`` 65 1006 ``ELFOSABI_AMDGPU_MESA3D`` 66 1007 ``ELFABIVERSION_AMDGPU_HSA_V2`` 0 1008 ``ELFABIVERSION_AMDGPU_HSA_V3`` 1 1009 ``ELFABIVERSION_AMDGPU_HSA_V4`` 2 1010 ``ELFABIVERSION_AMDGPU_HSA_V5`` 3 1011 ``ELFABIVERSION_AMDGPU_PAL`` 0 1012 ``ELFABIVERSION_AMDGPU_MESA3D`` 0 1013 =============================== ===== 1014 1015``e_ident[EI_CLASS]`` 1016 The ELF class is: 1017 1018 * ``ELFCLASS32`` for ``r600`` architecture. 1019 1020 * ``ELFCLASS64`` for ``amdgcn`` architecture which only supports 64-bit 1021 process address space applications. 1022 1023``e_ident[EI_DATA]`` 1024 All AMDGPU targets use ``ELFDATA2LSB`` for little-endian byte ordering. 1025 1026``e_ident[EI_OSABI]`` 1027 One of the following AMDGPU target architecture specific OS ABIs 1028 (see :ref:`amdgpu-os`): 1029 1030 * ``ELFOSABI_NONE`` for *unknown* OS. 1031 1032 * ``ELFOSABI_AMDGPU_HSA`` for ``amdhsa`` OS. 1033 1034 * ``ELFOSABI_AMDGPU_PAL`` for ``amdpal`` OS. 1035 1036 * ``ELFOSABI_AMDGPU_MESA3D`` for ``mesa3D`` OS. 1037 1038``e_ident[EI_ABIVERSION]`` 1039 The ABI version of the AMDGPU target architecture specific OS ABI to which the code 1040 object conforms: 1041 1042 * ``ELFABIVERSION_AMDGPU_HSA_V2`` is used to specify the version of AMD HSA 1043 runtime ABI for code object V2. Specify using the Clang option 1044 ``-mcode-object-version=2``. 1045 1046 * ``ELFABIVERSION_AMDGPU_HSA_V3`` is used to specify the version of AMD HSA 1047 runtime ABI for code object V3. Specify using the Clang option 1048 ``-mcode-object-version=3``. 1049 1050 * ``ELFABIVERSION_AMDGPU_HSA_V4`` is used to specify the version of AMD HSA 1051 runtime ABI for code object V4. Specify using the Clang option 1052 ``-mcode-object-version=4``. This is the default code object 1053 version if not specified. 1054 1055 * ``ELFABIVERSION_AMDGPU_HSA_V5`` is used to specify the version of AMD HSA 1056 runtime ABI for code object V5. Specify using the Clang option 1057 ``-mcode-object-version=5``. 1058 1059 * ``ELFABIVERSION_AMDGPU_PAL`` is used to specify the version of AMD PAL 1060 runtime ABI. 1061 1062 * ``ELFABIVERSION_AMDGPU_MESA3D`` is used to specify the version of AMD MESA 1063 3D runtime ABI. 1064 1065``e_type`` 1066 Can be one of the following values: 1067 1068 1069 ``ET_REL`` 1070 The type produced by the AMDGPU backend compiler as it is relocatable code 1071 object. 1072 1073 ``ET_DYN`` 1074 The type produced by the linker as it is a shared code object. 1075 1076 The AMD HSA runtime loader requires a ``ET_DYN`` code object. 1077 1078``e_machine`` 1079 The value ``EM_AMDGPU`` is used for the machine for all processors supported 1080 by the ``r600`` and ``amdgcn`` architectures (see 1081 :ref:`amdgpu-processor-table`). The specific processor is specified in the 1082 ``NT_AMD_HSA_ISA_VERSION`` note record for code object V2 (see 1083 :ref:`amdgpu-note-records-v2`) and in the ``EF_AMDGPU_MACH`` bit field of the 1084 ``e_flags`` for code object V3 and above (see 1085 :ref:`amdgpu-elf-header-e_flags-table-v3` and 1086 :ref:`amdgpu-elf-header-e_flags-table-v4-onwards`). 1087 1088``e_entry`` 1089 The entry point is 0 as the entry points for individual kernels must be 1090 selected in order to invoke them through AQL packets. 1091 1092``e_flags`` 1093 The AMDGPU backend uses the following ELF header flags: 1094 1095 .. table:: AMDGPU ELF Header ``e_flags`` for Code Object V2 1096 :name: amdgpu-elf-header-e_flags-v2-table 1097 1098 ===================================== ===== ============================= 1099 Name Value Description 1100 ===================================== ===== ============================= 1101 ``EF_AMDGPU_FEATURE_XNACK_V2`` 0x01 Indicates if the ``xnack`` 1102 target feature is 1103 enabled for all code 1104 contained in the code object. 1105 If the processor 1106 does not support the 1107 ``xnack`` target 1108 feature then must 1109 be 0. 1110 See 1111 :ref:`amdgpu-target-features`. 1112 ``EF_AMDGPU_FEATURE_TRAP_HANDLER_V2`` 0x02 Indicates if the trap 1113 handler is enabled for all 1114 code contained in the code 1115 object. If the processor 1116 does not support a trap 1117 handler then must be 0. 1118 See 1119 :ref:`amdgpu-target-features`. 1120 ===================================== ===== ============================= 1121 1122 .. table:: AMDGPU ELF Header ``e_flags`` for Code Object V3 1123 :name: amdgpu-elf-header-e_flags-table-v3 1124 1125 ================================= ===== ============================= 1126 Name Value Description 1127 ================================= ===== ============================= 1128 ``EF_AMDGPU_MACH`` 0x0ff AMDGPU processor selection 1129 mask for 1130 ``EF_AMDGPU_MACH_xxx`` values 1131 defined in 1132 :ref:`amdgpu-ef-amdgpu-mach-table`. 1133 ``EF_AMDGPU_FEATURE_XNACK_V3`` 0x100 Indicates if the ``xnack`` 1134 target feature is 1135 enabled for all code 1136 contained in the code object. 1137 If the processor 1138 does not support the 1139 ``xnack`` target 1140 feature then must 1141 be 0. 1142 See 1143 :ref:`amdgpu-target-features`. 1144 ``EF_AMDGPU_FEATURE_SRAMECC_V3`` 0x200 Indicates if the ``sramecc`` 1145 target feature is 1146 enabled for all code 1147 contained in the code object. 1148 If the processor 1149 does not support the 1150 ``sramecc`` target 1151 feature then must 1152 be 0. 1153 See 1154 :ref:`amdgpu-target-features`. 1155 ================================= ===== ============================= 1156 1157 .. table:: AMDGPU ELF Header ``e_flags`` for Code Object V4 and After 1158 :name: amdgpu-elf-header-e_flags-table-v4-onwards 1159 1160 ============================================ ===== =================================== 1161 Name Value Description 1162 ============================================ ===== =================================== 1163 ``EF_AMDGPU_MACH`` 0x0ff AMDGPU processor selection 1164 mask for 1165 ``EF_AMDGPU_MACH_xxx`` values 1166 defined in 1167 :ref:`amdgpu-ef-amdgpu-mach-table`. 1168 ``EF_AMDGPU_FEATURE_XNACK_V4`` 0x300 XNACK selection mask for 1169 ``EF_AMDGPU_FEATURE_XNACK_*_V4`` 1170 values. 1171 ``EF_AMDGPU_FEATURE_XNACK_UNSUPPORTED_V4`` 0x000 XNACK unsuppored. 1172 ``EF_AMDGPU_FEATURE_XNACK_ANY_V4`` 0x100 XNACK can have any value. 1173 ``EF_AMDGPU_FEATURE_XNACK_OFF_V4`` 0x200 XNACK disabled. 1174 ``EF_AMDGPU_FEATURE_XNACK_ON_V4`` 0x300 XNACK enabled. 1175 ``EF_AMDGPU_FEATURE_SRAMECC_V4`` 0xc00 SRAMECC selection mask for 1176 ``EF_AMDGPU_FEATURE_SRAMECC_*_V4`` 1177 values. 1178 ``EF_AMDGPU_FEATURE_SRAMECC_UNSUPPORTED_V4`` 0x000 SRAMECC unsuppored. 1179 ``EF_AMDGPU_FEATURE_SRAMECC_ANY_V4`` 0x400 SRAMECC can have any value. 1180 ``EF_AMDGPU_FEATURE_SRAMECC_OFF_V4`` 0x800 SRAMECC disabled, 1181 ``EF_AMDGPU_FEATURE_SRAMECC_ON_V4`` 0xc00 SRAMECC enabled. 1182 ============================================ ===== =================================== 1183 1184 .. table:: AMDGPU ``EF_AMDGPU_MACH`` Values 1185 :name: amdgpu-ef-amdgpu-mach-table 1186 1187 ==================================== ========== ============================= 1188 Name Value Description (see 1189 :ref:`amdgpu-processor-table`) 1190 ==================================== ========== ============================= 1191 ``EF_AMDGPU_MACH_NONE`` 0x000 *not specified* 1192 ``EF_AMDGPU_MACH_R600_R600`` 0x001 ``r600`` 1193 ``EF_AMDGPU_MACH_R600_R630`` 0x002 ``r630`` 1194 ``EF_AMDGPU_MACH_R600_RS880`` 0x003 ``rs880`` 1195 ``EF_AMDGPU_MACH_R600_RV670`` 0x004 ``rv670`` 1196 ``EF_AMDGPU_MACH_R600_RV710`` 0x005 ``rv710`` 1197 ``EF_AMDGPU_MACH_R600_RV730`` 0x006 ``rv730`` 1198 ``EF_AMDGPU_MACH_R600_RV770`` 0x007 ``rv770`` 1199 ``EF_AMDGPU_MACH_R600_CEDAR`` 0x008 ``cedar`` 1200 ``EF_AMDGPU_MACH_R600_CYPRESS`` 0x009 ``cypress`` 1201 ``EF_AMDGPU_MACH_R600_JUNIPER`` 0x00a ``juniper`` 1202 ``EF_AMDGPU_MACH_R600_REDWOOD`` 0x00b ``redwood`` 1203 ``EF_AMDGPU_MACH_R600_SUMO`` 0x00c ``sumo`` 1204 ``EF_AMDGPU_MACH_R600_BARTS`` 0x00d ``barts`` 1205 ``EF_AMDGPU_MACH_R600_CAICOS`` 0x00e ``caicos`` 1206 ``EF_AMDGPU_MACH_R600_CAYMAN`` 0x00f ``cayman`` 1207 ``EF_AMDGPU_MACH_R600_TURKS`` 0x010 ``turks`` 1208 *reserved* 0x011 - Reserved for ``r600`` 1209 0x01f architecture processors. 1210 ``EF_AMDGPU_MACH_AMDGCN_GFX600`` 0x020 ``gfx600`` 1211 ``EF_AMDGPU_MACH_AMDGCN_GFX601`` 0x021 ``gfx601`` 1212 ``EF_AMDGPU_MACH_AMDGCN_GFX700`` 0x022 ``gfx700`` 1213 ``EF_AMDGPU_MACH_AMDGCN_GFX701`` 0x023 ``gfx701`` 1214 ``EF_AMDGPU_MACH_AMDGCN_GFX702`` 0x024 ``gfx702`` 1215 ``EF_AMDGPU_MACH_AMDGCN_GFX703`` 0x025 ``gfx703`` 1216 ``EF_AMDGPU_MACH_AMDGCN_GFX704`` 0x026 ``gfx704`` 1217 *reserved* 0x027 Reserved. 1218 ``EF_AMDGPU_MACH_AMDGCN_GFX801`` 0x028 ``gfx801`` 1219 ``EF_AMDGPU_MACH_AMDGCN_GFX802`` 0x029 ``gfx802`` 1220 ``EF_AMDGPU_MACH_AMDGCN_GFX803`` 0x02a ``gfx803`` 1221 ``EF_AMDGPU_MACH_AMDGCN_GFX810`` 0x02b ``gfx810`` 1222 ``EF_AMDGPU_MACH_AMDGCN_GFX900`` 0x02c ``gfx900`` 1223 ``EF_AMDGPU_MACH_AMDGCN_GFX902`` 0x02d ``gfx902`` 1224 ``EF_AMDGPU_MACH_AMDGCN_GFX904`` 0x02e ``gfx904`` 1225 ``EF_AMDGPU_MACH_AMDGCN_GFX906`` 0x02f ``gfx906`` 1226 ``EF_AMDGPU_MACH_AMDGCN_GFX908`` 0x030 ``gfx908`` 1227 ``EF_AMDGPU_MACH_AMDGCN_GFX909`` 0x031 ``gfx909`` 1228 ``EF_AMDGPU_MACH_AMDGCN_GFX90C`` 0x032 ``gfx90c`` 1229 ``EF_AMDGPU_MACH_AMDGCN_GFX1010`` 0x033 ``gfx1010`` 1230 ``EF_AMDGPU_MACH_AMDGCN_GFX1011`` 0x034 ``gfx1011`` 1231 ``EF_AMDGPU_MACH_AMDGCN_GFX1012`` 0x035 ``gfx1012`` 1232 ``EF_AMDGPU_MACH_AMDGCN_GFX1030`` 0x036 ``gfx1030`` 1233 ``EF_AMDGPU_MACH_AMDGCN_GFX1031`` 0x037 ``gfx1031`` 1234 ``EF_AMDGPU_MACH_AMDGCN_GFX1032`` 0x038 ``gfx1032`` 1235 ``EF_AMDGPU_MACH_AMDGCN_GFX1033`` 0x039 ``gfx1033`` 1236 ``EF_AMDGPU_MACH_AMDGCN_GFX602`` 0x03a ``gfx602`` 1237 ``EF_AMDGPU_MACH_AMDGCN_GFX705`` 0x03b ``gfx705`` 1238 ``EF_AMDGPU_MACH_AMDGCN_GFX805`` 0x03c ``gfx805`` 1239 ``EF_AMDGPU_MACH_AMDGCN_GFX1035`` 0x03d ``gfx1035`` 1240 ``EF_AMDGPU_MACH_AMDGCN_GFX1034`` 0x03e ``gfx1034`` 1241 ``EF_AMDGPU_MACH_AMDGCN_GFX90A`` 0x03f ``gfx90a`` 1242 ``EF_AMDGPU_MACH_AMDGCN_GFX940`` 0x040 ``gfx940`` 1243 *reserved* 0x041 Reserved. 1244 ``EF_AMDGPU_MACH_AMDGCN_GFX1013`` 0x042 ``gfx1013`` 1245 *reserved* 0x043 Reserved. 1246 *reserved* 0x044 Reserved. 1247 ``EF_AMDGPU_MACH_AMDGCN_GFX1036`` 0x045 ``gfx1036`` 1248 ==================================== ========== ============================= 1249 1250Sections 1251-------- 1252 1253An AMDGPU target ELF code object has the standard ELF sections which include: 1254 1255 .. table:: AMDGPU ELF Sections 1256 :name: amdgpu-elf-sections-table 1257 1258 ================== ================ ================================= 1259 Name Type Attributes 1260 ================== ================ ================================= 1261 ``.bss`` ``SHT_NOBITS`` ``SHF_ALLOC`` + ``SHF_WRITE`` 1262 ``.data`` ``SHT_PROGBITS`` ``SHF_ALLOC`` + ``SHF_WRITE`` 1263 ``.debug_``\ *\** ``SHT_PROGBITS`` *none* 1264 ``.dynamic`` ``SHT_DYNAMIC`` ``SHF_ALLOC`` 1265 ``.dynstr`` ``SHT_PROGBITS`` ``SHF_ALLOC`` 1266 ``.dynsym`` ``SHT_PROGBITS`` ``SHF_ALLOC`` 1267 ``.got`` ``SHT_PROGBITS`` ``SHF_ALLOC`` + ``SHF_WRITE`` 1268 ``.hash`` ``SHT_HASH`` ``SHF_ALLOC`` 1269 ``.note`` ``SHT_NOTE`` *none* 1270 ``.rela``\ *name* ``SHT_RELA`` *none* 1271 ``.rela.dyn`` ``SHT_RELA`` *none* 1272 ``.rodata`` ``SHT_PROGBITS`` ``SHF_ALLOC`` 1273 ``.shstrtab`` ``SHT_STRTAB`` *none* 1274 ``.strtab`` ``SHT_STRTAB`` *none* 1275 ``.symtab`` ``SHT_SYMTAB`` *none* 1276 ``.text`` ``SHT_PROGBITS`` ``SHF_ALLOC`` + ``SHF_EXECINSTR`` 1277 ================== ================ ================================= 1278 1279These sections have their standard meanings (see [ELF]_) and are only generated 1280if needed. 1281 1282``.debug``\ *\** 1283 The standard DWARF sections. See :ref:`amdgpu-dwarf-debug-information` for 1284 information on the DWARF produced by the AMDGPU backend. 1285 1286``.dynamic``, ``.dynstr``, ``.dynsym``, ``.hash`` 1287 The standard sections used by a dynamic loader. 1288 1289``.note`` 1290 See :ref:`amdgpu-note-records` for the note records supported by the AMDGPU 1291 backend. 1292 1293``.rela``\ *name*, ``.rela.dyn`` 1294 For relocatable code objects, *name* is the name of the section that the 1295 relocation records apply. For example, ``.rela.text`` is the section name for 1296 relocation records associated with the ``.text`` section. 1297 1298 For linked shared code objects, ``.rela.dyn`` contains all the relocation 1299 records from each of the relocatable code object's ``.rela``\ *name* sections. 1300 1301 See :ref:`amdgpu-relocation-records` for the relocation records supported by 1302 the AMDGPU backend. 1303 1304``.text`` 1305 The executable machine code for the kernels and functions they call. Generated 1306 as position independent code. See :ref:`amdgpu-code-conventions` for 1307 information on conventions used in the isa generation. 1308 1309.. _amdgpu-note-records: 1310 1311Note Records 1312------------ 1313 1314The AMDGPU backend code object contains ELF note records in the ``.note`` 1315section. The set of generated notes and their semantics depend on the code 1316object version; see :ref:`amdgpu-note-records-v2` and 1317:ref:`amdgpu-note-records-v3-onwards`. 1318 1319As required by ``ELFCLASS32`` and ``ELFCLASS64``, minimal zero-byte padding 1320must be generated after the ``name`` field to ensure the ``desc`` field is 4 1321byte aligned. In addition, minimal zero-byte padding must be generated to 1322ensure the ``desc`` field size is a multiple of 4 bytes. The ``sh_addralign`` 1323field of the ``.note`` section must be at least 4 to indicate at least 8 byte 1324alignment. 1325 1326.. _amdgpu-note-records-v2: 1327 1328Code Object V2 Note Records 1329~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1330 1331.. warning:: 1332 Code object V2 is not the default code object version emitted by 1333 this version of LLVM. 1334 1335The AMDGPU backend code object uses the following ELF note record in the 1336``.note`` section when compiling for code object V2. 1337 1338The note record vendor field is "AMD". 1339 1340Additional note records may be present, but any which are not documented here 1341are deprecated and should not be used. 1342 1343 .. table:: AMDGPU Code Object V2 ELF Note Records 1344 :name: amdgpu-elf-note-records-v2-table 1345 1346 ===== ===================================== ====================================== 1347 Name Type Description 1348 ===== ===================================== ====================================== 1349 "AMD" ``NT_AMD_HSA_CODE_OBJECT_VERSION`` Code object version. 1350 "AMD" ``NT_AMD_HSA_HSAIL`` HSAIL properties generated by the HSAIL 1351 Finalizer and not the LLVM compiler. 1352 "AMD" ``NT_AMD_HSA_ISA_VERSION`` Target ISA version. 1353 "AMD" ``NT_AMD_HSA_METADATA`` Metadata null terminated string in 1354 YAML [YAML]_ textual format. 1355 "AMD" ``NT_AMD_HSA_ISA_NAME`` Target ISA name. 1356 ===== ===================================== ====================================== 1357 1358.. 1359 1360 .. table:: AMDGPU Code Object V2 ELF Note Record Enumeration Values 1361 :name: amdgpu-elf-note-record-enumeration-values-v2-table 1362 1363 ===================================== ===== 1364 Name Value 1365 ===================================== ===== 1366 ``NT_AMD_HSA_CODE_OBJECT_VERSION`` 1 1367 ``NT_AMD_HSA_HSAIL`` 2 1368 ``NT_AMD_HSA_ISA_VERSION`` 3 1369 *reserved* 4-9 1370 ``NT_AMD_HSA_METADATA`` 10 1371 ``NT_AMD_HSA_ISA_NAME`` 11 1372 ===================================== ===== 1373 1374``NT_AMD_HSA_CODE_OBJECT_VERSION`` 1375 Specifies the code object version number. The description field has the 1376 following layout: 1377 1378 .. code:: c 1379 1380 struct amdgpu_hsa_note_code_object_version_s { 1381 uint32_t major_version; 1382 uint32_t minor_version; 1383 }; 1384 1385 The ``major_version`` has a value less than or equal to 2. 1386 1387``NT_AMD_HSA_HSAIL`` 1388 Specifies the HSAIL properties used by the HSAIL Finalizer. The description 1389 field has the following layout: 1390 1391 .. code:: c 1392 1393 struct amdgpu_hsa_note_hsail_s { 1394 uint32_t hsail_major_version; 1395 uint32_t hsail_minor_version; 1396 uint8_t profile; 1397 uint8_t machine_model; 1398 uint8_t default_float_round; 1399 }; 1400 1401``NT_AMD_HSA_ISA_VERSION`` 1402 Specifies the target ISA version. The description field has the following layout: 1403 1404 .. code:: c 1405 1406 struct amdgpu_hsa_note_isa_s { 1407 uint16_t vendor_name_size; 1408 uint16_t architecture_name_size; 1409 uint32_t major; 1410 uint32_t minor; 1411 uint32_t stepping; 1412 char vendor_and_architecture_name[1]; 1413 }; 1414 1415 ``vendor_name_size`` and ``architecture_name_size`` are the length of the 1416 vendor and architecture names respectively, including the NUL character. 1417 1418 ``vendor_and_architecture_name`` contains the NUL terminates string for the 1419 vendor, immediately followed by the NUL terminated string for the 1420 architecture. 1421 1422 This note record is used by the HSA runtime loader. 1423 1424 Code object V2 only supports a limited number of processors and has fixed 1425 settings for target features. See 1426 :ref:`amdgpu-elf-note-record-supported_processors-v2-table` for a list of 1427 processors and the corresponding target ID. In the table the note record ISA 1428 name is a concatenation of the vendor name, architecture name, major, minor, 1429 and stepping separated by a ":". 1430 1431 The target ID column shows the processor name and fixed target features used 1432 by the LLVM compiler. The LLVM compiler does not generate a 1433 ``NT_AMD_HSA_HSAIL`` note record. 1434 1435 A code object generated by the Finalizer also uses code object V2 and always 1436 generates a ``NT_AMD_HSA_HSAIL`` note record. The processor name and 1437 ``sramecc`` target feature is as shown in 1438 :ref:`amdgpu-elf-note-record-supported_processors-v2-table` but the ``xnack`` 1439 target feature is specified by the ``EF_AMDGPU_FEATURE_XNACK_V2`` ``e_flags`` 1440 bit. 1441 1442``NT_AMD_HSA_ISA_NAME`` 1443 Specifies the target ISA name as a non-NUL terminated string. 1444 1445 This note record is not used by the HSA runtime loader. 1446 1447 See the ``NT_AMD_HSA_ISA_VERSION`` note record description of the code object 1448 V2's limited support of processors and fixed settings for target features. 1449 1450 See :ref:`amdgpu-elf-note-record-supported_processors-v2-table` for a mapping 1451 from the string to the corresponding target ID. If the ``xnack`` target 1452 feature is supported and enabled, the string produced by the LLVM compiler 1453 will may have a ``+xnack`` appended. The Finlizer did not do the appending and 1454 instead used the ``EF_AMDGPU_FEATURE_XNACK_V2`` ``e_flags`` bit. 1455 1456``NT_AMD_HSA_METADATA`` 1457 Specifies extensible metadata associated with the code objects executed on HSA 1458 [HSA]_ compatible runtimes (see :ref:`amdgpu-os`). It is required when the 1459 target triple OS is ``amdhsa`` (see :ref:`amdgpu-target-triples`). See 1460 :ref:`amdgpu-amdhsa-code-object-metadata-v2` for the syntax of the code object 1461 metadata string. 1462 1463 .. table:: AMDGPU Code Object V2 Supported Processors and Fixed Target Feature Settings 1464 :name: amdgpu-elf-note-record-supported_processors-v2-table 1465 1466 ===================== ========================== 1467 Note Record ISA Name Target ID 1468 ===================== ========================== 1469 ``AMD:AMDGPU:6:0:0`` ``gfx600`` 1470 ``AMD:AMDGPU:6:0:1`` ``gfx601`` 1471 ``AMD:AMDGPU:6:0:2`` ``gfx602`` 1472 ``AMD:AMDGPU:7:0:0`` ``gfx700`` 1473 ``AMD:AMDGPU:7:0:1`` ``gfx701`` 1474 ``AMD:AMDGPU:7:0:2`` ``gfx702`` 1475 ``AMD:AMDGPU:7:0:3`` ``gfx703`` 1476 ``AMD:AMDGPU:7:0:4`` ``gfx704`` 1477 ``AMD:AMDGPU:7:0:5`` ``gfx705`` 1478 ``AMD:AMDGPU:8:0:0`` ``gfx802`` 1479 ``AMD:AMDGPU:8:0:1`` ``gfx801:xnack+`` 1480 ``AMD:AMDGPU:8:0:2`` ``gfx802`` 1481 ``AMD:AMDGPU:8:0:3`` ``gfx803`` 1482 ``AMD:AMDGPU:8:0:4`` ``gfx803`` 1483 ``AMD:AMDGPU:8:0:5`` ``gfx805`` 1484 ``AMD:AMDGPU:8:1:0`` ``gfx810:xnack+`` 1485 ``AMD:AMDGPU:9:0:0`` ``gfx900:xnack-`` 1486 ``AMD:AMDGPU:9:0:1`` ``gfx900:xnack+`` 1487 ``AMD:AMDGPU:9:0:2`` ``gfx902:xnack-`` 1488 ``AMD:AMDGPU:9:0:3`` ``gfx902:xnack+`` 1489 ``AMD:AMDGPU:9:0:4`` ``gfx904:xnack-`` 1490 ``AMD:AMDGPU:9:0:5`` ``gfx904:xnack+`` 1491 ``AMD:AMDGPU:9:0:6`` ``gfx906:sramecc-:xnack-`` 1492 ``AMD:AMDGPU:9:0:7`` ``gfx906:sramecc-:xnack+`` 1493 ``AMD:AMDGPU:9:0:12`` ``gfx90c:xnack-`` 1494 ===================== ========================== 1495 1496.. _amdgpu-note-records-v3-onwards: 1497 1498Code Object V3 and Above Note Records 1499~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1500 1501The AMDGPU backend code object uses the following ELF note record in the 1502``.note`` section when compiling for code object V3 and above. 1503 1504The note record vendor field is "AMDGPU". 1505 1506Additional note records may be present, but any which are not documented here 1507are deprecated and should not be used. 1508 1509 .. table:: AMDGPU Code Object V3 and Above ELF Note Records 1510 :name: amdgpu-elf-note-records-table-v3-onwards 1511 1512 ======== ============================== ====================================== 1513 Name Type Description 1514 ======== ============================== ====================================== 1515 "AMDGPU" ``NT_AMDGPU_METADATA`` Metadata in Message Pack [MsgPack]_ 1516 binary format. 1517 ======== ============================== ====================================== 1518 1519.. 1520 1521 .. table:: AMDGPU Code Object V3 and Above ELF Note Record Enumeration Values 1522 :name: amdgpu-elf-note-record-enumeration-values-table-v3-onwards 1523 1524 ============================== ===== 1525 Name Value 1526 ============================== ===== 1527 *reserved* 0-31 1528 ``NT_AMDGPU_METADATA`` 32 1529 ============================== ===== 1530 1531``NT_AMDGPU_METADATA`` 1532 Specifies extensible metadata associated with an AMDGPU code object. It is 1533 encoded as a map in the Message Pack [MsgPack]_ binary data format. See 1534 :ref:`amdgpu-amdhsa-code-object-metadata-v3`, 1535 :ref:`amdgpu-amdhsa-code-object-metadata-v4` and 1536 :ref:`amdgpu-amdhsa-code-object-metadata-v5` for the map keys defined for the 1537 ``amdhsa`` OS. 1538 1539.. _amdgpu-symbols: 1540 1541Symbols 1542------- 1543 1544Symbols include the following: 1545 1546 .. table:: AMDGPU ELF Symbols 1547 :name: amdgpu-elf-symbols-table 1548 1549 ===================== ================== ================ ================== 1550 Name Type Section Description 1551 ===================== ================== ================ ================== 1552 *link-name* ``STT_OBJECT`` - ``.data`` Global variable 1553 - ``.rodata`` 1554 - ``.bss`` 1555 *link-name*\ ``.kd`` ``STT_OBJECT`` - ``.rodata`` Kernel descriptor 1556 *link-name* ``STT_FUNC`` - ``.text`` Kernel entry point 1557 *link-name* ``STT_OBJECT`` - SHN_AMDGPU_LDS Global variable in LDS 1558 ===================== ================== ================ ================== 1559 1560Global variable 1561 Global variables both used and defined by the compilation unit. 1562 1563 If the symbol is defined in the compilation unit then it is allocated in the 1564 appropriate section according to if it has initialized data or is readonly. 1565 1566 If the symbol is external then its section is ``STN_UNDEF`` and the loader 1567 will resolve relocations using the definition provided by another code object 1568 or explicitly defined by the runtime. 1569 1570 If the symbol resides in local/group memory (LDS) then its section is the 1571 special processor specific section name ``SHN_AMDGPU_LDS``, and the 1572 ``st_value`` field describes alignment requirements as it does for common 1573 symbols. 1574 1575 .. TODO:: 1576 1577 Add description of linked shared object symbols. Seems undefined symbols 1578 are marked as STT_NOTYPE. 1579 1580Kernel descriptor 1581 Every HSA kernel has an associated kernel descriptor. It is the address of the 1582 kernel descriptor that is used in the AQL dispatch packet used to invoke the 1583 kernel, not the kernel entry point. The layout of the HSA kernel descriptor is 1584 defined in :ref:`amdgpu-amdhsa-kernel-descriptor`. 1585 1586Kernel entry point 1587 Every HSA kernel also has a symbol for its machine code entry point. 1588 1589.. _amdgpu-relocation-records: 1590 1591Relocation Records 1592------------------ 1593 1594AMDGPU backend generates ``Elf64_Rela`` relocation records. Supported 1595relocatable fields are: 1596 1597``word32`` 1598 This specifies a 32-bit field occupying 4 bytes with arbitrary byte 1599 alignment. These values use the same byte order as other word values in the 1600 AMDGPU architecture. 1601 1602``word64`` 1603 This specifies a 64-bit field occupying 8 bytes with arbitrary byte 1604 alignment. These values use the same byte order as other word values in the 1605 AMDGPU architecture. 1606 1607Following notations are used for specifying relocation calculations: 1608 1609**A** 1610 Represents the addend used to compute the value of the relocatable field. 1611 1612**G** 1613 Represents the offset into the global offset table at which the relocation 1614 entry's symbol will reside during execution. 1615 1616**GOT** 1617 Represents the address of the global offset table. 1618 1619**P** 1620 Represents the place (section offset for ``et_rel`` or address for ``et_dyn``) 1621 of the storage unit being relocated (computed using ``r_offset``). 1622 1623**S** 1624 Represents the value of the symbol whose index resides in the relocation 1625 entry. Relocations not using this must specify a symbol index of 1626 ``STN_UNDEF``. 1627 1628**B** 1629 Represents the base address of a loaded executable or shared object which is 1630 the difference between the ELF address and the actual load address. 1631 Relocations using this are only valid in executable or shared objects. 1632 1633The following relocation types are supported: 1634 1635 .. table:: AMDGPU ELF Relocation Records 1636 :name: amdgpu-elf-relocation-records-table 1637 1638 ========================== ======= ===== ========== ============================== 1639 Relocation Type Kind Value Field Calculation 1640 ========================== ======= ===== ========== ============================== 1641 ``R_AMDGPU_NONE`` 0 *none* *none* 1642 ``R_AMDGPU_ABS32_LO`` Static, 1 ``word32`` (S + A) & 0xFFFFFFFF 1643 Dynamic 1644 ``R_AMDGPU_ABS32_HI`` Static, 2 ``word32`` (S + A) >> 32 1645 Dynamic 1646 ``R_AMDGPU_ABS64`` Static, 3 ``word64`` S + A 1647 Dynamic 1648 ``R_AMDGPU_REL32`` Static 4 ``word32`` S + A - P 1649 ``R_AMDGPU_REL64`` Static 5 ``word64`` S + A - P 1650 ``R_AMDGPU_ABS32`` Static, 6 ``word32`` S + A 1651 Dynamic 1652 ``R_AMDGPU_GOTPCREL`` Static 7 ``word32`` G + GOT + A - P 1653 ``R_AMDGPU_GOTPCREL32_LO`` Static 8 ``word32`` (G + GOT + A - P) & 0xFFFFFFFF 1654 ``R_AMDGPU_GOTPCREL32_HI`` Static 9 ``word32`` (G + GOT + A - P) >> 32 1655 ``R_AMDGPU_REL32_LO`` Static 10 ``word32`` (S + A - P) & 0xFFFFFFFF 1656 ``R_AMDGPU_REL32_HI`` Static 11 ``word32`` (S + A - P) >> 32 1657 *reserved* 12 1658 ``R_AMDGPU_RELATIVE64`` Dynamic 13 ``word64`` B + A 1659 ``R_AMDGPU_REL16`` Static 14 ``word16`` ((S + A - P) - 4) / 4 1660 ========================== ======= ===== ========== ============================== 1661 1662``R_AMDGPU_ABS32_LO`` and ``R_AMDGPU_ABS32_HI`` are only supported by 1663the ``mesa3d`` OS, which does not support ``R_AMDGPU_ABS64``. 1664 1665There is no current OS loader support for 32-bit programs and so 1666``R_AMDGPU_ABS32`` is not used. 1667 1668.. _amdgpu-loaded-code-object-path-uniform-resource-identifier: 1669 1670Loaded Code Object Path Uniform Resource Identifier (URI) 1671--------------------------------------------------------- 1672 1673The AMD GPU code object loader represents the path of the ELF shared object from 1674which the code object was loaded as a textual Uniform Resource Identifier (URI). 1675Note that the code object is the in memory loaded relocated form of the ELF 1676shared object. Multiple code objects may be loaded at different memory 1677addresses in the same process from the same ELF shared object. 1678 1679The loaded code object path URI syntax is defined by the following BNF syntax: 1680 1681.. code:: 1682 1683 code_object_uri ::== file_uri | memory_uri 1684 file_uri ::== "file://" file_path [ range_specifier ] 1685 memory_uri ::== "memory://" process_id range_specifier 1686 range_specifier ::== [ "#" | "?" ] "offset=" number "&" "size=" number 1687 file_path ::== URI_ENCODED_OS_FILE_PATH 1688 process_id ::== DECIMAL_NUMBER 1689 number ::== HEX_NUMBER | DECIMAL_NUMBER | OCTAL_NUMBER 1690 1691**number** 1692 Is a C integral literal where hexadecimal values are prefixed by "0x" or "0X", 1693 and octal values by "0". 1694 1695**file_path** 1696 Is the file's path specified as a URI encoded UTF-8 string. In URI encoding, 1697 every character that is not in the regular expression ``[a-zA-Z0-9/_.~-]`` is 1698 encoded as two uppercase hexadecimal digits proceeded by "%". Directories in 1699 the path are separated by "/". 1700 1701**offset** 1702 Is a 0-based byte offset to the start of the code object. For a file URI, it 1703 is from the start of the file specified by the ``file_path``, and if omitted 1704 defaults to 0. For a memory URI, it is the memory address and is required. 1705 1706**size** 1707 Is the number of bytes in the code object. For a file URI, if omitted it 1708 defaults to the size of the file. It is required for a memory URI. 1709 1710**process_id** 1711 Is the identity of the process owning the memory. For Linux it is the C 1712 unsigned integral decimal literal for the process ID (PID). 1713 1714For example: 1715 1716.. code:: 1717 1718 file:///dir1/dir2/file1 1719 file:///dir3/dir4/file2#offset=0x2000&size=3000 1720 memory://1234#offset=0x20000&size=3000 1721 1722.. _amdgpu-dwarf-debug-information: 1723 1724DWARF Debug Information 1725======================= 1726 1727.. warning:: 1728 1729 This section describes **provisional support** for AMDGPU DWARF [DWARF]_ that 1730 is not currently fully implemented and is subject to change. 1731 1732AMDGPU generates DWARF [DWARF]_ debugging information ELF sections (see 1733:ref:`amdgpu-elf-code-object`) which contain information that maps the code 1734object executable code and data to the source language constructs. It can be 1735used by tools such as debuggers and profilers. It uses features defined in 1736:doc:`AMDGPUDwarfExtensionsForHeterogeneousDebugging` that are made available in 1737DWARF Version 4 and DWARF Version 5 as an LLVM vendor extension. 1738 1739This section defines the AMDGPU target architecture specific DWARF mappings. 1740 1741.. _amdgpu-dwarf-register-identifier: 1742 1743Register Identifier 1744------------------- 1745 1746This section defines the AMDGPU target architecture register numbers used in 1747DWARF operation expressions (see DWARF Version 5 section 2.5 and 1748:ref:`amdgpu-dwarf-operation-expressions`) and Call Frame Information 1749instructions (see DWARF Version 5 section 6.4 and 1750:ref:`amdgpu-dwarf-call-frame-information`). 1751 1752A single code object can contain code for kernels that have different wavefront 1753sizes. The vector registers and some scalar registers are based on the wavefront 1754size. AMDGPU defines distinct DWARF registers for each wavefront size. This 1755simplifies the consumer of the DWARF so that each register has a fixed size, 1756rather than being dynamic according to the wavefront size mode. Similarly, 1757distinct DWARF registers are defined for those registers that vary in size 1758according to the process address size. This allows a consumer to treat a 1759specific AMDGPU processor as a single architecture regardless of how it is 1760configured at run time. The compiler explicitly specifies the DWARF registers 1761that match the mode in which the code it is generating will be executed. 1762 1763DWARF registers are encoded as numbers, which are mapped to architecture 1764registers. The mapping for AMDGPU is defined in 1765:ref:`amdgpu-dwarf-register-mapping-table`. All AMDGPU targets use the same 1766mapping. 1767 1768.. table:: AMDGPU DWARF Register Mapping 1769 :name: amdgpu-dwarf-register-mapping-table 1770 1771 ============== ================= ======== ================================== 1772 DWARF Register AMDGPU Register Bit Size Description 1773 ============== ================= ======== ================================== 1774 0 PC_32 32 Program Counter (PC) when 1775 executing in a 32-bit process 1776 address space. Used in the CFI to 1777 describe the PC of the calling 1778 frame. 1779 1 EXEC_MASK_32 32 Execution Mask Register when 1780 executing in wavefront 32 mode. 1781 2-15 *Reserved* *Reserved for highly accessed 1782 registers using DWARF shortcut.* 1783 16 PC_64 64 Program Counter (PC) when 1784 executing in a 64-bit process 1785 address space. Used in the CFI to 1786 describe the PC of the calling 1787 frame. 1788 17 EXEC_MASK_64 64 Execution Mask Register when 1789 executing in wavefront 64 mode. 1790 18-31 *Reserved* *Reserved for highly accessed 1791 registers using DWARF shortcut.* 1792 32-95 SGPR0-SGPR63 32 Scalar General Purpose 1793 Registers. 1794 96-127 *Reserved* *Reserved for frequently accessed 1795 registers using DWARF 1-byte ULEB.* 1796 128 STATUS 32 Status Register. 1797 129-511 *Reserved* *Reserved for future Scalar 1798 Architectural Registers.* 1799 512 VCC_32 32 Vector Condition Code Register 1800 when executing in wavefront 32 1801 mode. 1802 513-767 *Reserved* *Reserved for future Vector 1803 Architectural Registers when 1804 executing in wavefront 32 mode.* 1805 768 VCC_64 64 Vector Condition Code Register 1806 when executing in wavefront 64 1807 mode. 1808 769-1023 *Reserved* *Reserved for future Vector 1809 Architectural Registers when 1810 executing in wavefront 64 mode.* 1811 1024-1087 *Reserved* *Reserved for padding.* 1812 1088-1129 SGPR64-SGPR105 32 Scalar General Purpose Registers. 1813 1130-1535 *Reserved* *Reserved for future Scalar 1814 General Purpose Registers.* 1815 1536-1791 VGPR0-VGPR255 32*32 Vector General Purpose Registers 1816 when executing in wavefront 32 1817 mode. 1818 1792-2047 *Reserved* *Reserved for future Vector 1819 General Purpose Registers when 1820 executing in wavefront 32 mode.* 1821 2048-2303 AGPR0-AGPR255 32*32 Vector Accumulation Registers 1822 when executing in wavefront 32 1823 mode. 1824 2304-2559 *Reserved* *Reserved for future Vector 1825 Accumulation Registers when 1826 executing in wavefront 32 mode.* 1827 2560-2815 VGPR0-VGPR255 64*32 Vector General Purpose Registers 1828 when executing in wavefront 64 1829 mode. 1830 2816-3071 *Reserved* *Reserved for future Vector 1831 General Purpose Registers when 1832 executing in wavefront 64 mode.* 1833 3072-3327 AGPR0-AGPR255 64*32 Vector Accumulation Registers 1834 when executing in wavefront 64 1835 mode. 1836 3328-3583 *Reserved* *Reserved for future Vector 1837 Accumulation Registers when 1838 executing in wavefront 64 mode.* 1839 ============== ================= ======== ================================== 1840 1841The vector registers are represented as the full size for the wavefront. They 1842are organized as consecutive dwords (32-bits), one per lane, with the dword at 1843the least significant bit position corresponding to lane 0 and so forth. DWARF 1844location expressions involving the ``DW_OP_LLVM_offset`` and 1845``DW_OP_LLVM_push_lane`` operations are used to select the part of the vector 1846register corresponding to the lane that is executing the current thread of 1847execution in languages that are implemented using a SIMD or SIMT execution 1848model. 1849 1850If the wavefront size is 32 lanes then the wavefront 32 mode register 1851definitions are used. If the wavefront size is 64 lanes then the wavefront 64 1852mode register definitions are used. Some AMDGPU targets support executing in 1853both wavefront 32 and wavefront 64 mode. The register definitions corresponding 1854to the wavefront mode of the generated code will be used. 1855 1856If code is generated to execute in a 32-bit process address space, then the 185732-bit process address space register definitions are used. If code is generated 1858to execute in a 64-bit process address space, then the 64-bit process address 1859space register definitions are used. The ``amdgcn`` target only supports the 186064-bit process address space. 1861 1862.. _amdgpu-dwarf-address-class-identifier: 1863 1864Address Class Identifier 1865------------------------ 1866 1867The DWARF address class represents the source language memory space. See DWARF 1868Version 5 section 2.12 which is updated by the *DWARF Extensions For 1869Heterogeneous Debugging* section :ref:`amdgpu-dwarf-segment_addresses`. 1870 1871The DWARF address class mapping used for AMDGPU is defined in 1872:ref:`amdgpu-dwarf-address-class-mapping-table`. 1873 1874.. table:: AMDGPU DWARF Address Class Mapping 1875 :name: amdgpu-dwarf-address-class-mapping-table 1876 1877 ========================= ====== ================= 1878 DWARF AMDGPU 1879 -------------------------------- ----------------- 1880 Address Class Name Value Address Space 1881 ========================= ====== ================= 1882 ``DW_ADDR_none`` 0x0000 Generic (Flat) 1883 ``DW_ADDR_LLVM_global`` 0x0001 Global 1884 ``DW_ADDR_LLVM_constant`` 0x0002 Global 1885 ``DW_ADDR_LLVM_group`` 0x0003 Local (group/LDS) 1886 ``DW_ADDR_LLVM_private`` 0x0004 Private (Scratch) 1887 ``DW_ADDR_AMDGPU_region`` 0x8000 Region (GDS) 1888 ========================= ====== ================= 1889 1890The DWARF address class values defined in the *DWARF Extensions For 1891Heterogeneous Debugging* section :ref:`amdgpu-dwarf-segment_addresses` are used. 1892 1893In addition, ``DW_ADDR_AMDGPU_region`` is encoded as a vendor extension. This is 1894available for use for the AMD extension for access to the hardware GDS memory 1895which is scratchpad memory allocated per device. 1896 1897For AMDGPU if no ``DW_AT_address_class`` attribute is present, then the default 1898address class of ``DW_ADDR_none`` is used. 1899 1900See :ref:`amdgpu-dwarf-address-space-identifier` for information on the AMDGPU 1901mapping of DWARF address classes to DWARF address spaces, including address size 1902and NULL value. 1903 1904.. _amdgpu-dwarf-address-space-identifier: 1905 1906Address Space Identifier 1907------------------------ 1908 1909DWARF address spaces correspond to target architecture specific linear 1910addressable memory areas. See DWARF Version 5 section 2.12 and *DWARF Extensions 1911For Heterogeneous Debugging* section :ref:`amdgpu-dwarf-segment_addresses`. 1912 1913The DWARF address space mapping used for AMDGPU is defined in 1914:ref:`amdgpu-dwarf-address-space-mapping-table`. 1915 1916.. table:: AMDGPU DWARF Address Space Mapping 1917 :name: amdgpu-dwarf-address-space-mapping-table 1918 1919 ======================================= ===== ======= ======== ================= ======================= 1920 DWARF AMDGPU Notes 1921 --------------------------------------- ----- ---------------- ----------------- ----------------------- 1922 Address Space Name Value Address Bit Size Address Space 1923 --------------------------------------- ----- ------- -------- ----------------- ----------------------- 1924 .. 64-bit 32-bit 1925 process process 1926 address address 1927 space space 1928 ======================================= ===== ======= ======== ================= ======================= 1929 ``DW_ASPACE_none`` 0x00 64 32 Global *default address space* 1930 ``DW_ASPACE_AMDGPU_generic`` 0x01 64 32 Generic (Flat) 1931 ``DW_ASPACE_AMDGPU_region`` 0x02 32 32 Region (GDS) 1932 ``DW_ASPACE_AMDGPU_local`` 0x03 32 32 Local (group/LDS) 1933 *Reserved* 0x04 1934 ``DW_ASPACE_AMDGPU_private_lane`` 0x05 32 32 Private (Scratch) *focused lane* 1935 ``DW_ASPACE_AMDGPU_private_wave`` 0x06 32 32 Private (Scratch) *unswizzled wavefront* 1936 ======================================= ===== ======= ======== ================= ======================= 1937 1938See :ref:`amdgpu-address-spaces` for information on the AMDGPU address spaces 1939including address size and NULL value. 1940 1941The ``DW_ASPACE_none`` address space is the default target architecture address 1942space used in DWARF operations that do not specify an address space. It 1943therefore has to map to the global address space so that the ``DW_OP_addr*`` and 1944related operations can refer to addresses in the program code. 1945 1946The ``DW_ASPACE_AMDGPU_generic`` address space allows location expressions to 1947specify the flat address space. If the address corresponds to an address in the 1948local address space, then it corresponds to the wavefront that is executing the 1949focused thread of execution. If the address corresponds to an address in the 1950private address space, then it corresponds to the lane that is executing the 1951focused thread of execution for languages that are implemented using a SIMD or 1952SIMT execution model. 1953 1954.. note:: 1955 1956 CUDA-like languages such as HIP that do not have address spaces in the 1957 language type system, but do allow variables to be allocated in different 1958 address spaces, need to explicitly specify the ``DW_ASPACE_AMDGPU_generic`` 1959 address space in the DWARF expression operations as the default address space 1960 is the global address space. 1961 1962The ``DW_ASPACE_AMDGPU_local`` address space allows location expressions to 1963specify the local address space corresponding to the wavefront that is executing 1964the focused thread of execution. 1965 1966The ``DW_ASPACE_AMDGPU_private_lane`` address space allows location expressions 1967to specify the private address space corresponding to the lane that is executing 1968the focused thread of execution for languages that are implemented using a SIMD 1969or SIMT execution model. 1970 1971The ``DW_ASPACE_AMDGPU_private_wave`` address space allows location expressions 1972to specify the unswizzled private address space corresponding to the wavefront 1973that is executing the focused thread of execution. The wavefront view of private 1974memory is the per wavefront unswizzled backing memory layout defined in 1975:ref:`amdgpu-address-spaces`, such that address 0 corresponds to the first 1976location for the backing memory of the wavefront (namely the address is not 1977offset by ``wavefront-scratch-base``). The following formula can be used to 1978convert from a ``DW_ASPACE_AMDGPU_private_lane`` address to a 1979``DW_ASPACE_AMDGPU_private_wave`` address: 1980 1981:: 1982 1983 private-address-wavefront = 1984 ((private-address-lane / 4) * wavefront-size * 4) + 1985 (wavefront-lane-id * 4) + (private-address-lane % 4) 1986 1987If the ``DW_ASPACE_AMDGPU_private_lane`` address is dword aligned, and the start 1988of the dwords for each lane starting with lane 0 is required, then this 1989simplifies to: 1990 1991:: 1992 1993 private-address-wavefront = 1994 private-address-lane * wavefront-size 1995 1996A compiler can use the ``DW_ASPACE_AMDGPU_private_wave`` address space to read a 1997complete spilled vector register back into a complete vector register in the 1998CFI. The frame pointer can be a private lane address which is dword aligned, 1999which can be shifted to multiply by the wavefront size, and then used to form a 2000private wavefront address that gives a location for a contiguous set of dwords, 2001one per lane, where the vector register dwords are spilled. The compiler knows 2002the wavefront size since it generates the code. Note that the type of the 2003address may have to be converted as the size of a 2004``DW_ASPACE_AMDGPU_private_lane`` address may be smaller than the size of a 2005``DW_ASPACE_AMDGPU_private_wave`` address. 2006 2007.. _amdgpu-dwarf-lane-identifier: 2008 2009Lane identifier 2010--------------- 2011 2012DWARF lane identifies specify a target architecture lane position for hardware 2013that executes in a SIMD or SIMT manner, and on which a source language maps its 2014threads of execution onto those lanes. The DWARF lane identifier is pushed by 2015the ``DW_OP_LLVM_push_lane`` DWARF expression operation. See DWARF Version 5 2016section 2.5 which is updated by *DWARF Extensions For Heterogeneous Debugging* 2017section :ref:`amdgpu-dwarf-operation-expressions`. 2018 2019For AMDGPU, the lane identifier corresponds to the hardware lane ID of a 2020wavefront. It is numbered from 0 to the wavefront size minus 1. 2021 2022Operation Expressions 2023--------------------- 2024 2025DWARF expressions are used to compute program values and the locations of 2026program objects. See DWARF Version 5 section 2.5 and 2027:ref:`amdgpu-dwarf-operation-expressions`. 2028 2029DWARF location descriptions describe how to access storage which includes memory 2030and registers. When accessing storage on AMDGPU, bytes are ordered with least 2031significant bytes first, and bits are ordered within bytes with least 2032significant bits first. 2033 2034For AMDGPU CFI expressions, ``DW_OP_LLVM_select_bit_piece`` is used to describe 2035unwinding vector registers that are spilled under the execution mask to memory: 2036the zero-single location description is the vector register, and the one-single 2037location description is the spilled memory location description. The 2038``DW_OP_LLVM_form_aspace_address`` is used to specify the address space of the 2039memory location description. 2040 2041In AMDGPU expressions, ``DW_OP_LLVM_select_bit_piece`` is used by the 2042``DW_AT_LLVM_lane_pc`` attribute expression where divergent control flow is 2043controlled by the execution mask. An undefined location description together 2044with ``DW_OP_LLVM_extend`` is used to indicate the lane was not active on entry 2045to the subprogram. See :ref:`amdgpu-dwarf-dw-at-llvm-lane-pc` for an example. 2046 2047Debugger Information Entry Attributes 2048------------------------------------- 2049 2050This section describes how certain debugger information entry attributes are 2051used by AMDGPU. See the sections in DWARF Version 5 section 3.3.5 and 3.1.1 2052which are updated by *DWARF Extensions For Heterogeneous Debugging* section 2053:ref:`amdgpu-dwarf-low-level-information` and 2054:ref:`amdgpu-dwarf-full-and-partial-compilation-unit-entries`. 2055 2056.. _amdgpu-dwarf-dw-at-llvm-lane-pc: 2057 2058``DW_AT_LLVM_lane_pc`` 2059~~~~~~~~~~~~~~~~~~~~~~ 2060 2061For AMDGPU, the ``DW_AT_LLVM_lane_pc`` attribute is used to specify the program 2062location of the separate lanes of a SIMT thread. 2063 2064If the lane is an active lane then this will be the same as the current program 2065location. 2066 2067If the lane is inactive, but was active on entry to the subprogram, then this is 2068the program location in the subprogram at which execution of the lane is 2069conceptual positioned. 2070 2071If the lane was not active on entry to the subprogram, then this will be the 2072undefined location. A client debugger can check if the lane is part of a valid 2073work-group by checking that the lane is in the range of the associated 2074work-group within the grid, accounting for partial work-groups. If it is not, 2075then the debugger can omit any information for the lane. Otherwise, the debugger 2076may repeatedly unwind the stack and inspect the ``DW_AT_LLVM_lane_pc`` of the 2077calling subprogram until it finds a non-undefined location. Conceptually the 2078lane only has the call frames that it has a non-undefined 2079``DW_AT_LLVM_lane_pc``. 2080 2081The following example illustrates how the AMDGPU backend can generate a DWARF 2082location list expression for the nested ``IF/THEN/ELSE`` structures of the 2083following subprogram pseudo code for a target with 64 lanes per wavefront. 2084 2085.. code:: 2086 :number-lines: 2087 2088 SUBPROGRAM X 2089 BEGIN 2090 a; 2091 IF (c1) THEN 2092 b; 2093 IF (c2) THEN 2094 c; 2095 ELSE 2096 d; 2097 ENDIF 2098 e; 2099 ELSE 2100 f; 2101 ENDIF 2102 g; 2103 END 2104 2105The AMDGPU backend may generate the following pseudo LLVM MIR to manipulate the 2106execution mask (``EXEC``) to linearize the control flow. The condition is 2107evaluated to make a mask of the lanes for which the condition evaluates to true. 2108First the ``THEN`` region is executed by setting the ``EXEC`` mask to the 2109logical ``AND`` of the current ``EXEC`` mask with the condition mask. Then the 2110``ELSE`` region is executed by negating the ``EXEC`` mask and logical ``AND`` of 2111the saved ``EXEC`` mask at the start of the region. After the ``IF/THEN/ELSE`` 2112region the ``EXEC`` mask is restored to the value it had at the beginning of the 2113region. This is shown below. Other approaches are possible, but the basic 2114concept is the same. 2115 2116.. code:: 2117 :number-lines: 2118 2119 $lex_start: 2120 a; 2121 %1 = EXEC 2122 %2 = c1 2123 $lex_1_start: 2124 EXEC = %1 & %2 2125 $if_1_then: 2126 b; 2127 %3 = EXEC 2128 %4 = c2 2129 $lex_1_1_start: 2130 EXEC = %3 & %4 2131 $lex_1_1_then: 2132 c; 2133 EXEC = ~EXEC & %3 2134 $lex_1_1_else: 2135 d; 2136 EXEC = %3 2137 $lex_1_1_end: 2138 e; 2139 EXEC = ~EXEC & %1 2140 $lex_1_else: 2141 f; 2142 EXEC = %1 2143 $lex_1_end: 2144 g; 2145 $lex_end: 2146 2147To create the DWARF location list expression that defines the location 2148description of a vector of lane program locations, the LLVM MIR ``DBG_VALUE`` 2149pseudo instruction can be used to annotate the linearized control flow. This can 2150be done by defining an artificial variable for the lane PC. The DWARF location 2151list expression created for it is used as the value of the 2152``DW_AT_LLVM_lane_pc`` attribute on the subprogram's debugger information entry. 2153 2154A DWARF procedure is defined for each well nested structured control flow region 2155which provides the conceptual lane program location for a lane if it is not 2156active (namely it is divergent). The DWARF operation expression for each region 2157conceptually inherits the value of the immediately enclosing region and modifies 2158it according to the semantics of the region. 2159 2160For an ``IF/THEN/ELSE`` region the divergent program location is at the start of 2161the region for the ``THEN`` region since it is executed first. For the ``ELSE`` 2162region the divergent program location is at the end of the ``IF/THEN/ELSE`` 2163region since the ``THEN`` region has completed. 2164 2165The lane PC artificial variable is assigned at each region transition. It uses 2166the immediately enclosing region's DWARF procedure to compute the program 2167location for each lane assuming they are divergent, and then modifies the result 2168by inserting the current program location for each lane that the ``EXEC`` mask 2169indicates is active. 2170 2171By having separate DWARF procedures for each region, they can be reused to 2172define the value for any nested region. This reduces the total size of the DWARF 2173operation expressions. 2174 2175The following provides an example using pseudo LLVM MIR. 2176 2177.. code:: 2178 :number-lines: 2179 2180 $lex_start: 2181 DEFINE_DWARF %__uint_64 = DW_TAG_base_type[ 2182 DW_AT_name = "__uint64"; 2183 DW_AT_byte_size = 8; 2184 DW_AT_encoding = DW_ATE_unsigned; 2185 ]; 2186 DEFINE_DWARF %__active_lane_pc = DW_TAG_dwarf_procedure[ 2187 DW_AT_name = "__active_lane_pc"; 2188 DW_AT_location = [ 2189 DW_OP_regx PC; 2190 DW_OP_LLVM_extend 64, 64; 2191 DW_OP_regval_type EXEC, %uint_64; 2192 DW_OP_LLVM_select_bit_piece 64, 64; 2193 ]; 2194 ]; 2195 DEFINE_DWARF %__divergent_lane_pc = DW_TAG_dwarf_procedure[ 2196 DW_AT_name = "__divergent_lane_pc"; 2197 DW_AT_location = [ 2198 DW_OP_LLVM_undefined; 2199 DW_OP_LLVM_extend 64, 64; 2200 ]; 2201 ]; 2202 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 2203 DW_OP_call_ref %__divergent_lane_pc; 2204 DW_OP_call_ref %__active_lane_pc; 2205 ]; 2206 a; 2207 %1 = EXEC; 2208 DBG_VALUE %1, $noreg, %__lex_1_save_exec; 2209 %2 = c1; 2210 $lex_1_start: 2211 EXEC = %1 & %2; 2212 $lex_1_then: 2213 DEFINE_DWARF %__divergent_lane_pc_1_then = DW_TAG_dwarf_procedure[ 2214 DW_AT_name = "__divergent_lane_pc_1_then"; 2215 DW_AT_location = DIExpression[ 2216 DW_OP_call_ref %__divergent_lane_pc; 2217 DW_OP_addrx &lex_1_start; 2218 DW_OP_stack_value; 2219 DW_OP_LLVM_extend 64, 64; 2220 DW_OP_call_ref %__lex_1_save_exec; 2221 DW_OP_deref_type 64, %__uint_64; 2222 DW_OP_LLVM_select_bit_piece 64, 64; 2223 ]; 2224 ]; 2225 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 2226 DW_OP_call_ref %__divergent_lane_pc_1_then; 2227 DW_OP_call_ref %__active_lane_pc; 2228 ]; 2229 b; 2230 %3 = EXEC; 2231 DBG_VALUE %3, %__lex_1_1_save_exec; 2232 %4 = c2; 2233 $lex_1_1_start: 2234 EXEC = %3 & %4; 2235 $lex_1_1_then: 2236 DEFINE_DWARF %__divergent_lane_pc_1_1_then = DW_TAG_dwarf_procedure[ 2237 DW_AT_name = "__divergent_lane_pc_1_1_then"; 2238 DW_AT_location = DIExpression[ 2239 DW_OP_call_ref %__divergent_lane_pc_1_then; 2240 DW_OP_addrx &lex_1_1_start; 2241 DW_OP_stack_value; 2242 DW_OP_LLVM_extend 64, 64; 2243 DW_OP_call_ref %__lex_1_1_save_exec; 2244 DW_OP_deref_type 64, %__uint_64; 2245 DW_OP_LLVM_select_bit_piece 64, 64; 2246 ]; 2247 ]; 2248 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 2249 DW_OP_call_ref %__divergent_lane_pc_1_1_then; 2250 DW_OP_call_ref %__active_lane_pc; 2251 ]; 2252 c; 2253 EXEC = ~EXEC & %3; 2254 $lex_1_1_else: 2255 DEFINE_DWARF %__divergent_lane_pc_1_1_else = DW_TAG_dwarf_procedure[ 2256 DW_AT_name = "__divergent_lane_pc_1_1_else"; 2257 DW_AT_location = DIExpression[ 2258 DW_OP_call_ref %__divergent_lane_pc_1_then; 2259 DW_OP_addrx &lex_1_1_end; 2260 DW_OP_stack_value; 2261 DW_OP_LLVM_extend 64, 64; 2262 DW_OP_call_ref %__lex_1_1_save_exec; 2263 DW_OP_deref_type 64, %__uint_64; 2264 DW_OP_LLVM_select_bit_piece 64, 64; 2265 ]; 2266 ]; 2267 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 2268 DW_OP_call_ref %__divergent_lane_pc_1_1_else; 2269 DW_OP_call_ref %__active_lane_pc; 2270 ]; 2271 d; 2272 EXEC = %3; 2273 $lex_1_1_end: 2274 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 2275 DW_OP_call_ref %__divergent_lane_pc; 2276 DW_OP_call_ref %__active_lane_pc; 2277 ]; 2278 e; 2279 EXEC = ~EXEC & %1; 2280 $lex_1_else: 2281 DEFINE_DWARF %__divergent_lane_pc_1_else = DW_TAG_dwarf_procedure[ 2282 DW_AT_name = "__divergent_lane_pc_1_else"; 2283 DW_AT_location = DIExpression[ 2284 DW_OP_call_ref %__divergent_lane_pc; 2285 DW_OP_addrx &lex_1_end; 2286 DW_OP_stack_value; 2287 DW_OP_LLVM_extend 64, 64; 2288 DW_OP_call_ref %__lex_1_save_exec; 2289 DW_OP_deref_type 64, %__uint_64; 2290 DW_OP_LLVM_select_bit_piece 64, 64; 2291 ]; 2292 ]; 2293 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 2294 DW_OP_call_ref %__divergent_lane_pc_1_else; 2295 DW_OP_call_ref %__active_lane_pc; 2296 ]; 2297 f; 2298 EXEC = %1; 2299 $lex_1_end: 2300 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc DIExpression[ 2301 DW_OP_call_ref %__divergent_lane_pc; 2302 DW_OP_call_ref %__active_lane_pc; 2303 ]; 2304 g; 2305 $lex_end: 2306 2307The DWARF procedure ``%__active_lane_pc`` is used to update the lane pc elements 2308that are active, with the current program location. 2309 2310Artificial variables %__lex_1_save_exec and %__lex_1_1_save_exec are created for 2311the execution masks saved on entry to a region. Using the ``DBG_VALUE`` pseudo 2312instruction, location list entries will be created that describe where the 2313artificial variables are allocated at any given program location. The compiler 2314may allocate them to registers or spill them to memory. 2315 2316The DWARF procedures for each region use the values of the saved execution mask 2317artificial variables to only update the lanes that are active on entry to the 2318region. All other lanes retain the value of the enclosing region where they were 2319last active. If they were not active on entry to the subprogram, then will have 2320the undefined location description. 2321 2322Other structured control flow regions can be handled similarly. For example, 2323loops would set the divergent program location for the region at the end of the 2324loop. Any lanes active will be in the loop, and any lanes not active must have 2325exited the loop. 2326 2327An ``IF/THEN/ELSEIF/ELSEIF/...`` region can be treated as a nest of 2328``IF/THEN/ELSE`` regions. 2329 2330The DWARF procedures can use the active lane artificial variable described in 2331:ref:`amdgpu-dwarf-amdgpu-dw-at-llvm-active-lane` rather than the actual 2332``EXEC`` mask in order to support whole or quad wavefront mode. 2333 2334.. _amdgpu-dwarf-amdgpu-dw-at-llvm-active-lane: 2335 2336``DW_AT_LLVM_active_lane`` 2337~~~~~~~~~~~~~~~~~~~~~~~~~~ 2338 2339The ``DW_AT_LLVM_active_lane`` attribute on a subprogram debugger information 2340entry is used to specify the lanes that are conceptually active for a SIMT 2341thread. 2342 2343The execution mask may be modified to implement whole or quad wavefront mode 2344operations. For example, all lanes may need to temporarily be made active to 2345execute a whole wavefront operation. Such regions would save the ``EXEC`` mask, 2346update it to enable the necessary lanes, perform the operations, and then 2347restore the ``EXEC`` mask from the saved value. While executing the whole 2348wavefront region, the conceptual execution mask is the saved value, not the 2349``EXEC`` value. 2350 2351This is handled by defining an artificial variable for the active lane mask. The 2352active lane mask artificial variable would be the actual ``EXEC`` mask for 2353normal regions, and the saved execution mask for regions where the mask is 2354temporarily updated. The location list expression created for this artificial 2355variable is used to define the value of the ``DW_AT_LLVM_active_lane`` 2356attribute. 2357 2358``DW_AT_LLVM_augmentation`` 2359~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2360 2361For AMDGPU, the ``DW_AT_LLVM_augmentation`` attribute of a compilation unit 2362debugger information entry has the following value for the augmentation string: 2363 2364:: 2365 2366 [amdgpu:v0.0] 2367 2368The "vX.Y" specifies the major X and minor Y version number of the AMDGPU 2369extensions used in the DWARF of the compilation unit. The version number 2370conforms to [SEMVER]_. 2371 2372Call Frame Information 2373---------------------- 2374 2375DWARF Call Frame Information (CFI) describes how a consumer can virtually 2376*unwind* call frames in a running process or core dump. See DWARF Version 5 2377section 6.4 and :ref:`amdgpu-dwarf-call-frame-information`. 2378 2379For AMDGPU, the Common Information Entry (CIE) fields have the following values: 2380 23811. ``augmentation`` string contains the following null-terminated UTF-8 string: 2382 2383 :: 2384 2385 [amd:v0.0] 2386 2387 The ``vX.Y`` specifies the major X and minor Y version number of the AMDGPU 2388 extensions used in this CIE or to the FDEs that use it. The version number 2389 conforms to [SEMVER]_. 2390 23912. ``address_size`` for the ``Global`` address space is defined in 2392 :ref:`amdgpu-dwarf-address-space-identifier`. 2393 23943. ``segment_selector_size`` is 0 as AMDGPU does not use a segment selector. 2395 23964. ``code_alignment_factor`` is 4 bytes. 2397 2398 .. TODO:: 2399 2400 Add to :ref:`amdgpu-processor-table` table. 2401 24025. ``data_alignment_factor`` is 4 bytes. 2403 2404 .. TODO:: 2405 2406 Add to :ref:`amdgpu-processor-table` table. 2407 24086. ``return_address_register`` is ``PC_32`` for 32-bit processes and ``PC_64`` 2409 for 64-bit processes defined in :ref:`amdgpu-dwarf-register-identifier`. 2410 24117. ``initial_instructions`` Since a subprogram X with fewer registers can be 2412 called from subprogram Y that has more allocated, X will not change any of 2413 the extra registers as it cannot access them. Therefore, the default rule 2414 for all columns is ``same value``. 2415 2416For AMDGPU the register number follows the numbering defined in 2417:ref:`amdgpu-dwarf-register-identifier`. 2418 2419For AMDGPU the instructions are variable size. A consumer can subtract 1 from 2420the return address to get the address of a byte within the call site 2421instructions. See DWARF Version 5 section 6.4.4. 2422 2423Accelerated Access 2424------------------ 2425 2426See DWARF Version 5 section 6.1. 2427 2428Lookup By Name Section Header 2429~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2430 2431See DWARF Version 5 section 6.1.1.4.1 and :ref:`amdgpu-dwarf-lookup-by-name`. 2432 2433For AMDGPU the lookup by name section header table: 2434 2435``augmentation_string_size`` (uword) 2436 2437 Set to the length of the ``augmentation_string`` value which is always a 2438 multiple of 4. 2439 2440``augmentation_string`` (sequence of UTF-8 characters) 2441 2442 Contains the following UTF-8 string null padded to a multiple of 4 bytes: 2443 2444 :: 2445 2446 [amdgpu:v0.0] 2447 2448 The "vX.Y" specifies the major X and minor Y version number of the AMDGPU 2449 extensions used in the DWARF of this index. The version number conforms to 2450 [SEMVER]_. 2451 2452 .. note:: 2453 2454 This is different to the DWARF Version 5 definition that requires the first 2455 4 characters to be the vendor ID. But this is consistent with the other 2456 augmentation strings and does allow multiple vendor contributions. However, 2457 backwards compatibility may be more desirable. 2458 2459Lookup By Address Section Header 2460~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2461 2462See DWARF Version 5 section 6.1.2. 2463 2464For AMDGPU the lookup by address section header table: 2465 2466``address_size`` (ubyte) 2467 2468 Match the address size for the ``Global`` address space defined in 2469 :ref:`amdgpu-dwarf-address-space-identifier`. 2470 2471``segment_selector_size`` (ubyte) 2472 2473 AMDGPU does not use a segment selector so this is 0. The entries in the 2474 ``.debug_aranges`` do not have a segment selector. 2475 2476Line Number Information 2477----------------------- 2478 2479See DWARF Version 5 section 6.2 and :ref:`amdgpu-dwarf-line-number-information`. 2480 2481AMDGPU does not use the ``isa`` state machine registers and always sets it to 0. 2482The instruction set must be obtained from the ELF file header ``e_flags`` field 2483in the ``EF_AMDGPU_MACH`` bit position (see :ref:`ELF Header 2484<amdgpu-elf-header>`). See DWARF Version 5 section 6.2.2. 2485 2486.. TODO:: 2487 2488 Should the ``isa`` state machine register be used to indicate if the code is 2489 in wavefront32 or wavefront64 mode? Or used to specify the architecture ISA? 2490 2491For AMDGPU the line number program header fields have the following values (see 2492DWARF Version 5 section 6.2.4): 2493 2494``address_size`` (ubyte) 2495 Matches the address size for the ``Global`` address space defined in 2496 :ref:`amdgpu-dwarf-address-space-identifier`. 2497 2498``segment_selector_size`` (ubyte) 2499 AMDGPU does not use a segment selector so this is 0. 2500 2501``minimum_instruction_length`` (ubyte) 2502 For GFX9-GFX10 this is 4. 2503 2504``maximum_operations_per_instruction`` (ubyte) 2505 For GFX9-GFX10 this is 1. 2506 2507Source text for online-compiled programs (for example, those compiled by the 2508OpenCL language runtime) may be embedded into the DWARF Version 5 line table. 2509See DWARF Version 5 section 6.2.4.1 which is updated by *DWARF Extensions For 2510Heterogeneous Debugging* section :ref:`DW_LNCT_LLVM_source 2511<amdgpu-dwarf-line-number-information-dw-lnct-llvm-source>`. 2512 2513The Clang option used to control source embedding in AMDGPU is defined in 2514:ref:`amdgpu-clang-debug-options-table`. 2515 2516 .. table:: AMDGPU Clang Debug Options 2517 :name: amdgpu-clang-debug-options-table 2518 2519 ==================== ================================================== 2520 Debug Flag Description 2521 ==================== ================================================== 2522 -g[no-]embed-source Enable/disable embedding source text in DWARF 2523 debug sections. Useful for environments where 2524 source cannot be written to disk, such as 2525 when performing online compilation. 2526 ==================== ================================================== 2527 2528For example: 2529 2530``-gembed-source`` 2531 Enable the embedded source. 2532 2533``-gno-embed-source`` 2534 Disable the embedded source. 2535 253632-Bit and 64-Bit DWARF Formats 2537------------------------------- 2538 2539See DWARF Version 5 section 7.4 and 2540:ref:`amdgpu-dwarf-32-bit-and-64-bit-dwarf-formats`. 2541 2542For AMDGPU: 2543 2544* For the ``amdgcn`` target architecture only the 64-bit process address space 2545 is supported. 2546 2547* The producer can generate either 32-bit or 64-bit DWARF format. LLVM generates 2548 the 32-bit DWARF format. 2549 2550Unit Headers 2551------------ 2552 2553For AMDGPU the following values apply for each of the unit headers described in 2554DWARF Version 5 sections 7.5.1.1, 7.5.1.2, and 7.5.1.3: 2555 2556``address_size`` (ubyte) 2557 Matches the address size for the ``Global`` address space defined in 2558 :ref:`amdgpu-dwarf-address-space-identifier`. 2559 2560.. _amdgpu-code-conventions: 2561 2562Code Conventions 2563================ 2564 2565This section provides code conventions used for each supported target triple OS 2566(see :ref:`amdgpu-target-triples`). 2567 2568AMDHSA 2569------ 2570 2571This section provides code conventions used when the target triple OS is 2572``amdhsa`` (see :ref:`amdgpu-target-triples`). 2573 2574.. _amdgpu-amdhsa-code-object-metadata: 2575 2576Code Object Metadata 2577~~~~~~~~~~~~~~~~~~~~ 2578 2579The code object metadata specifies extensible metadata associated with the code 2580objects executed on HSA [HSA]_ compatible runtimes (see :ref:`amdgpu-os`). The 2581encoding and semantics of this metadata depends on the code object version; see 2582:ref:`amdgpu-amdhsa-code-object-metadata-v2`, 2583:ref:`amdgpu-amdhsa-code-object-metadata-v3`, 2584:ref:`amdgpu-amdhsa-code-object-metadata-v4` and 2585:ref:`amdgpu-amdhsa-code-object-metadata-v5`. 2586 2587Code object metadata is specified in a note record (see 2588:ref:`amdgpu-note-records`) and is required when the target triple OS is 2589``amdhsa`` (see :ref:`amdgpu-target-triples`). It must contain the minimum 2590information necessary to support the HSA compatible runtime kernel queries. For 2591example, the segment sizes needed in a dispatch packet. In addition, a 2592high-level language runtime may require other information to be included. For 2593example, the AMD OpenCL runtime records kernel argument information. 2594 2595.. _amdgpu-amdhsa-code-object-metadata-v2: 2596 2597Code Object V2 Metadata 2598+++++++++++++++++++++++ 2599 2600.. warning:: 2601 Code object V2 is not the default code object version emitted by this version 2602 of LLVM. 2603 2604Code object V2 metadata is specified by the ``NT_AMD_HSA_METADATA`` note record 2605(see :ref:`amdgpu-note-records-v2`). 2606 2607The metadata is specified as a YAML formatted string (see [YAML]_ and 2608:doc:`YamlIO`). 2609 2610.. TODO:: 2611 2612 Is the string null terminated? It probably should not if YAML allows it to 2613 contain null characters, otherwise it should be. 2614 2615The metadata is represented as a single YAML document comprised of the mapping 2616defined in table :ref:`amdgpu-amdhsa-code-object-metadata-map-v2-table` and 2617referenced tables. 2618 2619For boolean values, the string values of ``false`` and ``true`` are used for 2620false and true respectively. 2621 2622Additional information can be added to the mappings. To avoid conflicts, any 2623non-AMD key names should be prefixed by "*vendor-name*.". 2624 2625 .. table:: AMDHSA Code Object V2 Metadata Map 2626 :name: amdgpu-amdhsa-code-object-metadata-map-v2-table 2627 2628 ========== ============== ========= ======================================= 2629 String Key Value Type Required? Description 2630 ========== ============== ========= ======================================= 2631 "Version" sequence of Required - The first integer is the major 2632 2 integers version. Currently 1. 2633 - The second integer is the minor 2634 version. Currently 0. 2635 "Printf" sequence of Each string is encoded information 2636 strings about a printf function call. The 2637 encoded information is organized as 2638 fields separated by colon (':'): 2639 2640 ``ID:N:S[0]:S[1]:...:S[N-1]:FormatString`` 2641 2642 where: 2643 2644 ``ID`` 2645 A 32-bit integer as a unique id for 2646 each printf function call 2647 2648 ``N`` 2649 A 32-bit integer equal to the number 2650 of arguments of printf function call 2651 minus 1 2652 2653 ``S[i]`` (where i = 0, 1, ... , N-1) 2654 32-bit integers for the size in bytes 2655 of the i-th FormatString argument of 2656 the printf function call 2657 2658 FormatString 2659 The format string passed to the 2660 printf function call. 2661 "Kernels" sequence of Required Sequence of the mappings for each 2662 mapping kernel in the code object. See 2663 :ref:`amdgpu-amdhsa-code-object-kernel-metadata-map-v2-table` 2664 for the definition of the mapping. 2665 ========== ============== ========= ======================================= 2666 2667.. 2668 2669 .. table:: AMDHSA Code Object V2 Kernel Metadata Map 2670 :name: amdgpu-amdhsa-code-object-kernel-metadata-map-v2-table 2671 2672 ================= ============== ========= ================================ 2673 String Key Value Type Required? Description 2674 ================= ============== ========= ================================ 2675 "Name" string Required Source name of the kernel. 2676 "SymbolName" string Required Name of the kernel 2677 descriptor ELF symbol. 2678 "Language" string Source language of the kernel. 2679 Values include: 2680 2681 - "OpenCL C" 2682 - "OpenCL C++" 2683 - "HCC" 2684 - "OpenMP" 2685 2686 "LanguageVersion" sequence of - The first integer is the major 2687 2 integers version. 2688 - The second integer is the 2689 minor version. 2690 "Attrs" mapping Mapping of kernel attributes. 2691 See 2692 :ref:`amdgpu-amdhsa-code-object-kernel-attribute-metadata-map-v2-table` 2693 for the mapping definition. 2694 "Args" sequence of Sequence of mappings of the 2695 mapping kernel arguments. See 2696 :ref:`amdgpu-amdhsa-code-object-kernel-argument-metadata-map-v2-table` 2697 for the definition of the mapping. 2698 "CodeProps" mapping Mapping of properties related to 2699 the kernel code. See 2700 :ref:`amdgpu-amdhsa-code-object-kernel-code-properties-metadata-map-v2-table` 2701 for the mapping definition. 2702 ================= ============== ========= ================================ 2703 2704.. 2705 2706 .. table:: AMDHSA Code Object V2 Kernel Attribute Metadata Map 2707 :name: amdgpu-amdhsa-code-object-kernel-attribute-metadata-map-v2-table 2708 2709 =================== ============== ========= ============================== 2710 String Key Value Type Required? Description 2711 =================== ============== ========= ============================== 2712 "ReqdWorkGroupSize" sequence of If not 0, 0, 0 then all values 2713 3 integers must be >=1 and the dispatch 2714 work-group size X, Y, Z must 2715 correspond to the specified 2716 values. Defaults to 0, 0, 0. 2717 2718 Corresponds to the OpenCL 2719 ``reqd_work_group_size`` 2720 attribute. 2721 "WorkGroupSizeHint" sequence of The dispatch work-group size 2722 3 integers X, Y, Z is likely to be the 2723 specified values. 2724 2725 Corresponds to the OpenCL 2726 ``work_group_size_hint`` 2727 attribute. 2728 "VecTypeHint" string The name of a scalar or vector 2729 type. 2730 2731 Corresponds to the OpenCL 2732 ``vec_type_hint`` attribute. 2733 2734 "RuntimeHandle" string The external symbol name 2735 associated with a kernel. 2736 OpenCL runtime allocates a 2737 global buffer for the symbol 2738 and saves the kernel's address 2739 to it, which is used for 2740 device side enqueueing. Only 2741 available for device side 2742 enqueued kernels. 2743 =================== ============== ========= ============================== 2744 2745.. 2746 2747 .. table:: AMDHSA Code Object V2 Kernel Argument Metadata Map 2748 :name: amdgpu-amdhsa-code-object-kernel-argument-metadata-map-v2-table 2749 2750 ================= ============== ========= ================================ 2751 String Key Value Type Required? Description 2752 ================= ============== ========= ================================ 2753 "Name" string Kernel argument name. 2754 "TypeName" string Kernel argument type name. 2755 "Size" integer Required Kernel argument size in bytes. 2756 "Align" integer Required Kernel argument alignment in 2757 bytes. Must be a power of two. 2758 "ValueKind" string Required Kernel argument kind that 2759 specifies how to set up the 2760 corresponding argument. 2761 Values include: 2762 2763 "ByValue" 2764 The argument is copied 2765 directly into the kernarg. 2766 2767 "GlobalBuffer" 2768 A global address space pointer 2769 to the buffer data is passed 2770 in the kernarg. 2771 2772 "DynamicSharedPointer" 2773 A group address space pointer 2774 to dynamically allocated LDS 2775 is passed in the kernarg. 2776 2777 "Sampler" 2778 A global address space 2779 pointer to a S# is passed in 2780 the kernarg. 2781 2782 "Image" 2783 A global address space 2784 pointer to a T# is passed in 2785 the kernarg. 2786 2787 "Pipe" 2788 A global address space pointer 2789 to an OpenCL pipe is passed in 2790 the kernarg. 2791 2792 "Queue" 2793 A global address space pointer 2794 to an OpenCL device enqueue 2795 queue is passed in the 2796 kernarg. 2797 2798 "HiddenGlobalOffsetX" 2799 The OpenCL grid dispatch 2800 global offset for the X 2801 dimension is passed in the 2802 kernarg. 2803 2804 "HiddenGlobalOffsetY" 2805 The OpenCL grid dispatch 2806 global offset for the Y 2807 dimension is passed in the 2808 kernarg. 2809 2810 "HiddenGlobalOffsetZ" 2811 The OpenCL grid dispatch 2812 global offset for the Z 2813 dimension is passed in the 2814 kernarg. 2815 2816 "HiddenNone" 2817 An argument that is not used 2818 by the kernel. Space needs to 2819 be left for it, but it does 2820 not need to be set up. 2821 2822 "HiddenPrintfBuffer" 2823 A global address space pointer 2824 to the runtime printf buffer 2825 is passed in kernarg. 2826 2827 "HiddenHostcallBuffer" 2828 A global address space pointer 2829 to the runtime hostcall buffer 2830 is passed in kernarg. 2831 2832 "HiddenDefaultQueue" 2833 A global address space pointer 2834 to the OpenCL device enqueue 2835 queue that should be used by 2836 the kernel by default is 2837 passed in the kernarg. 2838 2839 "HiddenCompletionAction" 2840 A global address space pointer 2841 to help link enqueued kernels into 2842 the ancestor tree for determining 2843 when the parent kernel has finished. 2844 2845 "HiddenMultiGridSyncArg" 2846 A global address space pointer for 2847 multi-grid synchronization is 2848 passed in the kernarg. 2849 2850 "ValueType" string Unused and deprecated. This should no longer 2851 be emitted, but is accepted for compatibility. 2852 2853 2854 "PointeeAlign" integer Alignment in bytes of pointee 2855 type for pointer type kernel 2856 argument. Must be a power 2857 of 2. Only present if 2858 "ValueKind" is 2859 "DynamicSharedPointer". 2860 "AddrSpaceQual" string Kernel argument address space 2861 qualifier. Only present if 2862 "ValueKind" is "GlobalBuffer" or 2863 "DynamicSharedPointer". Values 2864 are: 2865 2866 - "Private" 2867 - "Global" 2868 - "Constant" 2869 - "Local" 2870 - "Generic" 2871 - "Region" 2872 2873 .. TODO:: 2874 2875 Is GlobalBuffer only Global 2876 or Constant? Is 2877 DynamicSharedPointer always 2878 Local? Can HCC allow Generic? 2879 How can Private or Region 2880 ever happen? 2881 2882 "AccQual" string Kernel argument access 2883 qualifier. Only present if 2884 "ValueKind" is "Image" or 2885 "Pipe". Values 2886 are: 2887 2888 - "ReadOnly" 2889 - "WriteOnly" 2890 - "ReadWrite" 2891 2892 .. TODO:: 2893 2894 Does this apply to 2895 GlobalBuffer? 2896 2897 "ActualAccQual" string The actual memory accesses 2898 performed by the kernel on the 2899 kernel argument. Only present if 2900 "ValueKind" is "GlobalBuffer", 2901 "Image", or "Pipe". This may be 2902 more restrictive than indicated 2903 by "AccQual" to reflect what the 2904 kernel actual does. If not 2905 present then the runtime must 2906 assume what is implied by 2907 "AccQual" and "IsConst". Values 2908 are: 2909 2910 - "ReadOnly" 2911 - "WriteOnly" 2912 - "ReadWrite" 2913 2914 "IsConst" boolean Indicates if the kernel argument 2915 is const qualified. Only present 2916 if "ValueKind" is 2917 "GlobalBuffer". 2918 2919 "IsRestrict" boolean Indicates if the kernel argument 2920 is restrict qualified. Only 2921 present if "ValueKind" is 2922 "GlobalBuffer". 2923 2924 "IsVolatile" boolean Indicates if the kernel argument 2925 is volatile qualified. Only 2926 present if "ValueKind" is 2927 "GlobalBuffer". 2928 2929 "IsPipe" boolean Indicates if the kernel argument 2930 is pipe qualified. Only present 2931 if "ValueKind" is "Pipe". 2932 2933 .. TODO:: 2934 2935 Can GlobalBuffer be pipe 2936 qualified? 2937 2938 ================= ============== ========= ================================ 2939 2940.. 2941 2942 .. table:: AMDHSA Code Object V2 Kernel Code Properties Metadata Map 2943 :name: amdgpu-amdhsa-code-object-kernel-code-properties-metadata-map-v2-table 2944 2945 ============================ ============== ========= ===================== 2946 String Key Value Type Required? Description 2947 ============================ ============== ========= ===================== 2948 "KernargSegmentSize" integer Required The size in bytes of 2949 the kernarg segment 2950 that holds the values 2951 of the arguments to 2952 the kernel. 2953 "GroupSegmentFixedSize" integer Required The amount of group 2954 segment memory 2955 required by a 2956 work-group in 2957 bytes. This does not 2958 include any 2959 dynamically allocated 2960 group segment memory 2961 that may be added 2962 when the kernel is 2963 dispatched. 2964 "PrivateSegmentFixedSize" integer Required The amount of fixed 2965 private address space 2966 memory required for a 2967 work-item in 2968 bytes. If the kernel 2969 uses a dynamic call 2970 stack then additional 2971 space must be added 2972 to this value for the 2973 call stack. 2974 "KernargSegmentAlign" integer Required The maximum byte 2975 alignment of 2976 arguments in the 2977 kernarg segment. Must 2978 be a power of 2. 2979 "WavefrontSize" integer Required Wavefront size. Must 2980 be a power of 2. 2981 "NumSGPRs" integer Required Number of scalar 2982 registers used by a 2983 wavefront for 2984 GFX6-GFX10. This 2985 includes the special 2986 SGPRs for VCC, Flat 2987 Scratch (GFX7-GFX10) 2988 and XNACK (for 2989 GFX8-GFX10). It does 2990 not include the 16 2991 SGPR added if a trap 2992 handler is 2993 enabled. It is not 2994 rounded up to the 2995 allocation 2996 granularity. 2997 "NumVGPRs" integer Required Number of vector 2998 registers used by 2999 each work-item for 3000 GFX6-GFX10 3001 "MaxFlatWorkGroupSize" integer Required Maximum flat 3002 work-group size 3003 supported by the 3004 kernel in work-items. 3005 Must be >=1 and 3006 consistent with 3007 ReqdWorkGroupSize if 3008 not 0, 0, 0. 3009 "NumSpilledSGPRs" integer Number of stores from 3010 a scalar register to 3011 a register allocator 3012 created spill 3013 location. 3014 "NumSpilledVGPRs" integer Number of stores from 3015 a vector register to 3016 a register allocator 3017 created spill 3018 location. 3019 ============================ ============== ========= ===================== 3020 3021.. _amdgpu-amdhsa-code-object-metadata-v3: 3022 3023Code Object V3 Metadata 3024+++++++++++++++++++++++ 3025 3026.. warning:: 3027 Code object V3 is not the default code object version emitted by this version 3028 of LLVM. 3029 3030Code object V3 and above metadata is specified by the ``NT_AMDGPU_METADATA`` note 3031record (see :ref:`amdgpu-note-records-v3-onwards`). 3032 3033The metadata is represented as Message Pack formatted binary data (see 3034[MsgPack]_). The top level is a Message Pack map that includes the 3035keys defined in table 3036:ref:`amdgpu-amdhsa-code-object-metadata-map-table-v3` and referenced 3037tables. 3038 3039Additional information can be added to the maps. To avoid conflicts, 3040any key names should be prefixed by "*vendor-name*." where 3041``vendor-name`` can be the name of the vendor and specific vendor 3042tool that generates the information. The prefix is abbreviated to 3043simply "." when it appears within a map that has been added by the 3044same *vendor-name*. 3045 3046 .. table:: AMDHSA Code Object V3 Metadata Map 3047 :name: amdgpu-amdhsa-code-object-metadata-map-table-v3 3048 3049 ================= ============== ========= ======================================= 3050 String Key Value Type Required? Description 3051 ================= ============== ========= ======================================= 3052 "amdhsa.version" sequence of Required - The first integer is the major 3053 2 integers version. Currently 1. 3054 - The second integer is the minor 3055 version. Currently 0. 3056 "amdhsa.printf" sequence of Each string is encoded information 3057 strings about a printf function call. The 3058 encoded information is organized as 3059 fields separated by colon (':'): 3060 3061 ``ID:N:S[0]:S[1]:...:S[N-1]:FormatString`` 3062 3063 where: 3064 3065 ``ID`` 3066 A 32-bit integer as a unique id for 3067 each printf function call 3068 3069 ``N`` 3070 A 32-bit integer equal to the number 3071 of arguments of printf function call 3072 minus 1 3073 3074 ``S[i]`` (where i = 0, 1, ... , N-1) 3075 32-bit integers for the size in bytes 3076 of the i-th FormatString argument of 3077 the printf function call 3078 3079 FormatString 3080 The format string passed to the 3081 printf function call. 3082 "amdhsa.kernels" sequence of Required Sequence of the maps for each 3083 map kernel in the code object. See 3084 :ref:`amdgpu-amdhsa-code-object-kernel-metadata-map-table-v3` 3085 for the definition of the keys included 3086 in that map. 3087 ================= ============== ========= ======================================= 3088 3089.. 3090 3091 .. table:: AMDHSA Code Object V3 Kernel Metadata Map 3092 :name: amdgpu-amdhsa-code-object-kernel-metadata-map-table-v3 3093 3094 =================================== ============== ========= ================================ 3095 String Key Value Type Required? Description 3096 =================================== ============== ========= ================================ 3097 ".name" string Required Source name of the kernel. 3098 ".symbol" string Required Name of the kernel 3099 descriptor ELF symbol. 3100 ".language" string Source language of the kernel. 3101 Values include: 3102 3103 - "OpenCL C" 3104 - "OpenCL C++" 3105 - "HCC" 3106 - "HIP" 3107 - "OpenMP" 3108 - "Assembler" 3109 3110 ".language_version" sequence of - The first integer is the major 3111 2 integers version. 3112 - The second integer is the 3113 minor version. 3114 ".args" sequence of Sequence of maps of the 3115 map kernel arguments. See 3116 :ref:`amdgpu-amdhsa-code-object-kernel-argument-metadata-map-table-v3` 3117 for the definition of the keys 3118 included in that map. 3119 ".reqd_workgroup_size" sequence of If not 0, 0, 0 then all values 3120 3 integers must be >=1 and the dispatch 3121 work-group size X, Y, Z must 3122 correspond to the specified 3123 values. Defaults to 0, 0, 0. 3124 3125 Corresponds to the OpenCL 3126 ``reqd_work_group_size`` 3127 attribute. 3128 ".workgroup_size_hint" sequence of The dispatch work-group size 3129 3 integers X, Y, Z is likely to be the 3130 specified values. 3131 3132 Corresponds to the OpenCL 3133 ``work_group_size_hint`` 3134 attribute. 3135 ".vec_type_hint" string The name of a scalar or vector 3136 type. 3137 3138 Corresponds to the OpenCL 3139 ``vec_type_hint`` attribute. 3140 3141 ".device_enqueue_symbol" string The external symbol name 3142 associated with a kernel. 3143 OpenCL runtime allocates a 3144 global buffer for the symbol 3145 and saves the kernel's address 3146 to it, which is used for 3147 device side enqueueing. Only 3148 available for device side 3149 enqueued kernels. 3150 ".kernarg_segment_size" integer Required The size in bytes of 3151 the kernarg segment 3152 that holds the values 3153 of the arguments to 3154 the kernel. 3155 ".group_segment_fixed_size" integer Required The amount of group 3156 segment memory 3157 required by a 3158 work-group in 3159 bytes. This does not 3160 include any 3161 dynamically allocated 3162 group segment memory 3163 that may be added 3164 when the kernel is 3165 dispatched. 3166 ".private_segment_fixed_size" integer Required The amount of fixed 3167 private address space 3168 memory required for a 3169 work-item in 3170 bytes. If the kernel 3171 uses a dynamic call 3172 stack then additional 3173 space must be added 3174 to this value for the 3175 call stack. 3176 ".kernarg_segment_align" integer Required The maximum byte 3177 alignment of 3178 arguments in the 3179 kernarg segment. Must 3180 be a power of 2. 3181 ".wavefront_size" integer Required Wavefront size. Must 3182 be a power of 2. 3183 ".sgpr_count" integer Required Number of scalar 3184 registers required by a 3185 wavefront for 3186 GFX6-GFX9. A register 3187 is required if it is 3188 used explicitly, or 3189 if a higher numbered 3190 register is used 3191 explicitly. This 3192 includes the special 3193 SGPRs for VCC, Flat 3194 Scratch (GFX7-GFX9) 3195 and XNACK (for 3196 GFX8-GFX9). It does 3197 not include the 16 3198 SGPR added if a trap 3199 handler is 3200 enabled. It is not 3201 rounded up to the 3202 allocation 3203 granularity. 3204 ".vgpr_count" integer Required Number of vector 3205 registers required by 3206 each work-item for 3207 GFX6-GFX9. A register 3208 is required if it is 3209 used explicitly, or 3210 if a higher numbered 3211 register is used 3212 explicitly. 3213 ".agpr_count" integer Required Number of accumulator 3214 registers required by 3215 each work-item for 3216 GFX90A, GFX908. 3217 ".max_flat_workgroup_size" integer Required Maximum flat 3218 work-group size 3219 supported by the 3220 kernel in work-items. 3221 Must be >=1 and 3222 consistent with 3223 ReqdWorkGroupSize if 3224 not 0, 0, 0. 3225 ".sgpr_spill_count" integer Number of stores from 3226 a scalar register to 3227 a register allocator 3228 created spill 3229 location. 3230 ".vgpr_spill_count" integer Number of stores from 3231 a vector register to 3232 a register allocator 3233 created spill 3234 location. 3235 ".kind" string The kind of the kernel 3236 with the following 3237 values: 3238 3239 "normal" 3240 Regular kernels. 3241 3242 "init" 3243 These kernels must be 3244 invoked after loading 3245 the containing code 3246 object and must 3247 complete before any 3248 normal and fini 3249 kernels in the same 3250 code object are 3251 invoked. 3252 3253 "fini" 3254 These kernels must be 3255 invoked before 3256 unloading the 3257 containing code object 3258 and after all init and 3259 normal kernels in the 3260 same code object have 3261 been invoked and 3262 completed. 3263 3264 If omitted, "normal" is 3265 assumed. 3266 =================================== ============== ========= ================================ 3267 3268.. 3269 3270 .. table:: AMDHSA Code Object V3 Kernel Argument Metadata Map 3271 :name: amdgpu-amdhsa-code-object-kernel-argument-metadata-map-table-v3 3272 3273 ====================== ============== ========= ================================ 3274 String Key Value Type Required? Description 3275 ====================== ============== ========= ================================ 3276 ".name" string Kernel argument name. 3277 ".type_name" string Kernel argument type name. 3278 ".size" integer Required Kernel argument size in bytes. 3279 ".offset" integer Required Kernel argument offset in 3280 bytes. The offset must be a 3281 multiple of the alignment 3282 required by the argument. 3283 ".value_kind" string Required Kernel argument kind that 3284 specifies how to set up the 3285 corresponding argument. 3286 Values include: 3287 3288 "by_value" 3289 The argument is copied 3290 directly into the kernarg. 3291 3292 "global_buffer" 3293 A global address space pointer 3294 to the buffer data is passed 3295 in the kernarg. 3296 3297 "dynamic_shared_pointer" 3298 A group address space pointer 3299 to dynamically allocated LDS 3300 is passed in the kernarg. 3301 3302 "sampler" 3303 A global address space 3304 pointer to a S# is passed in 3305 the kernarg. 3306 3307 "image" 3308 A global address space 3309 pointer to a T# is passed in 3310 the kernarg. 3311 3312 "pipe" 3313 A global address space pointer 3314 to an OpenCL pipe is passed in 3315 the kernarg. 3316 3317 "queue" 3318 A global address space pointer 3319 to an OpenCL device enqueue 3320 queue is passed in the 3321 kernarg. 3322 3323 "hidden_global_offset_x" 3324 The OpenCL grid dispatch 3325 global offset for the X 3326 dimension is passed in the 3327 kernarg. 3328 3329 "hidden_global_offset_y" 3330 The OpenCL grid dispatch 3331 global offset for the Y 3332 dimension is passed in the 3333 kernarg. 3334 3335 "hidden_global_offset_z" 3336 The OpenCL grid dispatch 3337 global offset for the Z 3338 dimension is passed in the 3339 kernarg. 3340 3341 "hidden_none" 3342 An argument that is not used 3343 by the kernel. Space needs to 3344 be left for it, but it does 3345 not need to be set up. 3346 3347 "hidden_printf_buffer" 3348 A global address space pointer 3349 to the runtime printf buffer 3350 is passed in kernarg. 3351 3352 "hidden_hostcall_buffer" 3353 A global address space pointer 3354 to the runtime hostcall buffer 3355 is passed in kernarg. 3356 3357 "hidden_default_queue" 3358 A global address space pointer 3359 to the OpenCL device enqueue 3360 queue that should be used by 3361 the kernel by default is 3362 passed in the kernarg. 3363 3364 "hidden_completion_action" 3365 A global address space pointer 3366 to help link enqueued kernels into 3367 the ancestor tree for determining 3368 when the parent kernel has finished. 3369 3370 "hidden_multigrid_sync_arg" 3371 A global address space pointer for 3372 multi-grid synchronization is 3373 passed in the kernarg. 3374 3375 ".value_type" string Unused and deprecated. This should no longer 3376 be emitted, but is accepted for compatibility. 3377 3378 ".pointee_align" integer Alignment in bytes of pointee 3379 type for pointer type kernel 3380 argument. Must be a power 3381 of 2. Only present if 3382 ".value_kind" is 3383 "dynamic_shared_pointer". 3384 ".address_space" string Kernel argument address space 3385 qualifier. Only present if 3386 ".value_kind" is "global_buffer" or 3387 "dynamic_shared_pointer". Values 3388 are: 3389 3390 - "private" 3391 - "global" 3392 - "constant" 3393 - "local" 3394 - "generic" 3395 - "region" 3396 3397 .. TODO:: 3398 3399 Is "global_buffer" only "global" 3400 or "constant"? Is 3401 "dynamic_shared_pointer" always 3402 "local"? Can HCC allow "generic"? 3403 How can "private" or "region" 3404 ever happen? 3405 3406 ".access" string Kernel argument access 3407 qualifier. Only present if 3408 ".value_kind" is "image" or 3409 "pipe". Values 3410 are: 3411 3412 - "read_only" 3413 - "write_only" 3414 - "read_write" 3415 3416 .. TODO:: 3417 3418 Does this apply to 3419 "global_buffer"? 3420 3421 ".actual_access" string The actual memory accesses 3422 performed by the kernel on the 3423 kernel argument. Only present if 3424 ".value_kind" is "global_buffer", 3425 "image", or "pipe". This may be 3426 more restrictive than indicated 3427 by ".access" to reflect what the 3428 kernel actual does. If not 3429 present then the runtime must 3430 assume what is implied by 3431 ".access" and ".is_const" . Values 3432 are: 3433 3434 - "read_only" 3435 - "write_only" 3436 - "read_write" 3437 3438 ".is_const" boolean Indicates if the kernel argument 3439 is const qualified. Only present 3440 if ".value_kind" is 3441 "global_buffer". 3442 3443 ".is_restrict" boolean Indicates if the kernel argument 3444 is restrict qualified. Only 3445 present if ".value_kind" is 3446 "global_buffer". 3447 3448 ".is_volatile" boolean Indicates if the kernel argument 3449 is volatile qualified. Only 3450 present if ".value_kind" is 3451 "global_buffer". 3452 3453 ".is_pipe" boolean Indicates if the kernel argument 3454 is pipe qualified. Only present 3455 if ".value_kind" is "pipe". 3456 3457 .. TODO:: 3458 3459 Can "global_buffer" be pipe 3460 qualified? 3461 3462 ====================== ============== ========= ================================ 3463 3464.. _amdgpu-amdhsa-code-object-metadata-v4: 3465 3466Code Object V4 Metadata 3467+++++++++++++++++++++++ 3468 3469Code object V4 metadata is the same as 3470:ref:`amdgpu-amdhsa-code-object-metadata-v3` with the changes and additions 3471defined in table :ref:`amdgpu-amdhsa-code-object-metadata-map-table-v4`. 3472 3473 .. table:: AMDHSA Code Object V4 Metadata Map Changes 3474 :name: amdgpu-amdhsa-code-object-metadata-map-table-v4 3475 3476 ================= ============== ========= ======================================= 3477 String Key Value Type Required? Description 3478 ================= ============== ========= ======================================= 3479 "amdhsa.version" sequence of Required - The first integer is the major 3480 2 integers version. Currently 1. 3481 - The second integer is the minor 3482 version. Currently 1. 3483 "amdhsa.target" string Required The target name of the code using the syntax: 3484 3485 .. code:: 3486 3487 <target-triple> [ "-" <target-id> ] 3488 3489 A canonical target ID must be 3490 used. See :ref:`amdgpu-target-triples` 3491 and :ref:`amdgpu-target-id`. 3492 ================= ============== ========= ======================================= 3493 3494.. _amdgpu-amdhsa-code-object-metadata-v5: 3495 3496Code Object V5 Metadata 3497+++++++++++++++++++++++ 3498 3499.. warning:: 3500 Code object V5 is not the default code object version emitted by this version 3501 of LLVM. 3502 3503 3504Code object V5 metadata is the same as 3505:ref:`amdgpu-amdhsa-code-object-metadata-v4` with the changes defined in table 3506:ref:`amdgpu-amdhsa-code-object-metadata-map-table-v5` and table 3507:ref:`amdgpu-amdhsa-code-object-kernel-argument-metadata-map-table-v5`. 3508 3509 .. table:: AMDHSA Code Object V5 Metadata Map Changes 3510 :name: amdgpu-amdhsa-code-object-metadata-map-table-v5 3511 3512 ================= ============== ========= ======================================= 3513 String Key Value Type Required? Description 3514 ================= ============== ========= ======================================= 3515 "amdhsa.version" sequence of Required - The first integer is the major 3516 2 integers version. Currently 1. 3517 - The second integer is the minor 3518 version. Currently 2. 3519 ================= ============== ========= ======================================= 3520 3521.. 3522 3523 .. table:: AMDHSA Code Object V5 Kernel Argument Metadata Map Additions and Changes 3524 :name: amdgpu-amdhsa-code-object-kernel-argument-metadata-map-table-v5 3525 3526 ====================== ============== ========= ================================ 3527 String Key Value Type Required? Description 3528 ====================== ============== ========= ================================ 3529 ".value_kind" string Required Kernel argument kind that 3530 specifies how to set up the 3531 corresponding argument. 3532 Values include: 3533 the same as code object V3 metadata 3534 (see :ref:`amdgpu-amdhsa-code-object-kernel-argument-metadata-map-table-v3`) 3535 with the following additions: 3536 3537 "hidden_block_count_x" 3538 The grid dispatch work-group count for the X dimension 3539 is passed in the kernarg. Some languages, such as OpenCL, 3540 support a last work-group in each dimension being partial. 3541 This count only includes the non-partial work-group count. 3542 This is not the same as the value in the AQL dispatch packet, 3543 which has the grid size in work-items. 3544 3545 "hidden_block_count_y" 3546 The grid dispatch work-group count for the Y dimension 3547 is passed in the kernarg. Some languages, such as OpenCL, 3548 support a last work-group in each dimension being partial. 3549 This count only includes the non-partial work-group count. 3550 This is not the same as the value in the AQL dispatch packet, 3551 which has the grid size in work-items. If the grid dimensionality 3552 is 1, then must be 1. 3553 3554 "hidden_block_count_z" 3555 The grid dispatch work-group count for the Z dimension 3556 is passed in the kernarg. Some languages, such as OpenCL, 3557 support a last work-group in each dimension being partial. 3558 This count only includes the non-partial work-group count. 3559 This is not the same as the value in the AQL dispatch packet, 3560 which has the grid size in work-items. If the grid dimensionality 3561 is 1 or 2, then must be 1. 3562 3563 "hidden_group_size_x" 3564 The grid dispatch work-group size for the X dimension is 3565 passed in the kernarg. This size only applies to the 3566 non-partial work-groups. This is the same value as the AQL 3567 dispatch packet work-group size. 3568 3569 "hidden_group_size_y" 3570 The grid dispatch work-group size for the Y dimension is 3571 passed in the kernarg. This size only applies to the 3572 non-partial work-groups. This is the same value as the AQL 3573 dispatch packet work-group size. If the grid dimensionality 3574 is 1, then must be 1. 3575 3576 "hidden_group_size_z" 3577 The grid dispatch work-group size for the Z dimension is 3578 passed in the kernarg. This size only applies to the 3579 non-partial work-groups. This is the same value as the AQL 3580 dispatch packet work-group size. If the grid dimensionality 3581 is 1 or 2, then must be 1. 3582 3583 "hidden_remainder_x" 3584 The grid dispatch work group size of the the partial work group 3585 of the X dimension, if it exists. Must be zero if a partial 3586 work group does not exist in the X dimension. 3587 3588 "hidden_remainder_y" 3589 The grid dispatch work group size of the the partial work group 3590 of the Y dimension, if it exists. Must be zero if a partial 3591 work group does not exist in the Y dimension. 3592 3593 "hidden_remainder_z" 3594 The grid dispatch work group size of the the partial work group 3595 of the Z dimension, if it exists. Must be zero if a partial 3596 work group does not exist in the Z dimension. 3597 3598 "hidden_grid_dims" 3599 The grid dispatch dimensionality. This is the same value 3600 as the AQL dispatch packet dimensionality. Must be a value 3601 between 1 and 3. 3602 3603 "hidden_heap_v1" 3604 A global address space pointer to an initialized memory 3605 buffer that conforms to the requirements of the malloc/free 3606 device library V1 version implementation. 3607 3608 "hidden_private_base" 3609 The high 32 bits of the flat addressing private aperture base. 3610 Only used by GFX8 to allow conversion between private segment 3611 and flat addresses. See :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`. 3612 3613 "hidden_shared_base" 3614 The high 32 bits of the flat addressing shared aperture base. 3615 Only used by GFX8 to allow conversion between shared segment 3616 and flat addresses. See :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`. 3617 3618 "hidden_queue_ptr" 3619 A global memory address space pointer to the ROCm runtime 3620 ``struct amd_queue_t`` structure for the HSA queue of the 3621 associated dispatch AQL packet. It is only required for pre-GFX9 3622 devices for the trap handler ABI (see :ref:`amdgpu-amdhsa-trap-handler-abi`). 3623 3624 ====================== ============== ========= ================================ 3625 3626.. 3627 3628Kernel Dispatch 3629~~~~~~~~~~~~~~~ 3630 3631The HSA architected queuing language (AQL) defines a user space memory interface 3632that can be used to control the dispatch of kernels, in an agent independent 3633way. An agent can have zero or more AQL queues created for it using an HSA 3634compatible runtime (see :ref:`amdgpu-os`), in which AQL packets (all of which 3635are 64 bytes) can be placed. See the *HSA Platform System Architecture 3636Specification* [HSA]_ for the AQL queue mechanics and packet layouts. 3637 3638The packet processor of a kernel agent is responsible for detecting and 3639dispatching HSA kernels from the AQL queues associated with it. For AMD GPUs the 3640packet processor is implemented by the hardware command processor (CP), 3641asynchronous dispatch controller (ADC) and shader processor input controller 3642(SPI). 3643 3644An HSA compatible runtime can be used to allocate an AQL queue object. It uses 3645the kernel mode driver to initialize and register the AQL queue with CP. 3646 3647To dispatch a kernel the following actions are performed. This can occur in the 3648CPU host program, or from an HSA kernel executing on a GPU. 3649 36501. A pointer to an AQL queue for the kernel agent on which the kernel is to be 3651 executed is obtained. 36522. A pointer to the kernel descriptor (see 3653 :ref:`amdgpu-amdhsa-kernel-descriptor`) of the kernel to execute is obtained. 3654 It must be for a kernel that is contained in a code object that was loaded 3655 by an HSA compatible runtime on the kernel agent with which the AQL queue is 3656 associated. 36573. Space is allocated for the kernel arguments using the HSA compatible runtime 3658 allocator for a memory region with the kernarg property for the kernel agent 3659 that will execute the kernel. It must be at least 16-byte aligned. 36604. Kernel argument values are assigned to the kernel argument memory 3661 allocation. The layout is defined in the *HSA Programmer's Language 3662 Reference* [HSA]_. For AMDGPU the kernel execution directly accesses the 3663 kernel argument memory in the same way constant memory is accessed. (Note 3664 that the HSA specification allows an implementation to copy the kernel 3665 argument contents to another location that is accessed by the kernel.) 36665. An AQL kernel dispatch packet is created on the AQL queue. The HSA compatible 3667 runtime api uses 64-bit atomic operations to reserve space in the AQL queue 3668 for the packet. The packet must be set up, and the final write must use an 3669 atomic store release to set the packet kind to ensure the packet contents are 3670 visible to the kernel agent. AQL defines a doorbell signal mechanism to 3671 notify the kernel agent that the AQL queue has been updated. These rules, and 3672 the layout of the AQL queue and kernel dispatch packet is defined in the *HSA 3673 System Architecture Specification* [HSA]_. 36746. A kernel dispatch packet includes information about the actual dispatch, 3675 such as grid and work-group size, together with information from the code 3676 object about the kernel, such as segment sizes. The HSA compatible runtime 3677 queries on the kernel symbol can be used to obtain the code object values 3678 which are recorded in the :ref:`amdgpu-amdhsa-code-object-metadata`. 36797. CP executes micro-code and is responsible for detecting and setting up the 3680 GPU to execute the wavefronts of a kernel dispatch. 36818. CP ensures that when the a wavefront starts executing the kernel machine 3682 code, the scalar general purpose registers (SGPR) and vector general purpose 3683 registers (VGPR) are set up as required by the machine code. The required 3684 setup is defined in the :ref:`amdgpu-amdhsa-kernel-descriptor`. The initial 3685 register state is defined in 3686 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`. 36879. The prolog of the kernel machine code (see 3688 :ref:`amdgpu-amdhsa-kernel-prolog`) sets up the machine state as necessary 3689 before continuing executing the machine code that corresponds to the kernel. 369010. When the kernel dispatch has completed execution, CP signals the completion 3691 signal specified in the kernel dispatch packet if not 0. 3692 3693.. _amdgpu-amdhsa-memory-spaces: 3694 3695Memory Spaces 3696~~~~~~~~~~~~~ 3697 3698The memory space properties are: 3699 3700 .. table:: AMDHSA Memory Spaces 3701 :name: amdgpu-amdhsa-memory-spaces-table 3702 3703 ================= =========== ======== ======= ================== 3704 Memory Space Name HSA Segment Hardware Address NULL Value 3705 Name Name Size 3706 ================= =========== ======== ======= ================== 3707 Private private scratch 32 0x00000000 3708 Local group LDS 32 0xFFFFFFFF 3709 Global global global 64 0x0000000000000000 3710 Constant constant *same as 64 0x0000000000000000 3711 global* 3712 Generic flat flat 64 0x0000000000000000 3713 Region N/A GDS 32 *not implemented 3714 for AMDHSA* 3715 ================= =========== ======== ======= ================== 3716 3717The global and constant memory spaces both use global virtual addresses, which 3718are the same virtual address space used by the CPU. However, some virtual 3719addresses may only be accessible to the CPU, some only accessible by the GPU, 3720and some by both. 3721 3722Using the constant memory space indicates that the data will not change during 3723the execution of the kernel. This allows scalar read instructions to be 3724used. The vector and scalar L1 caches are invalidated of volatile data before 3725each kernel dispatch execution to allow constant memory to change values between 3726kernel dispatches. 3727 3728The local memory space uses the hardware Local Data Store (LDS) which is 3729automatically allocated when the hardware creates work-groups of wavefronts, and 3730freed when all the wavefronts of a work-group have terminated. The data store 3731(DS) instructions can be used to access it. 3732 3733The private memory space uses the hardware scratch memory support. If the kernel 3734uses scratch, then the hardware allocates memory that is accessed using 3735wavefront lane dword (4 byte) interleaving. The mapping used from private 3736address to physical address is: 3737 3738 ``wavefront-scratch-base + 3739 (private-address * wavefront-size * 4) + 3740 (wavefront-lane-id * 4)`` 3741 3742There are different ways that the wavefront scratch base address is determined 3743by a wavefront (see :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). This 3744memory can be accessed in an interleaved manner using buffer instruction with 3745the scratch buffer descriptor and per wavefront scratch offset, by the scratch 3746instructions, or by flat instructions. If each lane of a wavefront accesses the 3747same private address, the interleaving results in adjacent dwords being accessed 3748and hence requires fewer cache lines to be fetched. Multi-dword access is not 3749supported except by flat and scratch instructions in GFX9-GFX10. 3750 3751The generic address space uses the hardware flat address support available in 3752GFX7-GFX10. This uses two fixed ranges of virtual addresses (the private and 3753local apertures), that are outside the range of addressible global memory, to 3754map from a flat address to a private or local address. 3755 3756FLAT instructions can take a flat address and access global, private (scratch) 3757and group (LDS) memory depending on if the address is within one of the 3758aperture ranges. Flat access to scratch requires hardware aperture setup and 3759setup in the kernel prologue (see 3760:ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`). Flat access to LDS requires 3761hardware aperture setup and M0 (GFX7-GFX8) register setup (see 3762:ref:`amdgpu-amdhsa-kernel-prolog-m0`). 3763 3764To convert between a segment address and a flat address the base address of the 3765apertures address can be used. For GFX7-GFX8 these are available in the 3766:ref:`amdgpu-amdhsa-hsa-aql-queue` the address of which can be obtained with 3767Queue Ptr SGPR (see :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). For 3768GFX9-GFX10 the aperture base addresses are directly available as inline constant 3769registers ``SRC_SHARED_BASE/LIMIT`` and ``SRC_PRIVATE_BASE/LIMIT``. In 64 bit 3770address mode the aperture sizes are 2^32 bytes and the base is aligned to 2^32 3771which makes it easier to convert from flat to segment or segment to flat. 3772 3773Image and Samplers 3774~~~~~~~~~~~~~~~~~~ 3775 3776Image and sample handles created by an HSA compatible runtime (see 3777:ref:`amdgpu-os`) are 64-bit addresses of a hardware 32-byte V# and 48 byte S# 3778object respectively. In order to support the HSA ``query_sampler`` operations 3779two extra dwords are used to store the HSA BRIG enumeration values for the 3780queries that are not trivially deducible from the S# representation. 3781 3782HSA Signals 3783~~~~~~~~~~~ 3784 3785HSA signal handles created by an HSA compatible runtime (see :ref:`amdgpu-os`) 3786are 64-bit addresses of a structure allocated in memory accessible from both the 3787CPU and GPU. The structure is defined by the runtime and subject to change 3788between releases. For example, see [AMD-ROCm-github]_. 3789 3790.. _amdgpu-amdhsa-hsa-aql-queue: 3791 3792HSA AQL Queue 3793~~~~~~~~~~~~~ 3794 3795The HSA AQL queue structure is defined by an HSA compatible runtime (see 3796:ref:`amdgpu-os`) and subject to change between releases. For example, see 3797[AMD-ROCm-github]_. For some processors it contains fields needed to implement 3798certain language features such as the flat address aperture bases. It also 3799contains fields used by CP such as managing the allocation of scratch memory. 3800 3801.. _amdgpu-amdhsa-kernel-descriptor: 3802 3803Kernel Descriptor 3804~~~~~~~~~~~~~~~~~ 3805 3806A kernel descriptor consists of the information needed by CP to initiate the 3807execution of a kernel, including the entry point address of the machine code 3808that implements the kernel. 3809 3810Code Object V3 Kernel Descriptor 3811++++++++++++++++++++++++++++++++ 3812 3813CP microcode requires the Kernel descriptor to be allocated on 64-byte 3814alignment. 3815 3816The fields used by CP for code objects before V3 also match those specified in 3817:ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 3818 3819 .. table:: Code Object V3 Kernel Descriptor 3820 :name: amdgpu-amdhsa-kernel-descriptor-v3-table 3821 3822 ======= ======= =============================== ============================ 3823 Bits Size Field Name Description 3824 ======= ======= =============================== ============================ 3825 31:0 4 bytes GROUP_SEGMENT_FIXED_SIZE The amount of fixed local 3826 address space memory 3827 required for a work-group 3828 in bytes. This does not 3829 include any dynamically 3830 allocated local address 3831 space memory that may be 3832 added when the kernel is 3833 dispatched. 3834 63:32 4 bytes PRIVATE_SEGMENT_FIXED_SIZE The amount of fixed 3835 private address space 3836 memory required for a 3837 work-item in bytes. 3838 Additional space may need to 3839 be added to this value if 3840 the call stack has 3841 non-inlined function calls. 3842 95:64 4 bytes KERNARG_SIZE The size of the kernarg 3843 memory pointed to by the 3844 AQL dispatch packet. The 3845 kernarg memory is used to 3846 pass arguments to the 3847 kernel. 3848 3849 * If the kernarg pointer in 3850 the dispatch packet is NULL 3851 then there are no kernel 3852 arguments. 3853 * If the kernarg pointer in 3854 the dispatch packet is 3855 not NULL and this value 3856 is 0 then the kernarg 3857 memory size is 3858 unspecified. 3859 * If the kernarg pointer in 3860 the dispatch packet is 3861 not NULL and this value 3862 is not 0 then the value 3863 specifies the kernarg 3864 memory size in bytes. It 3865 is recommended to provide 3866 a value as it may be used 3867 by CP to optimize making 3868 the kernarg memory 3869 visible to the kernel 3870 code. 3871 3872 127:96 4 bytes Reserved, must be 0. 3873 191:128 8 bytes KERNEL_CODE_ENTRY_BYTE_OFFSET Byte offset (possibly 3874 negative) from base 3875 address of kernel 3876 descriptor to kernel's 3877 entry point instruction 3878 which must be 256 byte 3879 aligned. 3880 351:272 20 Reserved, must be 0. 3881 bytes 3882 383:352 4 bytes COMPUTE_PGM_RSRC3 GFX6-GFX9 3883 Reserved, must be 0. 3884 GFX90A, GFX940 3885 Compute Shader (CS) 3886 program settings used by 3887 CP to set up 3888 ``COMPUTE_PGM_RSRC3`` 3889 configuration 3890 register. See 3891 :ref:`amdgpu-amdhsa-compute_pgm_rsrc3-gfx90a-table`. 3892 GFX10 3893 Compute Shader (CS) 3894 program settings used by 3895 CP to set up 3896 ``COMPUTE_PGM_RSRC3`` 3897 configuration 3898 register. See 3899 :ref:`amdgpu-amdhsa-compute_pgm_rsrc3-gfx10-table`. 3900 415:384 4 bytes COMPUTE_PGM_RSRC1 Compute Shader (CS) 3901 program settings used by 3902 CP to set up 3903 ``COMPUTE_PGM_RSRC1`` 3904 configuration 3905 register. See 3906 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 3907 447:416 4 bytes COMPUTE_PGM_RSRC2 Compute Shader (CS) 3908 program settings used by 3909 CP to set up 3910 ``COMPUTE_PGM_RSRC2`` 3911 configuration 3912 register. See 3913 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 3914 458:448 7 bits *See separate bits below.* Enable the setup of the 3915 SGPR user data registers 3916 (see 3917 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 3918 3919 The total number of SGPR 3920 user data registers 3921 requested must not exceed 3922 16 and match value in 3923 ``compute_pgm_rsrc2.user_sgpr.user_sgpr_count``. 3924 Any requests beyond 16 3925 will be ignored. 3926 >448 1 bit ENABLE_SGPR_PRIVATE_SEGMENT If the *Target Properties* 3927 _BUFFER column of 3928 :ref:`amdgpu-processor-table` 3929 specifies *Architected flat 3930 scratch* then not supported 3931 and must be 0, 3932 >449 1 bit ENABLE_SGPR_DISPATCH_PTR 3933 >450 1 bit ENABLE_SGPR_QUEUE_PTR 3934 >451 1 bit ENABLE_SGPR_KERNARG_SEGMENT_PTR 3935 >452 1 bit ENABLE_SGPR_DISPATCH_ID 3936 >453 1 bit ENABLE_SGPR_FLAT_SCRATCH_INIT If the *Target Properties* 3937 column of 3938 :ref:`amdgpu-processor-table` 3939 specifies *Architected flat 3940 scratch* then not supported 3941 and must be 0, 3942 >454 1 bit ENABLE_SGPR_PRIVATE_SEGMENT 3943 _SIZE 3944 457:455 3 bits Reserved, must be 0. 3945 458 1 bit ENABLE_WAVEFRONT_SIZE32 GFX6-GFX9 3946 Reserved, must be 0. 3947 GFX10 3948 - If 0 execute in 3949 wavefront size 64 mode. 3950 - If 1 execute in 3951 native wavefront size 3952 32 mode. 3953 463:459 1 bit Reserved, must be 0. 3954 464 1 bit RESERVED_464 Deprecated, must be 0. 3955 467:465 3 bits Reserved, must be 0. 3956 468 1 bit RESERVED_468 Deprecated, must be 0. 3957 469:471 3 bits Reserved, must be 0. 3958 511:472 5 bytes Reserved, must be 0. 3959 512 **Total size 64 bytes.** 3960 ======= ==================================================================== 3961 3962.. 3963 3964 .. table:: compute_pgm_rsrc1 for GFX6-GFX10 3965 :name: amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table 3966 3967 ======= ======= =============================== =========================================================================== 3968 Bits Size Field Name Description 3969 ======= ======= =============================== =========================================================================== 3970 5:0 6 bits GRANULATED_WORKITEM_VGPR_COUNT Number of vector register 3971 blocks used by each work-item; 3972 granularity is device 3973 specific: 3974 3975 GFX6-GFX9 3976 - vgprs_used 0..256 3977 - max(0, ceil(vgprs_used / 4) - 1) 3978 GFX90A, GFX940 3979 - vgprs_used 0..512 3980 - vgprs_used = align(arch_vgprs, 4) 3981 + acc_vgprs 3982 - max(0, ceil(vgprs_used / 8) - 1) 3983 GFX10 (wavefront size 64) 3984 - max_vgpr 1..256 3985 - max(0, ceil(vgprs_used / 4) - 1) 3986 GFX10 (wavefront size 32) 3987 - max_vgpr 1..256 3988 - max(0, ceil(vgprs_used / 8) - 1) 3989 3990 Where vgprs_used is defined 3991 as the highest VGPR number 3992 explicitly referenced plus 3993 one. 3994 3995 Used by CP to set up 3996 ``COMPUTE_PGM_RSRC1.VGPRS``. 3997 3998 The 3999 :ref:`amdgpu-assembler` 4000 calculates this 4001 automatically for the 4002 selected processor from 4003 values provided to the 4004 `.amdhsa_kernel` directive 4005 by the 4006 `.amdhsa_next_free_vgpr` 4007 nested directive (see 4008 :ref:`amdhsa-kernel-directives-table`). 4009 9:6 4 bits GRANULATED_WAVEFRONT_SGPR_COUNT Number of scalar register 4010 blocks used by a wavefront; 4011 granularity is device 4012 specific: 4013 4014 GFX6-GFX8 4015 - sgprs_used 0..112 4016 - max(0, ceil(sgprs_used / 8) - 1) 4017 GFX9 4018 - sgprs_used 0..112 4019 - 2 * max(0, ceil(sgprs_used / 16) - 1) 4020 GFX10 4021 Reserved, must be 0. 4022 (128 SGPRs always 4023 allocated.) 4024 4025 Where sgprs_used is 4026 defined as the highest 4027 SGPR number explicitly 4028 referenced plus one, plus 4029 a target specific number 4030 of additional special 4031 SGPRs for VCC, 4032 FLAT_SCRATCH (GFX7+) and 4033 XNACK_MASK (GFX8+), and 4034 any additional 4035 target specific 4036 limitations. It does not 4037 include the 16 SGPRs added 4038 if a trap handler is 4039 enabled. 4040 4041 The target specific 4042 limitations and special 4043 SGPR layout are defined in 4044 the hardware 4045 documentation, which can 4046 be found in the 4047 :ref:`amdgpu-processors` 4048 table. 4049 4050 Used by CP to set up 4051 ``COMPUTE_PGM_RSRC1.SGPRS``. 4052 4053 The 4054 :ref:`amdgpu-assembler` 4055 calculates this 4056 automatically for the 4057 selected processor from 4058 values provided to the 4059 `.amdhsa_kernel` directive 4060 by the 4061 `.amdhsa_next_free_sgpr` 4062 and `.amdhsa_reserve_*` 4063 nested directives (see 4064 :ref:`amdhsa-kernel-directives-table`). 4065 11:10 2 bits PRIORITY Must be 0. 4066 4067 Start executing wavefront 4068 at the specified priority. 4069 4070 CP is responsible for 4071 filling in 4072 ``COMPUTE_PGM_RSRC1.PRIORITY``. 4073 13:12 2 bits FLOAT_ROUND_MODE_32 Wavefront starts execution 4074 with specified rounding 4075 mode for single (32 4076 bit) floating point 4077 precision floating point 4078 operations. 4079 4080 Floating point rounding 4081 mode values are defined in 4082 :ref:`amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table`. 4083 4084 Used by CP to set up 4085 ``COMPUTE_PGM_RSRC1.FLOAT_MODE``. 4086 15:14 2 bits FLOAT_ROUND_MODE_16_64 Wavefront starts execution 4087 with specified rounding 4088 denorm mode for half/double (16 4089 and 64-bit) floating point 4090 precision floating point 4091 operations. 4092 4093 Floating point rounding 4094 mode values are defined in 4095 :ref:`amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table`. 4096 4097 Used by CP to set up 4098 ``COMPUTE_PGM_RSRC1.FLOAT_MODE``. 4099 17:16 2 bits FLOAT_DENORM_MODE_32 Wavefront starts execution 4100 with specified denorm mode 4101 for single (32 4102 bit) floating point 4103 precision floating point 4104 operations. 4105 4106 Floating point denorm mode 4107 values are defined in 4108 :ref:`amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table`. 4109 4110 Used by CP to set up 4111 ``COMPUTE_PGM_RSRC1.FLOAT_MODE``. 4112 19:18 2 bits FLOAT_DENORM_MODE_16_64 Wavefront starts execution 4113 with specified denorm mode 4114 for half/double (16 4115 and 64-bit) floating point 4116 precision floating point 4117 operations. 4118 4119 Floating point denorm mode 4120 values are defined in 4121 :ref:`amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table`. 4122 4123 Used by CP to set up 4124 ``COMPUTE_PGM_RSRC1.FLOAT_MODE``. 4125 20 1 bit PRIV Must be 0. 4126 4127 Start executing wavefront 4128 in privilege trap handler 4129 mode. 4130 4131 CP is responsible for 4132 filling in 4133 ``COMPUTE_PGM_RSRC1.PRIV``. 4134 21 1 bit ENABLE_DX10_CLAMP Wavefront starts execution 4135 with DX10 clamp mode 4136 enabled. Used by the vector 4137 ALU to force DX10 style 4138 treatment of NaN's (when 4139 set, clamp NaN to zero, 4140 otherwise pass NaN 4141 through). 4142 4143 Used by CP to set up 4144 ``COMPUTE_PGM_RSRC1.DX10_CLAMP``. 4145 22 1 bit DEBUG_MODE Must be 0. 4146 4147 Start executing wavefront 4148 in single step mode. 4149 4150 CP is responsible for 4151 filling in 4152 ``COMPUTE_PGM_RSRC1.DEBUG_MODE``. 4153 23 1 bit ENABLE_IEEE_MODE Wavefront starts execution 4154 with IEEE mode 4155 enabled. Floating point 4156 opcodes that support 4157 exception flag gathering 4158 will quiet and propagate 4159 signaling-NaN inputs per 4160 IEEE 754-2008. Min_dx10 and 4161 max_dx10 become IEEE 4162 754-2008 compliant due to 4163 signaling-NaN propagation 4164 and quieting. 4165 4166 Used by CP to set up 4167 ``COMPUTE_PGM_RSRC1.IEEE_MODE``. 4168 24 1 bit BULKY Must be 0. 4169 4170 Only one work-group allowed 4171 to execute on a compute 4172 unit. 4173 4174 CP is responsible for 4175 filling in 4176 ``COMPUTE_PGM_RSRC1.BULKY``. 4177 25 1 bit CDBG_USER Must be 0. 4178 4179 Flag that can be used to 4180 control debugging code. 4181 4182 CP is responsible for 4183 filling in 4184 ``COMPUTE_PGM_RSRC1.CDBG_USER``. 4185 26 1 bit FP16_OVFL GFX6-GFX8 4186 Reserved, must be 0. 4187 GFX9-GFX10 4188 Wavefront starts execution 4189 with specified fp16 overflow 4190 mode. 4191 4192 - If 0, fp16 overflow generates 4193 +/-INF values. 4194 - If 1, fp16 overflow that is the 4195 result of an +/-INF input value 4196 or divide by 0 produces a +/-INF, 4197 otherwise clamps computed 4198 overflow to +/-MAX_FP16 as 4199 appropriate. 4200 4201 Used by CP to set up 4202 ``COMPUTE_PGM_RSRC1.FP16_OVFL``. 4203 28:27 2 bits Reserved, must be 0. 4204 29 1 bit WGP_MODE GFX6-GFX9 4205 Reserved, must be 0. 4206 GFX10 4207 - If 0 execute work-groups in 4208 CU wavefront execution mode. 4209 - If 1 execute work-groups on 4210 in WGP wavefront execution mode. 4211 4212 See :ref:`amdgpu-amdhsa-memory-model`. 4213 4214 Used by CP to set up 4215 ``COMPUTE_PGM_RSRC1.WGP_MODE``. 4216 30 1 bit MEM_ORDERED GFX6-GFX9 4217 Reserved, must be 0. 4218 GFX10 4219 Controls the behavior of the 4220 s_waitcnt's vmcnt and vscnt 4221 counters. 4222 4223 - If 0 vmcnt reports completion 4224 of load and atomic with return 4225 out of order with sample 4226 instructions, and the vscnt 4227 reports the completion of 4228 store and atomic without 4229 return in order. 4230 - If 1 vmcnt reports completion 4231 of load, atomic with return 4232 and sample instructions in 4233 order, and the vscnt reports 4234 the completion of store and 4235 atomic without return in order. 4236 4237 Used by CP to set up 4238 ``COMPUTE_PGM_RSRC1.MEM_ORDERED``. 4239 31 1 bit FWD_PROGRESS GFX6-GFX9 4240 Reserved, must be 0. 4241 GFX10 4242 - If 0 execute SIMD wavefronts 4243 using oldest first policy. 4244 - If 1 execute SIMD wavefronts to 4245 ensure wavefronts will make some 4246 forward progress. 4247 4248 Used by CP to set up 4249 ``COMPUTE_PGM_RSRC1.FWD_PROGRESS``. 4250 32 **Total size 4 bytes** 4251 ======= =================================================================================================================== 4252 4253.. 4254 4255 .. table:: compute_pgm_rsrc2 for GFX6-GFX10 4256 :name: amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table 4257 4258 ======= ======= =============================== =========================================================================== 4259 Bits Size Field Name Description 4260 ======= ======= =============================== =========================================================================== 4261 0 1 bit ENABLE_PRIVATE_SEGMENT * Enable the setup of the 4262 private segment. 4263 * If the *Target Properties* 4264 column of 4265 :ref:`amdgpu-processor-table` 4266 does not specify 4267 *Architected flat 4268 scratch* then enable the 4269 setup of the SGPR 4270 wavefront scratch offset 4271 system register (see 4272 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4273 * If the *Target Properties* 4274 column of 4275 :ref:`amdgpu-processor-table` 4276 specifies *Architected 4277 flat scratch* then enable 4278 the setup of the 4279 FLAT_SCRATCH register 4280 pair (see 4281 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4282 4283 Used by CP to set up 4284 ``COMPUTE_PGM_RSRC2.SCRATCH_EN``. 4285 5:1 5 bits USER_SGPR_COUNT The total number of SGPR 4286 user data 4287 registers requested. This 4288 number must be greater than 4289 or equal to the number of user 4290 data registers enabled. 4291 4292 Used by CP to set up 4293 ``COMPUTE_PGM_RSRC2.USER_SGPR``. 4294 6 1 bit ENABLE_TRAP_HANDLER Must be 0. 4295 4296 This bit represents 4297 ``COMPUTE_PGM_RSRC2.TRAP_PRESENT``, 4298 which is set by the CP if 4299 the runtime has installed a 4300 trap handler. 4301 7 1 bit ENABLE_SGPR_WORKGROUP_ID_X Enable the setup of the 4302 system SGPR register for 4303 the work-group id in the X 4304 dimension (see 4305 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4306 4307 Used by CP to set up 4308 ``COMPUTE_PGM_RSRC2.TGID_X_EN``. 4309 8 1 bit ENABLE_SGPR_WORKGROUP_ID_Y Enable the setup of the 4310 system SGPR register for 4311 the work-group id in the Y 4312 dimension (see 4313 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4314 4315 Used by CP to set up 4316 ``COMPUTE_PGM_RSRC2.TGID_Y_EN``. 4317 9 1 bit ENABLE_SGPR_WORKGROUP_ID_Z Enable the setup of the 4318 system SGPR register for 4319 the work-group id in the Z 4320 dimension (see 4321 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4322 4323 Used by CP to set up 4324 ``COMPUTE_PGM_RSRC2.TGID_Z_EN``. 4325 10 1 bit ENABLE_SGPR_WORKGROUP_INFO Enable the setup of the 4326 system SGPR register for 4327 work-group information (see 4328 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4329 4330 Used by CP to set up 4331 ``COMPUTE_PGM_RSRC2.TGID_SIZE_EN``. 4332 12:11 2 bits ENABLE_VGPR_WORKITEM_ID Enable the setup of the 4333 VGPR system registers used 4334 for the work-item ID. 4335 :ref:`amdgpu-amdhsa-system-vgpr-work-item-id-enumeration-values-table` 4336 defines the values. 4337 4338 Used by CP to set up 4339 ``COMPUTE_PGM_RSRC2.TIDIG_CMP_CNT``. 4340 13 1 bit ENABLE_EXCEPTION_ADDRESS_WATCH Must be 0. 4341 4342 Wavefront starts execution 4343 with address watch 4344 exceptions enabled which 4345 are generated when L1 has 4346 witnessed a thread access 4347 an *address of 4348 interest*. 4349 4350 CP is responsible for 4351 filling in the address 4352 watch bit in 4353 ``COMPUTE_PGM_RSRC2.EXCP_EN_MSB`` 4354 according to what the 4355 runtime requests. 4356 14 1 bit ENABLE_EXCEPTION_MEMORY Must be 0. 4357 4358 Wavefront starts execution 4359 with memory violation 4360 exceptions exceptions 4361 enabled which are generated 4362 when a memory violation has 4363 occurred for this wavefront from 4364 L1 or LDS 4365 (write-to-read-only-memory, 4366 mis-aligned atomic, LDS 4367 address out of range, 4368 illegal address, etc.). 4369 4370 CP sets the memory 4371 violation bit in 4372 ``COMPUTE_PGM_RSRC2.EXCP_EN_MSB`` 4373 according to what the 4374 runtime requests. 4375 23:15 9 bits GRANULATED_LDS_SIZE Must be 0. 4376 4377 CP uses the rounded value 4378 from the dispatch packet, 4379 not this value, as the 4380 dispatch may contain 4381 dynamically allocated group 4382 segment memory. CP writes 4383 directly to 4384 ``COMPUTE_PGM_RSRC2.LDS_SIZE``. 4385 4386 Amount of group segment 4387 (LDS) to allocate for each 4388 work-group. Granularity is 4389 device specific: 4390 4391 GFX6 4392 roundup(lds-size / (64 * 4)) 4393 GFX7-GFX10 4394 roundup(lds-size / (128 * 4)) 4395 4396 24 1 bit ENABLE_EXCEPTION_IEEE_754_FP Wavefront starts execution 4397 _INVALID_OPERATION with specified exceptions 4398 enabled. 4399 4400 Used by CP to set up 4401 ``COMPUTE_PGM_RSRC2.EXCP_EN`` 4402 (set from bits 0..6). 4403 4404 IEEE 754 FP Invalid 4405 Operation 4406 25 1 bit ENABLE_EXCEPTION_FP_DENORMAL FP Denormal one or more 4407 _SOURCE input operands is a 4408 denormal number 4409 26 1 bit ENABLE_EXCEPTION_IEEE_754_FP IEEE 754 FP Division by 4410 _DIVISION_BY_ZERO Zero 4411 27 1 bit ENABLE_EXCEPTION_IEEE_754_FP IEEE 754 FP FP Overflow 4412 _OVERFLOW 4413 28 1 bit ENABLE_EXCEPTION_IEEE_754_FP IEEE 754 FP Underflow 4414 _UNDERFLOW 4415 29 1 bit ENABLE_EXCEPTION_IEEE_754_FP IEEE 754 FP Inexact 4416 _INEXACT 4417 30 1 bit ENABLE_EXCEPTION_INT_DIVIDE_BY Integer Division by Zero 4418 _ZERO (rcp_iflag_f32 instruction 4419 only) 4420 31 1 bit Reserved, must be 0. 4421 32 **Total size 4 bytes.** 4422 ======= =================================================================================================================== 4423 4424.. 4425 4426 .. table:: compute_pgm_rsrc3 for GFX90A, GFX940 4427 :name: amdgpu-amdhsa-compute_pgm_rsrc3-gfx90a-table 4428 4429 ======= ======= =============================== =========================================================================== 4430 Bits Size Field Name Description 4431 ======= ======= =============================== =========================================================================== 4432 5:0 6 bits ACCUM_OFFSET Offset of a first AccVGPR in the unified register file. Granularity 4. 4433 Value 0-63. 0 - accum-offset = 4, 1 - accum-offset = 8, ..., 4434 63 - accum-offset = 256. 4435 6:15 10 Reserved, must be 0. 4436 bits 4437 16 1 bit TG_SPLIT - If 0 the waves of a work-group are 4438 launched in the same CU. 4439 - If 1 the waves of a work-group can be 4440 launched in different CUs. The waves 4441 cannot use S_BARRIER or LDS. 4442 17:31 15 Reserved, must be 0. 4443 bits 4444 32 **Total size 4 bytes.** 4445 ======= =================================================================================================================== 4446 4447.. 4448 4449 .. table:: compute_pgm_rsrc3 for GFX10 4450 :name: amdgpu-amdhsa-compute_pgm_rsrc3-gfx10-table 4451 4452 ======= ======= =============================== =========================================================================== 4453 Bits Size Field Name Description 4454 ======= ======= =============================== =========================================================================== 4455 3:0 4 bits SHARED_VGPR_COUNT Number of shared VGPR blocks when executing in subvector mode. For 4456 wavefront size 64 the value is 0-15, representing 0-120 VGPRs (granularity 4457 of 8), such that (compute_pgm_rsrc1.vgprs +1)*4 + shared_vgpr_count*8 does 4458 not exceed 256. For wavefront size 32 shared_vgpr_count must be 0. 4459 31:4 28 Reserved, must be 0. 4460 bits 4461 32 **Total size 4 bytes.** 4462 ======= =================================================================================================================== 4463 4464.. 4465 4466 .. table:: Floating Point Rounding Mode Enumeration Values 4467 :name: amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table 4468 4469 ====================================== ===== ============================== 4470 Enumeration Name Value Description 4471 ====================================== ===== ============================== 4472 FLOAT_ROUND_MODE_NEAR_EVEN 0 Round Ties To Even 4473 FLOAT_ROUND_MODE_PLUS_INFINITY 1 Round Toward +infinity 4474 FLOAT_ROUND_MODE_MINUS_INFINITY 2 Round Toward -infinity 4475 FLOAT_ROUND_MODE_ZERO 3 Round Toward 0 4476 ====================================== ===== ============================== 4477 4478.. 4479 4480 .. table:: Floating Point Denorm Mode Enumeration Values 4481 :name: amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table 4482 4483 ====================================== ===== ============================== 4484 Enumeration Name Value Description 4485 ====================================== ===== ============================== 4486 FLOAT_DENORM_MODE_FLUSH_SRC_DST 0 Flush Source and Destination 4487 Denorms 4488 FLOAT_DENORM_MODE_FLUSH_DST 1 Flush Output Denorms 4489 FLOAT_DENORM_MODE_FLUSH_SRC 2 Flush Source Denorms 4490 FLOAT_DENORM_MODE_FLUSH_NONE 3 No Flush 4491 ====================================== ===== ============================== 4492 4493.. 4494 4495 .. table:: System VGPR Work-Item ID Enumeration Values 4496 :name: amdgpu-amdhsa-system-vgpr-work-item-id-enumeration-values-table 4497 4498 ======================================== ===== ============================ 4499 Enumeration Name Value Description 4500 ======================================== ===== ============================ 4501 SYSTEM_VGPR_WORKITEM_ID_X 0 Set work-item X dimension 4502 ID. 4503 SYSTEM_VGPR_WORKITEM_ID_X_Y 1 Set work-item X and Y 4504 dimensions ID. 4505 SYSTEM_VGPR_WORKITEM_ID_X_Y_Z 2 Set work-item X, Y and Z 4506 dimensions ID. 4507 SYSTEM_VGPR_WORKITEM_ID_UNDEFINED 3 Undefined. 4508 ======================================== ===== ============================ 4509 4510.. _amdgpu-amdhsa-initial-kernel-execution-state: 4511 4512Initial Kernel Execution State 4513~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 4514 4515This section defines the register state that will be set up by the packet 4516processor prior to the start of execution of every wavefront. This is limited by 4517the constraints of the hardware controllers of CP/ADC/SPI. 4518 4519The order of the SGPR registers is defined, but the compiler can specify which 4520ones are actually setup in the kernel descriptor using the ``enable_sgpr_*`` bit 4521fields (see :ref:`amdgpu-amdhsa-kernel-descriptor`). The register numbers used 4522for enabled registers are dense starting at SGPR0: the first enabled register is 4523SGPR0, the next enabled register is SGPR1 etc.; disabled registers do not have 4524an SGPR number. 4525 4526The initial SGPRs comprise up to 16 User SRGPs that are set by CP and apply to 4527all wavefronts of the grid. It is possible to specify more than 16 User SGPRs 4528using the ``enable_sgpr_*`` bit fields, in which case only the first 16 are 4529actually initialized. These are then immediately followed by the System SGPRs 4530that are set up by ADC/SPI and can have different values for each wavefront of 4531the grid dispatch. 4532 4533SGPR register initial state is defined in 4534:ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 4535 4536 .. table:: SGPR Register Set Up Order 4537 :name: amdgpu-amdhsa-sgpr-register-set-up-order-table 4538 4539 ========== ========================== ====== ============================== 4540 SGPR Order Name Number Description 4541 (kernel descriptor enable of 4542 field) SGPRs 4543 ========== ========================== ====== ============================== 4544 First Private Segment Buffer 4 See 4545 (enable_sgpr_private :ref:`amdgpu-amdhsa-kernel-prolog-private-segment-buffer`. 4546 _segment_buffer) 4547 then Dispatch Ptr 2 64-bit address of AQL dispatch 4548 (enable_sgpr_dispatch_ptr) packet for kernel dispatch 4549 actually executing. 4550 then Queue Ptr 2 64-bit address of amd_queue_t 4551 (enable_sgpr_queue_ptr) object for AQL queue on which 4552 the dispatch packet was 4553 queued. 4554 then Kernarg Segment Ptr 2 64-bit address of Kernarg 4555 (enable_sgpr_kernarg segment. This is directly 4556 _segment_ptr) copied from the 4557 kernarg_address in the kernel 4558 dispatch packet. 4559 4560 Having CP load it once avoids 4561 loading it at the beginning of 4562 every wavefront. 4563 then Dispatch Id 2 64-bit Dispatch ID of the 4564 (enable_sgpr_dispatch_id) dispatch packet being 4565 executed. 4566 then Flat Scratch Init 2 See 4567 (enable_sgpr_flat_scratch :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`. 4568 _init) 4569 then Private Segment Size 1 The 32-bit byte size of a 4570 (enable_sgpr_private single work-item's memory 4571 _segment_size) allocation. This is the 4572 value from the kernel 4573 dispatch packet Private 4574 Segment Byte Size rounded up 4575 by CP to a multiple of 4576 DWORD. 4577 4578 Having CP load it once avoids 4579 loading it at the beginning of 4580 every wavefront. 4581 4582 This is not used for 4583 GFX7-GFX8 since it is the same 4584 value as the second SGPR of 4585 Flat Scratch Init. However, it 4586 may be needed for GFX9-GFX10 which 4587 changes the meaning of the 4588 Flat Scratch Init value. 4589 then Work-Group Id X 1 32-bit work-group id in X 4590 (enable_sgpr_workgroup_id dimension of grid for 4591 _X) wavefront. 4592 then Work-Group Id Y 1 32-bit work-group id in Y 4593 (enable_sgpr_workgroup_id dimension of grid for 4594 _Y) wavefront. 4595 then Work-Group Id Z 1 32-bit work-group id in Z 4596 (enable_sgpr_workgroup_id dimension of grid for 4597 _Z) wavefront. 4598 then Work-Group Info 1 {first_wavefront, 14'b0000, 4599 (enable_sgpr_workgroup ordered_append_term[10:0], 4600 _info) threadgroup_size_in_wavefronts[5:0]} 4601 then Scratch Wavefront Offset 1 See 4602 (enable_sgpr_private :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`. 4603 _segment_wavefront_offset) and 4604 :ref:`amdgpu-amdhsa-kernel-prolog-private-segment-buffer`. 4605 ========== ========================== ====== ============================== 4606 4607The order of the VGPR registers is defined, but the compiler can specify which 4608ones are actually setup in the kernel descriptor using the ``enable_vgpr*`` bit 4609fields (see :ref:`amdgpu-amdhsa-kernel-descriptor`). The register numbers used 4610for enabled registers are dense starting at VGPR0: the first enabled register is 4611VGPR0, the next enabled register is VGPR1 etc.; disabled registers do not have a 4612VGPR number. 4613 4614There are different methods used for the VGPR initial state: 4615 4616* Unless the *Target Properties* column of :ref:`amdgpu-processor-table` 4617 specifies otherwise, a separate VGPR register is used per work-item ID. The 4618 VGPR register initial state for this method is defined in 4619 :ref:`amdgpu-amdhsa-vgpr-register-set-up-order-for-unpacked-work-item-id-method-table`. 4620* If *Target Properties* column of :ref:`amdgpu-processor-table` 4621 specifies *Packed work-item IDs*, the initial value of VGPR0 register is used 4622 for all work-item IDs. The register layout for this method is defined in 4623 :ref:`amdgpu-amdhsa-register-layout-for-packed-work-item-id-method-table`. 4624 4625 .. table:: VGPR Register Set Up Order for Unpacked Work-Item ID Method 4626 :name: amdgpu-amdhsa-vgpr-register-set-up-order-for-unpacked-work-item-id-method-table 4627 4628 ========== ========================== ====== ============================== 4629 VGPR Order Name Number Description 4630 (kernel descriptor enable of 4631 field) VGPRs 4632 ========== ========================== ====== ============================== 4633 First Work-Item Id X 1 32-bit work-item id in X 4634 (Always initialized) dimension of work-group for 4635 wavefront lane. 4636 then Work-Item Id Y 1 32-bit work-item id in Y 4637 (enable_vgpr_workitem_id dimension of work-group for 4638 > 0) wavefront lane. 4639 then Work-Item Id Z 1 32-bit work-item id in Z 4640 (enable_vgpr_workitem_id dimension of work-group for 4641 > 1) wavefront lane. 4642 ========== ========================== ====== ============================== 4643 4644.. 4645 4646 .. table:: Register Layout for Packed Work-Item ID Method 4647 :name: amdgpu-amdhsa-register-layout-for-packed-work-item-id-method-table 4648 4649 ======= ======= ================ ========================================= 4650 Bits Size Field Name Description 4651 ======= ======= ================ ========================================= 4652 0:9 10 bits Work-Item Id X Work-item id in X 4653 dimension of work-group for 4654 wavefront lane. 4655 4656 Always initialized. 4657 4658 10:19 10 bits Work-Item Id Y Work-item id in Y 4659 dimension of work-group for 4660 wavefront lane. 4661 4662 Initialized if enable_vgpr_workitem_id > 4663 0, otherwise set to 0. 4664 20:29 10 bits Work-Item Id Z Work-item id in Z 4665 dimension of work-group for 4666 wavefront lane. 4667 4668 Initialized if enable_vgpr_workitem_id > 4669 1, otherwise set to 0. 4670 30:31 2 bits Reserved, set to 0. 4671 ======= ======= ================ ========================================= 4672 4673The setting of registers is done by GPU CP/ADC/SPI hardware as follows: 4674 46751. SGPRs before the Work-Group Ids are set by CP using the 16 User Data 4676 registers. 46772. Work-group Id registers X, Y, Z are set by ADC which supports any 4678 combination including none. 46793. Scratch Wavefront Offset is set by SPI in a per wavefront basis which is why 4680 its value cannot be included with the flat scratch init value which is per 4681 queue (see :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`). 46824. The VGPRs are set by SPI which only supports specifying either (X), (X, Y) 4683 or (X, Y, Z). 46845. Flat Scratch register pair initialization is described in 4685 :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`. 4686 4687The global segment can be accessed either using buffer instructions (GFX6 which 4688has V# 64-bit address support), flat instructions (GFX7-GFX10), or global 4689instructions (GFX9-GFX10). 4690 4691If buffer operations are used, then the compiler can generate a V# with the 4692following properties: 4693 4694* base address of 0 4695* no swizzle 4696* ATC: 1 if IOMMU present (such as APU) 4697* ptr64: 1 4698* MTYPE set to support memory coherence that matches the runtime (such as CC for 4699 APU and NC for dGPU). 4700 4701.. _amdgpu-amdhsa-kernel-prolog: 4702 4703Kernel Prolog 4704~~~~~~~~~~~~~ 4705 4706The compiler performs initialization in the kernel prologue depending on the 4707target and information about things like stack usage in the kernel and called 4708functions. Some of this initialization requires the compiler to request certain 4709User and System SGPRs be present in the 4710:ref:`amdgpu-amdhsa-initial-kernel-execution-state` via the 4711:ref:`amdgpu-amdhsa-kernel-descriptor`. 4712 4713.. _amdgpu-amdhsa-kernel-prolog-cfi: 4714 4715CFI 4716+++ 4717 47181. The CFI return address is undefined. 4719 47202. The CFI CFA is defined using an expression which evaluates to a location 4721 description that comprises one memory location description for the 4722 ``DW_ASPACE_AMDGPU_private_lane`` address space address ``0``. 4723 4724.. _amdgpu-amdhsa-kernel-prolog-m0: 4725 4726M0 4727++ 4728 4729GFX6-GFX8 4730 The M0 register must be initialized with a value at least the total LDS size 4731 if the kernel may access LDS via DS or flat operations. Total LDS size is 4732 available in dispatch packet. For M0, it is also possible to use maximum 4733 possible value of LDS for given target (0x7FFF for GFX6 and 0xFFFF for 4734 GFX7-GFX8). 4735GFX9-GFX10 4736 The M0 register is not used for range checking LDS accesses and so does not 4737 need to be initialized in the prolog. 4738 4739.. _amdgpu-amdhsa-kernel-prolog-stack-pointer: 4740 4741Stack Pointer 4742+++++++++++++ 4743 4744If the kernel has function calls it must set up the ABI stack pointer described 4745in :ref:`amdgpu-amdhsa-function-call-convention-non-kernel-functions` by setting 4746SGPR32 to the unswizzled scratch offset of the address past the last local 4747allocation. 4748 4749.. _amdgpu-amdhsa-kernel-prolog-frame-pointer: 4750 4751Frame Pointer 4752+++++++++++++ 4753 4754If the kernel needs a frame pointer for the reasons defined in 4755``SIFrameLowering`` then SGPR33 is used and is always set to ``0`` in the 4756kernel prolog. If a frame pointer is not required then all uses of the frame 4757pointer are replaced with immediate ``0`` offsets. 4758 4759.. _amdgpu-amdhsa-kernel-prolog-flat-scratch: 4760 4761Flat Scratch 4762++++++++++++ 4763 4764There are different methods used for initializing flat scratch: 4765 4766* If the *Target Properties* column of :ref:`amdgpu-processor-table` 4767 specifies *Does not support generic address space*: 4768 4769 Flat scratch is not supported and there is no flat scratch register pair. 4770 4771* If the *Target Properties* column of :ref:`amdgpu-processor-table` 4772 specifies *Offset flat scratch*: 4773 4774 If the kernel or any function it calls may use flat operations to access 4775 scratch memory, the prolog code must set up the FLAT_SCRATCH register pair 4776 (FLAT_SCRATCH_LO/FLAT_SCRATCH_HI). Initialization uses Flat Scratch Init and 4777 Scratch Wavefront Offset SGPR registers (see 4778 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`): 4779 4780 1. The low word of Flat Scratch Init is the 32-bit byte offset from 4781 ``SH_HIDDEN_PRIVATE_BASE_VIMID`` to the base of scratch backing memory 4782 being managed by SPI for the queue executing the kernel dispatch. This is 4783 the same value used in the Scratch Segment Buffer V# base address. 4784 4785 CP obtains this from the runtime. (The Scratch Segment Buffer base address 4786 is ``SH_HIDDEN_PRIVATE_BASE_VIMID`` plus this offset.) 4787 4788 The prolog must add the value of Scratch Wavefront Offset to get the 4789 wavefront's byte scratch backing memory offset from 4790 ``SH_HIDDEN_PRIVATE_BASE_VIMID``. 4791 4792 The Scratch Wavefront Offset must also be used as an offset with Private 4793 segment address when using the Scratch Segment Buffer. 4794 4795 Since FLAT_SCRATCH_LO is in units of 256 bytes, the offset must be right 4796 shifted by 8 before moving into FLAT_SCRATCH_HI. 4797 4798 FLAT_SCRATCH_HI corresponds to SGPRn-4 on GFX7, and SGPRn-6 on GFX8 (where 4799 SGPRn is the highest numbered SGPR allocated to the wavefront). 4800 FLAT_SCRATCH_HI is multiplied by 256 (as it is in units of 256 bytes) and 4801 added to ``SH_HIDDEN_PRIVATE_BASE_VIMID`` to calculate the per wavefront 4802 FLAT SCRATCH BASE in flat memory instructions that access the scratch 4803 aperture. 4804 2. The second word of Flat Scratch Init is 32-bit byte size of a single 4805 work-items scratch memory usage. 4806 4807 CP obtains this from the runtime, and it is always a multiple of DWORD. CP 4808 checks that the value in the kernel dispatch packet Private Segment Byte 4809 Size is not larger and requests the runtime to increase the queue's scratch 4810 size if necessary. 4811 4812 CP directly loads from the kernel dispatch packet Private Segment Byte Size 4813 field and rounds up to a multiple of DWORD. Having CP load it once avoids 4814 loading it at the beginning of every wavefront. 4815 4816 The kernel prolog code must move it to FLAT_SCRATCH_LO which is SGPRn-3 on 4817 GFX7 and SGPRn-5 on GFX8. FLAT_SCRATCH_LO is used as the FLAT SCRATCH SIZE 4818 in flat memory instructions. 4819 4820* If the *Target Properties* column of :ref:`amdgpu-processor-table` 4821 specifies *Absolute flat scratch*: 4822 4823 If the kernel or any function it calls may use flat operations to access 4824 scratch memory, the prolog code must set up the FLAT_SCRATCH register pair 4825 (FLAT_SCRATCH_LO/FLAT_SCRATCH_HI which are in SGPRn-4/SGPRn-3). Initialization 4826 uses Flat Scratch Init and Scratch Wavefront Offset SGPR registers (see 4827 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`): 4828 4829 The Flat Scratch Init is the 64-bit address of the base of scratch backing 4830 memory being managed by SPI for the queue executing the kernel dispatch. 4831 4832 CP obtains this from the runtime. 4833 4834 The kernel prolog must add the value of the wave's Scratch Wavefront Offset 4835 and move the result as a 64-bit value to the FLAT_SCRATCH SGPR register pair 4836 which is SGPRn-6 and SGPRn-5. It is used as the FLAT SCRATCH BASE in flat 4837 memory instructions. 4838 4839 The Scratch Wavefront Offset must also be used as an offset with Private 4840 segment address when using the Scratch Segment Buffer (see 4841 :ref:`amdgpu-amdhsa-kernel-prolog-private-segment-buffer`). 4842 4843* If the *Target Properties* column of :ref:`amdgpu-processor-table` 4844 specifies *Architected flat scratch*: 4845 4846 If ENABLE_PRIVATE_SEGMENT is enabled in 4847 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table` then the FLAT_SCRATCH 4848 register pair will be initialized to the 64-bit address of the base of scratch 4849 backing memory being managed by SPI for the queue executing the kernel 4850 dispatch plus the value of the wave's Scratch Wavefront Offset for use as the 4851 flat scratch base in flat memory instructions. 4852 4853.. _amdgpu-amdhsa-kernel-prolog-private-segment-buffer: 4854 4855Private Segment Buffer 4856++++++++++++++++++++++ 4857 4858If the *Target Properties* column of :ref:`amdgpu-processor-table` specifies 4859*Architected flat scratch* then a Private Segment Buffer is not supported. 4860Instead the flat SCRATCH instructions are used. 4861 4862Otherwise, Private Segment Buffer SGPR register is used to initialize 4 SGPRs 4863that are used as a V# to access scratch. CP uses the value provided by the 4864runtime. It is used, together with Scratch Wavefront Offset as an offset, to 4865access the private memory space using a segment address. See 4866:ref:`amdgpu-amdhsa-initial-kernel-execution-state`. 4867 4868The scratch V# is a four-aligned SGPR and always selected for the kernel as 4869follows: 4870 4871 - If it is known during instruction selection that there is stack usage, 4872 SGPR0-3 is reserved for use as the scratch V#. Stack usage is assumed if 4873 optimizations are disabled (``-O0``), if stack objects already exist (for 4874 locals, etc.), or if there are any function calls. 4875 4876 - Otherwise, four high numbered SGPRs beginning at a four-aligned SGPR index 4877 are reserved for the tentative scratch V#. These will be used if it is 4878 determined that spilling is needed. 4879 4880 - If no use is made of the tentative scratch V#, then it is unreserved, 4881 and the register count is determined ignoring it. 4882 - If use is made of the tentative scratch V#, then its register numbers 4883 are shifted to the first four-aligned SGPR index after the highest one 4884 allocated by the register allocator, and all uses are updated. The 4885 register count includes them in the shifted location. 4886 - In either case, if the processor has the SGPR allocation bug, the 4887 tentative allocation is not shifted or unreserved in order to ensure 4888 the register count is higher to workaround the bug. 4889 4890 .. note:: 4891 4892 This approach of using a tentative scratch V# and shifting the register 4893 numbers if used avoids having to perform register allocation a second 4894 time if the tentative V# is eliminated. This is more efficient and 4895 avoids the problem that the second register allocation may perform 4896 spilling which will fail as there is no longer a scratch V#. 4897 4898When the kernel prolog code is being emitted it is known whether the scratch V# 4899described above is actually used. If it is, the prolog code must set it up by 4900copying the Private Segment Buffer to the scratch V# registers and then adding 4901the Private Segment Wavefront Offset to the queue base address in the V#. The 4902result is a V# with a base address pointing to the beginning of the wavefront 4903scratch backing memory. 4904 4905The Private Segment Buffer is always requested, but the Private Segment 4906Wavefront Offset is only requested if it is used (see 4907:ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4908 4909.. _amdgpu-amdhsa-memory-model: 4910 4911Memory Model 4912~~~~~~~~~~~~ 4913 4914This section describes the mapping of the LLVM memory model onto AMDGPU machine 4915code (see :ref:`memmodel`). 4916 4917The AMDGPU backend supports the memory synchronization scopes specified in 4918:ref:`amdgpu-memory-scopes`. 4919 4920The code sequences used to implement the memory model specify the order of 4921instructions that a single thread must execute. The ``s_waitcnt`` and cache 4922management instructions such as ``buffer_wbinvl1_vol`` are defined with respect 4923to other memory instructions executed by the same thread. This allows them to be 4924moved earlier or later which can allow them to be combined with other instances 4925of the same instruction, or hoisted/sunk out of loops to improve performance. 4926Only the instructions related to the memory model are given; additional 4927``s_waitcnt`` instructions are required to ensure registers are defined before 4928being used. These may be able to be combined with the memory model ``s_waitcnt`` 4929instructions as described above. 4930 4931The AMDGPU backend supports the following memory models: 4932 4933 HSA Memory Model [HSA]_ 4934 The HSA memory model uses a single happens-before relation for all address 4935 spaces (see :ref:`amdgpu-address-spaces`). 4936 OpenCL Memory Model [OpenCL]_ 4937 The OpenCL memory model which has separate happens-before relations for the 4938 global and local address spaces. Only a fence specifying both global and 4939 local address space, and seq_cst instructions join the relationships. Since 4940 the LLVM ``memfence`` instruction does not allow an address space to be 4941 specified the OpenCL fence has to conservatively assume both local and 4942 global address space was specified. However, optimizations can often be 4943 done to eliminate the additional ``s_waitcnt`` instructions when there are 4944 no intervening memory instructions which access the corresponding address 4945 space. The code sequences in the table indicate what can be omitted for the 4946 OpenCL memory. The target triple environment is used to determine if the 4947 source language is OpenCL (see :ref:`amdgpu-opencl`). 4948 4949``ds/flat_load/store/atomic`` instructions to local memory are termed LDS 4950operations. 4951 4952``buffer/global/flat_load/store/atomic`` instructions to global memory are 4953termed vector memory operations. 4954 4955Private address space uses ``buffer_load/store`` using the scratch V# 4956(GFX6-GFX8), or ``scratch_load/store`` (GFX9-GFX10). Since only a single thread 4957is accessing the memory, atomic memory orderings are not meaningful, and all 4958accesses are treated as non-atomic. 4959 4960Constant address space uses ``buffer/global_load`` instructions (or equivalent 4961scalar memory instructions). Since the constant address space contents do not 4962change during the execution of a kernel dispatch it is not legal to perform 4963stores, and atomic memory orderings are not meaningful, and all accesses are 4964treated as non-atomic. 4965 4966A memory synchronization scope wider than work-group is not meaningful for the 4967group (LDS) address space and is treated as work-group. 4968 4969The memory model does not support the region address space which is treated as 4970non-atomic. 4971 4972Acquire memory ordering is not meaningful on store atomic instructions and is 4973treated as non-atomic. 4974 4975Release memory ordering is not meaningful on load atomic instructions and is 4976treated a non-atomic. 4977 4978Acquire-release memory ordering is not meaningful on load or store atomic 4979instructions and is treated as acquire and release respectively. 4980 4981The memory order also adds the single thread optimization constraints defined in 4982table 4983:ref:`amdgpu-amdhsa-memory-model-single-thread-optimization-constraints-table`. 4984 4985 .. table:: AMDHSA Memory Model Single Thread Optimization Constraints 4986 :name: amdgpu-amdhsa-memory-model-single-thread-optimization-constraints-table 4987 4988 ============ ============================================================== 4989 LLVM Memory Optimization Constraints 4990 Ordering 4991 ============ ============================================================== 4992 unordered *none* 4993 monotonic *none* 4994 acquire - If a load atomic/atomicrmw then no following load/load 4995 atomic/store/store atomic/atomicrmw/fence instruction can be 4996 moved before the acquire. 4997 - If a fence then same as load atomic, plus no preceding 4998 associated fence-paired-atomic can be moved after the fence. 4999 release - If a store atomic/atomicrmw then no preceding load/load 5000 atomic/store/store atomic/atomicrmw/fence instruction can be 5001 moved after the release. 5002 - If a fence then same as store atomic, plus no following 5003 associated fence-paired-atomic can be moved before the 5004 fence. 5005 acq_rel Same constraints as both acquire and release. 5006 seq_cst - If a load atomic then same constraints as acquire, plus no 5007 preceding sequentially consistent load atomic/store 5008 atomic/atomicrmw/fence instruction can be moved after the 5009 seq_cst. 5010 - If a store atomic then the same constraints as release, plus 5011 no following sequentially consistent load atomic/store 5012 atomic/atomicrmw/fence instruction can be moved before the 5013 seq_cst. 5014 - If an atomicrmw/fence then same constraints as acq_rel. 5015 ============ ============================================================== 5016 5017The code sequences used to implement the memory model are defined in the 5018following sections: 5019 5020* :ref:`amdgpu-amdhsa-memory-model-gfx6-gfx9` 5021* :ref:`amdgpu-amdhsa-memory-model-gfx90a` 5022* :ref:`amdgpu-amdhsa-memory-model-gfx940` 5023* :ref:`amdgpu-amdhsa-memory-model-gfx10` 5024 5025.. _amdgpu-amdhsa-memory-model-gfx6-gfx9: 5026 5027Memory Model GFX6-GFX9 5028++++++++++++++++++++++ 5029 5030For GFX6-GFX9: 5031 5032* Each agent has multiple shader arrays (SA). 5033* Each SA has multiple compute units (CU). 5034* Each CU has multiple SIMDs that execute wavefronts. 5035* The wavefronts for a single work-group are executed in the same CU but may be 5036 executed by different SIMDs. 5037* Each CU has a single LDS memory shared by the wavefronts of the work-groups 5038 executing on it. 5039* All LDS operations of a CU are performed as wavefront wide operations in a 5040 global order and involve no caching. Completion is reported to a wavefront in 5041 execution order. 5042* The LDS memory has multiple request queues shared by the SIMDs of a 5043 CU. Therefore, the LDS operations performed by different wavefronts of a 5044 work-group can be reordered relative to each other, which can result in 5045 reordering the visibility of vector memory operations with respect to LDS 5046 operations of other wavefronts in the same work-group. A ``s_waitcnt 5047 lgkmcnt(0)`` is required to ensure synchronization between LDS operations and 5048 vector memory operations between wavefronts of a work-group, but not between 5049 operations performed by the same wavefront. 5050* The vector memory operations are performed as wavefront wide operations and 5051 completion is reported to a wavefront in execution order. The exception is 5052 that for GFX7-GFX9 ``flat_load/store/atomic`` instructions can report out of 5053 vector memory order if they access LDS memory, and out of LDS operation order 5054 if they access global memory. 5055* The vector memory operations access a single vector L1 cache shared by all 5056 SIMDs a CU. Therefore, no special action is required for coherence between the 5057 lanes of a single wavefront, or for coherence between wavefronts in the same 5058 work-group. A ``buffer_wbinvl1_vol`` is required for coherence between 5059 wavefronts executing in different work-groups as they may be executing on 5060 different CUs. 5061* The scalar memory operations access a scalar L1 cache shared by all wavefronts 5062 on a group of CUs. The scalar and vector L1 caches are not coherent. However, 5063 scalar operations are used in a restricted way so do not impact the memory 5064 model. See :ref:`amdgpu-amdhsa-memory-spaces`. 5065* The vector and scalar memory operations use an L2 cache shared by all CUs on 5066 the same agent. 5067* The L2 cache has independent channels to service disjoint ranges of virtual 5068 addresses. 5069* Each CU has a separate request queue per channel. Therefore, the vector and 5070 scalar memory operations performed by wavefronts executing in different 5071 work-groups (which may be executing on different CUs) of an agent can be 5072 reordered relative to each other. A ``s_waitcnt vmcnt(0)`` is required to 5073 ensure synchronization between vector memory operations of different CUs. It 5074 ensures a previous vector memory operation has completed before executing a 5075 subsequent vector memory or LDS operation and so can be used to meet the 5076 requirements of acquire and release. 5077* The L2 cache can be kept coherent with other agents on some targets, or ranges 5078 of virtual addresses can be set up to bypass it to ensure system coherence. 5079 5080Scalar memory operations are only used to access memory that is proven to not 5081change during the execution of the kernel dispatch. This includes constant 5082address space and global address space for program scope ``const`` variables. 5083Therefore, the kernel machine code does not have to maintain the scalar cache to 5084ensure it is coherent with the vector caches. The scalar and vector caches are 5085invalidated between kernel dispatches by CP since constant address space data 5086may change between kernel dispatch executions. See 5087:ref:`amdgpu-amdhsa-memory-spaces`. 5088 5089The one exception is if scalar writes are used to spill SGPR registers. In this 5090case the AMDGPU backend ensures the memory location used to spill is never 5091accessed by vector memory operations at the same time. If scalar writes are used 5092then a ``s_dcache_wb`` is inserted before the ``s_endpgm`` and before a function 5093return since the locations may be used for vector memory instructions by a 5094future wavefront that uses the same scratch area, or a function call that 5095creates a frame at the same address, respectively. There is no need for a 5096``s_dcache_inv`` as all scalar writes are write-before-read in the same thread. 5097 5098For kernarg backing memory: 5099 5100* CP invalidates the L1 cache at the start of each kernel dispatch. 5101* On dGPU the kernarg backing memory is allocated in host memory accessed as 5102 MTYPE UC (uncached) to avoid needing to invalidate the L2 cache. This also 5103 causes it to be treated as non-volatile and so is not invalidated by 5104 ``*_vol``. 5105* On APU the kernarg backing memory it is accessed as MTYPE CC (cache coherent) 5106 and so the L2 cache will be coherent with the CPU and other agents. 5107 5108Scratch backing memory (which is used for the private address space) is accessed 5109with MTYPE NC_NV (non-coherent non-volatile). Since the private address space is 5110only accessed by a single thread, and is always write-before-read, there is 5111never a need to invalidate these entries from the L1 cache. Hence all cache 5112invalidates are done as ``*_vol`` to only invalidate the volatile cache lines. 5113 5114The code sequences used to implement the memory model for GFX6-GFX9 are defined 5115in table :ref:`amdgpu-amdhsa-memory-model-code-sequences-gfx6-gfx9-table`. 5116 5117 .. table:: AMDHSA Memory Model Code Sequences GFX6-GFX9 5118 :name: amdgpu-amdhsa-memory-model-code-sequences-gfx6-gfx9-table 5119 5120 ============ ============ ============== ========== ================================ 5121 LLVM Instr LLVM Memory LLVM Memory AMDGPU AMDGPU Machine Code 5122 Ordering Sync Scope Address GFX6-GFX9 5123 Space 5124 ============ ============ ============== ========== ================================ 5125 **Non-Atomic** 5126 ------------------------------------------------------------------------------------ 5127 load *none* *none* - global - !volatile & !nontemporal 5128 - generic 5129 - private 1. buffer/global/flat_load 5130 - constant 5131 - !volatile & nontemporal 5132 5133 1. buffer/global/flat_load 5134 glc=1 slc=1 5135 5136 - volatile 5137 5138 1. buffer/global/flat_load 5139 glc=1 5140 2. s_waitcnt vmcnt(0) 5141 5142 - Must happen before 5143 any following volatile 5144 global/generic 5145 load/store. 5146 - Ensures that 5147 volatile 5148 operations to 5149 different 5150 addresses will not 5151 be reordered by 5152 hardware. 5153 5154 load *none* *none* - local 1. ds_load 5155 store *none* *none* - global - !volatile & !nontemporal 5156 - generic 5157 - private 1. buffer/global/flat_store 5158 - constant 5159 - !volatile & nontemporal 5160 5161 1. buffer/global/flat_store 5162 glc=1 slc=1 5163 5164 - volatile 5165 5166 1. buffer/global/flat_store 5167 2. s_waitcnt vmcnt(0) 5168 5169 - Must happen before 5170 any following volatile 5171 global/generic 5172 load/store. 5173 - Ensures that 5174 volatile 5175 operations to 5176 different 5177 addresses will not 5178 be reordered by 5179 hardware. 5180 5181 store *none* *none* - local 1. ds_store 5182 **Unordered Atomic** 5183 ------------------------------------------------------------------------------------ 5184 load atomic unordered *any* *any* *Same as non-atomic*. 5185 store atomic unordered *any* *any* *Same as non-atomic*. 5186 atomicrmw unordered *any* *any* *Same as monotonic atomic*. 5187 **Monotonic Atomic** 5188 ------------------------------------------------------------------------------------ 5189 load atomic monotonic - singlethread - global 1. buffer/global/ds/flat_load 5190 - wavefront - local 5191 - workgroup - generic 5192 load atomic monotonic - agent - global 1. buffer/global/flat_load 5193 - system - generic glc=1 5194 store atomic monotonic - singlethread - global 1. buffer/global/flat_store 5195 - wavefront - generic 5196 - workgroup 5197 - agent 5198 - system 5199 store atomic monotonic - singlethread - local 1. ds_store 5200 - wavefront 5201 - workgroup 5202 atomicrmw monotonic - singlethread - global 1. buffer/global/flat_atomic 5203 - wavefront - generic 5204 - workgroup 5205 - agent 5206 - system 5207 atomicrmw monotonic - singlethread - local 1. ds_atomic 5208 - wavefront 5209 - workgroup 5210 **Acquire Atomic** 5211 ------------------------------------------------------------------------------------ 5212 load atomic acquire - singlethread - global 1. buffer/global/ds/flat_load 5213 - wavefront - local 5214 - generic 5215 load atomic acquire - workgroup - global 1. buffer/global_load 5216 load atomic acquire - workgroup - local 1. ds/flat_load 5217 - generic 2. s_waitcnt lgkmcnt(0) 5218 5219 - If OpenCL, omit. 5220 - Must happen before 5221 any following 5222 global/generic 5223 load/load 5224 atomic/store/store 5225 atomic/atomicrmw. 5226 - Ensures any 5227 following global 5228 data read is no 5229 older than a local load 5230 atomic value being 5231 acquired. 5232 5233 load atomic acquire - agent - global 1. buffer/global_load 5234 - system glc=1 5235 2. s_waitcnt vmcnt(0) 5236 5237 - Must happen before 5238 following 5239 buffer_wbinvl1_vol. 5240 - Ensures the load 5241 has completed 5242 before invalidating 5243 the cache. 5244 5245 3. buffer_wbinvl1_vol 5246 5247 - Must happen before 5248 any following 5249 global/generic 5250 load/load 5251 atomic/atomicrmw. 5252 - Ensures that 5253 following 5254 loads will not see 5255 stale global data. 5256 5257 load atomic acquire - agent - generic 1. flat_load glc=1 5258 - system 2. s_waitcnt vmcnt(0) & 5259 lgkmcnt(0) 5260 5261 - If OpenCL omit 5262 lgkmcnt(0). 5263 - Must happen before 5264 following 5265 buffer_wbinvl1_vol. 5266 - Ensures the flat_load 5267 has completed 5268 before invalidating 5269 the cache. 5270 5271 3. buffer_wbinvl1_vol 5272 5273 - Must happen before 5274 any following 5275 global/generic 5276 load/load 5277 atomic/atomicrmw. 5278 - Ensures that 5279 following loads 5280 will not see stale 5281 global data. 5282 5283 atomicrmw acquire - singlethread - global 1. buffer/global/ds/flat_atomic 5284 - wavefront - local 5285 - generic 5286 atomicrmw acquire - workgroup - global 1. buffer/global_atomic 5287 atomicrmw acquire - workgroup - local 1. ds/flat_atomic 5288 - generic 2. s_waitcnt lgkmcnt(0) 5289 5290 - If OpenCL, omit. 5291 - Must happen before 5292 any following 5293 global/generic 5294 load/load 5295 atomic/store/store 5296 atomic/atomicrmw. 5297 - Ensures any 5298 following global 5299 data read is no 5300 older than a local 5301 atomicrmw value 5302 being acquired. 5303 5304 atomicrmw acquire - agent - global 1. buffer/global_atomic 5305 - system 2. s_waitcnt vmcnt(0) 5306 5307 - Must happen before 5308 following 5309 buffer_wbinvl1_vol. 5310 - Ensures the 5311 atomicrmw has 5312 completed before 5313 invalidating the 5314 cache. 5315 5316 3. buffer_wbinvl1_vol 5317 5318 - Must happen before 5319 any following 5320 global/generic 5321 load/load 5322 atomic/atomicrmw. 5323 - Ensures that 5324 following loads 5325 will not see stale 5326 global data. 5327 5328 atomicrmw acquire - agent - generic 1. flat_atomic 5329 - system 2. s_waitcnt vmcnt(0) & 5330 lgkmcnt(0) 5331 5332 - If OpenCL, omit 5333 lgkmcnt(0). 5334 - Must happen before 5335 following 5336 buffer_wbinvl1_vol. 5337 - Ensures the 5338 atomicrmw has 5339 completed before 5340 invalidating the 5341 cache. 5342 5343 3. buffer_wbinvl1_vol 5344 5345 - Must happen before 5346 any following 5347 global/generic 5348 load/load 5349 atomic/atomicrmw. 5350 - Ensures that 5351 following loads 5352 will not see stale 5353 global data. 5354 5355 fence acquire - singlethread *none* *none* 5356 - wavefront 5357 fence acquire - workgroup *none* 1. s_waitcnt lgkmcnt(0) 5358 5359 - If OpenCL and 5360 address space is 5361 not generic, omit. 5362 - However, since LLVM 5363 currently has no 5364 address space on 5365 the fence need to 5366 conservatively 5367 always generate. If 5368 fence had an 5369 address space then 5370 set to address 5371 space of OpenCL 5372 fence flag, or to 5373 generic if both 5374 local and global 5375 flags are 5376 specified. 5377 - Must happen after 5378 any preceding 5379 local/generic load 5380 atomic/atomicrmw 5381 with an equal or 5382 wider sync scope 5383 and memory ordering 5384 stronger than 5385 unordered (this is 5386 termed the 5387 fence-paired-atomic). 5388 - Must happen before 5389 any following 5390 global/generic 5391 load/load 5392 atomic/store/store 5393 atomic/atomicrmw. 5394 - Ensures any 5395 following global 5396 data read is no 5397 older than the 5398 value read by the 5399 fence-paired-atomic. 5400 5401 fence acquire - agent *none* 1. s_waitcnt lgkmcnt(0) & 5402 - system vmcnt(0) 5403 5404 - If OpenCL and 5405 address space is 5406 not generic, omit 5407 lgkmcnt(0). 5408 - However, since LLVM 5409 currently has no 5410 address space on 5411 the fence need to 5412 conservatively 5413 always generate 5414 (see comment for 5415 previous fence). 5416 - Could be split into 5417 separate s_waitcnt 5418 vmcnt(0) and 5419 s_waitcnt 5420 lgkmcnt(0) to allow 5421 them to be 5422 independently moved 5423 according to the 5424 following rules. 5425 - s_waitcnt vmcnt(0) 5426 must happen after 5427 any preceding 5428 global/generic load 5429 atomic/atomicrmw 5430 with an equal or 5431 wider sync scope 5432 and memory ordering 5433 stronger than 5434 unordered (this is 5435 termed the 5436 fence-paired-atomic). 5437 - s_waitcnt lgkmcnt(0) 5438 must happen after 5439 any preceding 5440 local/generic load 5441 atomic/atomicrmw 5442 with an equal or 5443 wider sync scope 5444 and memory ordering 5445 stronger than 5446 unordered (this is 5447 termed the 5448 fence-paired-atomic). 5449 - Must happen before 5450 the following 5451 buffer_wbinvl1_vol. 5452 - Ensures that the 5453 fence-paired atomic 5454 has completed 5455 before invalidating 5456 the 5457 cache. Therefore 5458 any following 5459 locations read must 5460 be no older than 5461 the value read by 5462 the 5463 fence-paired-atomic. 5464 5465 2. buffer_wbinvl1_vol 5466 5467 - Must happen before any 5468 following global/generic 5469 load/load 5470 atomic/store/store 5471 atomic/atomicrmw. 5472 - Ensures that 5473 following loads 5474 will not see stale 5475 global data. 5476 5477 **Release Atomic** 5478 ------------------------------------------------------------------------------------ 5479 store atomic release - singlethread - global 1. buffer/global/ds/flat_store 5480 - wavefront - local 5481 - generic 5482 store atomic release - workgroup - global 1. s_waitcnt lgkmcnt(0) 5483 - generic 5484 - If OpenCL, omit. 5485 - Must happen after 5486 any preceding 5487 local/generic 5488 load/store/load 5489 atomic/store 5490 atomic/atomicrmw. 5491 - Must happen before 5492 the following 5493 store. 5494 - Ensures that all 5495 memory operations 5496 to local have 5497 completed before 5498 performing the 5499 store that is being 5500 released. 5501 5502 2. buffer/global/flat_store 5503 store atomic release - workgroup - local 1. ds_store 5504 store atomic release - agent - global 1. s_waitcnt lgkmcnt(0) & 5505 - system - generic vmcnt(0) 5506 5507 - If OpenCL and 5508 address space is 5509 not generic, omit 5510 lgkmcnt(0). 5511 - Could be split into 5512 separate s_waitcnt 5513 vmcnt(0) and 5514 s_waitcnt 5515 lgkmcnt(0) to allow 5516 them to be 5517 independently moved 5518 according to the 5519 following rules. 5520 - s_waitcnt vmcnt(0) 5521 must happen after 5522 any preceding 5523 global/generic 5524 load/store/load 5525 atomic/store 5526 atomic/atomicrmw. 5527 - s_waitcnt lgkmcnt(0) 5528 must happen after 5529 any preceding 5530 local/generic 5531 load/store/load 5532 atomic/store 5533 atomic/atomicrmw. 5534 - Must happen before 5535 the following 5536 store. 5537 - Ensures that all 5538 memory operations 5539 to memory have 5540 completed before 5541 performing the 5542 store that is being 5543 released. 5544 5545 2. buffer/global/flat_store 5546 atomicrmw release - singlethread - global 1. buffer/global/ds/flat_atomic 5547 - wavefront - local 5548 - generic 5549 atomicrmw release - workgroup - global 1. s_waitcnt lgkmcnt(0) 5550 - generic 5551 - If OpenCL, omit. 5552 - Must happen after 5553 any preceding 5554 local/generic 5555 load/store/load 5556 atomic/store 5557 atomic/atomicrmw. 5558 - Must happen before 5559 the following 5560 atomicrmw. 5561 - Ensures that all 5562 memory operations 5563 to local have 5564 completed before 5565 performing the 5566 atomicrmw that is 5567 being released. 5568 5569 2. buffer/global/flat_atomic 5570 atomicrmw release - workgroup - local 1. ds_atomic 5571 atomicrmw release - agent - global 1. s_waitcnt lgkmcnt(0) & 5572 - system - generic vmcnt(0) 5573 5574 - If OpenCL, omit 5575 lgkmcnt(0). 5576 - Could be split into 5577 separate s_waitcnt 5578 vmcnt(0) and 5579 s_waitcnt 5580 lgkmcnt(0) to allow 5581 them to be 5582 independently moved 5583 according to the 5584 following rules. 5585 - s_waitcnt vmcnt(0) 5586 must happen after 5587 any preceding 5588 global/generic 5589 load/store/load 5590 atomic/store 5591 atomic/atomicrmw. 5592 - s_waitcnt lgkmcnt(0) 5593 must happen after 5594 any preceding 5595 local/generic 5596 load/store/load 5597 atomic/store 5598 atomic/atomicrmw. 5599 - Must happen before 5600 the following 5601 atomicrmw. 5602 - Ensures that all 5603 memory operations 5604 to global and local 5605 have completed 5606 before performing 5607 the atomicrmw that 5608 is being released. 5609 5610 2. buffer/global/flat_atomic 5611 fence release - singlethread *none* *none* 5612 - wavefront 5613 fence release - workgroup *none* 1. s_waitcnt lgkmcnt(0) 5614 5615 - If OpenCL and 5616 address space is 5617 not generic, omit. 5618 - However, since LLVM 5619 currently has no 5620 address space on 5621 the fence need to 5622 conservatively 5623 always generate. If 5624 fence had an 5625 address space then 5626 set to address 5627 space of OpenCL 5628 fence flag, or to 5629 generic if both 5630 local and global 5631 flags are 5632 specified. 5633 - Must happen after 5634 any preceding 5635 local/generic 5636 load/load 5637 atomic/store/store 5638 atomic/atomicrmw. 5639 - Must happen before 5640 any following store 5641 atomic/atomicrmw 5642 with an equal or 5643 wider sync scope 5644 and memory ordering 5645 stronger than 5646 unordered (this is 5647 termed the 5648 fence-paired-atomic). 5649 - Ensures that all 5650 memory operations 5651 to local have 5652 completed before 5653 performing the 5654 following 5655 fence-paired-atomic. 5656 5657 fence release - agent *none* 1. s_waitcnt lgkmcnt(0) & 5658 - system vmcnt(0) 5659 5660 - If OpenCL and 5661 address space is 5662 not generic, omit 5663 lgkmcnt(0). 5664 - If OpenCL and 5665 address space is 5666 local, omit 5667 vmcnt(0). 5668 - However, since LLVM 5669 currently has no 5670 address space on 5671 the fence need to 5672 conservatively 5673 always generate. If 5674 fence had an 5675 address space then 5676 set to address 5677 space of OpenCL 5678 fence flag, or to 5679 generic if both 5680 local and global 5681 flags are 5682 specified. 5683 - Could be split into 5684 separate s_waitcnt 5685 vmcnt(0) and 5686 s_waitcnt 5687 lgkmcnt(0) to allow 5688 them to be 5689 independently moved 5690 according to the 5691 following rules. 5692 - s_waitcnt vmcnt(0) 5693 must happen after 5694 any preceding 5695 global/generic 5696 load/store/load 5697 atomic/store 5698 atomic/atomicrmw. 5699 - s_waitcnt lgkmcnt(0) 5700 must happen after 5701 any preceding 5702 local/generic 5703 load/store/load 5704 atomic/store 5705 atomic/atomicrmw. 5706 - Must happen before 5707 any following store 5708 atomic/atomicrmw 5709 with an equal or 5710 wider sync scope 5711 and memory ordering 5712 stronger than 5713 unordered (this is 5714 termed the 5715 fence-paired-atomic). 5716 - Ensures that all 5717 memory operations 5718 have 5719 completed before 5720 performing the 5721 following 5722 fence-paired-atomic. 5723 5724 **Acquire-Release Atomic** 5725 ------------------------------------------------------------------------------------ 5726 atomicrmw acq_rel - singlethread - global 1. buffer/global/ds/flat_atomic 5727 - wavefront - local 5728 - generic 5729 atomicrmw acq_rel - workgroup - global 1. s_waitcnt lgkmcnt(0) 5730 5731 - If OpenCL, omit. 5732 - Must happen after 5733 any preceding 5734 local/generic 5735 load/store/load 5736 atomic/store 5737 atomic/atomicrmw. 5738 - Must happen before 5739 the following 5740 atomicrmw. 5741 - Ensures that all 5742 memory operations 5743 to local have 5744 completed before 5745 performing the 5746 atomicrmw that is 5747 being released. 5748 5749 2. buffer/global_atomic 5750 5751 atomicrmw acq_rel - workgroup - local 1. ds_atomic 5752 2. s_waitcnt lgkmcnt(0) 5753 5754 - If OpenCL, omit. 5755 - Must happen before 5756 any following 5757 global/generic 5758 load/load 5759 atomic/store/store 5760 atomic/atomicrmw. 5761 - Ensures any 5762 following global 5763 data read is no 5764 older than the local load 5765 atomic value being 5766 acquired. 5767 5768 atomicrmw acq_rel - workgroup - generic 1. s_waitcnt lgkmcnt(0) 5769 5770 - If OpenCL, omit. 5771 - Must happen after 5772 any preceding 5773 local/generic 5774 load/store/load 5775 atomic/store 5776 atomic/atomicrmw. 5777 - Must happen before 5778 the following 5779 atomicrmw. 5780 - Ensures that all 5781 memory operations 5782 to local have 5783 completed before 5784 performing the 5785 atomicrmw that is 5786 being released. 5787 5788 2. flat_atomic 5789 3. s_waitcnt lgkmcnt(0) 5790 5791 - If OpenCL, omit. 5792 - Must happen before 5793 any following 5794 global/generic 5795 load/load 5796 atomic/store/store 5797 atomic/atomicrmw. 5798 - Ensures any 5799 following global 5800 data read is no 5801 older than a local load 5802 atomic value being 5803 acquired. 5804 5805 atomicrmw acq_rel - agent - global 1. s_waitcnt lgkmcnt(0) & 5806 - system vmcnt(0) 5807 5808 - If OpenCL, omit 5809 lgkmcnt(0). 5810 - Could be split into 5811 separate s_waitcnt 5812 vmcnt(0) and 5813 s_waitcnt 5814 lgkmcnt(0) to allow 5815 them to be 5816 independently moved 5817 according to the 5818 following rules. 5819 - s_waitcnt vmcnt(0) 5820 must happen after 5821 any preceding 5822 global/generic 5823 load/store/load 5824 atomic/store 5825 atomic/atomicrmw. 5826 - s_waitcnt lgkmcnt(0) 5827 must happen after 5828 any preceding 5829 local/generic 5830 load/store/load 5831 atomic/store 5832 atomic/atomicrmw. 5833 - Must happen before 5834 the following 5835 atomicrmw. 5836 - Ensures that all 5837 memory operations 5838 to global have 5839 completed before 5840 performing the 5841 atomicrmw that is 5842 being released. 5843 5844 2. buffer/global_atomic 5845 3. s_waitcnt vmcnt(0) 5846 5847 - Must happen before 5848 following 5849 buffer_wbinvl1_vol. 5850 - Ensures the 5851 atomicrmw has 5852 completed before 5853 invalidating the 5854 cache. 5855 5856 4. buffer_wbinvl1_vol 5857 5858 - Must happen before 5859 any following 5860 global/generic 5861 load/load 5862 atomic/atomicrmw. 5863 - Ensures that 5864 following loads 5865 will not see stale 5866 global data. 5867 5868 atomicrmw acq_rel - agent - generic 1. s_waitcnt lgkmcnt(0) & 5869 - system vmcnt(0) 5870 5871 - If OpenCL, omit 5872 lgkmcnt(0). 5873 - Could be split into 5874 separate s_waitcnt 5875 vmcnt(0) and 5876 s_waitcnt 5877 lgkmcnt(0) to allow 5878 them to be 5879 independently moved 5880 according to the 5881 following rules. 5882 - s_waitcnt vmcnt(0) 5883 must happen after 5884 any preceding 5885 global/generic 5886 load/store/load 5887 atomic/store 5888 atomic/atomicrmw. 5889 - s_waitcnt lgkmcnt(0) 5890 must happen after 5891 any preceding 5892 local/generic 5893 load/store/load 5894 atomic/store 5895 atomic/atomicrmw. 5896 - Must happen before 5897 the following 5898 atomicrmw. 5899 - Ensures that all 5900 memory operations 5901 to global have 5902 completed before 5903 performing the 5904 atomicrmw that is 5905 being released. 5906 5907 2. flat_atomic 5908 3. s_waitcnt vmcnt(0) & 5909 lgkmcnt(0) 5910 5911 - If OpenCL, omit 5912 lgkmcnt(0). 5913 - Must happen before 5914 following 5915 buffer_wbinvl1_vol. 5916 - Ensures the 5917 atomicrmw has 5918 completed before 5919 invalidating the 5920 cache. 5921 5922 4. buffer_wbinvl1_vol 5923 5924 - Must happen before 5925 any following 5926 global/generic 5927 load/load 5928 atomic/atomicrmw. 5929 - Ensures that 5930 following loads 5931 will not see stale 5932 global data. 5933 5934 fence acq_rel - singlethread *none* *none* 5935 - wavefront 5936 fence acq_rel - workgroup *none* 1. s_waitcnt lgkmcnt(0) 5937 5938 - If OpenCL and 5939 address space is 5940 not generic, omit. 5941 - However, 5942 since LLVM 5943 currently has no 5944 address space on 5945 the fence need to 5946 conservatively 5947 always generate 5948 (see comment for 5949 previous fence). 5950 - Must happen after 5951 any preceding 5952 local/generic 5953 load/load 5954 atomic/store/store 5955 atomic/atomicrmw. 5956 - Must happen before 5957 any following 5958 global/generic 5959 load/load 5960 atomic/store/store 5961 atomic/atomicrmw. 5962 - Ensures that all 5963 memory operations 5964 to local have 5965 completed before 5966 performing any 5967 following global 5968 memory operations. 5969 - Ensures that the 5970 preceding 5971 local/generic load 5972 atomic/atomicrmw 5973 with an equal or 5974 wider sync scope 5975 and memory ordering 5976 stronger than 5977 unordered (this is 5978 termed the 5979 acquire-fence-paired-atomic) 5980 has completed 5981 before following 5982 global memory 5983 operations. This 5984 satisfies the 5985 requirements of 5986 acquire. 5987 - Ensures that all 5988 previous memory 5989 operations have 5990 completed before a 5991 following 5992 local/generic store 5993 atomic/atomicrmw 5994 with an equal or 5995 wider sync scope 5996 and memory ordering 5997 stronger than 5998 unordered (this is 5999 termed the 6000 release-fence-paired-atomic). 6001 This satisfies the 6002 requirements of 6003 release. 6004 6005 fence acq_rel - agent *none* 1. s_waitcnt lgkmcnt(0) & 6006 - system vmcnt(0) 6007 6008 - If OpenCL and 6009 address space is 6010 not generic, omit 6011 lgkmcnt(0). 6012 - However, since LLVM 6013 currently has no 6014 address space on 6015 the fence need to 6016 conservatively 6017 always generate 6018 (see comment for 6019 previous fence). 6020 - Could be split into 6021 separate s_waitcnt 6022 vmcnt(0) and 6023 s_waitcnt 6024 lgkmcnt(0) to allow 6025 them to be 6026 independently moved 6027 according to the 6028 following rules. 6029 - s_waitcnt vmcnt(0) 6030 must happen after 6031 any preceding 6032 global/generic 6033 load/store/load 6034 atomic/store 6035 atomic/atomicrmw. 6036 - s_waitcnt lgkmcnt(0) 6037 must happen after 6038 any preceding 6039 local/generic 6040 load/store/load 6041 atomic/store 6042 atomic/atomicrmw. 6043 - Must happen before 6044 the following 6045 buffer_wbinvl1_vol. 6046 - Ensures that the 6047 preceding 6048 global/local/generic 6049 load 6050 atomic/atomicrmw 6051 with an equal or 6052 wider sync scope 6053 and memory ordering 6054 stronger than 6055 unordered (this is 6056 termed the 6057 acquire-fence-paired-atomic) 6058 has completed 6059 before invalidating 6060 the cache. This 6061 satisfies the 6062 requirements of 6063 acquire. 6064 - Ensures that all 6065 previous memory 6066 operations have 6067 completed before a 6068 following 6069 global/local/generic 6070 store 6071 atomic/atomicrmw 6072 with an equal or 6073 wider sync scope 6074 and memory ordering 6075 stronger than 6076 unordered (this is 6077 termed the 6078 release-fence-paired-atomic). 6079 This satisfies the 6080 requirements of 6081 release. 6082 6083 2. buffer_wbinvl1_vol 6084 6085 - Must happen before 6086 any following 6087 global/generic 6088 load/load 6089 atomic/store/store 6090 atomic/atomicrmw. 6091 - Ensures that 6092 following loads 6093 will not see stale 6094 global data. This 6095 satisfies the 6096 requirements of 6097 acquire. 6098 6099 **Sequential Consistent Atomic** 6100 ------------------------------------------------------------------------------------ 6101 load atomic seq_cst - singlethread - global *Same as corresponding 6102 - wavefront - local load atomic acquire, 6103 - generic except must generate 6104 all instructions even 6105 for OpenCL.* 6106 load atomic seq_cst - workgroup - global 1. s_waitcnt lgkmcnt(0) 6107 - generic 6108 6109 - Must 6110 happen after 6111 preceding 6112 local/generic load 6113 atomic/store 6114 atomic/atomicrmw 6115 with memory 6116 ordering of seq_cst 6117 and with equal or 6118 wider sync scope. 6119 (Note that seq_cst 6120 fences have their 6121 own s_waitcnt 6122 lgkmcnt(0) and so do 6123 not need to be 6124 considered.) 6125 - Ensures any 6126 preceding 6127 sequential 6128 consistent local 6129 memory instructions 6130 have completed 6131 before executing 6132 this sequentially 6133 consistent 6134 instruction. This 6135 prevents reordering 6136 a seq_cst store 6137 followed by a 6138 seq_cst load. (Note 6139 that seq_cst is 6140 stronger than 6141 acquire/release as 6142 the reordering of 6143 load acquire 6144 followed by a store 6145 release is 6146 prevented by the 6147 s_waitcnt of 6148 the release, but 6149 there is nothing 6150 preventing a store 6151 release followed by 6152 load acquire from 6153 completing out of 6154 order. The s_waitcnt 6155 could be placed after 6156 seq_store or before 6157 the seq_load. We 6158 choose the load to 6159 make the s_waitcnt be 6160 as late as possible 6161 so that the store 6162 may have already 6163 completed.) 6164 6165 2. *Following 6166 instructions same as 6167 corresponding load 6168 atomic acquire, 6169 except must generate 6170 all instructions even 6171 for OpenCL.* 6172 load atomic seq_cst - workgroup - local *Same as corresponding 6173 load atomic acquire, 6174 except must generate 6175 all instructions even 6176 for OpenCL.* 6177 6178 load atomic seq_cst - agent - global 1. s_waitcnt lgkmcnt(0) & 6179 - system - generic vmcnt(0) 6180 6181 - Could be split into 6182 separate s_waitcnt 6183 vmcnt(0) 6184 and s_waitcnt 6185 lgkmcnt(0) to allow 6186 them to be 6187 independently moved 6188 according to the 6189 following rules. 6190 - s_waitcnt lgkmcnt(0) 6191 must happen after 6192 preceding 6193 global/generic load 6194 atomic/store 6195 atomic/atomicrmw 6196 with memory 6197 ordering of seq_cst 6198 and with equal or 6199 wider sync scope. 6200 (Note that seq_cst 6201 fences have their 6202 own s_waitcnt 6203 lgkmcnt(0) and so do 6204 not need to be 6205 considered.) 6206 - s_waitcnt vmcnt(0) 6207 must happen after 6208 preceding 6209 global/generic load 6210 atomic/store 6211 atomic/atomicrmw 6212 with memory 6213 ordering of seq_cst 6214 and with equal or 6215 wider sync scope. 6216 (Note that seq_cst 6217 fences have their 6218 own s_waitcnt 6219 vmcnt(0) and so do 6220 not need to be 6221 considered.) 6222 - Ensures any 6223 preceding 6224 sequential 6225 consistent global 6226 memory instructions 6227 have completed 6228 before executing 6229 this sequentially 6230 consistent 6231 instruction. This 6232 prevents reordering 6233 a seq_cst store 6234 followed by a 6235 seq_cst load. (Note 6236 that seq_cst is 6237 stronger than 6238 acquire/release as 6239 the reordering of 6240 load acquire 6241 followed by a store 6242 release is 6243 prevented by the 6244 s_waitcnt of 6245 the release, but 6246 there is nothing 6247 preventing a store 6248 release followed by 6249 load acquire from 6250 completing out of 6251 order. The s_waitcnt 6252 could be placed after 6253 seq_store or before 6254 the seq_load. We 6255 choose the load to 6256 make the s_waitcnt be 6257 as late as possible 6258 so that the store 6259 may have already 6260 completed.) 6261 6262 2. *Following 6263 instructions same as 6264 corresponding load 6265 atomic acquire, 6266 except must generate 6267 all instructions even 6268 for OpenCL.* 6269 store atomic seq_cst - singlethread - global *Same as corresponding 6270 - wavefront - local store atomic release, 6271 - workgroup - generic except must generate 6272 - agent all instructions even 6273 - system for OpenCL.* 6274 atomicrmw seq_cst - singlethread - global *Same as corresponding 6275 - wavefront - local atomicrmw acq_rel, 6276 - workgroup - generic except must generate 6277 - agent all instructions even 6278 - system for OpenCL.* 6279 fence seq_cst - singlethread *none* *Same as corresponding 6280 - wavefront fence acq_rel, 6281 - workgroup except must generate 6282 - agent all instructions even 6283 - system for OpenCL.* 6284 ============ ============ ============== ========== ================================ 6285 6286.. _amdgpu-amdhsa-memory-model-gfx90a: 6287 6288Memory Model GFX90A 6289+++++++++++++++++++ 6290 6291For GFX90A: 6292 6293* Each agent has multiple shader arrays (SA). 6294* Each SA has multiple compute units (CU). 6295* Each CU has multiple SIMDs that execute wavefronts. 6296* The wavefronts for a single work-group are executed in the same CU but may be 6297 executed by different SIMDs. The exception is when in tgsplit execution mode 6298 when the wavefronts may be executed by different SIMDs in different CUs. 6299* Each CU has a single LDS memory shared by the wavefronts of the work-groups 6300 executing on it. The exception is when in tgsplit execution mode when no LDS 6301 is allocated as wavefronts of the same work-group can be in different CUs. 6302* All LDS operations of a CU are performed as wavefront wide operations in a 6303 global order and involve no caching. Completion is reported to a wavefront in 6304 execution order. 6305* The LDS memory has multiple request queues shared by the SIMDs of a 6306 CU. Therefore, the LDS operations performed by different wavefronts of a 6307 work-group can be reordered relative to each other, which can result in 6308 reordering the visibility of vector memory operations with respect to LDS 6309 operations of other wavefronts in the same work-group. A ``s_waitcnt 6310 lgkmcnt(0)`` is required to ensure synchronization between LDS operations and 6311 vector memory operations between wavefronts of a work-group, but not between 6312 operations performed by the same wavefront. 6313* The vector memory operations are performed as wavefront wide operations and 6314 completion is reported to a wavefront in execution order. The exception is 6315 that ``flat_load/store/atomic`` instructions can report out of vector memory 6316 order if they access LDS memory, and out of LDS operation order if they access 6317 global memory. 6318* The vector memory operations access a single vector L1 cache shared by all 6319 SIMDs a CU. Therefore: 6320 6321 * No special action is required for coherence between the lanes of a single 6322 wavefront. 6323 6324 * No special action is required for coherence between wavefronts in the same 6325 work-group since they execute on the same CU. The exception is when in 6326 tgsplit execution mode as wavefronts of the same work-group can be in 6327 different CUs and so a ``buffer_wbinvl1_vol`` is required as described in 6328 the following item. 6329 6330 * A ``buffer_wbinvl1_vol`` is required for coherence between wavefronts 6331 executing in different work-groups as they may be executing on different 6332 CUs. 6333 6334* The scalar memory operations access a scalar L1 cache shared by all wavefronts 6335 on a group of CUs. The scalar and vector L1 caches are not coherent. However, 6336 scalar operations are used in a restricted way so do not impact the memory 6337 model. See :ref:`amdgpu-amdhsa-memory-spaces`. 6338* The vector and scalar memory operations use an L2 cache shared by all CUs on 6339 the same agent. 6340 6341 * The L2 cache has independent channels to service disjoint ranges of virtual 6342 addresses. 6343 * Each CU has a separate request queue per channel. Therefore, the vector and 6344 scalar memory operations performed by wavefronts executing in different 6345 work-groups (which may be executing on different CUs), or the same 6346 work-group if executing in tgsplit mode, of an agent can be reordered 6347 relative to each other. A ``s_waitcnt vmcnt(0)`` is required to ensure 6348 synchronization between vector memory operations of different CUs. It 6349 ensures a previous vector memory operation has completed before executing a 6350 subsequent vector memory or LDS operation and so can be used to meet the 6351 requirements of acquire and release. 6352 * The L2 cache of one agent can be kept coherent with other agents by: 6353 using the MTYPE RW (read-write) or MTYPE CC (cache-coherent) with the PTE 6354 C-bit for memory local to the L2; and using the MTYPE NC (non-coherent) with 6355 the PTE C-bit set or MTYPE UC (uncached) for memory not local to the L2. 6356 6357 * Any local memory cache lines will be automatically invalidated by writes 6358 from CUs associated with other L2 caches, or writes from the CPU, due to 6359 the cache probe caused by coherent requests. Coherent requests are caused 6360 by GPU accesses to pages with the PTE C-bit set, by CPU accesses over 6361 XGMI, and by PCIe requests that are configured to be coherent requests. 6362 * XGMI accesses from the CPU to local memory may be cached on the CPU. 6363 Subsequent access from the GPU will automatically invalidate or writeback 6364 the CPU cache due to the L2 probe filter and and the PTE C-bit being set. 6365 * Since all work-groups on the same agent share the same L2, no L2 6366 invalidation or writeback is required for coherence. 6367 * To ensure coherence of local and remote memory writes of work-groups in 6368 different agents a ``buffer_wbl2`` is required. It will writeback dirty L2 6369 cache lines of MTYPE RW (used for local coarse grain memory) and MTYPE NC 6370 ()used for remote coarse grain memory). Note that MTYPE CC (used for local 6371 fine grain memory) causes write through to DRAM, and MTYPE UC (used for 6372 remote fine grain memory) bypasses the L2, so both will never result in 6373 dirty L2 cache lines. 6374 * To ensure coherence of local and remote memory reads of work-groups in 6375 different agents a ``buffer_invl2`` is required. It will invalidate L2 6376 cache lines with MTYPE NC (used for remote coarse grain memory). Note that 6377 MTYPE CC (used for local fine grain memory) and MTYPE RW (used for local 6378 coarse memory) cause local reads to be invalidated by remote writes with 6379 with the PTE C-bit so these cache lines are not invalidated. Note that 6380 MTYPE UC (used for remote fine grain memory) bypasses the L2, so will 6381 never result in L2 cache lines that need to be invalidated. 6382 6383 * PCIe access from the GPU to the CPU memory is kept coherent by using the 6384 MTYPE UC (uncached) which bypasses the L2. 6385 6386Scalar memory operations are only used to access memory that is proven to not 6387change during the execution of the kernel dispatch. This includes constant 6388address space and global address space for program scope ``const`` variables. 6389Therefore, the kernel machine code does not have to maintain the scalar cache to 6390ensure it is coherent with the vector caches. The scalar and vector caches are 6391invalidated between kernel dispatches by CP since constant address space data 6392may change between kernel dispatch executions. See 6393:ref:`amdgpu-amdhsa-memory-spaces`. 6394 6395The one exception is if scalar writes are used to spill SGPR registers. In this 6396case the AMDGPU backend ensures the memory location used to spill is never 6397accessed by vector memory operations at the same time. If scalar writes are used 6398then a ``s_dcache_wb`` is inserted before the ``s_endpgm`` and before a function 6399return since the locations may be used for vector memory instructions by a 6400future wavefront that uses the same scratch area, or a function call that 6401creates a frame at the same address, respectively. There is no need for a 6402``s_dcache_inv`` as all scalar writes are write-before-read in the same thread. 6403 6404For kernarg backing memory: 6405 6406* CP invalidates the L1 cache at the start of each kernel dispatch. 6407* On dGPU over XGMI or PCIe the kernarg backing memory is allocated in host 6408 memory accessed as MTYPE UC (uncached) to avoid needing to invalidate the L2 6409 cache. This also causes it to be treated as non-volatile and so is not 6410 invalidated by ``*_vol``. 6411* On APU the kernarg backing memory is accessed as MTYPE CC (cache coherent) and 6412 so the L2 cache will be coherent with the CPU and other agents. 6413 6414Scratch backing memory (which is used for the private address space) is accessed 6415with MTYPE NC_NV (non-coherent non-volatile). Since the private address space is 6416only accessed by a single thread, and is always write-before-read, there is 6417never a need to invalidate these entries from the L1 cache. Hence all cache 6418invalidates are done as ``*_vol`` to only invalidate the volatile cache lines. 6419 6420The code sequences used to implement the memory model for GFX90A are defined 6421in table :ref:`amdgpu-amdhsa-memory-model-code-sequences-gfx90a-table`. 6422 6423 .. table:: AMDHSA Memory Model Code Sequences GFX90A 6424 :name: amdgpu-amdhsa-memory-model-code-sequences-gfx90a-table 6425 6426 ============ ============ ============== ========== ================================ 6427 LLVM Instr LLVM Memory LLVM Memory AMDGPU AMDGPU Machine Code 6428 Ordering Sync Scope Address GFX90A 6429 Space 6430 ============ ============ ============== ========== ================================ 6431 **Non-Atomic** 6432 ------------------------------------------------------------------------------------ 6433 load *none* *none* - global - !volatile & !nontemporal 6434 - generic 6435 - private 1. buffer/global/flat_load 6436 - constant 6437 - !volatile & nontemporal 6438 6439 1. buffer/global/flat_load 6440 glc=1 slc=1 6441 6442 - volatile 6443 6444 1. buffer/global/flat_load 6445 glc=1 6446 2. s_waitcnt vmcnt(0) 6447 6448 - Must happen before 6449 any following volatile 6450 global/generic 6451 load/store. 6452 - Ensures that 6453 volatile 6454 operations to 6455 different 6456 addresses will not 6457 be reordered by 6458 hardware. 6459 6460 load *none* *none* - local 1. ds_load 6461 store *none* *none* - global - !volatile & !nontemporal 6462 - generic 6463 - private 1. buffer/global/flat_store 6464 - constant 6465 - !volatile & nontemporal 6466 6467 1. buffer/global/flat_store 6468 glc=1 slc=1 6469 6470 - volatile 6471 6472 1. buffer/global/flat_store 6473 2. s_waitcnt vmcnt(0) 6474 6475 - Must happen before 6476 any following volatile 6477 global/generic 6478 load/store. 6479 - Ensures that 6480 volatile 6481 operations to 6482 different 6483 addresses will not 6484 be reordered by 6485 hardware. 6486 6487 store *none* *none* - local 1. ds_store 6488 **Unordered Atomic** 6489 ------------------------------------------------------------------------------------ 6490 load atomic unordered *any* *any* *Same as non-atomic*. 6491 store atomic unordered *any* *any* *Same as non-atomic*. 6492 atomicrmw unordered *any* *any* *Same as monotonic atomic*. 6493 **Monotonic Atomic** 6494 ------------------------------------------------------------------------------------ 6495 load atomic monotonic - singlethread - global 1. buffer/global/flat_load 6496 - wavefront - generic 6497 load atomic monotonic - workgroup - global 1. buffer/global/flat_load 6498 - generic glc=1 6499 6500 - If not TgSplit execution 6501 mode, omit glc=1. 6502 6503 load atomic monotonic - singlethread - local *If TgSplit execution mode, 6504 - wavefront local address space cannot 6505 - workgroup be used.* 6506 6507 1. ds_load 6508 load atomic monotonic - agent - global 1. buffer/global/flat_load 6509 - generic glc=1 6510 load atomic monotonic - system - global 1. buffer/global/flat_load 6511 - generic glc=1 6512 store atomic monotonic - singlethread - global 1. buffer/global/flat_store 6513 - wavefront - generic 6514 - workgroup 6515 - agent 6516 store atomic monotonic - system - global 1. buffer/global/flat_store 6517 - generic 6518 store atomic monotonic - singlethread - local *If TgSplit execution mode, 6519 - wavefront local address space cannot 6520 - workgroup be used.* 6521 6522 1. ds_store 6523 atomicrmw monotonic - singlethread - global 1. buffer/global/flat_atomic 6524 - wavefront - generic 6525 - workgroup 6526 - agent 6527 atomicrmw monotonic - system - global 1. buffer/global/flat_atomic 6528 - generic 6529 atomicrmw monotonic - singlethread - local *If TgSplit execution mode, 6530 - wavefront local address space cannot 6531 - workgroup be used.* 6532 6533 1. ds_atomic 6534 **Acquire Atomic** 6535 ------------------------------------------------------------------------------------ 6536 load atomic acquire - singlethread - global 1. buffer/global/ds/flat_load 6537 - wavefront - local 6538 - generic 6539 load atomic acquire - workgroup - global 1. buffer/global_load glc=1 6540 6541 - If not TgSplit execution 6542 mode, omit glc=1. 6543 6544 2. s_waitcnt vmcnt(0) 6545 6546 - If not TgSplit execution 6547 mode, omit. 6548 - Must happen before the 6549 following buffer_wbinvl1_vol. 6550 6551 3. buffer_wbinvl1_vol 6552 6553 - If not TgSplit execution 6554 mode, omit. 6555 - Must happen before 6556 any following 6557 global/generic 6558 load/load 6559 atomic/store/store 6560 atomic/atomicrmw. 6561 - Ensures that 6562 following 6563 loads will not see 6564 stale data. 6565 6566 load atomic acquire - workgroup - local *If TgSplit execution mode, 6567 local address space cannot 6568 be used.* 6569 6570 1. ds_load 6571 2. s_waitcnt lgkmcnt(0) 6572 6573 - If OpenCL, omit. 6574 - Must happen before 6575 any following 6576 global/generic 6577 load/load 6578 atomic/store/store 6579 atomic/atomicrmw. 6580 - Ensures any 6581 following global 6582 data read is no 6583 older than the local load 6584 atomic value being 6585 acquired. 6586 6587 load atomic acquire - workgroup - generic 1. flat_load glc=1 6588 6589 - If not TgSplit execution 6590 mode, omit glc=1. 6591 6592 2. s_waitcnt lgkm/vmcnt(0) 6593 6594 - Use lgkmcnt(0) if not 6595 TgSplit execution mode 6596 and vmcnt(0) if TgSplit 6597 execution mode. 6598 - If OpenCL, omit lgkmcnt(0). 6599 - Must happen before 6600 the following 6601 buffer_wbinvl1_vol and any 6602 following global/generic 6603 load/load 6604 atomic/store/store 6605 atomic/atomicrmw. 6606 - Ensures any 6607 following global 6608 data read is no 6609 older than a local load 6610 atomic value being 6611 acquired. 6612 6613 3. buffer_wbinvl1_vol 6614 6615 - If not TgSplit execution 6616 mode, omit. 6617 - Ensures that 6618 following 6619 loads will not see 6620 stale data. 6621 6622 load atomic acquire - agent - global 1. buffer/global_load 6623 glc=1 6624 2. s_waitcnt vmcnt(0) 6625 6626 - Must happen before 6627 following 6628 buffer_wbinvl1_vol. 6629 - Ensures the load 6630 has completed 6631 before invalidating 6632 the cache. 6633 6634 3. buffer_wbinvl1_vol 6635 6636 - Must happen before 6637 any following 6638 global/generic 6639 load/load 6640 atomic/atomicrmw. 6641 - Ensures that 6642 following 6643 loads will not see 6644 stale global data. 6645 6646 load atomic acquire - system - global 1. buffer/global/flat_load 6647 glc=1 6648 2. s_waitcnt vmcnt(0) 6649 6650 - Must happen before 6651 following buffer_invl2 and 6652 buffer_wbinvl1_vol. 6653 - Ensures the load 6654 has completed 6655 before invalidating 6656 the cache. 6657 6658 3. buffer_invl2; 6659 buffer_wbinvl1_vol 6660 6661 - Must happen before 6662 any following 6663 global/generic 6664 load/load 6665 atomic/atomicrmw. 6666 - Ensures that 6667 following 6668 loads will not see 6669 stale L1 global data, 6670 nor see stale L2 MTYPE 6671 NC global data. 6672 MTYPE RW and CC memory will 6673 never be stale in L2 due to 6674 the memory probes. 6675 6676 load atomic acquire - agent - generic 1. flat_load glc=1 6677 2. s_waitcnt vmcnt(0) & 6678 lgkmcnt(0) 6679 6680 - If TgSplit execution mode, 6681 omit lgkmcnt(0). 6682 - If OpenCL omit 6683 lgkmcnt(0). 6684 - Must happen before 6685 following 6686 buffer_wbinvl1_vol. 6687 - Ensures the flat_load 6688 has completed 6689 before invalidating 6690 the cache. 6691 6692 3. buffer_wbinvl1_vol 6693 6694 - Must happen before 6695 any following 6696 global/generic 6697 load/load 6698 atomic/atomicrmw. 6699 - Ensures that 6700 following loads 6701 will not see stale 6702 global data. 6703 6704 load atomic acquire - system - generic 1. flat_load glc=1 6705 2. s_waitcnt vmcnt(0) & 6706 lgkmcnt(0) 6707 6708 - If TgSplit execution mode, 6709 omit lgkmcnt(0). 6710 - If OpenCL omit 6711 lgkmcnt(0). 6712 - Must happen before 6713 following 6714 buffer_invl2 and 6715 buffer_wbinvl1_vol. 6716 - Ensures the flat_load 6717 has completed 6718 before invalidating 6719 the caches. 6720 6721 3. buffer_invl2; 6722 buffer_wbinvl1_vol 6723 6724 - Must happen before 6725 any following 6726 global/generic 6727 load/load 6728 atomic/atomicrmw. 6729 - Ensures that 6730 following 6731 loads will not see 6732 stale L1 global data, 6733 nor see stale L2 MTYPE 6734 NC global data. 6735 MTYPE RW and CC memory will 6736 never be stale in L2 due to 6737 the memory probes. 6738 6739 atomicrmw acquire - singlethread - global 1. buffer/global/flat_atomic 6740 - wavefront - generic 6741 atomicrmw acquire - singlethread - local *If TgSplit execution mode, 6742 - wavefront local address space cannot 6743 be used.* 6744 6745 1. ds_atomic 6746 atomicrmw acquire - workgroup - global 1. buffer/global_atomic 6747 2. s_waitcnt vmcnt(0) 6748 6749 - If not TgSplit execution 6750 mode, omit. 6751 - Must happen before the 6752 following buffer_wbinvl1_vol. 6753 - Ensures the atomicrmw 6754 has completed 6755 before invalidating 6756 the cache. 6757 6758 3. buffer_wbinvl1_vol 6759 6760 - If not TgSplit execution 6761 mode, omit. 6762 - Must happen before 6763 any following 6764 global/generic 6765 load/load 6766 atomic/atomicrmw. 6767 - Ensures that 6768 following loads 6769 will not see stale 6770 global data. 6771 6772 atomicrmw acquire - workgroup - local *If TgSplit execution mode, 6773 local address space cannot 6774 be used.* 6775 6776 1. ds_atomic 6777 2. s_waitcnt lgkmcnt(0) 6778 6779 - If OpenCL, omit. 6780 - Must happen before 6781 any following 6782 global/generic 6783 load/load 6784 atomic/store/store 6785 atomic/atomicrmw. 6786 - Ensures any 6787 following global 6788 data read is no 6789 older than the local 6790 atomicrmw value 6791 being acquired. 6792 6793 atomicrmw acquire - workgroup - generic 1. flat_atomic 6794 2. s_waitcnt lgkm/vmcnt(0) 6795 6796 - Use lgkmcnt(0) if not 6797 TgSplit execution mode 6798 and vmcnt(0) if TgSplit 6799 execution mode. 6800 - If OpenCL, omit lgkmcnt(0). 6801 - Must happen before 6802 the following 6803 buffer_wbinvl1_vol and 6804 any following 6805 global/generic 6806 load/load 6807 atomic/store/store 6808 atomic/atomicrmw. 6809 - Ensures any 6810 following global 6811 data read is no 6812 older than a local 6813 atomicrmw value 6814 being acquired. 6815 6816 3. buffer_wbinvl1_vol 6817 6818 - If not TgSplit execution 6819 mode, omit. 6820 - Ensures that 6821 following 6822 loads will not see 6823 stale data. 6824 6825 atomicrmw acquire - agent - global 1. buffer/global_atomic 6826 2. s_waitcnt vmcnt(0) 6827 6828 - Must happen before 6829 following 6830 buffer_wbinvl1_vol. 6831 - Ensures the 6832 atomicrmw has 6833 completed before 6834 invalidating the 6835 cache. 6836 6837 3. buffer_wbinvl1_vol 6838 6839 - Must happen before 6840 any following 6841 global/generic 6842 load/load 6843 atomic/atomicrmw. 6844 - Ensures that 6845 following loads 6846 will not see stale 6847 global data. 6848 6849 atomicrmw acquire - system - global 1. buffer/global_atomic 6850 2. s_waitcnt vmcnt(0) 6851 6852 - Must happen before 6853 following buffer_invl2 and 6854 buffer_wbinvl1_vol. 6855 - Ensures the 6856 atomicrmw has 6857 completed before 6858 invalidating the 6859 caches. 6860 6861 3. buffer_invl2; 6862 buffer_wbinvl1_vol 6863 6864 - Must happen before 6865 any following 6866 global/generic 6867 load/load 6868 atomic/atomicrmw. 6869 - Ensures that 6870 following 6871 loads will not see 6872 stale L1 global data, 6873 nor see stale L2 MTYPE 6874 NC global data. 6875 MTYPE RW and CC memory will 6876 never be stale in L2 due to 6877 the memory probes. 6878 6879 atomicrmw acquire - agent - generic 1. flat_atomic 6880 2. s_waitcnt vmcnt(0) & 6881 lgkmcnt(0) 6882 6883 - If TgSplit execution mode, 6884 omit lgkmcnt(0). 6885 - If OpenCL, omit 6886 lgkmcnt(0). 6887 - Must happen before 6888 following 6889 buffer_wbinvl1_vol. 6890 - Ensures the 6891 atomicrmw has 6892 completed before 6893 invalidating the 6894 cache. 6895 6896 3. buffer_wbinvl1_vol 6897 6898 - Must happen before 6899 any following 6900 global/generic 6901 load/load 6902 atomic/atomicrmw. 6903 - Ensures that 6904 following loads 6905 will not see stale 6906 global data. 6907 6908 atomicrmw acquire - system - generic 1. flat_atomic 6909 2. s_waitcnt vmcnt(0) & 6910 lgkmcnt(0) 6911 6912 - If TgSplit execution mode, 6913 omit lgkmcnt(0). 6914 - If OpenCL, omit 6915 lgkmcnt(0). 6916 - Must happen before 6917 following 6918 buffer_invl2 and 6919 buffer_wbinvl1_vol. 6920 - Ensures the 6921 atomicrmw has 6922 completed before 6923 invalidating the 6924 caches. 6925 6926 3. buffer_invl2; 6927 buffer_wbinvl1_vol 6928 6929 - Must happen before 6930 any following 6931 global/generic 6932 load/load 6933 atomic/atomicrmw. 6934 - Ensures that 6935 following 6936 loads will not see 6937 stale L1 global data, 6938 nor see stale L2 MTYPE 6939 NC global data. 6940 MTYPE RW and CC memory will 6941 never be stale in L2 due to 6942 the memory probes. 6943 6944 fence acquire - singlethread *none* *none* 6945 - wavefront 6946 fence acquire - workgroup *none* 1. s_waitcnt lgkm/vmcnt(0) 6947 6948 - Use lgkmcnt(0) if not 6949 TgSplit execution mode 6950 and vmcnt(0) if TgSplit 6951 execution mode. 6952 - If OpenCL and 6953 address space is 6954 not generic, omit 6955 lgkmcnt(0). 6956 - If OpenCL and 6957 address space is 6958 local, omit 6959 vmcnt(0). 6960 - However, since LLVM 6961 currently has no 6962 address space on 6963 the fence need to 6964 conservatively 6965 always generate. If 6966 fence had an 6967 address space then 6968 set to address 6969 space of OpenCL 6970 fence flag, or to 6971 generic if both 6972 local and global 6973 flags are 6974 specified. 6975 - s_waitcnt vmcnt(0) 6976 must happen after 6977 any preceding 6978 global/generic load 6979 atomic/ 6980 atomicrmw 6981 with an equal or 6982 wider sync scope 6983 and memory ordering 6984 stronger than 6985 unordered (this is 6986 termed the 6987 fence-paired-atomic). 6988 - s_waitcnt lgkmcnt(0) 6989 must happen after 6990 any preceding 6991 local/generic load 6992 atomic/atomicrmw 6993 with an equal or 6994 wider sync scope 6995 and memory ordering 6996 stronger than 6997 unordered (this is 6998 termed the 6999 fence-paired-atomic). 7000 - Must happen before 7001 the following 7002 buffer_wbinvl1_vol and 7003 any following 7004 global/generic 7005 load/load 7006 atomic/store/store 7007 atomic/atomicrmw. 7008 - Ensures any 7009 following global 7010 data read is no 7011 older than the 7012 value read by the 7013 fence-paired-atomic. 7014 7015 2. buffer_wbinvl1_vol 7016 7017 - If not TgSplit execution 7018 mode, omit. 7019 - Ensures that 7020 following 7021 loads will not see 7022 stale data. 7023 7024 fence acquire - agent *none* 1. s_waitcnt lgkmcnt(0) & 7025 vmcnt(0) 7026 7027 - If TgSplit execution mode, 7028 omit lgkmcnt(0). 7029 - If OpenCL and 7030 address space is 7031 not generic, omit 7032 lgkmcnt(0). 7033 - However, since LLVM 7034 currently has no 7035 address space on 7036 the fence need to 7037 conservatively 7038 always generate 7039 (see comment for 7040 previous fence). 7041 - Could be split into 7042 separate s_waitcnt 7043 vmcnt(0) and 7044 s_waitcnt 7045 lgkmcnt(0) to allow 7046 them to be 7047 independently moved 7048 according to the 7049 following rules. 7050 - s_waitcnt vmcnt(0) 7051 must happen after 7052 any preceding 7053 global/generic load 7054 atomic/atomicrmw 7055 with an equal or 7056 wider sync scope 7057 and memory ordering 7058 stronger than 7059 unordered (this is 7060 termed the 7061 fence-paired-atomic). 7062 - s_waitcnt lgkmcnt(0) 7063 must happen after 7064 any preceding 7065 local/generic load 7066 atomic/atomicrmw 7067 with an equal or 7068 wider sync scope 7069 and memory ordering 7070 stronger than 7071 unordered (this is 7072 termed the 7073 fence-paired-atomic). 7074 - Must happen before 7075 the following 7076 buffer_wbinvl1_vol. 7077 - Ensures that the 7078 fence-paired atomic 7079 has completed 7080 before invalidating 7081 the 7082 cache. Therefore 7083 any following 7084 locations read must 7085 be no older than 7086 the value read by 7087 the 7088 fence-paired-atomic. 7089 7090 2. buffer_wbinvl1_vol 7091 7092 - Must happen before any 7093 following global/generic 7094 load/load 7095 atomic/store/store 7096 atomic/atomicrmw. 7097 - Ensures that 7098 following loads 7099 will not see stale 7100 global data. 7101 7102 fence acquire - system *none* 1. s_waitcnt lgkmcnt(0) & 7103 vmcnt(0) 7104 7105 - If TgSplit execution mode, 7106 omit lgkmcnt(0). 7107 - If OpenCL and 7108 address space is 7109 not generic, omit 7110 lgkmcnt(0). 7111 - However, since LLVM 7112 currently has no 7113 address space on 7114 the fence need to 7115 conservatively 7116 always generate 7117 (see comment for 7118 previous fence). 7119 - Could be split into 7120 separate s_waitcnt 7121 vmcnt(0) and 7122 s_waitcnt 7123 lgkmcnt(0) to allow 7124 them to be 7125 independently moved 7126 according to the 7127 following rules. 7128 - s_waitcnt vmcnt(0) 7129 must happen after 7130 any preceding 7131 global/generic load 7132 atomic/atomicrmw 7133 with an equal or 7134 wider sync scope 7135 and memory ordering 7136 stronger than 7137 unordered (this is 7138 termed the 7139 fence-paired-atomic). 7140 - s_waitcnt lgkmcnt(0) 7141 must happen after 7142 any preceding 7143 local/generic load 7144 atomic/atomicrmw 7145 with an equal or 7146 wider sync scope 7147 and memory ordering 7148 stronger than 7149 unordered (this is 7150 termed the 7151 fence-paired-atomic). 7152 - Must happen before 7153 the following buffer_invl2 and 7154 buffer_wbinvl1_vol. 7155 - Ensures that the 7156 fence-paired atomic 7157 has completed 7158 before invalidating 7159 the 7160 cache. Therefore 7161 any following 7162 locations read must 7163 be no older than 7164 the value read by 7165 the 7166 fence-paired-atomic. 7167 7168 2. buffer_invl2; 7169 buffer_wbinvl1_vol 7170 7171 - Must happen before any 7172 following global/generic 7173 load/load 7174 atomic/store/store 7175 atomic/atomicrmw. 7176 - Ensures that 7177 following 7178 loads will not see 7179 stale L1 global data, 7180 nor see stale L2 MTYPE 7181 NC global data. 7182 MTYPE RW and CC memory will 7183 never be stale in L2 due to 7184 the memory probes. 7185 **Release Atomic** 7186 ------------------------------------------------------------------------------------ 7187 store atomic release - singlethread - global 1. buffer/global/flat_store 7188 - wavefront - generic 7189 store atomic release - singlethread - local *If TgSplit execution mode, 7190 - wavefront local address space cannot 7191 be used.* 7192 7193 1. ds_store 7194 store atomic release - workgroup - global 1. s_waitcnt lgkm/vmcnt(0) 7195 - generic 7196 - Use lgkmcnt(0) if not 7197 TgSplit execution mode 7198 and vmcnt(0) if TgSplit 7199 execution mode. 7200 - If OpenCL, omit lgkmcnt(0). 7201 - s_waitcnt vmcnt(0) 7202 must happen after 7203 any preceding 7204 global/generic load/store/ 7205 load atomic/store atomic/ 7206 atomicrmw. 7207 - s_waitcnt lgkmcnt(0) 7208 must happen after 7209 any preceding 7210 local/generic 7211 load/store/load 7212 atomic/store 7213 atomic/atomicrmw. 7214 - Must happen before 7215 the following 7216 store. 7217 - Ensures that all 7218 memory operations 7219 have 7220 completed before 7221 performing the 7222 store that is being 7223 released. 7224 7225 2. buffer/global/flat_store 7226 store atomic release - workgroup - local *If TgSplit execution mode, 7227 local address space cannot 7228 be used.* 7229 7230 1. ds_store 7231 store atomic release - agent - global 1. s_waitcnt lgkmcnt(0) & 7232 - generic vmcnt(0) 7233 7234 - If TgSplit execution mode, 7235 omit lgkmcnt(0). 7236 - If OpenCL and 7237 address space is 7238 not generic, omit 7239 lgkmcnt(0). 7240 - Could be split into 7241 separate s_waitcnt 7242 vmcnt(0) and 7243 s_waitcnt 7244 lgkmcnt(0) to allow 7245 them to be 7246 independently moved 7247 according to the 7248 following rules. 7249 - s_waitcnt vmcnt(0) 7250 must happen after 7251 any preceding 7252 global/generic 7253 load/store/load 7254 atomic/store 7255 atomic/atomicrmw. 7256 - s_waitcnt lgkmcnt(0) 7257 must happen after 7258 any preceding 7259 local/generic 7260 load/store/load 7261 atomic/store 7262 atomic/atomicrmw. 7263 - Must happen before 7264 the following 7265 store. 7266 - Ensures that all 7267 memory operations 7268 to memory have 7269 completed before 7270 performing the 7271 store that is being 7272 released. 7273 7274 2. buffer/global/flat_store 7275 store atomic release - system - global 1. buffer_wbl2 7276 - generic 7277 - Must happen before 7278 following s_waitcnt. 7279 - Performs L2 writeback to 7280 ensure previous 7281 global/generic 7282 store/atomicrmw are 7283 visible at system scope. 7284 7285 2. s_waitcnt lgkmcnt(0) & 7286 vmcnt(0) 7287 7288 - If TgSplit execution mode, 7289 omit lgkmcnt(0). 7290 - If OpenCL and 7291 address space is 7292 not generic, omit 7293 lgkmcnt(0). 7294 - Could be split into 7295 separate s_waitcnt 7296 vmcnt(0) and 7297 s_waitcnt 7298 lgkmcnt(0) to allow 7299 them to be 7300 independently moved 7301 according to the 7302 following rules. 7303 - s_waitcnt vmcnt(0) 7304 must happen after any 7305 preceding 7306 global/generic 7307 load/store/load 7308 atomic/store 7309 atomic/atomicrmw. 7310 - s_waitcnt lgkmcnt(0) 7311 must happen after any 7312 preceding 7313 local/generic 7314 load/store/load 7315 atomic/store 7316 atomic/atomicrmw. 7317 - Must happen before 7318 the following 7319 store. 7320 - Ensures that all 7321 memory operations 7322 to memory and the L2 7323 writeback have 7324 completed before 7325 performing the 7326 store that is being 7327 released. 7328 7329 3. buffer/global/flat_store 7330 atomicrmw release - singlethread - global 1. buffer/global/flat_atomic 7331 - wavefront - generic 7332 atomicrmw release - singlethread - local *If TgSplit execution mode, 7333 - wavefront local address space cannot 7334 be used.* 7335 7336 1. ds_atomic 7337 atomicrmw release - workgroup - global 1. s_waitcnt lgkm/vmcnt(0) 7338 - generic 7339 - Use lgkmcnt(0) if not 7340 TgSplit execution mode 7341 and vmcnt(0) if TgSplit 7342 execution mode. 7343 - If OpenCL, omit 7344 lgkmcnt(0). 7345 - s_waitcnt vmcnt(0) 7346 must happen after 7347 any preceding 7348 global/generic load/store/ 7349 load atomic/store atomic/ 7350 atomicrmw. 7351 - s_waitcnt lgkmcnt(0) 7352 must happen after 7353 any preceding 7354 local/generic 7355 load/store/load 7356 atomic/store 7357 atomic/atomicrmw. 7358 - Must happen before 7359 the following 7360 atomicrmw. 7361 - Ensures that all 7362 memory operations 7363 have 7364 completed before 7365 performing the 7366 atomicrmw that is 7367 being released. 7368 7369 2. buffer/global/flat_atomic 7370 atomicrmw release - workgroup - local *If TgSplit execution mode, 7371 local address space cannot 7372 be used.* 7373 7374 1. ds_atomic 7375 atomicrmw release - agent - global 1. s_waitcnt lgkmcnt(0) & 7376 - generic vmcnt(0) 7377 7378 - If TgSplit execution mode, 7379 omit lgkmcnt(0). 7380 - If OpenCL, omit 7381 lgkmcnt(0). 7382 - Could be split into 7383 separate s_waitcnt 7384 vmcnt(0) and 7385 s_waitcnt 7386 lgkmcnt(0) to allow 7387 them to be 7388 independently moved 7389 according to the 7390 following rules. 7391 - s_waitcnt vmcnt(0) 7392 must happen after 7393 any preceding 7394 global/generic 7395 load/store/load 7396 atomic/store 7397 atomic/atomicrmw. 7398 - s_waitcnt lgkmcnt(0) 7399 must happen after 7400 any preceding 7401 local/generic 7402 load/store/load 7403 atomic/store 7404 atomic/atomicrmw. 7405 - Must happen before 7406 the following 7407 atomicrmw. 7408 - Ensures that all 7409 memory operations 7410 to global and local 7411 have completed 7412 before performing 7413 the atomicrmw that 7414 is being released. 7415 7416 2. buffer/global/flat_atomic 7417 atomicrmw release - system - global 1. buffer_wbl2 7418 - generic 7419 - Must happen before 7420 following s_waitcnt. 7421 - Performs L2 writeback to 7422 ensure previous 7423 global/generic 7424 store/atomicrmw are 7425 visible at system scope. 7426 7427 2. s_waitcnt lgkmcnt(0) & 7428 vmcnt(0) 7429 7430 - If TgSplit execution mode, 7431 omit lgkmcnt(0). 7432 - If OpenCL, omit 7433 lgkmcnt(0). 7434 - Could be split into 7435 separate s_waitcnt 7436 vmcnt(0) and 7437 s_waitcnt 7438 lgkmcnt(0) to allow 7439 them to be 7440 independently moved 7441 according to the 7442 following rules. 7443 - s_waitcnt vmcnt(0) 7444 must happen after 7445 any preceding 7446 global/generic 7447 load/store/load 7448 atomic/store 7449 atomic/atomicrmw. 7450 - s_waitcnt lgkmcnt(0) 7451 must happen after 7452 any preceding 7453 local/generic 7454 load/store/load 7455 atomic/store 7456 atomic/atomicrmw. 7457 - Must happen before 7458 the following 7459 atomicrmw. 7460 - Ensures that all 7461 memory operations 7462 to memory and the L2 7463 writeback have 7464 completed before 7465 performing the 7466 store that is being 7467 released. 7468 7469 3. buffer/global/flat_atomic 7470 fence release - singlethread *none* *none* 7471 - wavefront 7472 fence release - workgroup *none* 1. s_waitcnt lgkm/vmcnt(0) 7473 7474 - Use lgkmcnt(0) if not 7475 TgSplit execution mode 7476 and vmcnt(0) if TgSplit 7477 execution mode. 7478 - If OpenCL and 7479 address space is 7480 not generic, omit 7481 lgkmcnt(0). 7482 - If OpenCL and 7483 address space is 7484 local, omit 7485 vmcnt(0). 7486 - However, since LLVM 7487 currently has no 7488 address space on 7489 the fence need to 7490 conservatively 7491 always generate. If 7492 fence had an 7493 address space then 7494 set to address 7495 space of OpenCL 7496 fence flag, or to 7497 generic if both 7498 local and global 7499 flags are 7500 specified. 7501 - s_waitcnt vmcnt(0) 7502 must happen after 7503 any preceding 7504 global/generic 7505 load/store/ 7506 load atomic/store atomic/ 7507 atomicrmw. 7508 - s_waitcnt lgkmcnt(0) 7509 must happen after 7510 any preceding 7511 local/generic 7512 load/load 7513 atomic/store/store 7514 atomic/atomicrmw. 7515 - Must happen before 7516 any following store 7517 atomic/atomicrmw 7518 with an equal or 7519 wider sync scope 7520 and memory ordering 7521 stronger than 7522 unordered (this is 7523 termed the 7524 fence-paired-atomic). 7525 - Ensures that all 7526 memory operations 7527 have 7528 completed before 7529 performing the 7530 following 7531 fence-paired-atomic. 7532 7533 fence release - agent *none* 1. s_waitcnt lgkmcnt(0) & 7534 vmcnt(0) 7535 7536 - If TgSplit execution mode, 7537 omit lgkmcnt(0). 7538 - If OpenCL and 7539 address space is 7540 not generic, omit 7541 lgkmcnt(0). 7542 - If OpenCL and 7543 address space is 7544 local, omit 7545 vmcnt(0). 7546 - However, since LLVM 7547 currently has no 7548 address space on 7549 the fence need to 7550 conservatively 7551 always generate. If 7552 fence had an 7553 address space then 7554 set to address 7555 space of OpenCL 7556 fence flag, or to 7557 generic if both 7558 local and global 7559 flags are 7560 specified. 7561 - Could be split into 7562 separate s_waitcnt 7563 vmcnt(0) and 7564 s_waitcnt 7565 lgkmcnt(0) to allow 7566 them to be 7567 independently moved 7568 according to the 7569 following rules. 7570 - s_waitcnt vmcnt(0) 7571 must happen after 7572 any preceding 7573 global/generic 7574 load/store/load 7575 atomic/store 7576 atomic/atomicrmw. 7577 - s_waitcnt lgkmcnt(0) 7578 must happen after 7579 any preceding 7580 local/generic 7581 load/store/load 7582 atomic/store 7583 atomic/atomicrmw. 7584 - Must happen before 7585 any following store 7586 atomic/atomicrmw 7587 with an equal or 7588 wider sync scope 7589 and memory ordering 7590 stronger than 7591 unordered (this is 7592 termed the 7593 fence-paired-atomic). 7594 - Ensures that all 7595 memory operations 7596 have 7597 completed before 7598 performing the 7599 following 7600 fence-paired-atomic. 7601 7602 fence release - system *none* 1. buffer_wbl2 7603 7604 - If OpenCL and 7605 address space is 7606 local, omit. 7607 - Must happen before 7608 following s_waitcnt. 7609 - Performs L2 writeback to 7610 ensure previous 7611 global/generic 7612 store/atomicrmw are 7613 visible at system scope. 7614 7615 2. s_waitcnt lgkmcnt(0) & 7616 vmcnt(0) 7617 7618 - If TgSplit execution mode, 7619 omit lgkmcnt(0). 7620 - If OpenCL and 7621 address space is 7622 not generic, omit 7623 lgkmcnt(0). 7624 - If OpenCL and 7625 address space is 7626 local, omit 7627 vmcnt(0). 7628 - However, since LLVM 7629 currently has no 7630 address space on 7631 the fence need to 7632 conservatively 7633 always generate. If 7634 fence had an 7635 address space then 7636 set to address 7637 space of OpenCL 7638 fence flag, or to 7639 generic if both 7640 local and global 7641 flags are 7642 specified. 7643 - Could be split into 7644 separate s_waitcnt 7645 vmcnt(0) and 7646 s_waitcnt 7647 lgkmcnt(0) to allow 7648 them to be 7649 independently moved 7650 according to the 7651 following rules. 7652 - s_waitcnt vmcnt(0) 7653 must happen after 7654 any preceding 7655 global/generic 7656 load/store/load 7657 atomic/store 7658 atomic/atomicrmw. 7659 - s_waitcnt lgkmcnt(0) 7660 must happen after 7661 any preceding 7662 local/generic 7663 load/store/load 7664 atomic/store 7665 atomic/atomicrmw. 7666 - Must happen before 7667 any following store 7668 atomic/atomicrmw 7669 with an equal or 7670 wider sync scope 7671 and memory ordering 7672 stronger than 7673 unordered (this is 7674 termed the 7675 fence-paired-atomic). 7676 - Ensures that all 7677 memory operations 7678 have 7679 completed before 7680 performing the 7681 following 7682 fence-paired-atomic. 7683 7684 **Acquire-Release Atomic** 7685 ------------------------------------------------------------------------------------ 7686 atomicrmw acq_rel - singlethread - global 1. buffer/global/flat_atomic 7687 - wavefront - generic 7688 atomicrmw acq_rel - singlethread - local *If TgSplit execution mode, 7689 - wavefront local address space cannot 7690 be used.* 7691 7692 1. ds_atomic 7693 atomicrmw acq_rel - workgroup - global 1. s_waitcnt lgkm/vmcnt(0) 7694 7695 - Use lgkmcnt(0) if not 7696 TgSplit execution mode 7697 and vmcnt(0) if TgSplit 7698 execution mode. 7699 - If OpenCL, omit 7700 lgkmcnt(0). 7701 - Must happen after 7702 any preceding 7703 local/generic 7704 load/store/load 7705 atomic/store 7706 atomic/atomicrmw. 7707 - s_waitcnt vmcnt(0) 7708 must happen after 7709 any preceding 7710 global/generic load/store/ 7711 load atomic/store atomic/ 7712 atomicrmw. 7713 - s_waitcnt lgkmcnt(0) 7714 must happen after 7715 any preceding 7716 local/generic 7717 load/store/load 7718 atomic/store 7719 atomic/atomicrmw. 7720 - Must happen before 7721 the following 7722 atomicrmw. 7723 - Ensures that all 7724 memory operations 7725 have 7726 completed before 7727 performing the 7728 atomicrmw that is 7729 being released. 7730 7731 2. buffer/global_atomic 7732 3. s_waitcnt vmcnt(0) 7733 7734 - If not TgSplit execution 7735 mode, omit. 7736 - Must happen before 7737 the following 7738 buffer_wbinvl1_vol. 7739 - Ensures any 7740 following global 7741 data read is no 7742 older than the 7743 atomicrmw value 7744 being acquired. 7745 7746 4. buffer_wbinvl1_vol 7747 7748 - If not TgSplit execution 7749 mode, omit. 7750 - Ensures that 7751 following 7752 loads will not see 7753 stale data. 7754 7755 atomicrmw acq_rel - workgroup - local *If TgSplit execution mode, 7756 local address space cannot 7757 be used.* 7758 7759 1. ds_atomic 7760 2. s_waitcnt lgkmcnt(0) 7761 7762 - If OpenCL, omit. 7763 - Must happen before 7764 any following 7765 global/generic 7766 load/load 7767 atomic/store/store 7768 atomic/atomicrmw. 7769 - Ensures any 7770 following global 7771 data read is no 7772 older than the local load 7773 atomic value being 7774 acquired. 7775 7776 atomicrmw acq_rel - workgroup - generic 1. s_waitcnt lgkm/vmcnt(0) 7777 7778 - Use lgkmcnt(0) if not 7779 TgSplit execution mode 7780 and vmcnt(0) if TgSplit 7781 execution mode. 7782 - If OpenCL, omit 7783 lgkmcnt(0). 7784 - s_waitcnt vmcnt(0) 7785 must happen after 7786 any preceding 7787 global/generic load/store/ 7788 load atomic/store atomic/ 7789 atomicrmw. 7790 - s_waitcnt lgkmcnt(0) 7791 must happen after 7792 any preceding 7793 local/generic 7794 load/store/load 7795 atomic/store 7796 atomic/atomicrmw. 7797 - Must happen before 7798 the following 7799 atomicrmw. 7800 - Ensures that all 7801 memory operations 7802 have 7803 completed before 7804 performing the 7805 atomicrmw that is 7806 being released. 7807 7808 2. flat_atomic 7809 3. s_waitcnt lgkmcnt(0) & 7810 vmcnt(0) 7811 7812 - If not TgSplit execution 7813 mode, omit vmcnt(0). 7814 - If OpenCL, omit 7815 lgkmcnt(0). 7816 - Must happen before 7817 the following 7818 buffer_wbinvl1_vol and 7819 any following 7820 global/generic 7821 load/load 7822 atomic/store/store 7823 atomic/atomicrmw. 7824 - Ensures any 7825 following global 7826 data read is no 7827 older than a local load 7828 atomic value being 7829 acquired. 7830 7831 3. buffer_wbinvl1_vol 7832 7833 - If not TgSplit execution 7834 mode, omit. 7835 - Ensures that 7836 following 7837 loads will not see 7838 stale data. 7839 7840 atomicrmw acq_rel - agent - global 1. s_waitcnt lgkmcnt(0) & 7841 vmcnt(0) 7842 7843 - If TgSplit execution mode, 7844 omit lgkmcnt(0). 7845 - If OpenCL, omit 7846 lgkmcnt(0). 7847 - Could be split into 7848 separate s_waitcnt 7849 vmcnt(0) and 7850 s_waitcnt 7851 lgkmcnt(0) to allow 7852 them to be 7853 independently moved 7854 according to the 7855 following rules. 7856 - s_waitcnt vmcnt(0) 7857 must happen after 7858 any preceding 7859 global/generic 7860 load/store/load 7861 atomic/store 7862 atomic/atomicrmw. 7863 - s_waitcnt lgkmcnt(0) 7864 must happen after 7865 any preceding 7866 local/generic 7867 load/store/load 7868 atomic/store 7869 atomic/atomicrmw. 7870 - Must happen before 7871 the following 7872 atomicrmw. 7873 - Ensures that all 7874 memory operations 7875 to global have 7876 completed before 7877 performing the 7878 atomicrmw that is 7879 being released. 7880 7881 2. buffer/global_atomic 7882 3. s_waitcnt vmcnt(0) 7883 7884 - Must happen before 7885 following 7886 buffer_wbinvl1_vol. 7887 - Ensures the 7888 atomicrmw has 7889 completed before 7890 invalidating the 7891 cache. 7892 7893 4. buffer_wbinvl1_vol 7894 7895 - Must happen before 7896 any following 7897 global/generic 7898 load/load 7899 atomic/atomicrmw. 7900 - Ensures that 7901 following loads 7902 will not see stale 7903 global data. 7904 7905 atomicrmw acq_rel - system - global 1. buffer_wbl2 7906 7907 - Must happen before 7908 following s_waitcnt. 7909 - Performs L2 writeback to 7910 ensure previous 7911 global/generic 7912 store/atomicrmw are 7913 visible at system scope. 7914 7915 2. s_waitcnt lgkmcnt(0) & 7916 vmcnt(0) 7917 7918 - If TgSplit execution mode, 7919 omit lgkmcnt(0). 7920 - If OpenCL, omit 7921 lgkmcnt(0). 7922 - Could be split into 7923 separate s_waitcnt 7924 vmcnt(0) and 7925 s_waitcnt 7926 lgkmcnt(0) to allow 7927 them to be 7928 independently moved 7929 according to the 7930 following rules. 7931 - s_waitcnt vmcnt(0) 7932 must happen after 7933 any preceding 7934 global/generic 7935 load/store/load 7936 atomic/store 7937 atomic/atomicrmw. 7938 - s_waitcnt lgkmcnt(0) 7939 must happen after 7940 any preceding 7941 local/generic 7942 load/store/load 7943 atomic/store 7944 atomic/atomicrmw. 7945 - Must happen before 7946 the following 7947 atomicrmw. 7948 - Ensures that all 7949 memory operations 7950 to global and L2 writeback 7951 have completed before 7952 performing the 7953 atomicrmw that is 7954 being released. 7955 7956 3. buffer/global_atomic 7957 4. s_waitcnt vmcnt(0) 7958 7959 - Must happen before 7960 following buffer_invl2 and 7961 buffer_wbinvl1_vol. 7962 - Ensures the 7963 atomicrmw has 7964 completed before 7965 invalidating the 7966 caches. 7967 7968 5. buffer_invl2; 7969 buffer_wbinvl1_vol 7970 7971 - Must happen before 7972 any following 7973 global/generic 7974 load/load 7975 atomic/atomicrmw. 7976 - Ensures that 7977 following 7978 loads will not see 7979 stale L1 global data, 7980 nor see stale L2 MTYPE 7981 NC global data. 7982 MTYPE RW and CC memory will 7983 never be stale in L2 due to 7984 the memory probes. 7985 7986 atomicrmw acq_rel - agent - generic 1. s_waitcnt lgkmcnt(0) & 7987 vmcnt(0) 7988 7989 - If TgSplit execution mode, 7990 omit lgkmcnt(0). 7991 - If OpenCL, omit 7992 lgkmcnt(0). 7993 - Could be split into 7994 separate s_waitcnt 7995 vmcnt(0) and 7996 s_waitcnt 7997 lgkmcnt(0) to allow 7998 them to be 7999 independently moved 8000 according to the 8001 following rules. 8002 - s_waitcnt vmcnt(0) 8003 must happen after 8004 any preceding 8005 global/generic 8006 load/store/load 8007 atomic/store 8008 atomic/atomicrmw. 8009 - s_waitcnt lgkmcnt(0) 8010 must happen after 8011 any preceding 8012 local/generic 8013 load/store/load 8014 atomic/store 8015 atomic/atomicrmw. 8016 - Must happen before 8017 the following 8018 atomicrmw. 8019 - Ensures that all 8020 memory operations 8021 to global have 8022 completed before 8023 performing the 8024 atomicrmw that is 8025 being released. 8026 8027 2. flat_atomic 8028 3. s_waitcnt vmcnt(0) & 8029 lgkmcnt(0) 8030 8031 - If TgSplit execution mode, 8032 omit lgkmcnt(0). 8033 - If OpenCL, omit 8034 lgkmcnt(0). 8035 - Must happen before 8036 following 8037 buffer_wbinvl1_vol. 8038 - Ensures the 8039 atomicrmw has 8040 completed before 8041 invalidating the 8042 cache. 8043 8044 4. buffer_wbinvl1_vol 8045 8046 - Must happen before 8047 any following 8048 global/generic 8049 load/load 8050 atomic/atomicrmw. 8051 - Ensures that 8052 following loads 8053 will not see stale 8054 global data. 8055 8056 atomicrmw acq_rel - system - generic 1. buffer_wbl2 8057 8058 - Must happen before 8059 following s_waitcnt. 8060 - Performs L2 writeback to 8061 ensure previous 8062 global/generic 8063 store/atomicrmw are 8064 visible at system scope. 8065 8066 2. s_waitcnt lgkmcnt(0) & 8067 vmcnt(0) 8068 8069 - If TgSplit execution mode, 8070 omit lgkmcnt(0). 8071 - If OpenCL, omit 8072 lgkmcnt(0). 8073 - Could be split into 8074 separate s_waitcnt 8075 vmcnt(0) and 8076 s_waitcnt 8077 lgkmcnt(0) to allow 8078 them to be 8079 independently moved 8080 according to the 8081 following rules. 8082 - s_waitcnt vmcnt(0) 8083 must happen after 8084 any preceding 8085 global/generic 8086 load/store/load 8087 atomic/store 8088 atomic/atomicrmw. 8089 - s_waitcnt lgkmcnt(0) 8090 must happen after 8091 any preceding 8092 local/generic 8093 load/store/load 8094 atomic/store 8095 atomic/atomicrmw. 8096 - Must happen before 8097 the following 8098 atomicrmw. 8099 - Ensures that all 8100 memory operations 8101 to global and L2 writeback 8102 have completed before 8103 performing the 8104 atomicrmw that is 8105 being released. 8106 8107 3. flat_atomic 8108 4. s_waitcnt vmcnt(0) & 8109 lgkmcnt(0) 8110 8111 - If TgSplit execution mode, 8112 omit lgkmcnt(0). 8113 - If OpenCL, omit 8114 lgkmcnt(0). 8115 - Must happen before 8116 following buffer_invl2 and 8117 buffer_wbinvl1_vol. 8118 - Ensures the 8119 atomicrmw has 8120 completed before 8121 invalidating the 8122 caches. 8123 8124 5. buffer_invl2; 8125 buffer_wbinvl1_vol 8126 8127 - Must happen before 8128 any following 8129 global/generic 8130 load/load 8131 atomic/atomicrmw. 8132 - Ensures that 8133 following 8134 loads will not see 8135 stale L1 global data, 8136 nor see stale L2 MTYPE 8137 NC global data. 8138 MTYPE RW and CC memory will 8139 never be stale in L2 due to 8140 the memory probes. 8141 8142 fence acq_rel - singlethread *none* *none* 8143 - wavefront 8144 fence acq_rel - workgroup *none* 1. s_waitcnt lgkm/vmcnt(0) 8145 8146 - Use lgkmcnt(0) if not 8147 TgSplit execution mode 8148 and vmcnt(0) if TgSplit 8149 execution mode. 8150 - If OpenCL and 8151 address space is 8152 not generic, omit 8153 lgkmcnt(0). 8154 - If OpenCL and 8155 address space is 8156 local, omit 8157 vmcnt(0). 8158 - However, 8159 since LLVM 8160 currently has no 8161 address space on 8162 the fence need to 8163 conservatively 8164 always generate 8165 (see comment for 8166 previous fence). 8167 - s_waitcnt vmcnt(0) 8168 must happen after 8169 any preceding 8170 global/generic 8171 load/store/ 8172 load atomic/store atomic/ 8173 atomicrmw. 8174 - s_waitcnt lgkmcnt(0) 8175 must happen after 8176 any preceding 8177 local/generic 8178 load/load 8179 atomic/store/store 8180 atomic/atomicrmw. 8181 - Must happen before 8182 any following 8183 global/generic 8184 load/load 8185 atomic/store/store 8186 atomic/atomicrmw. 8187 - Ensures that all 8188 memory operations 8189 have 8190 completed before 8191 performing any 8192 following global 8193 memory operations. 8194 - Ensures that the 8195 preceding 8196 local/generic load 8197 atomic/atomicrmw 8198 with an equal or 8199 wider sync scope 8200 and memory ordering 8201 stronger than 8202 unordered (this is 8203 termed the 8204 acquire-fence-paired-atomic) 8205 has completed 8206 before following 8207 global memory 8208 operations. This 8209 satisfies the 8210 requirements of 8211 acquire. 8212 - Ensures that all 8213 previous memory 8214 operations have 8215 completed before a 8216 following 8217 local/generic store 8218 atomic/atomicrmw 8219 with an equal or 8220 wider sync scope 8221 and memory ordering 8222 stronger than 8223 unordered (this is 8224 termed the 8225 release-fence-paired-atomic). 8226 This satisfies the 8227 requirements of 8228 release. 8229 - Must happen before 8230 the following 8231 buffer_wbinvl1_vol. 8232 - Ensures that the 8233 acquire-fence-paired 8234 atomic has completed 8235 before invalidating 8236 the 8237 cache. Therefore 8238 any following 8239 locations read must 8240 be no older than 8241 the value read by 8242 the 8243 acquire-fence-paired-atomic. 8244 8245 2. buffer_wbinvl1_vol 8246 8247 - If not TgSplit execution 8248 mode, omit. 8249 - Ensures that 8250 following 8251 loads will not see 8252 stale data. 8253 8254 fence acq_rel - agent *none* 1. s_waitcnt lgkmcnt(0) & 8255 vmcnt(0) 8256 8257 - If TgSplit execution mode, 8258 omit lgkmcnt(0). 8259 - If OpenCL and 8260 address space is 8261 not generic, omit 8262 lgkmcnt(0). 8263 - However, since LLVM 8264 currently has no 8265 address space on 8266 the fence need to 8267 conservatively 8268 always generate 8269 (see comment for 8270 previous fence). 8271 - Could be split into 8272 separate s_waitcnt 8273 vmcnt(0) and 8274 s_waitcnt 8275 lgkmcnt(0) to allow 8276 them to be 8277 independently moved 8278 according to the 8279 following rules. 8280 - s_waitcnt vmcnt(0) 8281 must happen after 8282 any preceding 8283 global/generic 8284 load/store/load 8285 atomic/store 8286 atomic/atomicrmw. 8287 - s_waitcnt lgkmcnt(0) 8288 must happen after 8289 any preceding 8290 local/generic 8291 load/store/load 8292 atomic/store 8293 atomic/atomicrmw. 8294 - Must happen before 8295 the following 8296 buffer_wbinvl1_vol. 8297 - Ensures that the 8298 preceding 8299 global/local/generic 8300 load 8301 atomic/atomicrmw 8302 with an equal or 8303 wider sync scope 8304 and memory ordering 8305 stronger than 8306 unordered (this is 8307 termed the 8308 acquire-fence-paired-atomic) 8309 has completed 8310 before invalidating 8311 the cache. This 8312 satisfies the 8313 requirements of 8314 acquire. 8315 - Ensures that all 8316 previous memory 8317 operations have 8318 completed before a 8319 following 8320 global/local/generic 8321 store 8322 atomic/atomicrmw 8323 with an equal or 8324 wider sync scope 8325 and memory ordering 8326 stronger than 8327 unordered (this is 8328 termed the 8329 release-fence-paired-atomic). 8330 This satisfies the 8331 requirements of 8332 release. 8333 8334 2. buffer_wbinvl1_vol 8335 8336 - Must happen before 8337 any following 8338 global/generic 8339 load/load 8340 atomic/store/store 8341 atomic/atomicrmw. 8342 - Ensures that 8343 following loads 8344 will not see stale 8345 global data. This 8346 satisfies the 8347 requirements of 8348 acquire. 8349 8350 fence acq_rel - system *none* 1. buffer_wbl2 8351 8352 - If OpenCL and 8353 address space is 8354 local, omit. 8355 - Must happen before 8356 following s_waitcnt. 8357 - Performs L2 writeback to 8358 ensure previous 8359 global/generic 8360 store/atomicrmw are 8361 visible at system scope. 8362 8363 2. s_waitcnt lgkmcnt(0) & 8364 vmcnt(0) 8365 8366 - If TgSplit execution mode, 8367 omit lgkmcnt(0). 8368 - If OpenCL and 8369 address space is 8370 not generic, omit 8371 lgkmcnt(0). 8372 - However, since LLVM 8373 currently has no 8374 address space on 8375 the fence need to 8376 conservatively 8377 always generate 8378 (see comment for 8379 previous fence). 8380 - Could be split into 8381 separate s_waitcnt 8382 vmcnt(0) and 8383 s_waitcnt 8384 lgkmcnt(0) to allow 8385 them to be 8386 independently moved 8387 according to the 8388 following rules. 8389 - s_waitcnt vmcnt(0) 8390 must happen after 8391 any preceding 8392 global/generic 8393 load/store/load 8394 atomic/store 8395 atomic/atomicrmw. 8396 - s_waitcnt lgkmcnt(0) 8397 must happen after 8398 any preceding 8399 local/generic 8400 load/store/load 8401 atomic/store 8402 atomic/atomicrmw. 8403 - Must happen before 8404 the following buffer_invl2 and 8405 buffer_wbinvl1_vol. 8406 - Ensures that the 8407 preceding 8408 global/local/generic 8409 load 8410 atomic/atomicrmw 8411 with an equal or 8412 wider sync scope 8413 and memory ordering 8414 stronger than 8415 unordered (this is 8416 termed the 8417 acquire-fence-paired-atomic) 8418 has completed 8419 before invalidating 8420 the cache. This 8421 satisfies the 8422 requirements of 8423 acquire. 8424 - Ensures that all 8425 previous memory 8426 operations have 8427 completed before a 8428 following 8429 global/local/generic 8430 store 8431 atomic/atomicrmw 8432 with an equal or 8433 wider sync scope 8434 and memory ordering 8435 stronger than 8436 unordered (this is 8437 termed the 8438 release-fence-paired-atomic). 8439 This satisfies the 8440 requirements of 8441 release. 8442 8443 3. buffer_invl2; 8444 buffer_wbinvl1_vol 8445 8446 - Must happen before 8447 any following 8448 global/generic 8449 load/load 8450 atomic/store/store 8451 atomic/atomicrmw. 8452 - Ensures that 8453 following 8454 loads will not see 8455 stale L1 global data, 8456 nor see stale L2 MTYPE 8457 NC global data. 8458 MTYPE RW and CC memory will 8459 never be stale in L2 due to 8460 the memory probes. 8461 8462 **Sequential Consistent Atomic** 8463 ------------------------------------------------------------------------------------ 8464 load atomic seq_cst - singlethread - global *Same as corresponding 8465 - wavefront - local load atomic acquire, 8466 - generic except must generate 8467 all instructions even 8468 for OpenCL.* 8469 load atomic seq_cst - workgroup - global 1. s_waitcnt lgkm/vmcnt(0) 8470 - generic 8471 - Use lgkmcnt(0) if not 8472 TgSplit execution mode 8473 and vmcnt(0) if TgSplit 8474 execution mode. 8475 - s_waitcnt lgkmcnt(0) must 8476 happen after 8477 preceding 8478 local/generic load 8479 atomic/store 8480 atomic/atomicrmw 8481 with memory 8482 ordering of seq_cst 8483 and with equal or 8484 wider sync scope. 8485 (Note that seq_cst 8486 fences have their 8487 own s_waitcnt 8488 lgkmcnt(0) and so do 8489 not need to be 8490 considered.) 8491 - s_waitcnt vmcnt(0) 8492 must happen after 8493 preceding 8494 global/generic load 8495 atomic/store 8496 atomic/atomicrmw 8497 with memory 8498 ordering of seq_cst 8499 and with equal or 8500 wider sync scope. 8501 (Note that seq_cst 8502 fences have their 8503 own s_waitcnt 8504 vmcnt(0) and so do 8505 not need to be 8506 considered.) 8507 - Ensures any 8508 preceding 8509 sequential 8510 consistent global/local 8511 memory instructions 8512 have completed 8513 before executing 8514 this sequentially 8515 consistent 8516 instruction. This 8517 prevents reordering 8518 a seq_cst store 8519 followed by a 8520 seq_cst load. (Note 8521 that seq_cst is 8522 stronger than 8523 acquire/release as 8524 the reordering of 8525 load acquire 8526 followed by a store 8527 release is 8528 prevented by the 8529 s_waitcnt of 8530 the release, but 8531 there is nothing 8532 preventing a store 8533 release followed by 8534 load acquire from 8535 completing out of 8536 order. The s_waitcnt 8537 could be placed after 8538 seq_store or before 8539 the seq_load. We 8540 choose the load to 8541 make the s_waitcnt be 8542 as late as possible 8543 so that the store 8544 may have already 8545 completed.) 8546 8547 2. *Following 8548 instructions same as 8549 corresponding load 8550 atomic acquire, 8551 except must generate 8552 all instructions even 8553 for OpenCL.* 8554 load atomic seq_cst - workgroup - local *If TgSplit execution mode, 8555 local address space cannot 8556 be used.* 8557 8558 *Same as corresponding 8559 load atomic acquire, 8560 except must generate 8561 all instructions even 8562 for OpenCL.* 8563 8564 load atomic seq_cst - agent - global 1. s_waitcnt lgkmcnt(0) & 8565 - system - generic vmcnt(0) 8566 8567 - If TgSplit execution mode, 8568 omit lgkmcnt(0). 8569 - Could be split into 8570 separate s_waitcnt 8571 vmcnt(0) 8572 and s_waitcnt 8573 lgkmcnt(0) to allow 8574 them to be 8575 independently moved 8576 according to the 8577 following rules. 8578 - s_waitcnt lgkmcnt(0) 8579 must happen after 8580 preceding 8581 global/generic load 8582 atomic/store 8583 atomic/atomicrmw 8584 with memory 8585 ordering of seq_cst 8586 and with equal or 8587 wider sync scope. 8588 (Note that seq_cst 8589 fences have their 8590 own s_waitcnt 8591 lgkmcnt(0) and so do 8592 not need to be 8593 considered.) 8594 - s_waitcnt vmcnt(0) 8595 must happen after 8596 preceding 8597 global/generic load 8598 atomic/store 8599 atomic/atomicrmw 8600 with memory 8601 ordering of seq_cst 8602 and with equal or 8603 wider sync scope. 8604 (Note that seq_cst 8605 fences have their 8606 own s_waitcnt 8607 vmcnt(0) and so do 8608 not need to be 8609 considered.) 8610 - Ensures any 8611 preceding 8612 sequential 8613 consistent global 8614 memory instructions 8615 have completed 8616 before executing 8617 this sequentially 8618 consistent 8619 instruction. This 8620 prevents reordering 8621 a seq_cst store 8622 followed by a 8623 seq_cst load. (Note 8624 that seq_cst is 8625 stronger than 8626 acquire/release as 8627 the reordering of 8628 load acquire 8629 followed by a store 8630 release is 8631 prevented by the 8632 s_waitcnt of 8633 the release, but 8634 there is nothing 8635 preventing a store 8636 release followed by 8637 load acquire from 8638 completing out of 8639 order. The s_waitcnt 8640 could be placed after 8641 seq_store or before 8642 the seq_load. We 8643 choose the load to 8644 make the s_waitcnt be 8645 as late as possible 8646 so that the store 8647 may have already 8648 completed.) 8649 8650 2. *Following 8651 instructions same as 8652 corresponding load 8653 atomic acquire, 8654 except must generate 8655 all instructions even 8656 for OpenCL.* 8657 store atomic seq_cst - singlethread - global *Same as corresponding 8658 - wavefront - local store atomic release, 8659 - workgroup - generic except must generate 8660 - agent all instructions even 8661 - system for OpenCL.* 8662 atomicrmw seq_cst - singlethread - global *Same as corresponding 8663 - wavefront - local atomicrmw acq_rel, 8664 - workgroup - generic except must generate 8665 - agent all instructions even 8666 - system for OpenCL.* 8667 fence seq_cst - singlethread *none* *Same as corresponding 8668 - wavefront fence acq_rel, 8669 - workgroup except must generate 8670 - agent all instructions even 8671 - system for OpenCL.* 8672 ============ ============ ============== ========== ================================ 8673 8674.. _amdgpu-amdhsa-memory-model-gfx940: 8675 8676Memory Model GFX940 8677+++++++++++++++++++ 8678 8679For GFX940: 8680 8681* Each agent has multiple shader arrays (SA). 8682* Each SA has multiple compute units (CU). 8683* Each CU has multiple SIMDs that execute wavefronts. 8684* The wavefronts for a single work-group are executed in the same CU but may be 8685 executed by different SIMDs. The exception is when in tgsplit execution mode 8686 when the wavefronts may be executed by different SIMDs in different CUs. 8687* Each CU has a single LDS memory shared by the wavefronts of the work-groups 8688 executing on it. The exception is when in tgsplit execution mode when no LDS 8689 is allocated as wavefronts of the same work-group can be in different CUs. 8690* All LDS operations of a CU are performed as wavefront wide operations in a 8691 global order and involve no caching. Completion is reported to a wavefront in 8692 execution order. 8693* The LDS memory has multiple request queues shared by the SIMDs of a 8694 CU. Therefore, the LDS operations performed by different wavefronts of a 8695 work-group can be reordered relative to each other, which can result in 8696 reordering the visibility of vector memory operations with respect to LDS 8697 operations of other wavefronts in the same work-group. A ``s_waitcnt 8698 lgkmcnt(0)`` is required to ensure synchronization between LDS operations and 8699 vector memory operations between wavefronts of a work-group, but not between 8700 operations performed by the same wavefront. 8701* The vector memory operations are performed as wavefront wide operations and 8702 completion is reported to a wavefront in execution order. The exception is 8703 that ``flat_load/store/atomic`` instructions can report out of vector memory 8704 order if they access LDS memory, and out of LDS operation order if they access 8705 global memory. 8706* The vector memory operations access a single vector L1 cache shared by all 8707 SIMDs a CU. Therefore: 8708 8709 * No special action is required for coherence between the lanes of a single 8710 wavefront. 8711 8712 * No special action is required for coherence between wavefronts in the same 8713 work-group since they execute on the same CU. The exception is when in 8714 tgsplit execution mode as wavefronts of the same work-group can be in 8715 different CUs and so a ``buffer_inv sc0`` is required which will invalidate 8716 the L1 cache. 8717 8718 * A ``buffer_inv sc0`` is required to invalidate the L1 cache for coherence 8719 between wavefronts executing in different work-groups as they may be 8720 executing on different CUs. 8721 8722 * Atomic read-modify-write instructions implicitly bypass the L1 cache. 8723 Therefore, they do not use the sc0 bit for coherence and instead use it to 8724 indicate if the instruction returns the original value being updated. They 8725 do use sc1 to indicate system or agent scope coherence. 8726 8727* The scalar memory operations access a scalar L1 cache shared by all wavefronts 8728 on a group of CUs. The scalar and vector L1 caches are not coherent. However, 8729 scalar operations are used in a restricted way so do not impact the memory 8730 model. See :ref:`amdgpu-amdhsa-memory-spaces`. 8731* The vector and scalar memory operations use an L2 cache. 8732 8733 * The gfx940 can be configured as a number of smaller agents with each having 8734 a single L2 shared by all CUs on the same agent, or as fewer (possibly one) 8735 larger agents with groups of CUs on each agent each sharing separate L2 8736 caches. 8737 * The L2 cache has independent channels to service disjoint ranges of virtual 8738 addresses. 8739 * Each CU has a separate request queue per channel for its associated L2. 8740 Therefore, the vector and scalar memory operations performed by wavefronts 8741 executing with different L1 caches and the same L2 cache can be reordered 8742 relative to each other. 8743 * A ``s_waitcnt vmcnt(0)`` is required to ensure synchronization between 8744 vector memory operations of different CUs. It ensures a previous vector 8745 memory operation has completed before executing a subsequent vector memory 8746 or LDS operation and so can be used to meet the requirements of acquire and 8747 release. 8748 * An L2 cache can be kept coherent with other L2 caches by using the MTYPE RW 8749 (read-write) for memory local to the L2, and MTYPE NC (non-coherent) with 8750 the PTE C-bit set for memory not local to the L2. 8751 8752 * Any local memory cache lines will be automatically invalidated by writes 8753 from CUs associated with other L2 caches, or writes from the CPU, due to 8754 the cache probe caused by the PTE C-bit. 8755 * XGMI accesses from the CPU to local memory may be cached on the CPU. 8756 Subsequent access from the GPU will automatically invalidate or writeback 8757 the CPU cache due to the L2 probe filter. 8758 * To ensure coherence of local memory writes of CUs with different L1 caches 8759 in the same agent a ``buffer_wbl2`` is required. It does nothing if the 8760 agent is configured to have a single L2, or will writeback dirty L2 cache 8761 lines if configured to have multiple L2 caches. 8762 * To ensure coherence of local memory writes of CUs in different agents a 8763 ``buffer_wbl2 sc1`` is required. It will writeback dirty L2 cache lines. 8764 * To ensure coherence of local memory reads of CUs with different L1 caches 8765 in the same agent a ``buffer_inv sc1`` is required. It does nothing if the 8766 agent is configured to have a single L2, or will invalidate non-local L2 8767 cache lines if configured to have multiple L2 caches. 8768 * To ensure coherence of local memory reads of CUs in different agents a 8769 ``buffer_inv sc0 sc1`` is required. It will invalidate non-local L2 cache 8770 lines if configured to have multiple L2 caches. 8771 8772 * PCIe access from the GPU to the CPU can be kept coherent by using the MTYPE 8773 UC (uncached) which bypasses the L2. 8774 8775Scalar memory operations are only used to access memory that is proven to not 8776change during the execution of the kernel dispatch. This includes constant 8777address space and global address space for program scope ``const`` variables. 8778Therefore, the kernel machine code does not have to maintain the scalar cache to 8779ensure it is coherent with the vector caches. The scalar and vector caches are 8780invalidated between kernel dispatches by CP since constant address space data 8781may change between kernel dispatch executions. See 8782:ref:`amdgpu-amdhsa-memory-spaces`. 8783 8784The one exception is if scalar writes are used to spill SGPR registers. In this 8785case the AMDGPU backend ensures the memory location used to spill is never 8786accessed by vector memory operations at the same time. If scalar writes are used 8787then a ``s_dcache_wb`` is inserted before the ``s_endpgm`` and before a function 8788return since the locations may be used for vector memory instructions by a 8789future wavefront that uses the same scratch area, or a function call that 8790creates a frame at the same address, respectively. There is no need for a 8791``s_dcache_inv`` as all scalar writes are write-before-read in the same thread. 8792 8793For kernarg backing memory: 8794 8795* CP invalidates the L1 cache at the start of each kernel dispatch. 8796* On dGPU over XGMI or PCIe the kernarg backing memory is allocated in host 8797 memory accessed as MTYPE UC (uncached) to avoid needing to invalidate the L2 8798 cache. This also causes it to be treated as non-volatile and so is not 8799 invalidated by ``*_vol``. 8800* On APU the kernarg backing memory is accessed as MTYPE CC (cache coherent) and 8801 so the L2 cache will be coherent with the CPU and other agents. 8802 8803Scratch backing memory (which is used for the private address space) is accessed 8804with MTYPE NC_NV (non-coherent non-volatile). Since the private address space is 8805only accessed by a single thread, and is always write-before-read, there is 8806never a need to invalidate these entries from the L1 cache. Hence all cache 8807invalidates are done as ``*_vol`` to only invalidate the volatile cache lines. 8808 8809The code sequences used to implement the memory model for GFX940 are defined 8810in table :ref:`amdgpu-amdhsa-memory-model-code-sequences-gfx940-table`. 8811 8812 .. table:: AMDHSA Memory Model Code Sequences GFX940 8813 :name: amdgpu-amdhsa-memory-model-code-sequences-gfx940-table 8814 8815 ============ ============ ============== ========== ================================ 8816 LLVM Instr LLVM Memory LLVM Memory AMDGPU AMDGPU Machine Code 8817 Ordering Sync Scope Address GFX940 8818 Space 8819 ============ ============ ============== ========== ================================ 8820 **Non-Atomic** 8821 ------------------------------------------------------------------------------------ 8822 load *none* *none* - global - !volatile & !nontemporal 8823 - generic 8824 - private 1. buffer/global/flat_load 8825 - constant 8826 - !volatile & nontemporal 8827 8828 1. buffer/global/flat_load 8829 nt=1 8830 8831 - volatile 8832 8833 1. buffer/global/flat_load 8834 sc0=1 sc1=1 8835 2. s_waitcnt vmcnt(0) 8836 8837 - Must happen before 8838 any following volatile 8839 global/generic 8840 load/store. 8841 - Ensures that 8842 volatile 8843 operations to 8844 different 8845 addresses will not 8846 be reordered by 8847 hardware. 8848 8849 load *none* *none* - local 1. ds_load 8850 store *none* *none* - global - !volatile & !nontemporal 8851 - generic 8852 - private 1. buffer/global/flat_store 8853 - constant 8854 - !volatile & nontemporal 8855 8856 1. buffer/global/flat_store 8857 nt=1 8858 8859 - volatile 8860 8861 1. buffer/global/flat_store 8862 sc0=1 sc1=1 8863 2. s_waitcnt vmcnt(0) 8864 8865 - Must happen before 8866 any following volatile 8867 global/generic 8868 load/store. 8869 - Ensures that 8870 volatile 8871 operations to 8872 different 8873 addresses will not 8874 be reordered by 8875 hardware. 8876 8877 store *none* *none* - local 1. ds_store 8878 **Unordered Atomic** 8879 ------------------------------------------------------------------------------------ 8880 load atomic unordered *any* *any* *Same as non-atomic*. 8881 store atomic unordered *any* *any* *Same as non-atomic*. 8882 atomicrmw unordered *any* *any* *Same as monotonic atomic*. 8883 **Monotonic Atomic** 8884 ------------------------------------------------------------------------------------ 8885 load atomic monotonic - singlethread - global 1. buffer/global/flat_load 8886 - wavefront - generic 8887 load atomic monotonic - workgroup - global 1. buffer/global/flat_load 8888 - generic sc0=1 8889 load atomic monotonic - singlethread - local *If TgSplit execution mode, 8890 - wavefront local address space cannot 8891 - workgroup be used.* 8892 8893 1. ds_load 8894 load atomic monotonic - agent - global 1. buffer/global/flat_load 8895 - generic sc1=1 8896 load atomic monotonic - system - global 1. buffer/global/flat_load 8897 - generic sc0=1 sc1=1 8898 store atomic monotonic - singlethread - global 1. buffer/global/flat_store 8899 - wavefront - generic 8900 store atomic monotonic - workgroup - global 1. buffer/global/flat_store 8901 - generic sc0=1 8902 store atomic monotonic - agent - global 1. buffer/global/flat_store 8903 - generic sc1=1 8904 store atomic monotonic - system - global 1. buffer/global/flat_store 8905 - generic sc0=1 sc1=1 8906 store atomic monotonic - singlethread - local *If TgSplit execution mode, 8907 - wavefront local address space cannot 8908 - workgroup be used.* 8909 8910 1. ds_store 8911 atomicrmw monotonic - singlethread - global 1. buffer/global/flat_atomic 8912 - wavefront - generic 8913 - workgroup 8914 - agent 8915 atomicrmw monotonic - system - global 1. buffer/global/flat_atomic 8916 - generic sc1=1 8917 atomicrmw monotonic - singlethread - local *If TgSplit execution mode, 8918 - wavefront local address space cannot 8919 - workgroup be used.* 8920 8921 1. ds_atomic 8922 **Acquire Atomic** 8923 ------------------------------------------------------------------------------------ 8924 load atomic acquire - singlethread - global 1. buffer/global/ds/flat_load 8925 - wavefront - local 8926 - generic 8927 load atomic acquire - workgroup - global 1. buffer/global_load sc0=1 8928 2. s_waitcnt vmcnt(0) 8929 8930 - If not TgSplit execution 8931 mode, omit. 8932 - Must happen before the 8933 following buffer_inv. 8934 8935 3. buffer_inv sc0=1 8936 8937 - If not TgSplit execution 8938 mode, omit. 8939 - Must happen before 8940 any following 8941 global/generic 8942 load/load 8943 atomic/store/store 8944 atomic/atomicrmw. 8945 - Ensures that 8946 following 8947 loads will not see 8948 stale data. 8949 8950 load atomic acquire - workgroup - local *If TgSplit execution mode, 8951 local address space cannot 8952 be used.* 8953 8954 1. ds_load 8955 2. s_waitcnt lgkmcnt(0) 8956 8957 - If OpenCL, omit. 8958 - Must happen before 8959 any following 8960 global/generic 8961 load/load 8962 atomic/store/store 8963 atomic/atomicrmw. 8964 - Ensures any 8965 following global 8966 data read is no 8967 older than the local load 8968 atomic value being 8969 acquired. 8970 8971 load atomic acquire - workgroup - generic 1. flat_load sc0=1 8972 2. s_waitcnt lgkm/vmcnt(0) 8973 8974 - Use lgkmcnt(0) if not 8975 TgSplit execution mode 8976 and vmcnt(0) if TgSplit 8977 execution mode. 8978 - If OpenCL, omit lgkmcnt(0). 8979 - Must happen before 8980 the following 8981 buffer_inv and any 8982 following global/generic 8983 load/load 8984 atomic/store/store 8985 atomic/atomicrmw. 8986 - Ensures any 8987 following global 8988 data read is no 8989 older than a local load 8990 atomic value being 8991 acquired. 8992 8993 3. buffer_inv sc0=1 8994 8995 - If not TgSplit execution 8996 mode, omit. 8997 - Ensures that 8998 following 8999 loads will not see 9000 stale data. 9001 9002 load atomic acquire - agent - global 1. buffer/global_load 9003 sc1=1 9004 2. s_waitcnt vmcnt(0) 9005 9006 - Must happen before 9007 following 9008 buffer_inv. 9009 - Ensures the load 9010 has completed 9011 before invalidating 9012 the cache. 9013 9014 3. buffer_inv sc1=1 9015 9016 - Must happen before 9017 any following 9018 global/generic 9019 load/load 9020 atomic/atomicrmw. 9021 - Ensures that 9022 following 9023 loads will not see 9024 stale global data. 9025 9026 load atomic acquire - system - global 1. buffer/global/flat_load 9027 sc0=1 sc1=1 9028 2. s_waitcnt vmcnt(0) 9029 9030 - Must happen before 9031 following 9032 buffer_inv. 9033 - Ensures the load 9034 has completed 9035 before invalidating 9036 the cache. 9037 9038 3. buffer_inv sc0=1 sc1=1 9039 9040 - Must happen before 9041 any following 9042 global/generic 9043 load/load 9044 atomic/atomicrmw. 9045 - Ensures that 9046 following 9047 loads will not see 9048 stale MTYPE NC global data. 9049 MTYPE RW and CC memory will 9050 never be stale due to the 9051 memory probes. 9052 9053 load atomic acquire - agent - generic 1. flat_load sc1=1 9054 2. s_waitcnt vmcnt(0) & 9055 lgkmcnt(0) 9056 9057 - If TgSplit execution mode, 9058 omit lgkmcnt(0). 9059 - If OpenCL omit 9060 lgkmcnt(0). 9061 - Must happen before 9062 following 9063 buffer_inv. 9064 - Ensures the flat_load 9065 has completed 9066 before invalidating 9067 the cache. 9068 9069 3. buffer_inv sc1=1 9070 9071 - Must happen before 9072 any following 9073 global/generic 9074 load/load 9075 atomic/atomicrmw. 9076 - Ensures that 9077 following loads 9078 will not see stale 9079 global data. 9080 9081 load atomic acquire - system - generic 1. flat_load sc0=1 sc1=1 9082 2. s_waitcnt vmcnt(0) & 9083 lgkmcnt(0) 9084 9085 - If TgSplit execution mode, 9086 omit lgkmcnt(0). 9087 - If OpenCL omit 9088 lgkmcnt(0). 9089 - Must happen before 9090 the following 9091 buffer_inv. 9092 - Ensures the flat_load 9093 has completed 9094 before invalidating 9095 the caches. 9096 9097 3. buffer_inv sc0=1 sc1=1 9098 9099 - Must happen before 9100 any following 9101 global/generic 9102 load/load 9103 atomic/atomicrmw. 9104 - Ensures that 9105 following 9106 loads will not see 9107 stale MTYPE NC global data. 9108 MTYPE RW and CC memory will 9109 never be stale due to the 9110 memory probes. 9111 9112 atomicrmw acquire - singlethread - global 1. buffer/global/flat_atomic 9113 - wavefront - generic 9114 atomicrmw acquire - singlethread - local *If TgSplit execution mode, 9115 - wavefront local address space cannot 9116 be used.* 9117 9118 1. ds_atomic 9119 atomicrmw acquire - workgroup - global 1. buffer/global_atomic 9120 2. s_waitcnt vmcnt(0) 9121 9122 - If not TgSplit execution 9123 mode, omit. 9124 - Must happen before the 9125 following buffer_inv. 9126 - Ensures the atomicrmw 9127 has completed 9128 before invalidating 9129 the cache. 9130 9131 3. buffer_inv sc0=1 9132 9133 - If not TgSplit execution 9134 mode, omit. 9135 - Must happen before 9136 any following 9137 global/generic 9138 load/load 9139 atomic/atomicrmw. 9140 - Ensures that 9141 following loads 9142 will not see stale 9143 global data. 9144 9145 atomicrmw acquire - workgroup - local *If TgSplit execution mode, 9146 local address space cannot 9147 be used.* 9148 9149 1. ds_atomic 9150 2. s_waitcnt lgkmcnt(0) 9151 9152 - If OpenCL, omit. 9153 - Must happen before 9154 any following 9155 global/generic 9156 load/load 9157 atomic/store/store 9158 atomic/atomicrmw. 9159 - Ensures any 9160 following global 9161 data read is no 9162 older than the local 9163 atomicrmw value 9164 being acquired. 9165 9166 atomicrmw acquire - workgroup - generic 1. flat_atomic 9167 2. s_waitcnt lgkm/vmcnt(0) 9168 9169 - Use lgkmcnt(0) if not 9170 TgSplit execution mode 9171 and vmcnt(0) if TgSplit 9172 execution mode. 9173 - If OpenCL, omit lgkmcnt(0). 9174 - Must happen before 9175 the following 9176 buffer_inv and 9177 any following 9178 global/generic 9179 load/load 9180 atomic/store/store 9181 atomic/atomicrmw. 9182 - Ensures any 9183 following global 9184 data read is no 9185 older than a local 9186 atomicrmw value 9187 being acquired. 9188 9189 3. buffer_inv sc0=1 9190 9191 - If not TgSplit execution 9192 mode, omit. 9193 - Ensures that 9194 following 9195 loads will not see 9196 stale data. 9197 9198 atomicrmw acquire - agent - global 1. buffer/global_atomic 9199 2. s_waitcnt vmcnt(0) 9200 9201 - Must happen before 9202 following 9203 buffer_inv. 9204 - Ensures the 9205 atomicrmw has 9206 completed before 9207 invalidating the 9208 cache. 9209 9210 3. buffer_inv sc1=1 9211 9212 - Must happen before 9213 any following 9214 global/generic 9215 load/load 9216 atomic/atomicrmw. 9217 - Ensures that 9218 following loads 9219 will not see stale 9220 global data. 9221 9222 atomicrmw acquire - system - global 1. buffer/global_atomic 9223 sc1=1 9224 2. s_waitcnt vmcnt(0) 9225 9226 - Must happen before 9227 following 9228 buffer_inv. 9229 - Ensures the 9230 atomicrmw has 9231 completed before 9232 invalidating the 9233 caches. 9234 9235 3. buffer_inv sc0=1 sc1=1 9236 9237 - Must happen before 9238 any following 9239 global/generic 9240 load/load 9241 atomic/atomicrmw. 9242 - Ensures that 9243 following 9244 loads will not see 9245 stale MTYPE NC global data. 9246 MTYPE RW and CC memory will 9247 never be stale due to the 9248 memory probes. 9249 9250 atomicrmw acquire - agent - generic 1. flat_atomic 9251 2. s_waitcnt vmcnt(0) & 9252 lgkmcnt(0) 9253 9254 - If TgSplit execution mode, 9255 omit lgkmcnt(0). 9256 - If OpenCL, omit 9257 lgkmcnt(0). 9258 - Must happen before 9259 following 9260 buffer_inv. 9261 - Ensures the 9262 atomicrmw has 9263 completed before 9264 invalidating the 9265 cache. 9266 9267 3. buffer_inv sc1=1 9268 9269 - Must happen before 9270 any following 9271 global/generic 9272 load/load 9273 atomic/atomicrmw. 9274 - Ensures that 9275 following loads 9276 will not see stale 9277 global data. 9278 9279 atomicrmw acquire - system - generic 1. flat_atomic sc1=1 9280 2. s_waitcnt vmcnt(0) & 9281 lgkmcnt(0) 9282 9283 - If TgSplit execution mode, 9284 omit lgkmcnt(0). 9285 - If OpenCL, omit 9286 lgkmcnt(0). 9287 - Must happen before 9288 following 9289 buffer_inv. 9290 - Ensures the 9291 atomicrmw has 9292 completed before 9293 invalidating the 9294 caches. 9295 9296 3. buffer_inv sc0=1 sc1=1 9297 9298 - Must happen before 9299 any following 9300 global/generic 9301 load/load 9302 atomic/atomicrmw. 9303 - Ensures that 9304 following 9305 loads will not see 9306 stale MTYPE NC global data. 9307 MTYPE RW and CC memory will 9308 never be stale due to the 9309 memory probes. 9310 9311 fence acquire - singlethread *none* *none* 9312 - wavefront 9313 fence acquire - workgroup *none* 1. s_waitcnt lgkm/vmcnt(0) 9314 9315 - Use lgkmcnt(0) if not 9316 TgSplit execution mode 9317 and vmcnt(0) if TgSplit 9318 execution mode. 9319 - If OpenCL and 9320 address space is 9321 not generic, omit 9322 lgkmcnt(0). 9323 - If OpenCL and 9324 address space is 9325 local, omit 9326 vmcnt(0). 9327 - However, since LLVM 9328 currently has no 9329 address space on 9330 the fence need to 9331 conservatively 9332 always generate. If 9333 fence had an 9334 address space then 9335 set to address 9336 space of OpenCL 9337 fence flag, or to 9338 generic if both 9339 local and global 9340 flags are 9341 specified. 9342 - s_waitcnt vmcnt(0) 9343 must happen after 9344 any preceding 9345 global/generic load 9346 atomic/ 9347 atomicrmw 9348 with an equal or 9349 wider sync scope 9350 and memory ordering 9351 stronger than 9352 unordered (this is 9353 termed the 9354 fence-paired-atomic). 9355 - s_waitcnt lgkmcnt(0) 9356 must happen after 9357 any preceding 9358 local/generic load 9359 atomic/atomicrmw 9360 with an equal or 9361 wider sync scope 9362 and memory ordering 9363 stronger than 9364 unordered (this is 9365 termed the 9366 fence-paired-atomic). 9367 - Must happen before 9368 the following 9369 buffer_inv and 9370 any following 9371 global/generic 9372 load/load 9373 atomic/store/store 9374 atomic/atomicrmw. 9375 - Ensures any 9376 following global 9377 data read is no 9378 older than the 9379 value read by the 9380 fence-paired-atomic. 9381 9382 3. buffer_inv sc0=1 9383 9384 - If not TgSplit execution 9385 mode, omit. 9386 - Ensures that 9387 following 9388 loads will not see 9389 stale data. 9390 9391 fence acquire - agent *none* 1. s_waitcnt lgkmcnt(0) & 9392 vmcnt(0) 9393 9394 - If TgSplit execution mode, 9395 omit lgkmcnt(0). 9396 - If OpenCL and 9397 address space is 9398 not generic, omit 9399 lgkmcnt(0). 9400 - However, since LLVM 9401 currently has no 9402 address space on 9403 the fence need to 9404 conservatively 9405 always generate 9406 (see comment for 9407 previous fence). 9408 - Could be split into 9409 separate s_waitcnt 9410 vmcnt(0) and 9411 s_waitcnt 9412 lgkmcnt(0) to allow 9413 them to be 9414 independently moved 9415 according to the 9416 following rules. 9417 - s_waitcnt vmcnt(0) 9418 must happen after 9419 any preceding 9420 global/generic load 9421 atomic/atomicrmw 9422 with an equal or 9423 wider sync scope 9424 and memory ordering 9425 stronger than 9426 unordered (this is 9427 termed the 9428 fence-paired-atomic). 9429 - s_waitcnt lgkmcnt(0) 9430 must happen after 9431 any preceding 9432 local/generic load 9433 atomic/atomicrmw 9434 with an equal or 9435 wider sync scope 9436 and memory ordering 9437 stronger than 9438 unordered (this is 9439 termed the 9440 fence-paired-atomic). 9441 - Must happen before 9442 the following 9443 buffer_inv. 9444 - Ensures that the 9445 fence-paired atomic 9446 has completed 9447 before invalidating 9448 the 9449 cache. Therefore 9450 any following 9451 locations read must 9452 be no older than 9453 the value read by 9454 the 9455 fence-paired-atomic. 9456 9457 2. buffer_inv sc1=1 9458 9459 - Must happen before any 9460 following global/generic 9461 load/load 9462 atomic/store/store 9463 atomic/atomicrmw. 9464 - Ensures that 9465 following loads 9466 will not see stale 9467 global data. 9468 9469 fence acquire - system *none* 1. s_waitcnt lgkmcnt(0) & 9470 vmcnt(0) 9471 9472 - If TgSplit execution mode, 9473 omit lgkmcnt(0). 9474 - If OpenCL and 9475 address space is 9476 not generic, omit 9477 lgkmcnt(0). 9478 - However, since LLVM 9479 currently has no 9480 address space on 9481 the fence need to 9482 conservatively 9483 always generate 9484 (see comment for 9485 previous fence). 9486 - Could be split into 9487 separate s_waitcnt 9488 vmcnt(0) and 9489 s_waitcnt 9490 lgkmcnt(0) to allow 9491 them to be 9492 independently moved 9493 according to the 9494 following rules. 9495 - s_waitcnt vmcnt(0) 9496 must happen after 9497 any preceding 9498 global/generic load 9499 atomic/atomicrmw 9500 with an equal or 9501 wider sync scope 9502 and memory ordering 9503 stronger than 9504 unordered (this is 9505 termed the 9506 fence-paired-atomic). 9507 - s_waitcnt lgkmcnt(0) 9508 must happen after 9509 any preceding 9510 local/generic load 9511 atomic/atomicrmw 9512 with an equal or 9513 wider sync scope 9514 and memory ordering 9515 stronger than 9516 unordered (this is 9517 termed the 9518 fence-paired-atomic). 9519 - Must happen before 9520 the following 9521 buffer_inv. 9522 - Ensures that the 9523 fence-paired atomic 9524 has completed 9525 before invalidating 9526 the 9527 cache. Therefore 9528 any following 9529 locations read must 9530 be no older than 9531 the value read by 9532 the 9533 fence-paired-atomic. 9534 9535 2. buffer_inv sc0=1 sc1=1 9536 9537 - Must happen before any 9538 following global/generic 9539 load/load 9540 atomic/store/store 9541 atomic/atomicrmw. 9542 - Ensures that 9543 following loads 9544 will not see stale 9545 global data. 9546 9547 **Release Atomic** 9548 ------------------------------------------------------------------------------------ 9549 store atomic release - singlethread - global 1. buffer/global/flat_store 9550 - wavefront - generic 9551 store atomic release - singlethread - local *If TgSplit execution mode, 9552 - wavefront local address space cannot 9553 be used.* 9554 9555 1. ds_store 9556 store atomic release - workgroup - global 1. s_waitcnt lgkm/vmcnt(0) 9557 - generic 9558 - Use lgkmcnt(0) if not 9559 TgSplit execution mode 9560 and vmcnt(0) if TgSplit 9561 execution mode. 9562 - If OpenCL, omit lgkmcnt(0). 9563 - s_waitcnt vmcnt(0) 9564 must happen after 9565 any preceding 9566 global/generic load/store/ 9567 load atomic/store atomic/ 9568 atomicrmw. 9569 - s_waitcnt lgkmcnt(0) 9570 must happen after 9571 any preceding 9572 local/generic 9573 load/store/load 9574 atomic/store 9575 atomic/atomicrmw. 9576 - Must happen before 9577 the following 9578 store. 9579 - Ensures that all 9580 memory operations 9581 have 9582 completed before 9583 performing the 9584 store that is being 9585 released. 9586 9587 2. buffer/global/flat_store sc0=1 9588 store atomic release - workgroup - local *If TgSplit execution mode, 9589 local address space cannot 9590 be used.* 9591 9592 1. ds_store 9593 store atomic release - agent - global 1. buffer_wbl2 sc1=1 9594 - generic 9595 - Must happen before 9596 following s_waitcnt. 9597 - Performs L2 writeback to 9598 ensure previous 9599 global/generic 9600 store/atomicrmw are 9601 visible at agent scope. 9602 9603 2. s_waitcnt lgkmcnt(0) & 9604 vmcnt(0) 9605 9606 - If TgSplit execution mode, 9607 omit lgkmcnt(0). 9608 - If OpenCL and 9609 address space is 9610 not generic, omit 9611 lgkmcnt(0). 9612 - Could be split into 9613 separate s_waitcnt 9614 vmcnt(0) and 9615 s_waitcnt 9616 lgkmcnt(0) to allow 9617 them to be 9618 independently moved 9619 according to the 9620 following rules. 9621 - s_waitcnt vmcnt(0) 9622 must happen after 9623 any preceding 9624 global/generic 9625 load/store/load 9626 atomic/store 9627 atomic/atomicrmw. 9628 - s_waitcnt lgkmcnt(0) 9629 must happen after 9630 any preceding 9631 local/generic 9632 load/store/load 9633 atomic/store 9634 atomic/atomicrmw. 9635 - Must happen before 9636 the following 9637 store. 9638 - Ensures that all 9639 memory operations 9640 to memory have 9641 completed before 9642 performing the 9643 store that is being 9644 released. 9645 9646 3. buffer/global/flat_store sc1=1 9647 store atomic release - system - global 1. buffer_wbl2 sc0=1 sc1=1 9648 - generic 9649 - Must happen before 9650 following s_waitcnt. 9651 - Performs L2 writeback to 9652 ensure previous 9653 global/generic 9654 store/atomicrmw are 9655 visible at system scope. 9656 9657 2. s_waitcnt lgkmcnt(0) & 9658 vmcnt(0) 9659 9660 - If TgSplit execution mode, 9661 omit lgkmcnt(0). 9662 - If OpenCL and 9663 address space is 9664 not generic, omit 9665 lgkmcnt(0). 9666 - Could be split into 9667 separate s_waitcnt 9668 vmcnt(0) and 9669 s_waitcnt 9670 lgkmcnt(0) to allow 9671 them to be 9672 independently moved 9673 according to the 9674 following rules. 9675 - s_waitcnt vmcnt(0) 9676 must happen after any 9677 preceding 9678 global/generic 9679 load/store/load 9680 atomic/store 9681 atomic/atomicrmw. 9682 - s_waitcnt lgkmcnt(0) 9683 must happen after any 9684 preceding 9685 local/generic 9686 load/store/load 9687 atomic/store 9688 atomic/atomicrmw. 9689 - Must happen before 9690 the following 9691 store. 9692 - Ensures that all 9693 memory operations 9694 to memory and the L2 9695 writeback have 9696 completed before 9697 performing the 9698 store that is being 9699 released. 9700 9701 3. buffer/global/flat_store 9702 sc0=1 sc1=1 9703 atomicrmw release - singlethread - global 1. buffer/global/flat_atomic 9704 - wavefront - generic 9705 atomicrmw release - singlethread - local *If TgSplit execution mode, 9706 - wavefront local address space cannot 9707 be used.* 9708 9709 1. ds_atomic 9710 atomicrmw release - workgroup - global 1. s_waitcnt lgkm/vmcnt(0) 9711 - generic 9712 - Use lgkmcnt(0) if not 9713 TgSplit execution mode 9714 and vmcnt(0) if TgSplit 9715 execution mode. 9716 - If OpenCL, omit 9717 lgkmcnt(0). 9718 - s_waitcnt vmcnt(0) 9719 must happen after 9720 any preceding 9721 global/generic load/store/ 9722 load atomic/store atomic/ 9723 atomicrmw. 9724 - s_waitcnt lgkmcnt(0) 9725 must happen after 9726 any preceding 9727 local/generic 9728 load/store/load 9729 atomic/store 9730 atomic/atomicrmw. 9731 - Must happen before 9732 the following 9733 atomicrmw. 9734 - Ensures that all 9735 memory operations 9736 have 9737 completed before 9738 performing the 9739 atomicrmw that is 9740 being released. 9741 9742 2. buffer/global/flat_atomic sc0=1 9743 atomicrmw release - workgroup - local *If TgSplit execution mode, 9744 local address space cannot 9745 be used.* 9746 9747 1. ds_atomic 9748 atomicrmw release - agent - global 1. buffer_wbl2 sc1=1 9749 - generic 9750 - Must happen before 9751 following s_waitcnt. 9752 - Performs L2 writeback to 9753 ensure previous 9754 global/generic 9755 store/atomicrmw are 9756 visible at agent scope. 9757 9758 2. s_waitcnt lgkmcnt(0) & 9759 vmcnt(0) 9760 9761 - If TgSplit execution mode, 9762 omit lgkmcnt(0). 9763 - If OpenCL, omit 9764 lgkmcnt(0). 9765 - Could be split into 9766 separate s_waitcnt 9767 vmcnt(0) and 9768 s_waitcnt 9769 lgkmcnt(0) to allow 9770 them to be 9771 independently moved 9772 according to the 9773 following rules. 9774 - s_waitcnt vmcnt(0) 9775 must happen after 9776 any preceding 9777 global/generic 9778 load/store/load 9779 atomic/store 9780 atomic/atomicrmw. 9781 - s_waitcnt lgkmcnt(0) 9782 must happen after 9783 any preceding 9784 local/generic 9785 load/store/load 9786 atomic/store 9787 atomic/atomicrmw. 9788 - Must happen before 9789 the following 9790 atomicrmw. 9791 - Ensures that all 9792 memory operations 9793 to global and local 9794 have completed 9795 before performing 9796 the atomicrmw that 9797 is being released. 9798 9799 3. buffer/global/flat_atomic sc1=1 9800 atomicrmw release - system - global 1. buffer_wbl2 sc0=1 sc1=1 9801 - generic 9802 - Must happen before 9803 following s_waitcnt. 9804 - Performs L2 writeback to 9805 ensure previous 9806 global/generic 9807 store/atomicrmw are 9808 visible at system scope. 9809 9810 2. s_waitcnt lgkmcnt(0) & 9811 vmcnt(0) 9812 9813 - If TgSplit execution mode, 9814 omit lgkmcnt(0). 9815 - If OpenCL, omit 9816 lgkmcnt(0). 9817 - Could be split into 9818 separate s_waitcnt 9819 vmcnt(0) and 9820 s_waitcnt 9821 lgkmcnt(0) to allow 9822 them to be 9823 independently moved 9824 according to the 9825 following rules. 9826 - s_waitcnt vmcnt(0) 9827 must happen after 9828 any preceding 9829 global/generic 9830 load/store/load 9831 atomic/store 9832 atomic/atomicrmw. 9833 - s_waitcnt lgkmcnt(0) 9834 must happen after 9835 any preceding 9836 local/generic 9837 load/store/load 9838 atomic/store 9839 atomic/atomicrmw. 9840 - Must happen before 9841 the following 9842 atomicrmw. 9843 - Ensures that all 9844 memory operations 9845 to memory and the L2 9846 writeback have 9847 completed before 9848 performing the 9849 store that is being 9850 released. 9851 9852 3. buffer/global/flat_atomic 9853 sc0=1 sc1=1 9854 fence release - singlethread *none* *none* 9855 - wavefront 9856 fence release - workgroup *none* 1. s_waitcnt lgkm/vmcnt(0) 9857 9858 - Use lgkmcnt(0) if not 9859 TgSplit execution mode 9860 and vmcnt(0) if TgSplit 9861 execution mode. 9862 - If OpenCL and 9863 address space is 9864 not generic, omit 9865 lgkmcnt(0). 9866 - If OpenCL and 9867 address space is 9868 local, omit 9869 vmcnt(0). 9870 - However, since LLVM 9871 currently has no 9872 address space on 9873 the fence need to 9874 conservatively 9875 always generate. If 9876 fence had an 9877 address space then 9878 set to address 9879 space of OpenCL 9880 fence flag, or to 9881 generic if both 9882 local and global 9883 flags are 9884 specified. 9885 - s_waitcnt vmcnt(0) 9886 must happen after 9887 any preceding 9888 global/generic 9889 load/store/ 9890 load atomic/store atomic/ 9891 atomicrmw. 9892 - s_waitcnt lgkmcnt(0) 9893 must happen after 9894 any preceding 9895 local/generic 9896 load/load 9897 atomic/store/store 9898 atomic/atomicrmw. 9899 - Must happen before 9900 any following store 9901 atomic/atomicrmw 9902 with an equal or 9903 wider sync scope 9904 and memory ordering 9905 stronger than 9906 unordered (this is 9907 termed the 9908 fence-paired-atomic). 9909 - Ensures that all 9910 memory operations 9911 have 9912 completed before 9913 performing the 9914 following 9915 fence-paired-atomic. 9916 9917 fence release - agent *none* 1. buffer_wbl2 sc1=1 9918 9919 - If OpenCL and 9920 address space is 9921 local, omit. 9922 - Must happen before 9923 following s_waitcnt. 9924 - Performs L2 writeback to 9925 ensure previous 9926 global/generic 9927 store/atomicrmw are 9928 visible at agent scope. 9929 9930 2. s_waitcnt lgkmcnt(0) & 9931 vmcnt(0) 9932 9933 - If TgSplit execution mode, 9934 omit lgkmcnt(0). 9935 - If OpenCL and 9936 address space is 9937 not generic, omit 9938 lgkmcnt(0). 9939 - If OpenCL and 9940 address space is 9941 local, omit 9942 vmcnt(0). 9943 - However, since LLVM 9944 currently has no 9945 address space on 9946 the fence need to 9947 conservatively 9948 always generate. If 9949 fence had an 9950 address space then 9951 set to address 9952 space of OpenCL 9953 fence flag, or to 9954 generic if both 9955 local and global 9956 flags are 9957 specified. 9958 - Could be split into 9959 separate s_waitcnt 9960 vmcnt(0) and 9961 s_waitcnt 9962 lgkmcnt(0) to allow 9963 them to be 9964 independently moved 9965 according to the 9966 following rules. 9967 - s_waitcnt vmcnt(0) 9968 must happen after 9969 any preceding 9970 global/generic 9971 load/store/load 9972 atomic/store 9973 atomic/atomicrmw. 9974 - s_waitcnt lgkmcnt(0) 9975 must happen after 9976 any preceding 9977 local/generic 9978 load/store/load 9979 atomic/store 9980 atomic/atomicrmw. 9981 - Must happen before 9982 any following store 9983 atomic/atomicrmw 9984 with an equal or 9985 wider sync scope 9986 and memory ordering 9987 stronger than 9988 unordered (this is 9989 termed the 9990 fence-paired-atomic). 9991 - Ensures that all 9992 memory operations 9993 have 9994 completed before 9995 performing the 9996 following 9997 fence-paired-atomic. 9998 9999 fence release - system *none* 1. buffer_wbl2 sc0=1 sc1=1 10000 10001 - Must happen before 10002 following s_waitcnt. 10003 - Performs L2 writeback to 10004 ensure previous 10005 global/generic 10006 store/atomicrmw are 10007 visible at system scope. 10008 10009 2. s_waitcnt lgkmcnt(0) & 10010 vmcnt(0) 10011 10012 - If TgSplit execution mode, 10013 omit lgkmcnt(0). 10014 - If OpenCL and 10015 address space is 10016 not generic, omit 10017 lgkmcnt(0). 10018 - If OpenCL and 10019 address space is 10020 local, omit 10021 vmcnt(0). 10022 - However, since LLVM 10023 currently has no 10024 address space on 10025 the fence need to 10026 conservatively 10027 always generate. If 10028 fence had an 10029 address space then 10030 set to address 10031 space of OpenCL 10032 fence flag, or to 10033 generic if both 10034 local and global 10035 flags are 10036 specified. 10037 - Could be split into 10038 separate s_waitcnt 10039 vmcnt(0) and 10040 s_waitcnt 10041 lgkmcnt(0) to allow 10042 them to be 10043 independently moved 10044 according to the 10045 following rules. 10046 - s_waitcnt vmcnt(0) 10047 must happen after 10048 any preceding 10049 global/generic 10050 load/store/load 10051 atomic/store 10052 atomic/atomicrmw. 10053 - s_waitcnt lgkmcnt(0) 10054 must happen after 10055 any preceding 10056 local/generic 10057 load/store/load 10058 atomic/store 10059 atomic/atomicrmw. 10060 - Must happen before 10061 any following store 10062 atomic/atomicrmw 10063 with an equal or 10064 wider sync scope 10065 and memory ordering 10066 stronger than 10067 unordered (this is 10068 termed the 10069 fence-paired-atomic). 10070 - Ensures that all 10071 memory operations 10072 have 10073 completed before 10074 performing the 10075 following 10076 fence-paired-atomic. 10077 10078 **Acquire-Release Atomic** 10079 ------------------------------------------------------------------------------------ 10080 atomicrmw acq_rel - singlethread - global 1. buffer/global/flat_atomic 10081 - wavefront - generic 10082 atomicrmw acq_rel - singlethread - local *If TgSplit execution mode, 10083 - wavefront local address space cannot 10084 be used.* 10085 10086 1. ds_atomic 10087 atomicrmw acq_rel - workgroup - global 1. s_waitcnt lgkm/vmcnt(0) 10088 10089 - Use lgkmcnt(0) if not 10090 TgSplit execution mode 10091 and vmcnt(0) if TgSplit 10092 execution mode. 10093 - If OpenCL, omit 10094 lgkmcnt(0). 10095 - Must happen after 10096 any preceding 10097 local/generic 10098 load/store/load 10099 atomic/store 10100 atomic/atomicrmw. 10101 - s_waitcnt vmcnt(0) 10102 must happen after 10103 any preceding 10104 global/generic load/store/ 10105 load atomic/store atomic/ 10106 atomicrmw. 10107 - s_waitcnt lgkmcnt(0) 10108 must happen after 10109 any preceding 10110 local/generic 10111 load/store/load 10112 atomic/store 10113 atomic/atomicrmw. 10114 - Must happen before 10115 the following 10116 atomicrmw. 10117 - Ensures that all 10118 memory operations 10119 have 10120 completed before 10121 performing the 10122 atomicrmw that is 10123 being released. 10124 10125 2. buffer/global_atomic 10126 3. s_waitcnt vmcnt(0) 10127 10128 - If not TgSplit execution 10129 mode, omit. 10130 - Must happen before 10131 the following 10132 buffer_inv. 10133 - Ensures any 10134 following global 10135 data read is no 10136 older than the 10137 atomicrmw value 10138 being acquired. 10139 10140 4. buffer_inv sc0=1 10141 10142 - If not TgSplit execution 10143 mode, omit. 10144 - Ensures that 10145 following 10146 loads will not see 10147 stale data. 10148 10149 atomicrmw acq_rel - workgroup - local *If TgSplit execution mode, 10150 local address space cannot 10151 be used.* 10152 10153 1. ds_atomic 10154 2. s_waitcnt lgkmcnt(0) 10155 10156 - If OpenCL, omit. 10157 - Must happen before 10158 any following 10159 global/generic 10160 load/load 10161 atomic/store/store 10162 atomic/atomicrmw. 10163 - Ensures any 10164 following global 10165 data read is no 10166 older than the local load 10167 atomic value being 10168 acquired. 10169 10170 atomicrmw acq_rel - workgroup - generic 1. s_waitcnt lgkm/vmcnt(0) 10171 10172 - Use lgkmcnt(0) if not 10173 TgSplit execution mode 10174 and vmcnt(0) if TgSplit 10175 execution mode. 10176 - If OpenCL, omit 10177 lgkmcnt(0). 10178 - s_waitcnt vmcnt(0) 10179 must happen after 10180 any preceding 10181 global/generic load/store/ 10182 load atomic/store atomic/ 10183 atomicrmw. 10184 - s_waitcnt lgkmcnt(0) 10185 must happen after 10186 any preceding 10187 local/generic 10188 load/store/load 10189 atomic/store 10190 atomic/atomicrmw. 10191 - Must happen before 10192 the following 10193 atomicrmw. 10194 - Ensures that all 10195 memory operations 10196 have 10197 completed before 10198 performing the 10199 atomicrmw that is 10200 being released. 10201 10202 2. flat_atomic 10203 3. s_waitcnt lgkmcnt(0) & 10204 vmcnt(0) 10205 10206 - If not TgSplit execution 10207 mode, omit vmcnt(0). 10208 - If OpenCL, omit 10209 lgkmcnt(0). 10210 - Must happen before 10211 the following 10212 buffer_inv and 10213 any following 10214 global/generic 10215 load/load 10216 atomic/store/store 10217 atomic/atomicrmw. 10218 - Ensures any 10219 following global 10220 data read is no 10221 older than a local load 10222 atomic value being 10223 acquired. 10224 10225 3. buffer_inv sc0=1 10226 10227 - If not TgSplit execution 10228 mode, omit. 10229 - Ensures that 10230 following 10231 loads will not see 10232 stale data. 10233 10234 atomicrmw acq_rel - agent - global 1. buffer_wbl2 sc1=1 10235 10236 - Must happen before 10237 following s_waitcnt. 10238 - Performs L2 writeback to 10239 ensure previous 10240 global/generic 10241 store/atomicrmw are 10242 visible at agent scope. 10243 10244 2. s_waitcnt lgkmcnt(0) & 10245 vmcnt(0) 10246 10247 - If TgSplit execution mode, 10248 omit lgkmcnt(0). 10249 - If OpenCL, omit 10250 lgkmcnt(0). 10251 - Could be split into 10252 separate s_waitcnt 10253 vmcnt(0) and 10254 s_waitcnt 10255 lgkmcnt(0) to allow 10256 them to be 10257 independently moved 10258 according to the 10259 following rules. 10260 - s_waitcnt vmcnt(0) 10261 must happen after 10262 any preceding 10263 global/generic 10264 load/store/load 10265 atomic/store 10266 atomic/atomicrmw. 10267 - s_waitcnt lgkmcnt(0) 10268 must happen after 10269 any preceding 10270 local/generic 10271 load/store/load 10272 atomic/store 10273 atomic/atomicrmw. 10274 - Must happen before 10275 the following 10276 atomicrmw. 10277 - Ensures that all 10278 memory operations 10279 to global have 10280 completed before 10281 performing the 10282 atomicrmw that is 10283 being released. 10284 10285 3. buffer/global_atomic 10286 4. s_waitcnt vmcnt(0) 10287 10288 - Must happen before 10289 following 10290 buffer_inv. 10291 - Ensures the 10292 atomicrmw has 10293 completed before 10294 invalidating the 10295 cache. 10296 10297 5. buffer_inv sc1=1 10298 10299 - Must happen before 10300 any following 10301 global/generic 10302 load/load 10303 atomic/atomicrmw. 10304 - Ensures that 10305 following loads 10306 will not see stale 10307 global data. 10308 10309 atomicrmw acq_rel - system - global 1. buffer_wbl2 sc0=1 sc1=1 10310 10311 - Must happen before 10312 following s_waitcnt. 10313 - Performs L2 writeback to 10314 ensure previous 10315 global/generic 10316 store/atomicrmw are 10317 visible at system scope. 10318 10319 2. s_waitcnt lgkmcnt(0) & 10320 vmcnt(0) 10321 10322 - If TgSplit execution mode, 10323 omit lgkmcnt(0). 10324 - If OpenCL, omit 10325 lgkmcnt(0). 10326 - Could be split into 10327 separate s_waitcnt 10328 vmcnt(0) and 10329 s_waitcnt 10330 lgkmcnt(0) to allow 10331 them to be 10332 independently moved 10333 according to the 10334 following rules. 10335 - s_waitcnt vmcnt(0) 10336 must happen after 10337 any preceding 10338 global/generic 10339 load/store/load 10340 atomic/store 10341 atomic/atomicrmw. 10342 - s_waitcnt lgkmcnt(0) 10343 must happen after 10344 any preceding 10345 local/generic 10346 load/store/load 10347 atomic/store 10348 atomic/atomicrmw. 10349 - Must happen before 10350 the following 10351 atomicrmw. 10352 - Ensures that all 10353 memory operations 10354 to global and L2 writeback 10355 have completed before 10356 performing the 10357 atomicrmw that is 10358 being released. 10359 10360 3. buffer/global_atomic 10361 sc1=1 10362 4. s_waitcnt vmcnt(0) 10363 10364 - Must happen before 10365 following 10366 buffer_inv. 10367 - Ensures the 10368 atomicrmw has 10369 completed before 10370 invalidating the 10371 caches. 10372 10373 5. buffer_inv sc0=1 sc1=1 10374 10375 - Must happen before 10376 any following 10377 global/generic 10378 load/load 10379 atomic/atomicrmw. 10380 - Ensures that 10381 following loads 10382 will not see stale 10383 MTYPE NC global data. 10384 MTYPE RW and CC memory will 10385 never be stale due to the 10386 memory probes. 10387 10388 atomicrmw acq_rel - agent - generic 1. buffer_wbl2 sc1=1 10389 10390 - Must happen before 10391 following s_waitcnt. 10392 - Performs L2 writeback to 10393 ensure previous 10394 global/generic 10395 store/atomicrmw are 10396 visible at agent scope. 10397 10398 2. s_waitcnt lgkmcnt(0) & 10399 vmcnt(0) 10400 10401 - If TgSplit execution mode, 10402 omit lgkmcnt(0). 10403 - If OpenCL, omit 10404 lgkmcnt(0). 10405 - Could be split into 10406 separate s_waitcnt 10407 vmcnt(0) and 10408 s_waitcnt 10409 lgkmcnt(0) to allow 10410 them to be 10411 independently moved 10412 according to the 10413 following rules. 10414 - s_waitcnt vmcnt(0) 10415 must happen after 10416 any preceding 10417 global/generic 10418 load/store/load 10419 atomic/store 10420 atomic/atomicrmw. 10421 - s_waitcnt lgkmcnt(0) 10422 must happen after 10423 any preceding 10424 local/generic 10425 load/store/load 10426 atomic/store 10427 atomic/atomicrmw. 10428 - Must happen before 10429 the following 10430 atomicrmw. 10431 - Ensures that all 10432 memory operations 10433 to global have 10434 completed before 10435 performing the 10436 atomicrmw that is 10437 being released. 10438 10439 3. flat_atomic 10440 4. s_waitcnt vmcnt(0) & 10441 lgkmcnt(0) 10442 10443 - If TgSplit execution mode, 10444 omit lgkmcnt(0). 10445 - If OpenCL, omit 10446 lgkmcnt(0). 10447 - Must happen before 10448 following 10449 buffer_inv. 10450 - Ensures the 10451 atomicrmw has 10452 completed before 10453 invalidating the 10454 cache. 10455 10456 5. buffer_inv sc1=1 10457 10458 - Must happen before 10459 any following 10460 global/generic 10461 load/load 10462 atomic/atomicrmw. 10463 - Ensures that 10464 following loads 10465 will not see stale 10466 global data. 10467 10468 atomicrmw acq_rel - system - generic 1. buffer_wbl2 sc0=1 sc1=1 10469 10470 - Must happen before 10471 following s_waitcnt. 10472 - Performs L2 writeback to 10473 ensure previous 10474 global/generic 10475 store/atomicrmw are 10476 visible at system scope. 10477 10478 2. s_waitcnt lgkmcnt(0) & 10479 vmcnt(0) 10480 10481 - If TgSplit execution mode, 10482 omit lgkmcnt(0). 10483 - If OpenCL, omit 10484 lgkmcnt(0). 10485 - Could be split into 10486 separate s_waitcnt 10487 vmcnt(0) and 10488 s_waitcnt 10489 lgkmcnt(0) to allow 10490 them to be 10491 independently moved 10492 according to the 10493 following rules. 10494 - s_waitcnt vmcnt(0) 10495 must happen after 10496 any preceding 10497 global/generic 10498 load/store/load 10499 atomic/store 10500 atomic/atomicrmw. 10501 - s_waitcnt lgkmcnt(0) 10502 must happen after 10503 any preceding 10504 local/generic 10505 load/store/load 10506 atomic/store 10507 atomic/atomicrmw. 10508 - Must happen before 10509 the following 10510 atomicrmw. 10511 - Ensures that all 10512 memory operations 10513 to global and L2 writeback 10514 have completed before 10515 performing the 10516 atomicrmw that is 10517 being released. 10518 10519 3. flat_atomic sc1=1 10520 4. s_waitcnt vmcnt(0) & 10521 lgkmcnt(0) 10522 10523 - If TgSplit execution mode, 10524 omit lgkmcnt(0). 10525 - If OpenCL, omit 10526 lgkmcnt(0). 10527 - Must happen before 10528 following 10529 buffer_inv. 10530 - Ensures the 10531 atomicrmw has 10532 completed before 10533 invalidating the 10534 caches. 10535 10536 5. buffer_inv sc0=1 sc1=1 10537 10538 - Must happen before 10539 any following 10540 global/generic 10541 load/load 10542 atomic/atomicrmw. 10543 - Ensures that 10544 following loads 10545 will not see stale 10546 MTYPE NC global data. 10547 MTYPE RW and CC memory will 10548 never be stale due to the 10549 memory probes. 10550 10551 fence acq_rel - singlethread *none* *none* 10552 - wavefront 10553 fence acq_rel - workgroup *none* 1. s_waitcnt lgkm/vmcnt(0) 10554 10555 - Use lgkmcnt(0) if not 10556 TgSplit execution mode 10557 and vmcnt(0) if TgSplit 10558 execution mode. 10559 - If OpenCL and 10560 address space is 10561 not generic, omit 10562 lgkmcnt(0). 10563 - If OpenCL and 10564 address space is 10565 local, omit 10566 vmcnt(0). 10567 - However, 10568 since LLVM 10569 currently has no 10570 address space on 10571 the fence need to 10572 conservatively 10573 always generate 10574 (see comment for 10575 previous fence). 10576 - s_waitcnt vmcnt(0) 10577 must happen after 10578 any preceding 10579 global/generic 10580 load/store/ 10581 load atomic/store atomic/ 10582 atomicrmw. 10583 - s_waitcnt lgkmcnt(0) 10584 must happen after 10585 any preceding 10586 local/generic 10587 load/load 10588 atomic/store/store 10589 atomic/atomicrmw. 10590 - Must happen before 10591 any following 10592 global/generic 10593 load/load 10594 atomic/store/store 10595 atomic/atomicrmw. 10596 - Ensures that all 10597 memory operations 10598 have 10599 completed before 10600 performing any 10601 following global 10602 memory operations. 10603 - Ensures that the 10604 preceding 10605 local/generic load 10606 atomic/atomicrmw 10607 with an equal or 10608 wider sync scope 10609 and memory ordering 10610 stronger than 10611 unordered (this is 10612 termed the 10613 acquire-fence-paired-atomic) 10614 has completed 10615 before following 10616 global memory 10617 operations. This 10618 satisfies the 10619 requirements of 10620 acquire. 10621 - Ensures that all 10622 previous memory 10623 operations have 10624 completed before a 10625 following 10626 local/generic store 10627 atomic/atomicrmw 10628 with an equal or 10629 wider sync scope 10630 and memory ordering 10631 stronger than 10632 unordered (this is 10633 termed the 10634 release-fence-paired-atomic). 10635 This satisfies the 10636 requirements of 10637 release. 10638 - Must happen before 10639 the following 10640 buffer_inv. 10641 - Ensures that the 10642 acquire-fence-paired 10643 atomic has completed 10644 before invalidating 10645 the 10646 cache. Therefore 10647 any following 10648 locations read must 10649 be no older than 10650 the value read by 10651 the 10652 acquire-fence-paired-atomic. 10653 10654 3. buffer_inv sc0=1 10655 10656 - If not TgSplit execution 10657 mode, omit. 10658 - Ensures that 10659 following 10660 loads will not see 10661 stale data. 10662 10663 fence acq_rel - agent *none* 1. buffer_wbl2 sc1=1 10664 10665 - If OpenCL and 10666 address space is 10667 local, omit. 10668 - Must happen before 10669 following s_waitcnt. 10670 - Performs L2 writeback to 10671 ensure previous 10672 global/generic 10673 store/atomicrmw are 10674 visible at agent scope. 10675 10676 2. s_waitcnt lgkmcnt(0) & 10677 vmcnt(0) 10678 10679 - If TgSplit execution mode, 10680 omit lgkmcnt(0). 10681 - If OpenCL and 10682 address space is 10683 not generic, omit 10684 lgkmcnt(0). 10685 - However, since LLVM 10686 currently has no 10687 address space on 10688 the fence need to 10689 conservatively 10690 always generate 10691 (see comment for 10692 previous fence). 10693 - Could be split into 10694 separate s_waitcnt 10695 vmcnt(0) and 10696 s_waitcnt 10697 lgkmcnt(0) to allow 10698 them to be 10699 independently moved 10700 according to the 10701 following rules. 10702 - s_waitcnt vmcnt(0) 10703 must happen after 10704 any preceding 10705 global/generic 10706 load/store/load 10707 atomic/store 10708 atomic/atomicrmw. 10709 - s_waitcnt lgkmcnt(0) 10710 must happen after 10711 any preceding 10712 local/generic 10713 load/store/load 10714 atomic/store 10715 atomic/atomicrmw. 10716 - Must happen before 10717 the following 10718 buffer_inv. 10719 - Ensures that the 10720 preceding 10721 global/local/generic 10722 load 10723 atomic/atomicrmw 10724 with an equal or 10725 wider sync scope 10726 and memory ordering 10727 stronger than 10728 unordered (this is 10729 termed the 10730 acquire-fence-paired-atomic) 10731 has completed 10732 before invalidating 10733 the cache. This 10734 satisfies the 10735 requirements of 10736 acquire. 10737 - Ensures that all 10738 previous memory 10739 operations have 10740 completed before a 10741 following 10742 global/local/generic 10743 store 10744 atomic/atomicrmw 10745 with an equal or 10746 wider sync scope 10747 and memory ordering 10748 stronger than 10749 unordered (this is 10750 termed the 10751 release-fence-paired-atomic). 10752 This satisfies the 10753 requirements of 10754 release. 10755 10756 3. buffer_inv sc1=1 10757 10758 - Must happen before 10759 any following 10760 global/generic 10761 load/load 10762 atomic/store/store 10763 atomic/atomicrmw. 10764 - Ensures that 10765 following loads 10766 will not see stale 10767 global data. This 10768 satisfies the 10769 requirements of 10770 acquire. 10771 10772 fence acq_rel - system *none* 1. buffer_wbl2 sc0=1 sc1=1 10773 10774 - If OpenCL and 10775 address space is 10776 local, omit. 10777 - Must happen before 10778 following s_waitcnt. 10779 - Performs L2 writeback to 10780 ensure previous 10781 global/generic 10782 store/atomicrmw are 10783 visible at system scope. 10784 10785 1. s_waitcnt lgkmcnt(0) & 10786 vmcnt(0) 10787 10788 - If TgSplit execution mode, 10789 omit lgkmcnt(0). 10790 - If OpenCL and 10791 address space is 10792 not generic, omit 10793 lgkmcnt(0). 10794 - However, since LLVM 10795 currently has no 10796 address space on 10797 the fence need to 10798 conservatively 10799 always generate 10800 (see comment for 10801 previous fence). 10802 - Could be split into 10803 separate s_waitcnt 10804 vmcnt(0) and 10805 s_waitcnt 10806 lgkmcnt(0) to allow 10807 them to be 10808 independently moved 10809 according to the 10810 following rules. 10811 - s_waitcnt vmcnt(0) 10812 must happen after 10813 any preceding 10814 global/generic 10815 load/store/load 10816 atomic/store 10817 atomic/atomicrmw. 10818 - s_waitcnt lgkmcnt(0) 10819 must happen after 10820 any preceding 10821 local/generic 10822 load/store/load 10823 atomic/store 10824 atomic/atomicrmw. 10825 - Must happen before 10826 the following 10827 buffer_inv. 10828 - Ensures that the 10829 preceding 10830 global/local/generic 10831 load 10832 atomic/atomicrmw 10833 with an equal or 10834 wider sync scope 10835 and memory ordering 10836 stronger than 10837 unordered (this is 10838 termed the 10839 acquire-fence-paired-atomic) 10840 has completed 10841 before invalidating 10842 the cache. This 10843 satisfies the 10844 requirements of 10845 acquire. 10846 - Ensures that all 10847 previous memory 10848 operations have 10849 completed before a 10850 following 10851 global/local/generic 10852 store 10853 atomic/atomicrmw 10854 with an equal or 10855 wider sync scope 10856 and memory ordering 10857 stronger than 10858 unordered (this is 10859 termed the 10860 release-fence-paired-atomic). 10861 This satisfies the 10862 requirements of 10863 release. 10864 10865 2. buffer_inv sc0=1 sc1=1 10866 10867 - Must happen before 10868 any following 10869 global/generic 10870 load/load 10871 atomic/store/store 10872 atomic/atomicrmw. 10873 - Ensures that 10874 following loads 10875 will not see stale 10876 MTYPE NC global data. 10877 MTYPE RW and CC memory will 10878 never be stale due to the 10879 memory probes. 10880 10881 **Sequential Consistent Atomic** 10882 ------------------------------------------------------------------------------------ 10883 load atomic seq_cst - singlethread - global *Same as corresponding 10884 - wavefront - local load atomic acquire, 10885 - generic except must generate 10886 all instructions even 10887 for OpenCL.* 10888 load atomic seq_cst - workgroup - global 1. s_waitcnt lgkm/vmcnt(0) 10889 - generic 10890 - Use lgkmcnt(0) if not 10891 TgSplit execution mode 10892 and vmcnt(0) if TgSplit 10893 execution mode. 10894 - s_waitcnt lgkmcnt(0) must 10895 happen after 10896 preceding 10897 local/generic load 10898 atomic/store 10899 atomic/atomicrmw 10900 with memory 10901 ordering of seq_cst 10902 and with equal or 10903 wider sync scope. 10904 (Note that seq_cst 10905 fences have their 10906 own s_waitcnt 10907 lgkmcnt(0) and so do 10908 not need to be 10909 considered.) 10910 - s_waitcnt vmcnt(0) 10911 must happen after 10912 preceding 10913 global/generic load 10914 atomic/store 10915 atomic/atomicrmw 10916 with memory 10917 ordering of seq_cst 10918 and with equal or 10919 wider sync scope. 10920 (Note that seq_cst 10921 fences have their 10922 own s_waitcnt 10923 vmcnt(0) and so do 10924 not need to be 10925 considered.) 10926 - Ensures any 10927 preceding 10928 sequential 10929 consistent global/local 10930 memory instructions 10931 have completed 10932 before executing 10933 this sequentially 10934 consistent 10935 instruction. This 10936 prevents reordering 10937 a seq_cst store 10938 followed by a 10939 seq_cst load. (Note 10940 that seq_cst is 10941 stronger than 10942 acquire/release as 10943 the reordering of 10944 load acquire 10945 followed by a store 10946 release is 10947 prevented by the 10948 s_waitcnt of 10949 the release, but 10950 there is nothing 10951 preventing a store 10952 release followed by 10953 load acquire from 10954 completing out of 10955 order. The s_waitcnt 10956 could be placed after 10957 seq_store or before 10958 the seq_load. We 10959 choose the load to 10960 make the s_waitcnt be 10961 as late as possible 10962 so that the store 10963 may have already 10964 completed.) 10965 10966 2. *Following 10967 instructions same as 10968 corresponding load 10969 atomic acquire, 10970 except must generate 10971 all instructions even 10972 for OpenCL.* 10973 load atomic seq_cst - workgroup - local *If TgSplit execution mode, 10974 local address space cannot 10975 be used.* 10976 10977 *Same as corresponding 10978 load atomic acquire, 10979 except must generate 10980 all instructions even 10981 for OpenCL.* 10982 10983 load atomic seq_cst - agent - global 1. s_waitcnt lgkmcnt(0) & 10984 - system - generic vmcnt(0) 10985 10986 - If TgSplit execution mode, 10987 omit lgkmcnt(0). 10988 - Could be split into 10989 separate s_waitcnt 10990 vmcnt(0) 10991 and s_waitcnt 10992 lgkmcnt(0) to allow 10993 them to be 10994 independently moved 10995 according to the 10996 following rules. 10997 - s_waitcnt lgkmcnt(0) 10998 must happen after 10999 preceding 11000 global/generic load 11001 atomic/store 11002 atomic/atomicrmw 11003 with memory 11004 ordering of seq_cst 11005 and with equal or 11006 wider sync scope. 11007 (Note that seq_cst 11008 fences have their 11009 own s_waitcnt 11010 lgkmcnt(0) and so do 11011 not need to be 11012 considered.) 11013 - s_waitcnt vmcnt(0) 11014 must happen after 11015 preceding 11016 global/generic load 11017 atomic/store 11018 atomic/atomicrmw 11019 with memory 11020 ordering of seq_cst 11021 and with equal or 11022 wider sync scope. 11023 (Note that seq_cst 11024 fences have their 11025 own s_waitcnt 11026 vmcnt(0) and so do 11027 not need to be 11028 considered.) 11029 - Ensures any 11030 preceding 11031 sequential 11032 consistent global 11033 memory instructions 11034 have completed 11035 before executing 11036 this sequentially 11037 consistent 11038 instruction. This 11039 prevents reordering 11040 a seq_cst store 11041 followed by a 11042 seq_cst load. (Note 11043 that seq_cst is 11044 stronger than 11045 acquire/release as 11046 the reordering of 11047 load acquire 11048 followed by a store 11049 release is 11050 prevented by the 11051 s_waitcnt of 11052 the release, but 11053 there is nothing 11054 preventing a store 11055 release followed by 11056 load acquire from 11057 completing out of 11058 order. The s_waitcnt 11059 could be placed after 11060 seq_store or before 11061 the seq_load. We 11062 choose the load to 11063 make the s_waitcnt be 11064 as late as possible 11065 so that the store 11066 may have already 11067 completed.) 11068 11069 2. *Following 11070 instructions same as 11071 corresponding load 11072 atomic acquire, 11073 except must generate 11074 all instructions even 11075 for OpenCL.* 11076 store atomic seq_cst - singlethread - global *Same as corresponding 11077 - wavefront - local store atomic release, 11078 - workgroup - generic except must generate 11079 - agent all instructions even 11080 - system for OpenCL.* 11081 atomicrmw seq_cst - singlethread - global *Same as corresponding 11082 - wavefront - local atomicrmw acq_rel, 11083 - workgroup - generic except must generate 11084 - agent all instructions even 11085 - system for OpenCL.* 11086 fence seq_cst - singlethread *none* *Same as corresponding 11087 - wavefront fence acq_rel, 11088 - workgroup except must generate 11089 - agent all instructions even 11090 - system for OpenCL.* 11091 ============ ============ ============== ========== ================================ 11092 11093.. _amdgpu-amdhsa-memory-model-gfx10: 11094 11095Memory Model GFX10 11096++++++++++++++++++ 11097 11098For GFX10: 11099 11100* Each agent has multiple shader arrays (SA). 11101* Each SA has multiple work-group processors (WGP). 11102* Each WGP has multiple compute units (CU). 11103* Each CU has multiple SIMDs that execute wavefronts. 11104* The wavefronts for a single work-group are executed in the same 11105 WGP. In CU wavefront execution mode the wavefronts may be executed by 11106 different SIMDs in the same CU. In WGP wavefront execution mode the 11107 wavefronts may be executed by different SIMDs in different CUs in the same 11108 WGP. 11109* Each WGP has a single LDS memory shared by the wavefronts of the work-groups 11110 executing on it. 11111* All LDS operations of a WGP are performed as wavefront wide operations in a 11112 global order and involve no caching. Completion is reported to a wavefront in 11113 execution order. 11114* The LDS memory has multiple request queues shared by the SIMDs of a 11115 WGP. Therefore, the LDS operations performed by different wavefronts of a 11116 work-group can be reordered relative to each other, which can result in 11117 reordering the visibility of vector memory operations with respect to LDS 11118 operations of other wavefronts in the same work-group. A ``s_waitcnt 11119 lgkmcnt(0)`` is required to ensure synchronization between LDS operations and 11120 vector memory operations between wavefronts of a work-group, but not between 11121 operations performed by the same wavefront. 11122* The vector memory operations are performed as wavefront wide operations. 11123 Completion of load/store/sample operations are reported to a wavefront in 11124 execution order of other load/store/sample operations performed by that 11125 wavefront. 11126* The vector memory operations access a vector L0 cache. There is a single L0 11127 cache per CU. Each SIMD of a CU accesses the same L0 cache. Therefore, no 11128 special action is required for coherence between the lanes of a single 11129 wavefront. However, a ``buffer_gl0_inv`` is required for coherence between 11130 wavefronts executing in the same work-group as they may be executing on SIMDs 11131 of different CUs that access different L0s. A ``buffer_gl0_inv`` is also 11132 required for coherence between wavefronts executing in different work-groups 11133 as they may be executing on different WGPs. 11134* The scalar memory operations access a scalar L0 cache shared by all wavefronts 11135 on a WGP. The scalar and vector L0 caches are not coherent. However, scalar 11136 operations are used in a restricted way so do not impact the memory model. See 11137 :ref:`amdgpu-amdhsa-memory-spaces`. 11138* The vector and scalar memory L0 caches use an L1 cache shared by all WGPs on 11139 the same SA. Therefore, no special action is required for coherence between 11140 the wavefronts of a single work-group. However, a ``buffer_gl1_inv`` is 11141 required for coherence between wavefronts executing in different work-groups 11142 as they may be executing on different SAs that access different L1s. 11143* The L1 caches have independent quadrants to service disjoint ranges of virtual 11144 addresses. 11145* Each L0 cache has a separate request queue per L1 quadrant. Therefore, the 11146 vector and scalar memory operations performed by different wavefronts, whether 11147 executing in the same or different work-groups (which may be executing on 11148 different CUs accessing different L0s), can be reordered relative to each 11149 other. A ``s_waitcnt vmcnt(0) & vscnt(0)`` is required to ensure 11150 synchronization between vector memory operations of different wavefronts. It 11151 ensures a previous vector memory operation has completed before executing a 11152 subsequent vector memory or LDS operation and so can be used to meet the 11153 requirements of acquire, release and sequential consistency. 11154* The L1 caches use an L2 cache shared by all SAs on the same agent. 11155* The L2 cache has independent channels to service disjoint ranges of virtual 11156 addresses. 11157* Each L1 quadrant of a single SA accesses a different L2 channel. Each L1 11158 quadrant has a separate request queue per L2 channel. Therefore, the vector 11159 and scalar memory operations performed by wavefronts executing in different 11160 work-groups (which may be executing on different SAs) of an agent can be 11161 reordered relative to each other. A ``s_waitcnt vmcnt(0) & vscnt(0)`` is 11162 required to ensure synchronization between vector memory operations of 11163 different SAs. It ensures a previous vector memory operation has completed 11164 before executing a subsequent vector memory and so can be used to meet the 11165 requirements of acquire, release and sequential consistency. 11166* The L2 cache can be kept coherent with other agents on some targets, or ranges 11167 of virtual addresses can be set up to bypass it to ensure system coherence. 11168* On GFX10.3 a memory attached last level (MALL) cache exists for GPU memory. 11169 The MALL cache is fully coherent with GPU memory and has no impact on system 11170 coherence. All agents (GPU and CPU) access GPU memory through the MALL cache. 11171 11172Scalar memory operations are only used to access memory that is proven to not 11173change during the execution of the kernel dispatch. This includes constant 11174address space and global address space for program scope ``const`` variables. 11175Therefore, the kernel machine code does not have to maintain the scalar cache to 11176ensure it is coherent with the vector caches. The scalar and vector caches are 11177invalidated between kernel dispatches by CP since constant address space data 11178may change between kernel dispatch executions. See 11179:ref:`amdgpu-amdhsa-memory-spaces`. 11180 11181The one exception is if scalar writes are used to spill SGPR registers. In this 11182case the AMDGPU backend ensures the memory location used to spill is never 11183accessed by vector memory operations at the same time. If scalar writes are used 11184then a ``s_dcache_wb`` is inserted before the ``s_endpgm`` and before a function 11185return since the locations may be used for vector memory instructions by a 11186future wavefront that uses the same scratch area, or a function call that 11187creates a frame at the same address, respectively. There is no need for a 11188``s_dcache_inv`` as all scalar writes are write-before-read in the same thread. 11189 11190For kernarg backing memory: 11191 11192* CP invalidates the L0 and L1 caches at the start of each kernel dispatch. 11193* On dGPU the kernarg backing memory is accessed as MTYPE UC (uncached) to avoid 11194 needing to invalidate the L2 cache. 11195* On APU the kernarg backing memory is accessed as MTYPE CC (cache coherent) and 11196 so the L2 cache will be coherent with the CPU and other agents. 11197 11198Scratch backing memory (which is used for the private address space) is accessed 11199with MTYPE NC (non-coherent). Since the private address space is only accessed 11200by a single thread, and is always write-before-read, there is never a need to 11201invalidate these entries from the L0 or L1 caches. 11202 11203Wavefronts are executed in native mode with in-order reporting of loads and 11204sample instructions. In this mode vmcnt reports completion of load, atomic with 11205return and sample instructions in order, and the vscnt reports the completion of 11206store and atomic without return in order. See ``MEM_ORDERED`` field in 11207:ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 11208 11209Wavefronts can be executed in WGP or CU wavefront execution mode: 11210 11211* In WGP wavefront execution mode the wavefronts of a work-group are executed 11212 on the SIMDs of both CUs of the WGP. Therefore, explicit management of the per 11213 CU L0 caches is required for work-group synchronization. Also accesses to L1 11214 at work-group scope need to be explicitly ordered as the accesses from 11215 different CUs are not ordered. 11216* In CU wavefront execution mode the wavefronts of a work-group are executed on 11217 the SIMDs of a single CU of the WGP. Therefore, all global memory access by 11218 the work-group access the same L0 which in turn ensures L1 accesses are 11219 ordered and so do not require explicit management of the caches for 11220 work-group synchronization. 11221 11222See ``WGP_MODE`` field in 11223:ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table` and 11224:ref:`amdgpu-target-features`. 11225 11226The code sequences used to implement the memory model for GFX10 are defined in 11227table :ref:`amdgpu-amdhsa-memory-model-code-sequences-gfx10-table`. 11228 11229 .. table:: AMDHSA Memory Model Code Sequences GFX10 11230 :name: amdgpu-amdhsa-memory-model-code-sequences-gfx10-table 11231 11232 ============ ============ ============== ========== ================================ 11233 LLVM Instr LLVM Memory LLVM Memory AMDGPU AMDGPU Machine Code 11234 Ordering Sync Scope Address GFX10 11235 Space 11236 ============ ============ ============== ========== ================================ 11237 **Non-Atomic** 11238 ------------------------------------------------------------------------------------ 11239 load *none* *none* - global - !volatile & !nontemporal 11240 - generic 11241 - private 1. buffer/global/flat_load 11242 - constant 11243 - !volatile & nontemporal 11244 11245 1. buffer/global/flat_load 11246 slc=1 11247 11248 - volatile 11249 11250 1. buffer/global/flat_load 11251 glc=1 dlc=1 11252 2. s_waitcnt vmcnt(0) 11253 11254 - Must happen before 11255 any following volatile 11256 global/generic 11257 load/store. 11258 - Ensures that 11259 volatile 11260 operations to 11261 different 11262 addresses will not 11263 be reordered by 11264 hardware. 11265 11266 load *none* *none* - local 1. ds_load 11267 store *none* *none* - global - !volatile & !nontemporal 11268 - generic 11269 - private 1. buffer/global/flat_store 11270 - constant 11271 - !volatile & nontemporal 11272 11273 1. buffer/global/flat_store 11274 glc=1 slc=1 11275 11276 - volatile 11277 11278 1. buffer/global/flat_store 11279 2. s_waitcnt vscnt(0) 11280 11281 - Must happen before 11282 any following volatile 11283 global/generic 11284 load/store. 11285 - Ensures that 11286 volatile 11287 operations to 11288 different 11289 addresses will not 11290 be reordered by 11291 hardware. 11292 11293 store *none* *none* - local 1. ds_store 11294 **Unordered Atomic** 11295 ------------------------------------------------------------------------------------ 11296 load atomic unordered *any* *any* *Same as non-atomic*. 11297 store atomic unordered *any* *any* *Same as non-atomic*. 11298 atomicrmw unordered *any* *any* *Same as monotonic atomic*. 11299 **Monotonic Atomic** 11300 ------------------------------------------------------------------------------------ 11301 load atomic monotonic - singlethread - global 1. buffer/global/flat_load 11302 - wavefront - generic 11303 load atomic monotonic - workgroup - global 1. buffer/global/flat_load 11304 - generic glc=1 11305 11306 - If CU wavefront execution 11307 mode, omit glc=1. 11308 11309 load atomic monotonic - singlethread - local 1. ds_load 11310 - wavefront 11311 - workgroup 11312 load atomic monotonic - agent - global 1. buffer/global/flat_load 11313 - system - generic glc=1 dlc=1 11314 store atomic monotonic - singlethread - global 1. buffer/global/flat_store 11315 - wavefront - generic 11316 - workgroup 11317 - agent 11318 - system 11319 store atomic monotonic - singlethread - local 1. ds_store 11320 - wavefront 11321 - workgroup 11322 atomicrmw monotonic - singlethread - global 1. buffer/global/flat_atomic 11323 - wavefront - generic 11324 - workgroup 11325 - agent 11326 - system 11327 atomicrmw monotonic - singlethread - local 1. ds_atomic 11328 - wavefront 11329 - workgroup 11330 **Acquire Atomic** 11331 ------------------------------------------------------------------------------------ 11332 load atomic acquire - singlethread - global 1. buffer/global/ds/flat_load 11333 - wavefront - local 11334 - generic 11335 load atomic acquire - workgroup - global 1. buffer/global_load glc=1 11336 11337 - If CU wavefront execution 11338 mode, omit glc=1. 11339 11340 2. s_waitcnt vmcnt(0) 11341 11342 - If CU wavefront execution 11343 mode, omit. 11344 - Must happen before 11345 the following buffer_gl0_inv 11346 and before any following 11347 global/generic 11348 load/load 11349 atomic/store/store 11350 atomic/atomicrmw. 11351 11352 3. buffer_gl0_inv 11353 11354 - If CU wavefront execution 11355 mode, omit. 11356 - Ensures that 11357 following 11358 loads will not see 11359 stale data. 11360 11361 load atomic acquire - workgroup - local 1. ds_load 11362 2. s_waitcnt lgkmcnt(0) 11363 11364 - If OpenCL, omit. 11365 - Must happen before 11366 the following buffer_gl0_inv 11367 and before any following 11368 global/generic load/load 11369 atomic/store/store 11370 atomic/atomicrmw. 11371 - Ensures any 11372 following global 11373 data read is no 11374 older than the local load 11375 atomic value being 11376 acquired. 11377 11378 3. buffer_gl0_inv 11379 11380 - If CU wavefront execution 11381 mode, omit. 11382 - If OpenCL, omit. 11383 - Ensures that 11384 following 11385 loads will not see 11386 stale data. 11387 11388 load atomic acquire - workgroup - generic 1. flat_load glc=1 11389 11390 - If CU wavefront execution 11391 mode, omit glc=1. 11392 11393 2. s_waitcnt lgkmcnt(0) & 11394 vmcnt(0) 11395 11396 - If CU wavefront execution 11397 mode, omit vmcnt(0). 11398 - If OpenCL, omit 11399 lgkmcnt(0). 11400 - Must happen before 11401 the following 11402 buffer_gl0_inv and any 11403 following global/generic 11404 load/load 11405 atomic/store/store 11406 atomic/atomicrmw. 11407 - Ensures any 11408 following global 11409 data read is no 11410 older than a local load 11411 atomic value being 11412 acquired. 11413 11414 3. buffer_gl0_inv 11415 11416 - If CU wavefront execution 11417 mode, omit. 11418 - Ensures that 11419 following 11420 loads will not see 11421 stale data. 11422 11423 load atomic acquire - agent - global 1. buffer/global_load 11424 - system glc=1 dlc=1 11425 2. s_waitcnt vmcnt(0) 11426 11427 - Must happen before 11428 following 11429 buffer_gl*_inv. 11430 - Ensures the load 11431 has completed 11432 before invalidating 11433 the caches. 11434 11435 3. buffer_gl0_inv; 11436 buffer_gl1_inv 11437 11438 - Must happen before 11439 any following 11440 global/generic 11441 load/load 11442 atomic/atomicrmw. 11443 - Ensures that 11444 following 11445 loads will not see 11446 stale global data. 11447 11448 load atomic acquire - agent - generic 1. flat_load glc=1 dlc=1 11449 - system 2. s_waitcnt vmcnt(0) & 11450 lgkmcnt(0) 11451 11452 - If OpenCL omit 11453 lgkmcnt(0). 11454 - Must happen before 11455 following 11456 buffer_gl*_invl. 11457 - Ensures the flat_load 11458 has completed 11459 before invalidating 11460 the caches. 11461 11462 3. buffer_gl0_inv; 11463 buffer_gl1_inv 11464 11465 - Must happen before 11466 any following 11467 global/generic 11468 load/load 11469 atomic/atomicrmw. 11470 - Ensures that 11471 following loads 11472 will not see stale 11473 global data. 11474 11475 atomicrmw acquire - singlethread - global 1. buffer/global/ds/flat_atomic 11476 - wavefront - local 11477 - generic 11478 atomicrmw acquire - workgroup - global 1. buffer/global_atomic 11479 2. s_waitcnt vm/vscnt(0) 11480 11481 - If CU wavefront execution 11482 mode, omit. 11483 - Use vmcnt(0) if atomic with 11484 return and vscnt(0) if 11485 atomic with no-return. 11486 - Must happen before 11487 the following buffer_gl0_inv 11488 and before any following 11489 global/generic 11490 load/load 11491 atomic/store/store 11492 atomic/atomicrmw. 11493 11494 3. buffer_gl0_inv 11495 11496 - If CU wavefront execution 11497 mode, omit. 11498 - Ensures that 11499 following 11500 loads will not see 11501 stale data. 11502 11503 atomicrmw acquire - workgroup - local 1. ds_atomic 11504 2. s_waitcnt lgkmcnt(0) 11505 11506 - If OpenCL, omit. 11507 - Must happen before 11508 the following 11509 buffer_gl0_inv. 11510 - Ensures any 11511 following global 11512 data read is no 11513 older than the local 11514 atomicrmw value 11515 being acquired. 11516 11517 3. buffer_gl0_inv 11518 11519 - If OpenCL omit. 11520 - Ensures that 11521 following 11522 loads will not see 11523 stale data. 11524 11525 atomicrmw acquire - workgroup - generic 1. flat_atomic 11526 2. s_waitcnt lgkmcnt(0) & 11527 vm/vscnt(0) 11528 11529 - If CU wavefront execution 11530 mode, omit vm/vscnt(0). 11531 - If OpenCL, omit lgkmcnt(0). 11532 - Use vmcnt(0) if atomic with 11533 return and vscnt(0) if 11534 atomic with no-return. 11535 - Must happen before 11536 the following 11537 buffer_gl0_inv. 11538 - Ensures any 11539 following global 11540 data read is no 11541 older than a local 11542 atomicrmw value 11543 being acquired. 11544 11545 3. buffer_gl0_inv 11546 11547 - If CU wavefront execution 11548 mode, omit. 11549 - Ensures that 11550 following 11551 loads will not see 11552 stale data. 11553 11554 atomicrmw acquire - agent - global 1. buffer/global_atomic 11555 - system 2. s_waitcnt vm/vscnt(0) 11556 11557 - Use vmcnt(0) if atomic with 11558 return and vscnt(0) if 11559 atomic with no-return. 11560 - Must happen before 11561 following 11562 buffer_gl*_inv. 11563 - Ensures the 11564 atomicrmw has 11565 completed before 11566 invalidating the 11567 caches. 11568 11569 3. buffer_gl0_inv; 11570 buffer_gl1_inv 11571 11572 - Must happen before 11573 any following 11574 global/generic 11575 load/load 11576 atomic/atomicrmw. 11577 - Ensures that 11578 following loads 11579 will not see stale 11580 global data. 11581 11582 atomicrmw acquire - agent - generic 1. flat_atomic 11583 - system 2. s_waitcnt vm/vscnt(0) & 11584 lgkmcnt(0) 11585 11586 - If OpenCL, omit 11587 lgkmcnt(0). 11588 - Use vmcnt(0) if atomic with 11589 return and vscnt(0) if 11590 atomic with no-return. 11591 - Must happen before 11592 following 11593 buffer_gl*_inv. 11594 - Ensures the 11595 atomicrmw has 11596 completed before 11597 invalidating the 11598 caches. 11599 11600 3. buffer_gl0_inv; 11601 buffer_gl1_inv 11602 11603 - Must happen before 11604 any following 11605 global/generic 11606 load/load 11607 atomic/atomicrmw. 11608 - Ensures that 11609 following loads 11610 will not see stale 11611 global data. 11612 11613 fence acquire - singlethread *none* *none* 11614 - wavefront 11615 fence acquire - workgroup *none* 1. s_waitcnt lgkmcnt(0) & 11616 vmcnt(0) & vscnt(0) 11617 11618 - If CU wavefront execution 11619 mode, omit vmcnt(0) and 11620 vscnt(0). 11621 - If OpenCL and 11622 address space is 11623 not generic, omit 11624 lgkmcnt(0). 11625 - If OpenCL and 11626 address space is 11627 local, omit 11628 vmcnt(0) and vscnt(0). 11629 - However, since LLVM 11630 currently has no 11631 address space on 11632 the fence need to 11633 conservatively 11634 always generate. If 11635 fence had an 11636 address space then 11637 set to address 11638 space of OpenCL 11639 fence flag, or to 11640 generic if both 11641 local and global 11642 flags are 11643 specified. 11644 - Could be split into 11645 separate s_waitcnt 11646 vmcnt(0), s_waitcnt 11647 vscnt(0) and s_waitcnt 11648 lgkmcnt(0) to allow 11649 them to be 11650 independently moved 11651 according to the 11652 following rules. 11653 - s_waitcnt vmcnt(0) 11654 must happen after 11655 any preceding 11656 global/generic load 11657 atomic/ 11658 atomicrmw-with-return-value 11659 with an equal or 11660 wider sync scope 11661 and memory ordering 11662 stronger than 11663 unordered (this is 11664 termed the 11665 fence-paired-atomic). 11666 - s_waitcnt vscnt(0) 11667 must happen after 11668 any preceding 11669 global/generic 11670 atomicrmw-no-return-value 11671 with an equal or 11672 wider sync scope 11673 and memory ordering 11674 stronger than 11675 unordered (this is 11676 termed the 11677 fence-paired-atomic). 11678 - s_waitcnt lgkmcnt(0) 11679 must happen after 11680 any preceding 11681 local/generic load 11682 atomic/atomicrmw 11683 with an equal or 11684 wider sync scope 11685 and memory ordering 11686 stronger than 11687 unordered (this is 11688 termed the 11689 fence-paired-atomic). 11690 - Must happen before 11691 the following 11692 buffer_gl0_inv. 11693 - Ensures that the 11694 fence-paired atomic 11695 has completed 11696 before invalidating 11697 the 11698 cache. Therefore 11699 any following 11700 locations read must 11701 be no older than 11702 the value read by 11703 the 11704 fence-paired-atomic. 11705 11706 3. buffer_gl0_inv 11707 11708 - If CU wavefront execution 11709 mode, omit. 11710 - Ensures that 11711 following 11712 loads will not see 11713 stale data. 11714 11715 fence acquire - agent *none* 1. s_waitcnt lgkmcnt(0) & 11716 - system vmcnt(0) & vscnt(0) 11717 11718 - If OpenCL and 11719 address space is 11720 not generic, omit 11721 lgkmcnt(0). 11722 - If OpenCL and 11723 address space is 11724 local, omit 11725 vmcnt(0) and vscnt(0). 11726 - However, since LLVM 11727 currently has no 11728 address space on 11729 the fence need to 11730 conservatively 11731 always generate 11732 (see comment for 11733 previous fence). 11734 - Could be split into 11735 separate s_waitcnt 11736 vmcnt(0), s_waitcnt 11737 vscnt(0) and s_waitcnt 11738 lgkmcnt(0) to allow 11739 them to be 11740 independently moved 11741 according to the 11742 following rules. 11743 - s_waitcnt vmcnt(0) 11744 must happen after 11745 any preceding 11746 global/generic load 11747 atomic/ 11748 atomicrmw-with-return-value 11749 with an equal or 11750 wider sync scope 11751 and memory ordering 11752 stronger than 11753 unordered (this is 11754 termed the 11755 fence-paired-atomic). 11756 - s_waitcnt vscnt(0) 11757 must happen after 11758 any preceding 11759 global/generic 11760 atomicrmw-no-return-value 11761 with an equal or 11762 wider sync scope 11763 and memory ordering 11764 stronger than 11765 unordered (this is 11766 termed the 11767 fence-paired-atomic). 11768 - s_waitcnt lgkmcnt(0) 11769 must happen after 11770 any preceding 11771 local/generic load 11772 atomic/atomicrmw 11773 with an equal or 11774 wider sync scope 11775 and memory ordering 11776 stronger than 11777 unordered (this is 11778 termed the 11779 fence-paired-atomic). 11780 - Must happen before 11781 the following 11782 buffer_gl*_inv. 11783 - Ensures that the 11784 fence-paired atomic 11785 has completed 11786 before invalidating 11787 the 11788 caches. Therefore 11789 any following 11790 locations read must 11791 be no older than 11792 the value read by 11793 the 11794 fence-paired-atomic. 11795 11796 2. buffer_gl0_inv; 11797 buffer_gl1_inv 11798 11799 - Must happen before any 11800 following global/generic 11801 load/load 11802 atomic/store/store 11803 atomic/atomicrmw. 11804 - Ensures that 11805 following loads 11806 will not see stale 11807 global data. 11808 11809 **Release Atomic** 11810 ------------------------------------------------------------------------------------ 11811 store atomic release - singlethread - global 1. buffer/global/ds/flat_store 11812 - wavefront - local 11813 - generic 11814 store atomic release - workgroup - global 1. s_waitcnt lgkmcnt(0) & 11815 - generic vmcnt(0) & vscnt(0) 11816 11817 - If CU wavefront execution 11818 mode, omit vmcnt(0) and 11819 vscnt(0). 11820 - If OpenCL, omit 11821 lgkmcnt(0). 11822 - Could be split into 11823 separate s_waitcnt 11824 vmcnt(0), s_waitcnt 11825 vscnt(0) and s_waitcnt 11826 lgkmcnt(0) to allow 11827 them to be 11828 independently moved 11829 according to the 11830 following rules. 11831 - s_waitcnt vmcnt(0) 11832 must happen after 11833 any preceding 11834 global/generic load/load 11835 atomic/ 11836 atomicrmw-with-return-value. 11837 - s_waitcnt vscnt(0) 11838 must happen after 11839 any preceding 11840 global/generic 11841 store/store 11842 atomic/ 11843 atomicrmw-no-return-value. 11844 - s_waitcnt lgkmcnt(0) 11845 must happen after 11846 any preceding 11847 local/generic 11848 load/store/load 11849 atomic/store 11850 atomic/atomicrmw. 11851 - Must happen before 11852 the following 11853 store. 11854 - Ensures that all 11855 memory operations 11856 have 11857 completed before 11858 performing the 11859 store that is being 11860 released. 11861 11862 2. buffer/global/flat_store 11863 store atomic release - workgroup - local 1. s_waitcnt vmcnt(0) & vscnt(0) 11864 11865 - If CU wavefront execution 11866 mode, omit. 11867 - If OpenCL, omit. 11868 - Could be split into 11869 separate s_waitcnt 11870 vmcnt(0) and s_waitcnt 11871 vscnt(0) to allow 11872 them to be 11873 independently moved 11874 according to the 11875 following rules. 11876 - s_waitcnt vmcnt(0) 11877 must happen after 11878 any preceding 11879 global/generic load/load 11880 atomic/ 11881 atomicrmw-with-return-value. 11882 - s_waitcnt vscnt(0) 11883 must happen after 11884 any preceding 11885 global/generic 11886 store/store atomic/ 11887 atomicrmw-no-return-value. 11888 - Must happen before 11889 the following 11890 store. 11891 - Ensures that all 11892 global memory 11893 operations have 11894 completed before 11895 performing the 11896 store that is being 11897 released. 11898 11899 2. ds_store 11900 store atomic release - agent - global 1. s_waitcnt lgkmcnt(0) & 11901 - system - generic vmcnt(0) & vscnt(0) 11902 11903 - If OpenCL and 11904 address space is 11905 not generic, omit 11906 lgkmcnt(0). 11907 - Could be split into 11908 separate s_waitcnt 11909 vmcnt(0), s_waitcnt vscnt(0) 11910 and s_waitcnt 11911 lgkmcnt(0) to allow 11912 them to be 11913 independently moved 11914 according to the 11915 following rules. 11916 - s_waitcnt vmcnt(0) 11917 must happen after 11918 any preceding 11919 global/generic 11920 load/load 11921 atomic/ 11922 atomicrmw-with-return-value. 11923 - s_waitcnt vscnt(0) 11924 must happen after 11925 any preceding 11926 global/generic 11927 store/store atomic/ 11928 atomicrmw-no-return-value. 11929 - s_waitcnt lgkmcnt(0) 11930 must happen after 11931 any preceding 11932 local/generic 11933 load/store/load 11934 atomic/store 11935 atomic/atomicrmw. 11936 - Must happen before 11937 the following 11938 store. 11939 - Ensures that all 11940 memory operations 11941 have 11942 completed before 11943 performing the 11944 store that is being 11945 released. 11946 11947 2. buffer/global/flat_store 11948 atomicrmw release - singlethread - global 1. buffer/global/ds/flat_atomic 11949 - wavefront - local 11950 - generic 11951 atomicrmw release - workgroup - global 1. s_waitcnt lgkmcnt(0) & 11952 - generic vmcnt(0) & vscnt(0) 11953 11954 - If CU wavefront execution 11955 mode, omit vmcnt(0) and 11956 vscnt(0). 11957 - If OpenCL, omit lgkmcnt(0). 11958 - Could be split into 11959 separate s_waitcnt 11960 vmcnt(0), s_waitcnt 11961 vscnt(0) and s_waitcnt 11962 lgkmcnt(0) to allow 11963 them to be 11964 independently moved 11965 according to the 11966 following rules. 11967 - s_waitcnt vmcnt(0) 11968 must happen after 11969 any preceding 11970 global/generic load/load 11971 atomic/ 11972 atomicrmw-with-return-value. 11973 - s_waitcnt vscnt(0) 11974 must happen after 11975 any preceding 11976 global/generic 11977 store/store 11978 atomic/ 11979 atomicrmw-no-return-value. 11980 - s_waitcnt lgkmcnt(0) 11981 must happen after 11982 any preceding 11983 local/generic 11984 load/store/load 11985 atomic/store 11986 atomic/atomicrmw. 11987 - Must happen before 11988 the following 11989 atomicrmw. 11990 - Ensures that all 11991 memory operations 11992 have 11993 completed before 11994 performing the 11995 atomicrmw that is 11996 being released. 11997 11998 2. buffer/global/flat_atomic 11999 atomicrmw release - workgroup - local 1. s_waitcnt vmcnt(0) & vscnt(0) 12000 12001 - If CU wavefront execution 12002 mode, omit. 12003 - If OpenCL, omit. 12004 - Could be split into 12005 separate s_waitcnt 12006 vmcnt(0) and s_waitcnt 12007 vscnt(0) to allow 12008 them to be 12009 independently moved 12010 according to the 12011 following rules. 12012 - s_waitcnt vmcnt(0) 12013 must happen after 12014 any preceding 12015 global/generic load/load 12016 atomic/ 12017 atomicrmw-with-return-value. 12018 - s_waitcnt vscnt(0) 12019 must happen after 12020 any preceding 12021 global/generic 12022 store/store atomic/ 12023 atomicrmw-no-return-value. 12024 - Must happen before 12025 the following 12026 store. 12027 - Ensures that all 12028 global memory 12029 operations have 12030 completed before 12031 performing the 12032 store that is being 12033 released. 12034 12035 2. ds_atomic 12036 atomicrmw release - agent - global 1. s_waitcnt lgkmcnt(0) & 12037 - system - generic vmcnt(0) & vscnt(0) 12038 12039 - If OpenCL, omit 12040 lgkmcnt(0). 12041 - Could be split into 12042 separate s_waitcnt 12043 vmcnt(0), s_waitcnt 12044 vscnt(0) and s_waitcnt 12045 lgkmcnt(0) to allow 12046 them to be 12047 independently moved 12048 according to the 12049 following rules. 12050 - s_waitcnt vmcnt(0) 12051 must happen after 12052 any preceding 12053 global/generic 12054 load/load atomic/ 12055 atomicrmw-with-return-value. 12056 - s_waitcnt vscnt(0) 12057 must happen after 12058 any preceding 12059 global/generic 12060 store/store atomic/ 12061 atomicrmw-no-return-value. 12062 - s_waitcnt lgkmcnt(0) 12063 must happen after 12064 any preceding 12065 local/generic 12066 load/store/load 12067 atomic/store 12068 atomic/atomicrmw. 12069 - Must happen before 12070 the following 12071 atomicrmw. 12072 - Ensures that all 12073 memory operations 12074 to global and local 12075 have completed 12076 before performing 12077 the atomicrmw that 12078 is being released. 12079 12080 2. buffer/global/flat_atomic 12081 fence release - singlethread *none* *none* 12082 - wavefront 12083 fence release - workgroup *none* 1. s_waitcnt lgkmcnt(0) & 12084 vmcnt(0) & vscnt(0) 12085 12086 - If CU wavefront execution 12087 mode, omit vmcnt(0) and 12088 vscnt(0). 12089 - If OpenCL and 12090 address space is 12091 not generic, omit 12092 lgkmcnt(0). 12093 - If OpenCL and 12094 address space is 12095 local, omit 12096 vmcnt(0) and vscnt(0). 12097 - However, since LLVM 12098 currently has no 12099 address space on 12100 the fence need to 12101 conservatively 12102 always generate. If 12103 fence had an 12104 address space then 12105 set to address 12106 space of OpenCL 12107 fence flag, or to 12108 generic if both 12109 local and global 12110 flags are 12111 specified. 12112 - Could be split into 12113 separate s_waitcnt 12114 vmcnt(0), s_waitcnt 12115 vscnt(0) and s_waitcnt 12116 lgkmcnt(0) to allow 12117 them to be 12118 independently moved 12119 according to the 12120 following rules. 12121 - s_waitcnt vmcnt(0) 12122 must happen after 12123 any preceding 12124 global/generic 12125 load/load 12126 atomic/ 12127 atomicrmw-with-return-value. 12128 - s_waitcnt vscnt(0) 12129 must happen after 12130 any preceding 12131 global/generic 12132 store/store atomic/ 12133 atomicrmw-no-return-value. 12134 - s_waitcnt lgkmcnt(0) 12135 must happen after 12136 any preceding 12137 local/generic 12138 load/store/load 12139 atomic/store atomic/ 12140 atomicrmw. 12141 - Must happen before 12142 any following store 12143 atomic/atomicrmw 12144 with an equal or 12145 wider sync scope 12146 and memory ordering 12147 stronger than 12148 unordered (this is 12149 termed the 12150 fence-paired-atomic). 12151 - Ensures that all 12152 memory operations 12153 have 12154 completed before 12155 performing the 12156 following 12157 fence-paired-atomic. 12158 12159 fence release - agent *none* 1. s_waitcnt lgkmcnt(0) & 12160 - system vmcnt(0) & vscnt(0) 12161 12162 - If OpenCL and 12163 address space is 12164 not generic, omit 12165 lgkmcnt(0). 12166 - If OpenCL and 12167 address space is 12168 local, omit 12169 vmcnt(0) and vscnt(0). 12170 - However, since LLVM 12171 currently has no 12172 address space on 12173 the fence need to 12174 conservatively 12175 always generate. If 12176 fence had an 12177 address space then 12178 set to address 12179 space of OpenCL 12180 fence flag, or to 12181 generic if both 12182 local and global 12183 flags are 12184 specified. 12185 - Could be split into 12186 separate s_waitcnt 12187 vmcnt(0), s_waitcnt 12188 vscnt(0) and s_waitcnt 12189 lgkmcnt(0) to allow 12190 them to be 12191 independently moved 12192 according to the 12193 following rules. 12194 - s_waitcnt vmcnt(0) 12195 must happen after 12196 any preceding 12197 global/generic 12198 load/load atomic/ 12199 atomicrmw-with-return-value. 12200 - s_waitcnt vscnt(0) 12201 must happen after 12202 any preceding 12203 global/generic 12204 store/store atomic/ 12205 atomicrmw-no-return-value. 12206 - s_waitcnt lgkmcnt(0) 12207 must happen after 12208 any preceding 12209 local/generic 12210 load/store/load 12211 atomic/store 12212 atomic/atomicrmw. 12213 - Must happen before 12214 any following store 12215 atomic/atomicrmw 12216 with an equal or 12217 wider sync scope 12218 and memory ordering 12219 stronger than 12220 unordered (this is 12221 termed the 12222 fence-paired-atomic). 12223 - Ensures that all 12224 memory operations 12225 have 12226 completed before 12227 performing the 12228 following 12229 fence-paired-atomic. 12230 12231 **Acquire-Release Atomic** 12232 ------------------------------------------------------------------------------------ 12233 atomicrmw acq_rel - singlethread - global 1. buffer/global/ds/flat_atomic 12234 - wavefront - local 12235 - generic 12236 atomicrmw acq_rel - workgroup - global 1. s_waitcnt lgkmcnt(0) & 12237 vmcnt(0) & vscnt(0) 12238 12239 - If CU wavefront execution 12240 mode, omit vmcnt(0) and 12241 vscnt(0). 12242 - If OpenCL, omit 12243 lgkmcnt(0). 12244 - Must happen after 12245 any preceding 12246 local/generic 12247 load/store/load 12248 atomic/store 12249 atomic/atomicrmw. 12250 - Could be split into 12251 separate s_waitcnt 12252 vmcnt(0), s_waitcnt 12253 vscnt(0), and s_waitcnt 12254 lgkmcnt(0) to allow 12255 them to be 12256 independently moved 12257 according to the 12258 following rules. 12259 - s_waitcnt vmcnt(0) 12260 must happen after 12261 any preceding 12262 global/generic load/load 12263 atomic/ 12264 atomicrmw-with-return-value. 12265 - s_waitcnt vscnt(0) 12266 must happen after 12267 any preceding 12268 global/generic 12269 store/store 12270 atomic/ 12271 atomicrmw-no-return-value. 12272 - s_waitcnt lgkmcnt(0) 12273 must happen after 12274 any preceding 12275 local/generic 12276 load/store/load 12277 atomic/store 12278 atomic/atomicrmw. 12279 - Must happen before 12280 the following 12281 atomicrmw. 12282 - Ensures that all 12283 memory operations 12284 have 12285 completed before 12286 performing the 12287 atomicrmw that is 12288 being released. 12289 12290 2. buffer/global_atomic 12291 3. s_waitcnt vm/vscnt(0) 12292 12293 - If CU wavefront execution 12294 mode, omit. 12295 - Use vmcnt(0) if atomic with 12296 return and vscnt(0) if 12297 atomic with no-return. 12298 - Must happen before 12299 the following 12300 buffer_gl0_inv. 12301 - Ensures any 12302 following global 12303 data read is no 12304 older than the 12305 atomicrmw value 12306 being acquired. 12307 12308 4. buffer_gl0_inv 12309 12310 - If CU wavefront execution 12311 mode, omit. 12312 - Ensures that 12313 following 12314 loads will not see 12315 stale data. 12316 12317 atomicrmw acq_rel - workgroup - local 1. s_waitcnt vmcnt(0) & vscnt(0) 12318 12319 - If CU wavefront execution 12320 mode, omit. 12321 - If OpenCL, omit. 12322 - Could be split into 12323 separate s_waitcnt 12324 vmcnt(0) and s_waitcnt 12325 vscnt(0) to allow 12326 them to be 12327 independently moved 12328 according to the 12329 following rules. 12330 - s_waitcnt vmcnt(0) 12331 must happen after 12332 any preceding 12333 global/generic load/load 12334 atomic/ 12335 atomicrmw-with-return-value. 12336 - s_waitcnt vscnt(0) 12337 must happen after 12338 any preceding 12339 global/generic 12340 store/store atomic/ 12341 atomicrmw-no-return-value. 12342 - Must happen before 12343 the following 12344 store. 12345 - Ensures that all 12346 global memory 12347 operations have 12348 completed before 12349 performing the 12350 store that is being 12351 released. 12352 12353 2. ds_atomic 12354 3. s_waitcnt lgkmcnt(0) 12355 12356 - If OpenCL, omit. 12357 - Must happen before 12358 the following 12359 buffer_gl0_inv. 12360 - Ensures any 12361 following global 12362 data read is no 12363 older than the local load 12364 atomic value being 12365 acquired. 12366 12367 4. buffer_gl0_inv 12368 12369 - If CU wavefront execution 12370 mode, omit. 12371 - If OpenCL omit. 12372 - Ensures that 12373 following 12374 loads will not see 12375 stale data. 12376 12377 atomicrmw acq_rel - workgroup - generic 1. s_waitcnt lgkmcnt(0) & 12378 vmcnt(0) & vscnt(0) 12379 12380 - If CU wavefront execution 12381 mode, omit vmcnt(0) and 12382 vscnt(0). 12383 - If OpenCL, omit lgkmcnt(0). 12384 - Could be split into 12385 separate s_waitcnt 12386 vmcnt(0), s_waitcnt 12387 vscnt(0) and s_waitcnt 12388 lgkmcnt(0) to allow 12389 them to be 12390 independently moved 12391 according to the 12392 following rules. 12393 - s_waitcnt vmcnt(0) 12394 must happen after 12395 any preceding 12396 global/generic load/load 12397 atomic/ 12398 atomicrmw-with-return-value. 12399 - s_waitcnt vscnt(0) 12400 must happen after 12401 any preceding 12402 global/generic 12403 store/store 12404 atomic/ 12405 atomicrmw-no-return-value. 12406 - s_waitcnt lgkmcnt(0) 12407 must happen after 12408 any preceding 12409 local/generic 12410 load/store/load 12411 atomic/store 12412 atomic/atomicrmw. 12413 - Must happen before 12414 the following 12415 atomicrmw. 12416 - Ensures that all 12417 memory operations 12418 have 12419 completed before 12420 performing the 12421 atomicrmw that is 12422 being released. 12423 12424 2. flat_atomic 12425 3. s_waitcnt lgkmcnt(0) & 12426 vmcnt(0) & vscnt(0) 12427 12428 - If CU wavefront execution 12429 mode, omit vmcnt(0) and 12430 vscnt(0). 12431 - If OpenCL, omit lgkmcnt(0). 12432 - Must happen before 12433 the following 12434 buffer_gl0_inv. 12435 - Ensures any 12436 following global 12437 data read is no 12438 older than the load 12439 atomic value being 12440 acquired. 12441 12442 3. buffer_gl0_inv 12443 12444 - If CU wavefront execution 12445 mode, omit. 12446 - Ensures that 12447 following 12448 loads will not see 12449 stale data. 12450 12451 atomicrmw acq_rel - agent - global 1. s_waitcnt lgkmcnt(0) & 12452 - system vmcnt(0) & vscnt(0) 12453 12454 - If OpenCL, omit 12455 lgkmcnt(0). 12456 - Could be split into 12457 separate s_waitcnt 12458 vmcnt(0), s_waitcnt 12459 vscnt(0) and s_waitcnt 12460 lgkmcnt(0) to allow 12461 them to be 12462 independently moved 12463 according to the 12464 following rules. 12465 - s_waitcnt vmcnt(0) 12466 must happen after 12467 any preceding 12468 global/generic 12469 load/load atomic/ 12470 atomicrmw-with-return-value. 12471 - s_waitcnt vscnt(0) 12472 must happen after 12473 any preceding 12474 global/generic 12475 store/store atomic/ 12476 atomicrmw-no-return-value. 12477 - s_waitcnt lgkmcnt(0) 12478 must happen after 12479 any preceding 12480 local/generic 12481 load/store/load 12482 atomic/store 12483 atomic/atomicrmw. 12484 - Must happen before 12485 the following 12486 atomicrmw. 12487 - Ensures that all 12488 memory operations 12489 to global have 12490 completed before 12491 performing the 12492 atomicrmw that is 12493 being released. 12494 12495 2. buffer/global_atomic 12496 3. s_waitcnt vm/vscnt(0) 12497 12498 - Use vmcnt(0) if atomic with 12499 return and vscnt(0) if 12500 atomic with no-return. 12501 - Must happen before 12502 following 12503 buffer_gl*_inv. 12504 - Ensures the 12505 atomicrmw has 12506 completed before 12507 invalidating the 12508 caches. 12509 12510 4. buffer_gl0_inv; 12511 buffer_gl1_inv 12512 12513 - Must happen before 12514 any following 12515 global/generic 12516 load/load 12517 atomic/atomicrmw. 12518 - Ensures that 12519 following loads 12520 will not see stale 12521 global data. 12522 12523 atomicrmw acq_rel - agent - generic 1. s_waitcnt lgkmcnt(0) & 12524 - system vmcnt(0) & vscnt(0) 12525 12526 - If OpenCL, omit 12527 lgkmcnt(0). 12528 - Could be split into 12529 separate s_waitcnt 12530 vmcnt(0), s_waitcnt 12531 vscnt(0), and s_waitcnt 12532 lgkmcnt(0) to allow 12533 them to be 12534 independently moved 12535 according to the 12536 following rules. 12537 - s_waitcnt vmcnt(0) 12538 must happen after 12539 any preceding 12540 global/generic 12541 load/load atomic 12542 atomicrmw-with-return-value. 12543 - s_waitcnt vscnt(0) 12544 must happen after 12545 any preceding 12546 global/generic 12547 store/store atomic/ 12548 atomicrmw-no-return-value. 12549 - s_waitcnt lgkmcnt(0) 12550 must happen after 12551 any preceding 12552 local/generic 12553 load/store/load 12554 atomic/store 12555 atomic/atomicrmw. 12556 - Must happen before 12557 the following 12558 atomicrmw. 12559 - Ensures that all 12560 memory operations 12561 have 12562 completed before 12563 performing the 12564 atomicrmw that is 12565 being released. 12566 12567 2. flat_atomic 12568 3. s_waitcnt vm/vscnt(0) & 12569 lgkmcnt(0) 12570 12571 - If OpenCL, omit 12572 lgkmcnt(0). 12573 - Use vmcnt(0) if atomic with 12574 return and vscnt(0) if 12575 atomic with no-return. 12576 - Must happen before 12577 following 12578 buffer_gl*_inv. 12579 - Ensures the 12580 atomicrmw has 12581 completed before 12582 invalidating the 12583 caches. 12584 12585 4. buffer_gl0_inv; 12586 buffer_gl1_inv 12587 12588 - Must happen before 12589 any following 12590 global/generic 12591 load/load 12592 atomic/atomicrmw. 12593 - Ensures that 12594 following loads 12595 will not see stale 12596 global data. 12597 12598 fence acq_rel - singlethread *none* *none* 12599 - wavefront 12600 fence acq_rel - workgroup *none* 1. s_waitcnt lgkmcnt(0) & 12601 vmcnt(0) & vscnt(0) 12602 12603 - If CU wavefront execution 12604 mode, omit vmcnt(0) and 12605 vscnt(0). 12606 - If OpenCL and 12607 address space is 12608 not generic, omit 12609 lgkmcnt(0). 12610 - If OpenCL and 12611 address space is 12612 local, omit 12613 vmcnt(0) and vscnt(0). 12614 - However, 12615 since LLVM 12616 currently has no 12617 address space on 12618 the fence need to 12619 conservatively 12620 always generate 12621 (see comment for 12622 previous fence). 12623 - Could be split into 12624 separate s_waitcnt 12625 vmcnt(0), s_waitcnt 12626 vscnt(0) and s_waitcnt 12627 lgkmcnt(0) to allow 12628 them to be 12629 independently moved 12630 according to the 12631 following rules. 12632 - s_waitcnt vmcnt(0) 12633 must happen after 12634 any preceding 12635 global/generic 12636 load/load 12637 atomic/ 12638 atomicrmw-with-return-value. 12639 - s_waitcnt vscnt(0) 12640 must happen after 12641 any preceding 12642 global/generic 12643 store/store atomic/ 12644 atomicrmw-no-return-value. 12645 - s_waitcnt lgkmcnt(0) 12646 must happen after 12647 any preceding 12648 local/generic 12649 load/store/load 12650 atomic/store atomic/ 12651 atomicrmw. 12652 - Must happen before 12653 any following 12654 global/generic 12655 load/load 12656 atomic/store/store 12657 atomic/atomicrmw. 12658 - Ensures that all 12659 memory operations 12660 have 12661 completed before 12662 performing any 12663 following global 12664 memory operations. 12665 - Ensures that the 12666 preceding 12667 local/generic load 12668 atomic/atomicrmw 12669 with an equal or 12670 wider sync scope 12671 and memory ordering 12672 stronger than 12673 unordered (this is 12674 termed the 12675 acquire-fence-paired-atomic) 12676 has completed 12677 before following 12678 global memory 12679 operations. This 12680 satisfies the 12681 requirements of 12682 acquire. 12683 - Ensures that all 12684 previous memory 12685 operations have 12686 completed before a 12687 following 12688 local/generic store 12689 atomic/atomicrmw 12690 with an equal or 12691 wider sync scope 12692 and memory ordering 12693 stronger than 12694 unordered (this is 12695 termed the 12696 release-fence-paired-atomic). 12697 This satisfies the 12698 requirements of 12699 release. 12700 - Must happen before 12701 the following 12702 buffer_gl0_inv. 12703 - Ensures that the 12704 acquire-fence-paired 12705 atomic has completed 12706 before invalidating 12707 the 12708 cache. Therefore 12709 any following 12710 locations read must 12711 be no older than 12712 the value read by 12713 the 12714 acquire-fence-paired-atomic. 12715 12716 3. buffer_gl0_inv 12717 12718 - If CU wavefront execution 12719 mode, omit. 12720 - Ensures that 12721 following 12722 loads will not see 12723 stale data. 12724 12725 fence acq_rel - agent *none* 1. s_waitcnt lgkmcnt(0) & 12726 - system vmcnt(0) & vscnt(0) 12727 12728 - If OpenCL and 12729 address space is 12730 not generic, omit 12731 lgkmcnt(0). 12732 - If OpenCL and 12733 address space is 12734 local, omit 12735 vmcnt(0) and vscnt(0). 12736 - However, since LLVM 12737 currently has no 12738 address space on 12739 the fence need to 12740 conservatively 12741 always generate 12742 (see comment for 12743 previous fence). 12744 - Could be split into 12745 separate s_waitcnt 12746 vmcnt(0), s_waitcnt 12747 vscnt(0) and s_waitcnt 12748 lgkmcnt(0) to allow 12749 them to be 12750 independently moved 12751 according to the 12752 following rules. 12753 - s_waitcnt vmcnt(0) 12754 must happen after 12755 any preceding 12756 global/generic 12757 load/load 12758 atomic/ 12759 atomicrmw-with-return-value. 12760 - s_waitcnt vscnt(0) 12761 must happen after 12762 any preceding 12763 global/generic 12764 store/store atomic/ 12765 atomicrmw-no-return-value. 12766 - s_waitcnt lgkmcnt(0) 12767 must happen after 12768 any preceding 12769 local/generic 12770 load/store/load 12771 atomic/store 12772 atomic/atomicrmw. 12773 - Must happen before 12774 the following 12775 buffer_gl*_inv. 12776 - Ensures that the 12777 preceding 12778 global/local/generic 12779 load 12780 atomic/atomicrmw 12781 with an equal or 12782 wider sync scope 12783 and memory ordering 12784 stronger than 12785 unordered (this is 12786 termed the 12787 acquire-fence-paired-atomic) 12788 has completed 12789 before invalidating 12790 the caches. This 12791 satisfies the 12792 requirements of 12793 acquire. 12794 - Ensures that all 12795 previous memory 12796 operations have 12797 completed before a 12798 following 12799 global/local/generic 12800 store 12801 atomic/atomicrmw 12802 with an equal or 12803 wider sync scope 12804 and memory ordering 12805 stronger than 12806 unordered (this is 12807 termed the 12808 release-fence-paired-atomic). 12809 This satisfies the 12810 requirements of 12811 release. 12812 12813 2. buffer_gl0_inv; 12814 buffer_gl1_inv 12815 12816 - Must happen before 12817 any following 12818 global/generic 12819 load/load 12820 atomic/store/store 12821 atomic/atomicrmw. 12822 - Ensures that 12823 following loads 12824 will not see stale 12825 global data. This 12826 satisfies the 12827 requirements of 12828 acquire. 12829 12830 **Sequential Consistent Atomic** 12831 ------------------------------------------------------------------------------------ 12832 load atomic seq_cst - singlethread - global *Same as corresponding 12833 - wavefront - local load atomic acquire, 12834 - generic except must generate 12835 all instructions even 12836 for OpenCL.* 12837 load atomic seq_cst - workgroup - global 1. s_waitcnt lgkmcnt(0) & 12838 - generic vmcnt(0) & vscnt(0) 12839 12840 - If CU wavefront execution 12841 mode, omit vmcnt(0) and 12842 vscnt(0). 12843 - Could be split into 12844 separate s_waitcnt 12845 vmcnt(0), s_waitcnt 12846 vscnt(0), and s_waitcnt 12847 lgkmcnt(0) to allow 12848 them to be 12849 independently moved 12850 according to the 12851 following rules. 12852 - s_waitcnt lgkmcnt(0) must 12853 happen after 12854 preceding 12855 local/generic load 12856 atomic/store 12857 atomic/atomicrmw 12858 with memory 12859 ordering of seq_cst 12860 and with equal or 12861 wider sync scope. 12862 (Note that seq_cst 12863 fences have their 12864 own s_waitcnt 12865 lgkmcnt(0) and so do 12866 not need to be 12867 considered.) 12868 - s_waitcnt vmcnt(0) 12869 must happen after 12870 preceding 12871 global/generic load 12872 atomic/ 12873 atomicrmw-with-return-value 12874 with memory 12875 ordering of seq_cst 12876 and with equal or 12877 wider sync scope. 12878 (Note that seq_cst 12879 fences have their 12880 own s_waitcnt 12881 vmcnt(0) and so do 12882 not need to be 12883 considered.) 12884 - s_waitcnt vscnt(0) 12885 Must happen after 12886 preceding 12887 global/generic store 12888 atomic/ 12889 atomicrmw-no-return-value 12890 with memory 12891 ordering of seq_cst 12892 and with equal or 12893 wider sync scope. 12894 (Note that seq_cst 12895 fences have their 12896 own s_waitcnt 12897 vscnt(0) and so do 12898 not need to be 12899 considered.) 12900 - Ensures any 12901 preceding 12902 sequential 12903 consistent global/local 12904 memory instructions 12905 have completed 12906 before executing 12907 this sequentially 12908 consistent 12909 instruction. This 12910 prevents reordering 12911 a seq_cst store 12912 followed by a 12913 seq_cst load. (Note 12914 that seq_cst is 12915 stronger than 12916 acquire/release as 12917 the reordering of 12918 load acquire 12919 followed by a store 12920 release is 12921 prevented by the 12922 s_waitcnt of 12923 the release, but 12924 there is nothing 12925 preventing a store 12926 release followed by 12927 load acquire from 12928 completing out of 12929 order. The s_waitcnt 12930 could be placed after 12931 seq_store or before 12932 the seq_load. We 12933 choose the load to 12934 make the s_waitcnt be 12935 as late as possible 12936 so that the store 12937 may have already 12938 completed.) 12939 12940 2. *Following 12941 instructions same as 12942 corresponding load 12943 atomic acquire, 12944 except must generate 12945 all instructions even 12946 for OpenCL.* 12947 load atomic seq_cst - workgroup - local 12948 12949 1. s_waitcnt vmcnt(0) & vscnt(0) 12950 12951 - If CU wavefront execution 12952 mode, omit. 12953 - Could be split into 12954 separate s_waitcnt 12955 vmcnt(0) and s_waitcnt 12956 vscnt(0) to allow 12957 them to be 12958 independently moved 12959 according to the 12960 following rules. 12961 - s_waitcnt vmcnt(0) 12962 Must happen after 12963 preceding 12964 global/generic load 12965 atomic/ 12966 atomicrmw-with-return-value 12967 with memory 12968 ordering of seq_cst 12969 and with equal or 12970 wider sync scope. 12971 (Note that seq_cst 12972 fences have their 12973 own s_waitcnt 12974 vmcnt(0) and so do 12975 not need to be 12976 considered.) 12977 - s_waitcnt vscnt(0) 12978 Must happen after 12979 preceding 12980 global/generic store 12981 atomic/ 12982 atomicrmw-no-return-value 12983 with memory 12984 ordering of seq_cst 12985 and with equal or 12986 wider sync scope. 12987 (Note that seq_cst 12988 fences have their 12989 own s_waitcnt 12990 vscnt(0) and so do 12991 not need to be 12992 considered.) 12993 - Ensures any 12994 preceding 12995 sequential 12996 consistent global 12997 memory instructions 12998 have completed 12999 before executing 13000 this sequentially 13001 consistent 13002 instruction. This 13003 prevents reordering 13004 a seq_cst store 13005 followed by a 13006 seq_cst load. (Note 13007 that seq_cst is 13008 stronger than 13009 acquire/release as 13010 the reordering of 13011 load acquire 13012 followed by a store 13013 release is 13014 prevented by the 13015 s_waitcnt of 13016 the release, but 13017 there is nothing 13018 preventing a store 13019 release followed by 13020 load acquire from 13021 completing out of 13022 order. The s_waitcnt 13023 could be placed after 13024 seq_store or before 13025 the seq_load. We 13026 choose the load to 13027 make the s_waitcnt be 13028 as late as possible 13029 so that the store 13030 may have already 13031 completed.) 13032 13033 2. *Following 13034 instructions same as 13035 corresponding load 13036 atomic acquire, 13037 except must generate 13038 all instructions even 13039 for OpenCL.* 13040 13041 load atomic seq_cst - agent - global 1. s_waitcnt lgkmcnt(0) & 13042 - system - generic vmcnt(0) & vscnt(0) 13043 13044 - Could be split into 13045 separate s_waitcnt 13046 vmcnt(0), s_waitcnt 13047 vscnt(0) and s_waitcnt 13048 lgkmcnt(0) to allow 13049 them to be 13050 independently moved 13051 according to the 13052 following rules. 13053 - s_waitcnt lgkmcnt(0) 13054 must happen after 13055 preceding 13056 local load 13057 atomic/store 13058 atomic/atomicrmw 13059 with memory 13060 ordering of seq_cst 13061 and with equal or 13062 wider sync scope. 13063 (Note that seq_cst 13064 fences have their 13065 own s_waitcnt 13066 lgkmcnt(0) and so do 13067 not need to be 13068 considered.) 13069 - s_waitcnt vmcnt(0) 13070 must happen after 13071 preceding 13072 global/generic load 13073 atomic/ 13074 atomicrmw-with-return-value 13075 with memory 13076 ordering of seq_cst 13077 and with equal or 13078 wider sync scope. 13079 (Note that seq_cst 13080 fences have their 13081 own s_waitcnt 13082 vmcnt(0) and so do 13083 not need to be 13084 considered.) 13085 - s_waitcnt vscnt(0) 13086 Must happen after 13087 preceding 13088 global/generic store 13089 atomic/ 13090 atomicrmw-no-return-value 13091 with memory 13092 ordering of seq_cst 13093 and with equal or 13094 wider sync scope. 13095 (Note that seq_cst 13096 fences have their 13097 own s_waitcnt 13098 vscnt(0) and so do 13099 not need to be 13100 considered.) 13101 - Ensures any 13102 preceding 13103 sequential 13104 consistent global 13105 memory instructions 13106 have completed 13107 before executing 13108 this sequentially 13109 consistent 13110 instruction. This 13111 prevents reordering 13112 a seq_cst store 13113 followed by a 13114 seq_cst load. (Note 13115 that seq_cst is 13116 stronger than 13117 acquire/release as 13118 the reordering of 13119 load acquire 13120 followed by a store 13121 release is 13122 prevented by the 13123 s_waitcnt of 13124 the release, but 13125 there is nothing 13126 preventing a store 13127 release followed by 13128 load acquire from 13129 completing out of 13130 order. The s_waitcnt 13131 could be placed after 13132 seq_store or before 13133 the seq_load. We 13134 choose the load to 13135 make the s_waitcnt be 13136 as late as possible 13137 so that the store 13138 may have already 13139 completed.) 13140 13141 2. *Following 13142 instructions same as 13143 corresponding load 13144 atomic acquire, 13145 except must generate 13146 all instructions even 13147 for OpenCL.* 13148 store atomic seq_cst - singlethread - global *Same as corresponding 13149 - wavefront - local store atomic release, 13150 - workgroup - generic except must generate 13151 - agent all instructions even 13152 - system for OpenCL.* 13153 atomicrmw seq_cst - singlethread - global *Same as corresponding 13154 - wavefront - local atomicrmw acq_rel, 13155 - workgroup - generic except must generate 13156 - agent all instructions even 13157 - system for OpenCL.* 13158 fence seq_cst - singlethread *none* *Same as corresponding 13159 - wavefront fence acq_rel, 13160 - workgroup except must generate 13161 - agent all instructions even 13162 - system for OpenCL.* 13163 ============ ============ ============== ========== ================================ 13164 13165.. _amdgpu-amdhsa-trap-handler-abi: 13166 13167Trap Handler ABI 13168~~~~~~~~~~~~~~~~ 13169 13170For code objects generated by the AMDGPU backend for HSA [HSA]_ compatible 13171runtimes (see :ref:`amdgpu-os`), the runtime installs a trap handler that 13172supports the ``s_trap`` instruction. For usage see: 13173 13174- :ref:`amdgpu-trap-handler-for-amdhsa-os-v2-table` 13175- :ref:`amdgpu-trap-handler-for-amdhsa-os-v3-table` 13176- :ref:`amdgpu-trap-handler-for-amdhsa-os-v4-onwards-table` 13177 13178 .. table:: AMDGPU Trap Handler for AMDHSA OS Code Object V2 13179 :name: amdgpu-trap-handler-for-amdhsa-os-v2-table 13180 13181 =================== =============== =============== ======================================= 13182 Usage Code Sequence Trap Handler Description 13183 Inputs 13184 =================== =============== =============== ======================================= 13185 reserved ``s_trap 0x00`` Reserved by hardware. 13186 ``debugtrap(arg)`` ``s_trap 0x01`` ``SGPR0-1``: Reserved for Finalizer HSA ``debugtrap`` 13187 ``queue_ptr`` intrinsic (not implemented). 13188 ``VGPR0``: 13189 ``arg`` 13190 ``llvm.trap`` ``s_trap 0x02`` ``SGPR0-1``: Causes wave to be halted with the PC at 13191 ``queue_ptr`` the trap instruction. The associated 13192 queue is signalled to put it into the 13193 error state. When the queue is put in 13194 the error state, the waves executing 13195 dispatches on the queue will be 13196 terminated. 13197 ``llvm.debugtrap`` ``s_trap 0x03`` *none* - If debugger not enabled then behaves 13198 as a no-operation. The trap handler 13199 is entered and immediately returns to 13200 continue execution of the wavefront. 13201 - If the debugger is enabled, causes 13202 the debug trap to be reported by the 13203 debugger and the wavefront is put in 13204 the halt state with the PC at the 13205 instruction. The debugger must 13206 increment the PC and resume the wave. 13207 reserved ``s_trap 0x04`` Reserved. 13208 reserved ``s_trap 0x05`` Reserved. 13209 reserved ``s_trap 0x06`` Reserved. 13210 reserved ``s_trap 0x07`` Reserved. 13211 reserved ``s_trap 0x08`` Reserved. 13212 reserved ``s_trap 0xfe`` Reserved. 13213 reserved ``s_trap 0xff`` Reserved. 13214 =================== =============== =============== ======================================= 13215 13216.. 13217 13218 .. table:: AMDGPU Trap Handler for AMDHSA OS Code Object V3 13219 :name: amdgpu-trap-handler-for-amdhsa-os-v3-table 13220 13221 =================== =============== =============== ======================================= 13222 Usage Code Sequence Trap Handler Description 13223 Inputs 13224 =================== =============== =============== ======================================= 13225 reserved ``s_trap 0x00`` Reserved by hardware. 13226 debugger breakpoint ``s_trap 0x01`` *none* Reserved for debugger to use for 13227 breakpoints. Causes wave to be halted 13228 with the PC at the trap instruction. 13229 The debugger is responsible to resume 13230 the wave, including the instruction 13231 that the breakpoint overwrote. 13232 ``llvm.trap`` ``s_trap 0x02`` ``SGPR0-1``: Causes wave to be halted with the PC at 13233 ``queue_ptr`` the trap instruction. The associated 13234 queue is signalled to put it into the 13235 error state. When the queue is put in 13236 the error state, the waves executing 13237 dispatches on the queue will be 13238 terminated. 13239 ``llvm.debugtrap`` ``s_trap 0x03`` *none* - If debugger not enabled then behaves 13240 as a no-operation. The trap handler 13241 is entered and immediately returns to 13242 continue execution of the wavefront. 13243 - If the debugger is enabled, causes 13244 the debug trap to be reported by the 13245 debugger and the wavefront is put in 13246 the halt state with the PC at the 13247 instruction. The debugger must 13248 increment the PC and resume the wave. 13249 reserved ``s_trap 0x04`` Reserved. 13250 reserved ``s_trap 0x05`` Reserved. 13251 reserved ``s_trap 0x06`` Reserved. 13252 reserved ``s_trap 0x07`` Reserved. 13253 reserved ``s_trap 0x08`` Reserved. 13254 reserved ``s_trap 0xfe`` Reserved. 13255 reserved ``s_trap 0xff`` Reserved. 13256 =================== =============== =============== ======================================= 13257 13258.. 13259 13260 .. table:: AMDGPU Trap Handler for AMDHSA OS Code Object V4 and Above 13261 :name: amdgpu-trap-handler-for-amdhsa-os-v4-onwards-table 13262 13263 =================== =============== ================ ================= ======================================= 13264 Usage Code Sequence GFX6-GFX8 Inputs GFX9-GFX10 Inputs Description 13265 =================== =============== ================ ================= ======================================= 13266 reserved ``s_trap 0x00`` Reserved by hardware. 13267 debugger breakpoint ``s_trap 0x01`` *none* *none* Reserved for debugger to use for 13268 breakpoints. Causes wave to be halted 13269 with the PC at the trap instruction. 13270 The debugger is responsible to resume 13271 the wave, including the instruction 13272 that the breakpoint overwrote. 13273 ``llvm.trap`` ``s_trap 0x02`` ``SGPR0-1``: *none* Causes wave to be halted with the PC at 13274 ``queue_ptr`` the trap instruction. The associated 13275 queue is signalled to put it into the 13276 error state. When the queue is put in 13277 the error state, the waves executing 13278 dispatches on the queue will be 13279 terminated. 13280 ``llvm.debugtrap`` ``s_trap 0x03`` *none* *none* - If debugger not enabled then behaves 13281 as a no-operation. The trap handler 13282 is entered and immediately returns to 13283 continue execution of the wavefront. 13284 - If the debugger is enabled, causes 13285 the debug trap to be reported by the 13286 debugger and the wavefront is put in 13287 the halt state with the PC at the 13288 instruction. The debugger must 13289 increment the PC and resume the wave. 13290 reserved ``s_trap 0x04`` Reserved. 13291 reserved ``s_trap 0x05`` Reserved. 13292 reserved ``s_trap 0x06`` Reserved. 13293 reserved ``s_trap 0x07`` Reserved. 13294 reserved ``s_trap 0x08`` Reserved. 13295 reserved ``s_trap 0xfe`` Reserved. 13296 reserved ``s_trap 0xff`` Reserved. 13297 =================== =============== ================ ================= ======================================= 13298 13299.. _amdgpu-amdhsa-function-call-convention: 13300 13301Call Convention 13302~~~~~~~~~~~~~~~ 13303 13304.. note:: 13305 13306 This section is currently incomplete and has inaccuracies. It is WIP that will 13307 be updated as information is determined. 13308 13309See :ref:`amdgpu-dwarf-address-space-identifier` for information on swizzled 13310addresses. Unswizzled addresses are normal linear addresses. 13311 13312.. _amdgpu-amdhsa-function-call-convention-kernel-functions: 13313 13314Kernel Functions 13315++++++++++++++++ 13316 13317This section describes the call convention ABI for the outer kernel function. 13318 13319See :ref:`amdgpu-amdhsa-initial-kernel-execution-state` for the kernel call 13320convention. 13321 13322The following is not part of the AMDGPU kernel calling convention but describes 13323how the AMDGPU implements function calls: 13324 133251. Clang decides the kernarg layout to match the *HSA Programmer's Language 13326 Reference* [HSA]_. 13327 13328 - All structs are passed directly. 13329 - Lambda values are passed *TBA*. 13330 13331 .. TODO:: 13332 13333 - Does this really follow HSA rules? Or are structs >16 bytes passed 13334 by-value struct? 13335 - What is ABI for lambda values? 13336 133374. The kernel performs certain setup in its prolog, as described in 13338 :ref:`amdgpu-amdhsa-kernel-prolog`. 13339 13340.. _amdgpu-amdhsa-function-call-convention-non-kernel-functions: 13341 13342Non-Kernel Functions 13343++++++++++++++++++++ 13344 13345This section describes the call convention ABI for functions other than the 13346outer kernel function. 13347 13348If a kernel has function calls then scratch is always allocated and used for 13349the call stack which grows from low address to high address using the swizzled 13350scratch address space. 13351 13352On entry to a function: 13353 133541. SGPR0-3 contain a V# with the following properties (see 13355 :ref:`amdgpu-amdhsa-kernel-prolog-private-segment-buffer`): 13356 13357 * Base address pointing to the beginning of the wavefront scratch backing 13358 memory. 13359 * Swizzled with dword element size and stride of wavefront size elements. 13360 133612. The FLAT_SCRATCH register pair is setup. See 13362 :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`. 133633. GFX6-GFX8: M0 register set to the size of LDS in bytes. See 13364 :ref:`amdgpu-amdhsa-kernel-prolog-m0`. 133654. The EXEC register is set to the lanes active on entry to the function. 133665. MODE register: *TBD* 133676. VGPR0-31 and SGPR4-29 are used to pass function input arguments as described 13368 below. 133697. SGPR30-31 return address (RA). The code address that the function must 13370 return to when it completes. The value is undefined if the function is *no 13371 return*. 133728. SGPR32 is used for the stack pointer (SP). It is an unswizzled scratch 13373 offset relative to the beginning of the wavefront scratch backing memory. 13374 13375 The unswizzled SP can be used with buffer instructions as an unswizzled SGPR 13376 offset with the scratch V# in SGPR0-3 to access the stack in a swizzled 13377 manner. 13378 13379 The unswizzled SP value can be converted into the swizzled SP value by: 13380 13381 | swizzled SP = unswizzled SP / wavefront size 13382 13383 This may be used to obtain the private address space address of stack 13384 objects and to convert this address to a flat address by adding the flat 13385 scratch aperture base address. 13386 13387 The swizzled SP value is always 4 bytes aligned for the ``r600`` 13388 architecture and 16 byte aligned for the ``amdgcn`` architecture. 13389 13390 .. note:: 13391 13392 The ``amdgcn`` value is selected to avoid dynamic stack alignment for the 13393 OpenCL language which has the largest base type defined as 16 bytes. 13394 13395 On entry, the swizzled SP value is the address of the first function 13396 argument passed on the stack. Other stack passed arguments are positive 13397 offsets from the entry swizzled SP value. 13398 13399 The function may use positive offsets beyond the last stack passed argument 13400 for stack allocated local variables and register spill slots. If necessary, 13401 the function may align these to greater alignment than 16 bytes. After these 13402 the function may dynamically allocate space for such things as runtime sized 13403 ``alloca`` local allocations. 13404 13405 If the function calls another function, it will place any stack allocated 13406 arguments after the last local allocation and adjust SGPR32 to the address 13407 after the last local allocation. 13408 134099. All other registers are unspecified. 1341010. Any necessary ``s_waitcnt`` has been performed to ensure memory is available 13411 to the function. 13412 13413On exit from a function: 13414 134151. VGPR0-31 and SGPR4-29 are used to pass function result arguments as 13416 described below. Any registers used are considered clobbered registers. 134172. The following registers are preserved and have the same value as on entry: 13418 13419 * FLAT_SCRATCH 13420 * EXEC 13421 * GFX6-GFX8: M0 13422 * All SGPR registers except the clobbered registers of SGPR4-31. 13423 * VGPR40-47 13424 * VGPR56-63 13425 * VGPR72-79 13426 * VGPR88-95 13427 * VGPR104-111 13428 * VGPR120-127 13429 * VGPR136-143 13430 * VGPR152-159 13431 * VGPR168-175 13432 * VGPR184-191 13433 * VGPR200-207 13434 * VGPR216-223 13435 * VGPR232-239 13436 * VGPR248-255 13437 13438 .. note:: 13439 13440 Except the argument registers, the VGPRs clobbered and the preserved 13441 registers are intermixed at regular intervals in order to keep a 13442 similar ratio independent of the number of allocated VGPRs. 13443 13444 * GFX90A: All AGPR registers except the clobbered registers AGPR0-31. 13445 * Lanes of all VGPRs that are inactive at the call site. 13446 13447 For the AMDGPU backend, an inter-procedural register allocation (IPRA) 13448 optimization may mark some of clobbered SGPR and VGPR registers as 13449 preserved if it can be determined that the called function does not change 13450 their value. 13451 134522. The PC is set to the RA provided on entry. 134533. MODE register: *TBD*. 134544. All other registers are clobbered. 134555. Any necessary ``s_waitcnt`` has been performed to ensure memory accessed by 13456 function is available to the caller. 13457 13458.. TODO:: 13459 13460 - How are function results returned? The address of structured types is passed 13461 by reference, but what about other types? 13462 13463The function input arguments are made up of the formal arguments explicitly 13464declared by the source language function plus the implicit input arguments used 13465by the implementation. 13466 13467The source language input arguments are: 13468 134691. Any source language implicit ``this`` or ``self`` argument comes first as a 13470 pointer type. 134712. Followed by the function formal arguments in left to right source order. 13472 13473The source language result arguments are: 13474 134751. The function result argument. 13476 13477The source language input or result struct type arguments that are less than or 13478equal to 16 bytes, are decomposed recursively into their base type fields, and 13479each field is passed as if a separate argument. For input arguments, if the 13480called function requires the struct to be in memory, for example because its 13481address is taken, then the function body is responsible for allocating a stack 13482location and copying the field arguments into it. Clang terms this *direct 13483struct*. 13484 13485The source language input struct type arguments that are greater than 16 bytes, 13486are passed by reference. The caller is responsible for allocating a stack 13487location to make a copy of the struct value and pass the address as the input 13488argument. The called function is responsible to perform the dereference when 13489accessing the input argument. Clang terms this *by-value struct*. 13490 13491A source language result struct type argument that is greater than 16 bytes, is 13492returned by reference. The caller is responsible for allocating a stack location 13493to hold the result value and passes the address as the last input argument 13494(before the implicit input arguments). In this case there are no result 13495arguments. The called function is responsible to perform the dereference when 13496storing the result value. Clang terms this *structured return (sret)*. 13497 13498*TODO: correct the ``sret`` definition.* 13499 13500.. TODO:: 13501 13502 Is this definition correct? Or is ``sret`` only used if passing in registers, and 13503 pass as non-decomposed struct as stack argument? Or something else? Is the 13504 memory location in the caller stack frame, or a stack memory argument and so 13505 no address is passed as the caller can directly write to the argument stack 13506 location? But then the stack location is still live after return. If an 13507 argument stack location is it the first stack argument or the last one? 13508 13509Lambda argument types are treated as struct types with an implementation defined 13510set of fields. 13511 13512.. TODO:: 13513 13514 Need to specify the ABI for lambda types for AMDGPU. 13515 13516For AMDGPU backend all source language arguments (including the decomposed 13517struct type arguments) are passed in VGPRs unless marked ``inreg`` in which case 13518they are passed in SGPRs. 13519 13520The AMDGPU backend walks the function call graph from the leaves to determine 13521which implicit input arguments are used, propagating to each caller of the 13522function. The used implicit arguments are appended to the function arguments 13523after the source language arguments in the following order: 13524 13525.. TODO:: 13526 13527 Is recursion or external functions supported? 13528 135291. Work-Item ID (1 VGPR) 13530 13531 The X, Y and Z work-item ID are packed into a single VGRP with the following 13532 layout. Only fields actually used by the function are set. The other bits 13533 are undefined. 13534 13535 The values come from the initial kernel execution state. See 13536 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`. 13537 13538 .. table:: Work-item implicit argument layout 13539 :name: amdgpu-amdhsa-workitem-implicit-argument-layout-table 13540 13541 ======= ======= ============== 13542 Bits Size Field Name 13543 ======= ======= ============== 13544 9:0 10 bits X Work-Item ID 13545 19:10 10 bits Y Work-Item ID 13546 29:20 10 bits Z Work-Item ID 13547 31:30 2 bits Unused 13548 ======= ======= ============== 13549 135502. Dispatch Ptr (2 SGPRs) 13551 13552 The value comes from the initial kernel execution state. See 13553 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 13554 135553. Queue Ptr (2 SGPRs) 13556 13557 The value comes from the initial kernel execution state. See 13558 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 13559 135604. Kernarg Segment Ptr (2 SGPRs) 13561 13562 The value comes from the initial kernel execution state. See 13563 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 13564 135655. Dispatch id (2 SGPRs) 13566 13567 The value comes from the initial kernel execution state. See 13568 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 13569 135706. Work-Group ID X (1 SGPR) 13571 13572 The value comes from the initial kernel execution state. See 13573 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 13574 135757. Work-Group ID Y (1 SGPR) 13576 13577 The value comes from the initial kernel execution state. See 13578 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 13579 135808. Work-Group ID Z (1 SGPR) 13581 13582 The value comes from the initial kernel execution state. See 13583 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 13584 135859. Implicit Argument Ptr (2 SGPRs) 13586 13587 The value is computed by adding an offset to Kernarg Segment Ptr to get the 13588 global address space pointer to the first kernarg implicit argument. 13589 13590The input and result arguments are assigned in order in the following manner: 13591 13592.. note:: 13593 13594 There are likely some errors and omissions in the following description that 13595 need correction. 13596 13597 .. TODO:: 13598 13599 Check the Clang source code to decipher how function arguments and return 13600 results are handled. Also see the AMDGPU specific values used. 13601 13602* VGPR arguments are assigned to consecutive VGPRs starting at VGPR0 up to 13603 VGPR31. 13604 13605 If there are more arguments than will fit in these registers, the remaining 13606 arguments are allocated on the stack in order on naturally aligned 13607 addresses. 13608 13609 .. TODO:: 13610 13611 How are overly aligned structures allocated on the stack? 13612 13613* SGPR arguments are assigned to consecutive SGPRs starting at SGPR0 up to 13614 SGPR29. 13615 13616 If there are more arguments than will fit in these registers, the remaining 13617 arguments are allocated on the stack in order on naturally aligned 13618 addresses. 13619 13620Note that decomposed struct type arguments may have some fields passed in 13621registers and some in memory. 13622 13623.. TODO:: 13624 13625 So, a struct which can pass some fields as decomposed register arguments, will 13626 pass the rest as decomposed stack elements? But an argument that will not start 13627 in registers will not be decomposed and will be passed as a non-decomposed 13628 stack value? 13629 13630The following is not part of the AMDGPU function calling convention but 13631describes how the AMDGPU implements function calls: 13632 136331. SGPR33 is used as a frame pointer (FP) if necessary. Like the SP it is an 13634 unswizzled scratch address. It is only needed if runtime sized ``alloca`` 13635 are used, or for the reasons defined in ``SIFrameLowering``. 136362. Runtime stack alignment is supported. SGPR34 is used as a base pointer (BP) 13637 to access the incoming stack arguments in the function. The BP is needed 13638 only when the function requires the runtime stack alignment. 13639 136403. Allocating SGPR arguments on the stack are not supported. 13641 136424. No CFI is currently generated. See 13643 :ref:`amdgpu-dwarf-call-frame-information`. 13644 13645 .. note:: 13646 13647 CFI will be generated that defines the CFA as the unswizzled address 13648 relative to the wave scratch base in the unswizzled private address space 13649 of the lowest address stack allocated local variable. 13650 13651 ``DW_AT_frame_base`` will be defined as the swizzled address in the 13652 swizzled private address space by dividing the CFA by the wavefront size 13653 (since CFA is always at least dword aligned which matches the scratch 13654 swizzle element size). 13655 13656 If no dynamic stack alignment was performed, the stack allocated arguments 13657 are accessed as negative offsets relative to ``DW_AT_frame_base``, and the 13658 local variables and register spill slots are accessed as positive offsets 13659 relative to ``DW_AT_frame_base``. 13660 136615. Function argument passing is implemented by copying the input physical 13662 registers to virtual registers on entry. The register allocator can spill if 13663 necessary. These are copied back to physical registers at call sites. The 13664 net effect is that each function call can have these values in entirely 13665 distinct locations. The IPRA can help avoid shuffling argument registers. 136666. Call sites are implemented by setting up the arguments at positive offsets 13667 from SP. Then SP is incremented to account for the known frame size before 13668 the call and decremented after the call. 13669 13670 .. note:: 13671 13672 The CFI will reflect the changed calculation needed to compute the CFA 13673 from SP. 13674 136757. 4 byte spill slots are used in the stack frame. One slot is allocated for an 13676 emergency spill slot. Buffer instructions are used for stack accesses and 13677 not the ``flat_scratch`` instruction. 13678 13679 .. TODO:: 13680 13681 Explain when the emergency spill slot is used. 13682 13683.. TODO:: 13684 13685 Possible broken issues: 13686 13687 - Stack arguments must be aligned to required alignment. 13688 - Stack is aligned to max(16, max formal argument alignment) 13689 - Direct argument < 64 bits should check register budget. 13690 - Register budget calculation should respect ``inreg`` for SGPR. 13691 - SGPR overflow is not handled. 13692 - struct with 1 member unpeeling is not checking size of member. 13693 - ``sret`` is after ``this`` pointer. 13694 - Caller is not implementing stack realignment: need an extra pointer. 13695 - Should say AMDGPU passes FP rather than SP. 13696 - Should CFI define CFA as address of locals or arguments. Difference is 13697 apparent when have implemented dynamic alignment. 13698 - If ``SCRATCH`` instruction could allow negative offsets, then can make FP be 13699 highest address of stack frame and use negative offset for locals. Would 13700 allow SP to be the same as FP and could support signal-handler-like as now 13701 have a real SP for the top of the stack. 13702 - How is ``sret`` passed on the stack? In argument stack area? Can it overlay 13703 arguments? 13704 13705AMDPAL 13706------ 13707 13708This section provides code conventions used when the target triple OS is 13709``amdpal`` (see :ref:`amdgpu-target-triples`). 13710 13711.. _amdgpu-amdpal-code-object-metadata-section: 13712 13713Code Object Metadata 13714~~~~~~~~~~~~~~~~~~~~ 13715 13716.. note:: 13717 13718 The metadata is currently in development and is subject to major 13719 changes. Only the current version is supported. *When this document 13720 was generated the version was 2.6.* 13721 13722Code object metadata is specified by the ``NT_AMDGPU_METADATA`` note 13723record (see :ref:`amdgpu-note-records-v3-onwards`). 13724 13725The metadata is represented as Message Pack formatted binary data (see 13726[MsgPack]_). The top level is a Message Pack map that includes the keys 13727defined in table :ref:`amdgpu-amdpal-code-object-metadata-map-table` 13728and referenced tables. 13729 13730Additional information can be added to the maps. To avoid conflicts, any 13731key names should be prefixed by "*vendor-name*." where ``vendor-name`` 13732can be the name of the vendor and specific vendor tool that generates the 13733information. The prefix is abbreviated to simply "." when it appears 13734within a map that has been added by the same *vendor-name*. 13735 13736 .. table:: AMDPAL Code Object Metadata Map 13737 :name: amdgpu-amdpal-code-object-metadata-map-table 13738 13739 =================== ============== ========= ====================================================================== 13740 String Key Value Type Required? Description 13741 =================== ============== ========= ====================================================================== 13742 "amdpal.version" sequence of Required PAL code object metadata (major, minor) version. The current values 13743 2 integers are defined by *Util::Abi::PipelineMetadata(Major|Minor)Version*. 13744 "amdpal.pipelines" sequence of Required Per-pipeline metadata. See 13745 map :ref:`amdgpu-amdpal-code-object-pipeline-metadata-map-table` for the 13746 definition of the keys included in that map. 13747 =================== ============== ========= ====================================================================== 13748 13749.. 13750 13751 .. table:: AMDPAL Code Object Pipeline Metadata Map 13752 :name: amdgpu-amdpal-code-object-pipeline-metadata-map-table 13753 13754 ====================================== ============== ========= =================================================== 13755 String Key Value Type Required? Description 13756 ====================================== ============== ========= =================================================== 13757 ".name" string Source name of the pipeline. 13758 ".type" string Pipeline type, e.g. VsPs. Values include: 13759 13760 - "VsPs" 13761 - "Gs" 13762 - "Cs" 13763 - "Ngg" 13764 - "Tess" 13765 - "GsTess" 13766 - "NggTess" 13767 13768 ".internal_pipeline_hash" sequence of Required Internal compiler hash for this pipeline. Lower 13769 2 integers 64 bits is the "stable" portion of the hash, used 13770 for e.g. shader replacement lookup. Upper 64 bits 13771 is the "unique" portion of the hash, used for 13772 e.g. pipeline cache lookup. The value is 13773 implementation defined, and can not be relied on 13774 between different builds of the compiler. 13775 ".shaders" map Per-API shader metadata. See 13776 :ref:`amdgpu-amdpal-code-object-shader-map-table` 13777 for the definition of the keys included in that 13778 map. 13779 ".hardware_stages" map Per-hardware stage metadata. See 13780 :ref:`amdgpu-amdpal-code-object-hardware-stage-map-table` 13781 for the definition of the keys included in that 13782 map. 13783 ".shader_functions" map Per-shader function metadata. See 13784 :ref:`amdgpu-amdpal-code-object-shader-function-map-table` 13785 for the definition of the keys included in that 13786 map. 13787 ".registers" map Required Hardware register configuration. See 13788 :ref:`amdgpu-amdpal-code-object-register-map-table` 13789 for the definition of the keys included in that 13790 map. 13791 ".user_data_limit" integer Number of user data entries accessed by this 13792 pipeline. 13793 ".spill_threshold" integer The user data spill threshold. 0xFFFF for 13794 NoUserDataSpilling. 13795 ".uses_viewport_array_index" boolean Indicates whether or not the pipeline uses the 13796 viewport array index feature. Pipelines which use 13797 this feature can render into all 16 viewports, 13798 whereas pipelines which do not use it are 13799 restricted to viewport #0. 13800 ".es_gs_lds_size" integer Size in bytes of LDS space used internally for 13801 handling data-passing between the ES and GS 13802 shader stages. This can be zero if the data is 13803 passed using off-chip buffers. This value should 13804 be used to program all user-SGPRs which have been 13805 marked with "UserDataMapping::EsGsLdsSize" 13806 (typically only the GS and VS HW stages will ever 13807 have a user-SGPR so marked). 13808 ".nggSubgroupSize" integer Explicit maximum subgroup size for NGG shaders 13809 (maximum number of threads in a subgroup). 13810 ".num_interpolants" integer Graphics only. Number of PS interpolants. 13811 ".mesh_scratch_memory_size" integer Max mesh shader scratch memory used. 13812 ".api" string Name of the client graphics API. 13813 ".api_create_info" binary Graphics API shader create info binary blob. Can 13814 be defined by the driver using the compiler if 13815 they want to be able to correlate API-specific 13816 information used during creation at a later time. 13817 ====================================== ============== ========= =================================================== 13818 13819.. 13820 13821 .. table:: AMDPAL Code Object Shader Map 13822 :name: amdgpu-amdpal-code-object-shader-map-table 13823 13824 13825 +-------------+--------------+-------------------------------------------------------------------+ 13826 |String Key |Value Type |Description | 13827 +=============+==============+===================================================================+ 13828 |- ".compute" |map |See :ref:`amdgpu-amdpal-code-object-api-shader-metadata-map-table` | 13829 |- ".vertex" | |for the definition of the keys included in that map. | 13830 |- ".hull" | | | 13831 |- ".domain" | | | 13832 |- ".geometry"| | | 13833 |- ".pixel" | | | 13834 +-------------+--------------+-------------------------------------------------------------------+ 13835 13836.. 13837 13838 .. table:: AMDPAL Code Object API Shader Metadata Map 13839 :name: amdgpu-amdpal-code-object-api-shader-metadata-map-table 13840 13841 ==================== ============== ========= ===================================================================== 13842 String Key Value Type Required? Description 13843 ==================== ============== ========= ===================================================================== 13844 ".api_shader_hash" sequence of Required Input shader hash, typically passed in from the client. The value 13845 2 integers is implementation defined, and can not be relied on between 13846 different builds of the compiler. 13847 ".hardware_mapping" sequence of Required Flags indicating the HW stages this API shader maps to. Values 13848 string include: 13849 13850 - ".ls" 13851 - ".hs" 13852 - ".es" 13853 - ".gs" 13854 - ".vs" 13855 - ".ps" 13856 - ".cs" 13857 13858 ==================== ============== ========= ===================================================================== 13859 13860.. 13861 13862 .. table:: AMDPAL Code Object Hardware Stage Map 13863 :name: amdgpu-amdpal-code-object-hardware-stage-map-table 13864 13865 +-------------+--------------+-----------------------------------------------------------------------+ 13866 |String Key |Value Type |Description | 13867 +=============+==============+=======================================================================+ 13868 |- ".ls" |map |See :ref:`amdgpu-amdpal-code-object-hardware-stage-metadata-map-table` | 13869 |- ".hs" | |for the definition of the keys included in that map. | 13870 |- ".es" | | | 13871 |- ".gs" | | | 13872 |- ".vs" | | | 13873 |- ".ps" | | | 13874 |- ".cs" | | | 13875 +-------------+--------------+-----------------------------------------------------------------------+ 13876 13877.. 13878 13879 .. table:: AMDPAL Code Object Hardware Stage Metadata Map 13880 :name: amdgpu-amdpal-code-object-hardware-stage-metadata-map-table 13881 13882 ========================== ============== ========= =============================================================== 13883 String Key Value Type Required? Description 13884 ========================== ============== ========= =============================================================== 13885 ".entry_point" string The ELF symbol pointing to this pipeline's stage entry point. 13886 ".scratch_memory_size" integer Scratch memory size in bytes. 13887 ".lds_size" integer Local Data Share size in bytes. 13888 ".perf_data_buffer_size" integer Performance data buffer size in bytes. 13889 ".vgpr_count" integer Number of VGPRs used. 13890 ".agpr_count" integer Number of AGPRs used. 13891 ".sgpr_count" integer Number of SGPRs used. 13892 ".vgpr_limit" integer If non-zero, indicates the shader was compiled with a 13893 directive to instruct the compiler to limit the VGPR usage to 13894 be less than or equal to the specified value (only set if 13895 different from HW default). 13896 ".sgpr_limit" integer SGPR count upper limit (only set if different from HW 13897 default). 13898 ".threadgroup_dimensions" sequence of Thread-group X/Y/Z dimensions (Compute only). 13899 3 integers 13900 ".wavefront_size" integer Wavefront size (only set if different from HW default). 13901 ".uses_uavs" boolean The shader reads or writes UAVs. 13902 ".uses_rovs" boolean The shader reads or writes ROVs. 13903 ".writes_uavs" boolean The shader writes to one or more UAVs. 13904 ".writes_depth" boolean The shader writes out a depth value. 13905 ".uses_append_consume" boolean The shader uses append and/or consume operations, either 13906 memory or GDS. 13907 ".uses_prim_id" boolean The shader uses PrimID. 13908 ========================== ============== ========= =============================================================== 13909 13910.. 13911 13912 .. table:: AMDPAL Code Object Shader Function Map 13913 :name: amdgpu-amdpal-code-object-shader-function-map-table 13914 13915 =============== ============== ==================================================================== 13916 String Key Value Type Description 13917 =============== ============== ==================================================================== 13918 *symbol name* map *symbol name* is the ELF symbol name of the shader function code 13919 entry address. The value is the function's metadata. See 13920 :ref:`amdgpu-amdpal-code-object-shader-function-metadata-map-table`. 13921 =============== ============== ==================================================================== 13922 13923.. 13924 13925 .. table:: AMDPAL Code Object Shader Function Metadata Map 13926 :name: amdgpu-amdpal-code-object-shader-function-metadata-map-table 13927 13928 ============================= ============== ================================================================= 13929 String Key Value Type Description 13930 ============================= ============== ================================================================= 13931 ".api_shader_hash" sequence of Input shader hash, typically passed in from the client. The value 13932 2 integers is implementation defined, and can not be relied on between 13933 different builds of the compiler. 13934 ".scratch_memory_size" integer Size in bytes of scratch memory used by the shader. 13935 ".lds_size" integer Size in bytes of LDS memory. 13936 ".vgpr_count" integer Number of VGPRs used by the shader. 13937 ".sgpr_count" integer Number of SGPRs used by the shader. 13938 ".stack_frame_size_in_bytes" integer Amount of stack size used by the shader. 13939 ".shader_subtype" string Shader subtype/kind. Values include: 13940 13941 - "Unknown" 13942 13943 ============================= ============== ================================================================= 13944 13945.. 13946 13947 .. table:: AMDPAL Code Object Register Map 13948 :name: amdgpu-amdpal-code-object-register-map-table 13949 13950 ========================== ============== ==================================================================== 13951 32-bit Integer Key Value Type Description 13952 ========================== ============== ==================================================================== 13953 ``reg offset`` 32-bit integer ``reg offset`` is the dword offset into the GFXIP register space of 13954 a GRBM register (i.e., driver accessible GPU register number, not 13955 shader GPR register number). The driver is required to program each 13956 specified register to the corresponding specified value when 13957 executing this pipeline. Typically, the ``reg offsets`` are the 13958 ``uint16_t`` offsets to each register as defined by the hardware 13959 chip headers. The register is set to the provided value. However, a 13960 ``reg offset`` that specifies a user data register (e.g., 13961 COMPUTE_USER_DATA_0) needs special treatment. See 13962 :ref:`amdgpu-amdpal-code-object-user-data-section` section for more 13963 information. 13964 ========================== ============== ==================================================================== 13965 13966.. _amdgpu-amdpal-code-object-user-data-section: 13967 13968User Data 13969+++++++++ 13970 13971Each hardware stage has a set of 32-bit physical SPI *user data registers* 13972(either 16 or 32 based on graphics IP and the stage) which can be 13973written from a command buffer and then loaded into SGPRs when waves are 13974launched via a subsequent dispatch or draw operation. This is the way 13975most arguments are passed from the application/runtime to a hardware 13976shader. 13977 13978PAL abstracts this functionality by exposing a set of 128 *user data 13979entries* per pipeline a client can use to pass arguments from a command 13980buffer to one or more shaders in that pipeline. The ELF code object must 13981specify a mapping from virtualized *user data entries* to physical *user 13982data registers*, and PAL is responsible for implementing that mapping, 13983including spilling overflow *user data entries* to memory if needed. 13984 13985Since the *user data registers* are GRBM-accessible SPI registers, this 13986mapping is actually embedded in the ``.registers`` metadata entry. For 13987most registers, the value in that map is a literal 32-bit value that 13988should be written to the register by the driver. However, when the 13989register is a *user data register* (any USER_DATA register e.g., 13990SPI_SHADER_USER_DATA_PS_5), the value is instead an encoding that tells 13991the driver to write either a *user data entry* value or one of several 13992driver-internal values to the register. This encoding is described in 13993the following table: 13994 13995.. note:: 13996 13997 Currently, *user data registers* 0 and 1 (e.g., SPI_SHADER_USER_DATA_PS_0, 13998 and SPI_SHADER_USER_DATA_PS_1) are reserved. *User data register* 0 must 13999 always be programmed to the address of the GlobalTable, and *user data 14000 register* 1 must always be programmed to the address of the PerShaderTable. 14001 14002.. 14003 14004 .. table:: AMDPAL User Data Mapping 14005 :name: amdgpu-amdpal-code-object-metadata-user-data-mapping-table 14006 14007 ========== ================= =============================================================================== 14008 Value Name Description 14009 ========== ================= =============================================================================== 14010 0..127 *User Data Entry* 32-bit value of user_data_entry[N] as specified via *CmdSetUserData()* 14011 0x10000000 GlobalTable 32-bit pointer to GPU memory containing the global internal table (should 14012 always point to *user data register* 0). 14013 0x10000001 PerShaderTable 32-bit pointer to GPU memory containing the per-shader internal table. See 14014 :ref:`amdgpu-amdpal-code-object-metadata-user-data-per-shader-table-section` 14015 for more detail (should always point to *user data register* 1). 14016 0x10000002 SpillTable 32-bit pointer to GPU memory containing the user data spill table. See 14017 :ref:`amdgpu-amdpal-code-object-metadata-user-data-spill-table-section` for 14018 more detail. 14019 0x10000003 BaseVertex Vertex offset (32-bit unsigned integer). Not needed if the pipeline doesn't 14020 reference the draw index in the vertex shader. Only supported by the first 14021 stage in a graphics pipeline. 14022 0x10000004 BaseInstance Instance offset (32-bit unsigned integer). Only supported by the first stage in 14023 a graphics pipeline. 14024 0x10000005 DrawIndex Draw index (32-bit unsigned integer). Only supported by the first stage in a 14025 graphics pipeline. 14026 0x10000006 Workgroup Thread group count (32-bit unsigned integer). Low half of a 64-bit address of 14027 a buffer containing the grid dimensions for a Compute dispatch operation. The 14028 high half of the address is stored in the next sequential user-SGPR. Only 14029 supported by compute pipelines. 14030 0x1000000A EsGsLdsSize Indicates that PAL will program this user-SGPR to contain the amount of LDS 14031 space used for the ES/GS pseudo-ring-buffer for passing data between shader 14032 stages. 14033 0x1000000B ViewId View id (32-bit unsigned integer) identifies a view of graphic 14034 pipeline instancing. 14035 0x1000000C StreamOutTable 32-bit pointer to GPU memory containing the stream out target SRD table. This 14036 can only appear for one shader stage per pipeline. 14037 0x1000000D PerShaderPerfData 32-bit pointer to GPU memory containing the per-shader performance data buffer. 14038 0x1000000F VertexBufferTable 32-bit pointer to GPU memory containing the vertex buffer SRD table. This can 14039 only appear for one shader stage per pipeline. 14040 0x10000010 UavExportTable 32-bit pointer to GPU memory containing the UAV export SRD table. This can 14041 only appear for one shader stage per pipeline (PS). These replace color targets 14042 and are completely separate from any UAVs used by the shader. This is optional, 14043 and only used by the PS when UAV exports are used to replace color-target 14044 exports to optimize specific shaders. 14045 0x10000011 NggCullingData 64-bit pointer to GPU memory containing the hardware register data needed by 14046 some NGG pipelines to perform culling. This value contains the address of the 14047 first of two consecutive registers which provide the full GPU address. 14048 0x10000015 FetchShaderPtr 64-bit pointer to GPU memory containing the fetch shader subroutine. 14049 ========== ================= =============================================================================== 14050 14051.. _amdgpu-amdpal-code-object-metadata-user-data-per-shader-table-section: 14052 14053Per-Shader Table 14054################ 14055 14056Low 32 bits of the GPU address for an optional buffer in the ``.data`` 14057section of the ELF. The high 32 bits of the address match the high 32 bits 14058of the shader's program counter. 14059 14060The buffer can be anything the shader compiler needs it for, and 14061allows each shader to have its own region of the ``.data`` section. 14062Typically, this could be a table of buffer SRD's and the data pointed to 14063by the buffer SRD's, but it could be a flat-address region of memory as 14064well. Its layout and usage are defined by the shader compiler. 14065 14066Each shader's table in the ``.data`` section is referenced by the symbol 14067``_amdgpu_``\ *xs*\ ``_shdr_intrl_data`` where *xs* corresponds with the 14068hardware shader stage the data is for. E.g., 14069``_amdgpu_cs_shdr_intrl_data`` for the compute shader hardware stage. 14070 14071.. _amdgpu-amdpal-code-object-metadata-user-data-spill-table-section: 14072 14073Spill Table 14074########### 14075 14076It is possible for a hardware shader to need access to more *user data 14077entries* than there are slots available in user data registers for one 14078or more hardware shader stages. In that case, the PAL runtime expects 14079the necessary *user data entries* to be spilled to GPU memory and use 14080one user data register to point to the spilled user data memory. The 14081value of the *user data entry* must then represent the location where 14082a shader expects to read the low 32-bits of the table's GPU virtual 14083address. The *spill table* itself represents a set of 32-bit values 14084managed by the PAL runtime in GPU-accessible memory that can be made 14085indirectly accessible to a hardware shader. 14086 14087Unspecified OS 14088-------------- 14089 14090This section provides code conventions used when the target triple OS is 14091empty (see :ref:`amdgpu-target-triples`). 14092 14093Trap Handler ABI 14094~~~~~~~~~~~~~~~~ 14095 14096For code objects generated by AMDGPU backend for non-amdhsa OS, the runtime does 14097not install a trap handler. The ``llvm.trap`` and ``llvm.debugtrap`` 14098instructions are handled as follows: 14099 14100 .. table:: AMDGPU Trap Handler for Non-AMDHSA OS 14101 :name: amdgpu-trap-handler-for-non-amdhsa-os-table 14102 14103 =============== =============== =========================================== 14104 Usage Code Sequence Description 14105 =============== =============== =========================================== 14106 llvm.trap s_endpgm Causes wavefront to be terminated. 14107 llvm.debugtrap *none* Compiler warning given that there is no 14108 trap handler installed. 14109 =============== =============== =========================================== 14110 14111Source Languages 14112================ 14113 14114.. _amdgpu-opencl: 14115 14116OpenCL 14117------ 14118 14119When the language is OpenCL the following differences occur: 14120 141211. The OpenCL memory model is used (see :ref:`amdgpu-amdhsa-memory-model`). 141222. The AMDGPU backend appends additional arguments to the kernel's explicit 14123 arguments for the AMDHSA OS (see 14124 :ref:`opencl-kernel-implicit-arguments-appended-for-amdhsa-os-table`). 141253. Additional metadata is generated 14126 (see :ref:`amdgpu-amdhsa-code-object-metadata`). 14127 14128 .. table:: OpenCL kernel implicit arguments appended for AMDHSA OS 14129 :name: opencl-kernel-implicit-arguments-appended-for-amdhsa-os-table 14130 14131 ======== ==== ========= =========================================== 14132 Position Byte Byte Description 14133 Size Alignment 14134 ======== ==== ========= =========================================== 14135 1 8 8 OpenCL Global Offset X 14136 2 8 8 OpenCL Global Offset Y 14137 3 8 8 OpenCL Global Offset Z 14138 4 8 8 OpenCL address of printf buffer 14139 5 8 8 OpenCL address of virtual queue used by 14140 enqueue_kernel. 14141 6 8 8 OpenCL address of AqlWrap struct used by 14142 enqueue_kernel. 14143 7 8 8 Pointer argument used for Multi-gird 14144 synchronization. 14145 ======== ==== ========= =========================================== 14146 14147.. _amdgpu-hcc: 14148 14149HCC 14150--- 14151 14152When the language is HCC the following differences occur: 14153 141541. The HSA memory model is used (see :ref:`amdgpu-amdhsa-memory-model`). 14155 14156.. _amdgpu-assembler: 14157 14158Assembler 14159--------- 14160 14161AMDGPU backend has LLVM-MC based assembler which is currently in development. 14162It supports AMDGCN GFX6-GFX10. 14163 14164This section describes general syntax for instructions and operands. 14165 14166Instructions 14167~~~~~~~~~~~~ 14168 14169An instruction has the following :doc:`syntax<AMDGPUInstructionSyntax>`: 14170 14171 | ``<``\ *opcode*\ ``> <``\ *operand0*\ ``>, <``\ *operand1*\ ``>,... 14172 <``\ *modifier0*\ ``> <``\ *modifier1*\ ``>...`` 14173 14174:doc:`Operands<AMDGPUOperandSyntax>` are comma-separated while 14175:doc:`modifiers<AMDGPUModifierSyntax>` are space-separated. 14176 14177The order of operands and modifiers is fixed. 14178Most modifiers are optional and may be omitted. 14179 14180Links to detailed instruction syntax description may be found in the following 14181table. Note that features under development are not included 14182in this description. 14183 14184 ============= ============================================= ======================================= 14185 Architecture Core ISA ISA Variants and Extensions 14186 ============= ============================================= ======================================= 14187 GCN 2 :doc:`GFX7<AMDGPU/AMDGPUAsmGFX7>` \- 14188 GCN 3, GCN 4 :doc:`GFX8<AMDGPU/AMDGPUAsmGFX8>` \- 14189 GCN 5 :doc:`GFX9<AMDGPU/AMDGPUAsmGFX9>` :doc:`gfx900<AMDGPU/AMDGPUAsmGFX900>` 14190 14191 :doc:`gfx902<AMDGPU/AMDGPUAsmGFX900>` 14192 14193 :doc:`gfx904<AMDGPU/AMDGPUAsmGFX904>` 14194 14195 :doc:`gfx906<AMDGPU/AMDGPUAsmGFX906>` 14196 14197 :doc:`gfx909<AMDGPU/AMDGPUAsmGFX900>` 14198 14199 CDNA 1 :doc:`GFX9<AMDGPU/AMDGPUAsmGFX9>` :doc:`gfx908<AMDGPU/AMDGPUAsmGFX908>` 14200 CDNA 2 :doc:`GFX9<AMDGPU/AMDGPUAsmGFX9>` :doc:`gfx90a<AMDGPU/AMDGPUAsmGFX90a>` 14201 RDNA 1 :doc:`GFX10 RDNA1<AMDGPU/AMDGPUAsmGFX10>` :doc:`gfx1010<AMDGPU/AMDGPUAsmGFX10>` 14202 14203 :doc:`gfx1011<AMDGPU/AMDGPUAsmGFX1011>` 14204 14205 :doc:`gfx1012<AMDGPU/AMDGPUAsmGFX1011>` 14206 14207 RDNA 2 :doc:`GFX10 RDNA2<AMDGPU/AMDGPUAsmGFX1030>` :doc:`gfx1030<AMDGPU/AMDGPUAsmGFX1030>` 14208 14209 :doc:`gfx1031<AMDGPU/AMDGPUAsmGFX1030>` 14210 14211 :doc:`gfx1032<AMDGPU/AMDGPUAsmGFX1030>` 14212 14213 :doc:`gfx1033<AMDGPU/AMDGPUAsmGFX1030>` 14214 14215 :doc:`gfx1034<AMDGPU/AMDGPUAsmGFX1030>` 14216 14217 :doc:`gfx1035<AMDGPU/AMDGPUAsmGFX1030>` 14218 14219 :doc:`gfx1036<AMDGPU/AMDGPUAsmGFX1030>` 14220 ============= ============================================= ======================================= 14221 14222For more information about instructions, their semantics and supported 14223combinations of operands, refer to one of instruction set architecture manuals 14224[AMD-GCN-GFX6]_, [AMD-GCN-GFX7]_, [AMD-GCN-GFX8]_, 14225[AMD-GCN-GFX900-GFX904-VEGA]_, [AMD-GCN-GFX906-VEGA7NM]_, 14226[AMD-GCN-GFX908-CDNA1]_, [AMD-GCN-GFX90A-CDNA2]_, [AMD-GCN-GFX10-RDNA1]_ and 14227[AMD-GCN-GFX10-RDNA2]_. 14228 14229Operands 14230~~~~~~~~ 14231 14232Detailed description of operands may be found :doc:`here<AMDGPUOperandSyntax>`. 14233 14234Modifiers 14235~~~~~~~~~ 14236 14237Detailed description of modifiers may be found 14238:doc:`here<AMDGPUModifierSyntax>`. 14239 14240Instruction Examples 14241~~~~~~~~~~~~~~~~~~~~ 14242 14243DS 14244++ 14245 14246.. code-block:: nasm 14247 14248 ds_add_u32 v2, v4 offset:16 14249 ds_write_src2_b64 v2 offset0:4 offset1:8 14250 ds_cmpst_f32 v2, v4, v6 14251 ds_min_rtn_f64 v[8:9], v2, v[4:5] 14252 14253For full list of supported instructions, refer to "LDS/GDS instructions" in ISA 14254Manual. 14255 14256FLAT 14257++++ 14258 14259.. code-block:: nasm 14260 14261 flat_load_dword v1, v[3:4] 14262 flat_store_dwordx3 v[3:4], v[5:7] 14263 flat_atomic_swap v1, v[3:4], v5 glc 14264 flat_atomic_cmpswap v1, v[3:4], v[5:6] glc slc 14265 flat_atomic_fmax_x2 v[1:2], v[3:4], v[5:6] glc 14266 14267For full list of supported instructions, refer to "FLAT instructions" in ISA 14268Manual. 14269 14270MUBUF 14271+++++ 14272 14273.. code-block:: nasm 14274 14275 buffer_load_dword v1, off, s[4:7], s1 14276 buffer_store_dwordx4 v[1:4], v2, ttmp[4:7], s1 offen offset:4 glc tfe 14277 buffer_store_format_xy v[1:2], off, s[4:7], s1 14278 buffer_wbinvl1 14279 buffer_atomic_inc v1, v2, s[8:11], s4 idxen offset:4 slc 14280 14281For full list of supported instructions, refer to "MUBUF Instructions" in ISA 14282Manual. 14283 14284SMRD/SMEM 14285+++++++++ 14286 14287.. code-block:: nasm 14288 14289 s_load_dword s1, s[2:3], 0xfc 14290 s_load_dwordx8 s[8:15], s[2:3], s4 14291 s_load_dwordx16 s[88:103], s[2:3], s4 14292 s_dcache_inv_vol 14293 s_memtime s[4:5] 14294 14295For full list of supported instructions, refer to "Scalar Memory Operations" in 14296ISA Manual. 14297 14298SOP1 14299++++ 14300 14301.. code-block:: nasm 14302 14303 s_mov_b32 s1, s2 14304 s_mov_b64 s[0:1], 0x80000000 14305 s_cmov_b32 s1, 200 14306 s_wqm_b64 s[2:3], s[4:5] 14307 s_bcnt0_i32_b64 s1, s[2:3] 14308 s_swappc_b64 s[2:3], s[4:5] 14309 s_cbranch_join s[4:5] 14310 14311For full list of supported instructions, refer to "SOP1 Instructions" in ISA 14312Manual. 14313 14314SOP2 14315++++ 14316 14317.. code-block:: nasm 14318 14319 s_add_u32 s1, s2, s3 14320 s_and_b64 s[2:3], s[4:5], s[6:7] 14321 s_cselect_b32 s1, s2, s3 14322 s_andn2_b32 s2, s4, s6 14323 s_lshr_b64 s[2:3], s[4:5], s6 14324 s_ashr_i32 s2, s4, s6 14325 s_bfm_b64 s[2:3], s4, s6 14326 s_bfe_i64 s[2:3], s[4:5], s6 14327 s_cbranch_g_fork s[4:5], s[6:7] 14328 14329For full list of supported instructions, refer to "SOP2 Instructions" in ISA 14330Manual. 14331 14332SOPC 14333++++ 14334 14335.. code-block:: nasm 14336 14337 s_cmp_eq_i32 s1, s2 14338 s_bitcmp1_b32 s1, s2 14339 s_bitcmp0_b64 s[2:3], s4 14340 s_setvskip s3, s5 14341 14342For full list of supported instructions, refer to "SOPC Instructions" in ISA 14343Manual. 14344 14345SOPP 14346++++ 14347 14348.. code-block:: nasm 14349 14350 s_barrier 14351 s_nop 2 14352 s_endpgm 14353 s_waitcnt 0 ; Wait for all counters to be 0 14354 s_waitcnt vmcnt(0) & expcnt(0) & lgkmcnt(0) ; Equivalent to above 14355 s_waitcnt vmcnt(1) ; Wait for vmcnt counter to be 1. 14356 s_sethalt 9 14357 s_sleep 10 14358 s_sendmsg 0x1 14359 s_sendmsg sendmsg(MSG_INTERRUPT) 14360 s_trap 1 14361 14362For full list of supported instructions, refer to "SOPP Instructions" in ISA 14363Manual. 14364 14365Unless otherwise mentioned, little verification is performed on the operands 14366of SOPP Instructions, so it is up to the programmer to be familiar with the 14367range or acceptable values. 14368 14369VALU 14370++++ 14371 14372For vector ALU instruction opcodes (VOP1, VOP2, VOP3, VOPC, VOP_DPP, VOP_SDWA), 14373the assembler will automatically use optimal encoding based on its operands. To 14374force specific encoding, one can add a suffix to the opcode of the instruction: 14375 14376* _e32 for 32-bit VOP1/VOP2/VOPC 14377* _e64 for 64-bit VOP3 14378* _dpp for VOP_DPP 14379* _sdwa for VOP_SDWA 14380 14381VOP1/VOP2/VOP3/VOPC examples: 14382 14383.. code-block:: nasm 14384 14385 v_mov_b32 v1, v2 14386 v_mov_b32_e32 v1, v2 14387 v_nop 14388 v_cvt_f64_i32_e32 v[1:2], v2 14389 v_floor_f32_e32 v1, v2 14390 v_bfrev_b32_e32 v1, v2 14391 v_add_f32_e32 v1, v2, v3 14392 v_mul_i32_i24_e64 v1, v2, 3 14393 v_mul_i32_i24_e32 v1, -3, v3 14394 v_mul_i32_i24_e32 v1, -100, v3 14395 v_addc_u32 v1, s[0:1], v2, v3, s[2:3] 14396 v_max_f16_e32 v1, v2, v3 14397 14398VOP_DPP examples: 14399 14400.. code-block:: nasm 14401 14402 v_mov_b32 v0, v0 quad_perm:[0,2,1,1] 14403 v_sin_f32 v0, v0 row_shl:1 row_mask:0xa bank_mask:0x1 bound_ctrl:0 14404 v_mov_b32 v0, v0 wave_shl:1 14405 v_mov_b32 v0, v0 row_mirror 14406 v_mov_b32 v0, v0 row_bcast:31 14407 v_mov_b32 v0, v0 quad_perm:[1,3,0,1] row_mask:0xa bank_mask:0x1 bound_ctrl:0 14408 v_add_f32 v0, v0, |v0| row_shl:1 row_mask:0xa bank_mask:0x1 bound_ctrl:0 14409 v_max_f16 v1, v2, v3 row_shl:1 row_mask:0xa bank_mask:0x1 bound_ctrl:0 14410 14411VOP_SDWA examples: 14412 14413.. code-block:: nasm 14414 14415 v_mov_b32 v1, v2 dst_sel:BYTE_0 dst_unused:UNUSED_PRESERVE src0_sel:DWORD 14416 v_min_u32 v200, v200, v1 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:BYTE_1 src1_sel:DWORD 14417 v_sin_f32 v0, v0 dst_unused:UNUSED_PAD src0_sel:WORD_1 14418 v_fract_f32 v0, |v0| dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 14419 v_cmpx_le_u32 vcc, v1, v2 src0_sel:BYTE_2 src1_sel:WORD_0 14420 14421For full list of supported instructions, refer to "Vector ALU instructions". 14422 14423.. _amdgpu-amdhsa-assembler-predefined-symbols-v2: 14424 14425Code Object V2 Predefined Symbols 14426~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 14427 14428.. warning:: 14429 Code object V2 is not the default code object version emitted by 14430 this version of LLVM. 14431 14432The AMDGPU assembler defines and updates some symbols automatically. These 14433symbols do not affect code generation. 14434 14435.option.machine_version_major 14436+++++++++++++++++++++++++++++ 14437 14438Set to the GFX major generation number of the target being assembled for. For 14439example, when assembling for a "GFX9" target this will be set to the integer 14440value "9". The possible GFX major generation numbers are presented in 14441:ref:`amdgpu-processors`. 14442 14443.option.machine_version_minor 14444+++++++++++++++++++++++++++++ 14445 14446Set to the GFX minor generation number of the target being assembled for. For 14447example, when assembling for a "GFX810" target this will be set to the integer 14448value "1". The possible GFX minor generation numbers are presented in 14449:ref:`amdgpu-processors`. 14450 14451.option.machine_version_stepping 14452++++++++++++++++++++++++++++++++ 14453 14454Set to the GFX stepping generation number of the target being assembled for. 14455For example, when assembling for a "GFX704" target this will be set to the 14456integer value "4". The possible GFX stepping generation numbers are presented 14457in :ref:`amdgpu-processors`. 14458 14459.kernel.vgpr_count 14460++++++++++++++++++ 14461 14462Set to zero each time a 14463:ref:`amdgpu-amdhsa-assembler-directive-amdgpu_hsa_kernel` directive is 14464encountered. At each instruction, if the current value of this symbol is less 14465than or equal to the maximum VGPR number explicitly referenced within that 14466instruction then the symbol value is updated to equal that VGPR number plus 14467one. 14468 14469.kernel.sgpr_count 14470++++++++++++++++++ 14471 14472Set to zero each time a 14473:ref:`amdgpu-amdhsa-assembler-directive-amdgpu_hsa_kernel` directive is 14474encountered. At each instruction, if the current value of this symbol is less 14475than or equal to the maximum VGPR number explicitly referenced within that 14476instruction then the symbol value is updated to equal that SGPR number plus 14477one. 14478 14479.. _amdgpu-amdhsa-assembler-directives-v2: 14480 14481Code Object V2 Directives 14482~~~~~~~~~~~~~~~~~~~~~~~~~ 14483 14484.. warning:: 14485 Code object V2 is not the default code object version emitted by 14486 this version of LLVM. 14487 14488AMDGPU ABI defines auxiliary data in output code object. In assembly source, 14489one can specify them with assembler directives. 14490 14491.hsa_code_object_version major, minor 14492+++++++++++++++++++++++++++++++++++++ 14493 14494*major* and *minor* are integers that specify the version of the HSA code 14495object that will be generated by the assembler. 14496 14497.hsa_code_object_isa [major, minor, stepping, vendor, arch] 14498+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 14499 14500 14501*major*, *minor*, and *stepping* are all integers that describe the instruction 14502set architecture (ISA) version of the assembly program. 14503 14504*vendor* and *arch* are quoted strings. *vendor* should always be equal to 14505"AMD" and *arch* should always be equal to "AMDGPU". 14506 14507By default, the assembler will derive the ISA version, *vendor*, and *arch* 14508from the value of the -mcpu option that is passed to the assembler. 14509 14510.. _amdgpu-amdhsa-assembler-directive-amdgpu_hsa_kernel: 14511 14512.amdgpu_hsa_kernel (name) 14513+++++++++++++++++++++++++ 14514 14515This directives specifies that the symbol with given name is a kernel entry 14516point (label) and the object should contain corresponding symbol of type 14517STT_AMDGPU_HSA_KERNEL. 14518 14519.amd_kernel_code_t 14520++++++++++++++++++ 14521 14522This directive marks the beginning of a list of key / value pairs that are used 14523to specify the amd_kernel_code_t object that will be emitted by the assembler. 14524The list must be terminated by the *.end_amd_kernel_code_t* directive. For any 14525amd_kernel_code_t values that are unspecified a default value will be used. The 14526default value for all keys is 0, with the following exceptions: 14527 14528- *amd_code_version_major* defaults to 1. 14529- *amd_kernel_code_version_minor* defaults to 2. 14530- *amd_machine_kind* defaults to 1. 14531- *amd_machine_version_major*, *machine_version_minor*, and 14532 *amd_machine_version_stepping* are derived from the value of the -mcpu option 14533 that is passed to the assembler. 14534- *kernel_code_entry_byte_offset* defaults to 256. 14535- *wavefront_size* defaults 6 for all targets before GFX10. For GFX10 onwards 14536 defaults to 6 if target feature ``wavefrontsize64`` is enabled, otherwise 5. 14537 Note that wavefront size is specified as a power of two, so a value of **n** 14538 means a size of 2^ **n**. 14539- *call_convention* defaults to -1. 14540- *kernarg_segment_alignment*, *group_segment_alignment*, and 14541 *private_segment_alignment* default to 4. Note that alignments are specified 14542 as a power of 2, so a value of **n** means an alignment of 2^ **n**. 14543- *enable_tg_split* defaults to 1 if target feature ``tgsplit`` is enabled for 14544 GFX90A onwards. 14545- *enable_wgp_mode* defaults to 1 if target feature ``cumode`` is disabled for 14546 GFX10 onwards. 14547- *enable_mem_ordered* defaults to 1 for GFX10 onwards. 14548 14549The *.amd_kernel_code_t* directive must be placed immediately after the 14550function label and before any instructions. 14551 14552For a full list of amd_kernel_code_t keys, refer to AMDGPU ABI document, 14553comments in lib/Target/AMDGPU/AmdKernelCodeT.h and test/CodeGen/AMDGPU/hsa.s. 14554 14555.. _amdgpu-amdhsa-assembler-example-v2: 14556 14557Code Object V2 Example Source Code 14558~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 14559 14560.. warning:: 14561 Code Object V2 is not the default code object version emitted by 14562 this version of LLVM. 14563 14564Here is an example of a minimal assembly source file, defining one HSA kernel: 14565 14566.. code:: 14567 :number-lines: 14568 14569 .hsa_code_object_version 1,0 14570 .hsa_code_object_isa 14571 14572 .hsatext 14573 .globl hello_world 14574 .p2align 8 14575 .amdgpu_hsa_kernel hello_world 14576 14577 hello_world: 14578 14579 .amd_kernel_code_t 14580 enable_sgpr_kernarg_segment_ptr = 1 14581 is_ptr64 = 1 14582 compute_pgm_rsrc1_vgprs = 0 14583 compute_pgm_rsrc1_sgprs = 0 14584 compute_pgm_rsrc2_user_sgpr = 2 14585 compute_pgm_rsrc1_wgp_mode = 0 14586 compute_pgm_rsrc1_mem_ordered = 0 14587 compute_pgm_rsrc1_fwd_progress = 1 14588 .end_amd_kernel_code_t 14589 14590 s_load_dwordx2 s[0:1], s[0:1] 0x0 14591 v_mov_b32 v0, 3.14159 14592 s_waitcnt lgkmcnt(0) 14593 v_mov_b32 v1, s0 14594 v_mov_b32 v2, s1 14595 flat_store_dword v[1:2], v0 14596 s_endpgm 14597 .Lfunc_end0: 14598 .size hello_world, .Lfunc_end0-hello_world 14599 14600.. _amdgpu-amdhsa-assembler-predefined-symbols-v3-onwards: 14601 14602Code Object V3 and Above Predefined Symbols 14603~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 14604 14605The AMDGPU assembler defines and updates some symbols automatically. These 14606symbols do not affect code generation. 14607 14608.amdgcn.gfx_generation_number 14609+++++++++++++++++++++++++++++ 14610 14611Set to the GFX major generation number of the target being assembled for. For 14612example, when assembling for a "GFX9" target this will be set to the integer 14613value "9". The possible GFX major generation numbers are presented in 14614:ref:`amdgpu-processors`. 14615 14616.amdgcn.gfx_generation_minor 14617++++++++++++++++++++++++++++ 14618 14619Set to the GFX minor generation number of the target being assembled for. For 14620example, when assembling for a "GFX810" target this will be set to the integer 14621value "1". The possible GFX minor generation numbers are presented in 14622:ref:`amdgpu-processors`. 14623 14624.amdgcn.gfx_generation_stepping 14625+++++++++++++++++++++++++++++++ 14626 14627Set to the GFX stepping generation number of the target being assembled for. 14628For example, when assembling for a "GFX704" target this will be set to the 14629integer value "4". The possible GFX stepping generation numbers are presented 14630in :ref:`amdgpu-processors`. 14631 14632.. _amdgpu-amdhsa-assembler-symbol-next_free_vgpr: 14633 14634.amdgcn.next_free_vgpr 14635++++++++++++++++++++++ 14636 14637Set to zero before assembly begins. At each instruction, if the current value 14638of this symbol is less than or equal to the maximum VGPR number explicitly 14639referenced within that instruction then the symbol value is updated to equal 14640that VGPR number plus one. 14641 14642May be used to set the `.amdhsa_next_free_vgpr` directive in 14643:ref:`amdhsa-kernel-directives-table`. 14644 14645May be set at any time, e.g. manually set to zero at the start of each kernel. 14646 14647.. _amdgpu-amdhsa-assembler-symbol-next_free_sgpr: 14648 14649.amdgcn.next_free_sgpr 14650++++++++++++++++++++++ 14651 14652Set to zero before assembly begins. At each instruction, if the current value 14653of this symbol is less than or equal the maximum SGPR number explicitly 14654referenced within that instruction then the symbol value is updated to equal 14655that SGPR number plus one. 14656 14657May be used to set the `.amdhsa_next_free_spgr` directive in 14658:ref:`amdhsa-kernel-directives-table`. 14659 14660May be set at any time, e.g. manually set to zero at the start of each kernel. 14661 14662.. _amdgpu-amdhsa-assembler-directives-v3-onwards: 14663 14664Code Object V3 and Above Directives 14665~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 14666 14667Directives which begin with ``.amdgcn`` are valid for all ``amdgcn`` 14668architecture processors, and are not OS-specific. Directives which begin with 14669``.amdhsa`` are specific to ``amdgcn`` architecture processors when the 14670``amdhsa`` OS is specified. See :ref:`amdgpu-target-triples` and 14671:ref:`amdgpu-processors`. 14672 14673.. _amdgpu-assembler-directive-amdgcn-target: 14674 14675.amdgcn_target <target-triple> "-" <target-id> 14676++++++++++++++++++++++++++++++++++++++++++++++ 14677 14678Optional directive which declares the ``<target-triple>-<target-id>`` supported 14679by the containing assembler source file. Used by the assembler to validate 14680command-line options such as ``-triple``, ``-mcpu``, and 14681``--offload-arch=<target-id>``. A non-canonical target ID is allowed. See 14682:ref:`amdgpu-target-triples` and :ref:`amdgpu-target-id`. 14683 14684.. note:: 14685 14686 The target ID syntax used for code object V2 to V3 for this directive differs 14687 from that used elsewhere. See :ref:`amdgpu-target-id-v2-v3`. 14688 14689.amdhsa_kernel <name> 14690+++++++++++++++++++++ 14691 14692Creates a correctly aligned AMDHSA kernel descriptor and a symbol, 14693``<name>.kd``, in the current location of the current section. Only valid when 14694the OS is ``amdhsa``. ``<name>`` must be a symbol that labels the first 14695instruction to execute, and does not need to be previously defined. 14696 14697Marks the beginning of a list of directives used to generate the bytes of a 14698kernel descriptor, as described in :ref:`amdgpu-amdhsa-kernel-descriptor`. 14699Directives which may appear in this list are described in 14700:ref:`amdhsa-kernel-directives-table`. Directives may appear in any order, must 14701be valid for the target being assembled for, and cannot be repeated. Directives 14702support the range of values specified by the field they reference in 14703:ref:`amdgpu-amdhsa-kernel-descriptor`. If a directive is not specified, it is 14704assumed to have its default value, unless it is marked as "Required", in which 14705case it is an error to omit the directive. This list of directives is 14706terminated by an ``.end_amdhsa_kernel`` directive. 14707 14708 .. table:: AMDHSA Kernel Assembler Directives 14709 :name: amdhsa-kernel-directives-table 14710 14711 ======================================================== =================== ============ =================== 14712 Directive Default Supported On Description 14713 ======================================================== =================== ============ =================== 14714 ``.amdhsa_group_segment_fixed_size`` 0 GFX6-GFX10 Controls GROUP_SEGMENT_FIXED_SIZE in 14715 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 14716 ``.amdhsa_private_segment_fixed_size`` 0 GFX6-GFX10 Controls PRIVATE_SEGMENT_FIXED_SIZE in 14717 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 14718 ``.amdhsa_kernarg_size`` 0 GFX6-GFX10 Controls KERNARG_SIZE in 14719 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 14720 ``.amdhsa_user_sgpr_count`` 0 GFX6-GFX10 Controls USER_SGPR_COUNT in COMPUTE_PGM_RSRC2 14721 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table` 14722 ``.amdhsa_user_sgpr_private_segment_buffer`` 0 GFX6-GFX10 Controls ENABLE_SGPR_PRIVATE_SEGMENT_BUFFER in 14723 (except :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 14724 GFX940) 14725 ``.amdhsa_user_sgpr_dispatch_ptr`` 0 GFX6-GFX10 Controls ENABLE_SGPR_DISPATCH_PTR in 14726 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 14727 ``.amdhsa_user_sgpr_queue_ptr`` 0 GFX6-GFX10 Controls ENABLE_SGPR_QUEUE_PTR in 14728 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 14729 ``.amdhsa_user_sgpr_kernarg_segment_ptr`` 0 GFX6-GFX10 Controls ENABLE_SGPR_KERNARG_SEGMENT_PTR in 14730 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 14731 ``.amdhsa_user_sgpr_dispatch_id`` 0 GFX6-GFX10 Controls ENABLE_SGPR_DISPATCH_ID in 14732 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 14733 ``.amdhsa_user_sgpr_flat_scratch_init`` 0 GFX6-GFX10 Controls ENABLE_SGPR_FLAT_SCRATCH_INIT in 14734 (except :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 14735 GFX940) 14736 ``.amdhsa_user_sgpr_private_segment_size`` 0 GFX6-GFX10 Controls ENABLE_SGPR_PRIVATE_SEGMENT_SIZE in 14737 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 14738 ``.amdhsa_wavefront_size32`` Target GFX10 Controls ENABLE_WAVEFRONT_SIZE32 in 14739 Feature :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 14740 Specific 14741 (wavefrontsize64) 14742 ``.amdhsa_system_sgpr_private_segment_wavefront_offset`` 0 GFX6-GFX10 Controls ENABLE_PRIVATE_SEGMENT in 14743 (except :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 14744 GFX940) 14745 ``.amdhsa_enable_private_segment`` 0 GFX940 Controls ENABLE_PRIVATE_SEGMENT in 14746 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 14747 ``.amdhsa_system_sgpr_workgroup_id_x`` 1 GFX6-GFX10 Controls ENABLE_SGPR_WORKGROUP_ID_X in 14748 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 14749 ``.amdhsa_system_sgpr_workgroup_id_y`` 0 GFX6-GFX10 Controls ENABLE_SGPR_WORKGROUP_ID_Y in 14750 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 14751 ``.amdhsa_system_sgpr_workgroup_id_z`` 0 GFX6-GFX10 Controls ENABLE_SGPR_WORKGROUP_ID_Z in 14752 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 14753 ``.amdhsa_system_sgpr_workgroup_info`` 0 GFX6-GFX10 Controls ENABLE_SGPR_WORKGROUP_INFO in 14754 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 14755 ``.amdhsa_system_vgpr_workitem_id`` 0 GFX6-GFX10 Controls ENABLE_VGPR_WORKITEM_ID in 14756 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 14757 Possible values are defined in 14758 :ref:`amdgpu-amdhsa-system-vgpr-work-item-id-enumeration-values-table`. 14759 ``.amdhsa_next_free_vgpr`` Required GFX6-GFX10 Maximum VGPR number explicitly referenced, plus one. 14760 Used to calculate GRANULATED_WORKITEM_VGPR_COUNT in 14761 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 14762 ``.amdhsa_next_free_sgpr`` Required GFX6-GFX10 Maximum SGPR number explicitly referenced, plus one. 14763 Used to calculate GRANULATED_WAVEFRONT_SGPR_COUNT in 14764 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 14765 ``.amdhsa_accum_offset`` Required GFX90A, Offset of a first AccVGPR in the unified register file. 14766 GFX940 Used to calculate ACCUM_OFFSET in 14767 :ref:`amdgpu-amdhsa-compute_pgm_rsrc3-gfx90a-table`. 14768 ``.amdhsa_reserve_vcc`` 1 GFX6-GFX10 Whether the kernel may use the special VCC SGPR. 14769 Used to calculate GRANULATED_WAVEFRONT_SGPR_COUNT in 14770 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 14771 ``.amdhsa_reserve_flat_scratch`` 1 GFX7-GFX10 Whether the kernel may use flat instructions to access 14772 (except scratch memory. Used to calculate 14773 GFX940) GRANULATED_WAVEFRONT_SGPR_COUNT in 14774 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 14775 ``.amdhsa_reserve_xnack_mask`` Target GFX8-GFX10 Whether the kernel may trigger XNACK replay. 14776 Feature Used to calculate GRANULATED_WAVEFRONT_SGPR_COUNT in 14777 Specific :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 14778 (xnack) 14779 ``.amdhsa_float_round_mode_32`` 0 GFX6-GFX10 Controls FLOAT_ROUND_MODE_32 in 14780 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 14781 Possible values are defined in 14782 :ref:`amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table`. 14783 ``.amdhsa_float_round_mode_16_64`` 0 GFX6-GFX10 Controls FLOAT_ROUND_MODE_16_64 in 14784 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 14785 Possible values are defined in 14786 :ref:`amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table`. 14787 ``.amdhsa_float_denorm_mode_32`` 0 GFX6-GFX10 Controls FLOAT_DENORM_MODE_32 in 14788 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 14789 Possible values are defined in 14790 :ref:`amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table`. 14791 ``.amdhsa_float_denorm_mode_16_64`` 3 GFX6-GFX10 Controls FLOAT_DENORM_MODE_16_64 in 14792 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 14793 Possible values are defined in 14794 :ref:`amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table`. 14795 ``.amdhsa_dx10_clamp`` 1 GFX6-GFX10 Controls ENABLE_DX10_CLAMP in 14796 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 14797 ``.amdhsa_ieee_mode`` 1 GFX6-GFX10 Controls ENABLE_IEEE_MODE in 14798 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 14799 ``.amdhsa_fp16_overflow`` 0 GFX9-GFX10 Controls FP16_OVFL in 14800 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 14801 ``.amdhsa_tg_split`` Target GFX90A, Controls TG_SPLIT in 14802 Feature GFX940 :ref:`amdgpu-amdhsa-compute_pgm_rsrc3-gfx90a-table`. 14803 Specific 14804 (tgsplit) 14805 ``.amdhsa_workgroup_processor_mode`` Target GFX10 Controls ENABLE_WGP_MODE in 14806 Feature :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 14807 Specific 14808 (cumode) 14809 ``.amdhsa_memory_ordered`` 1 GFX10 Controls MEM_ORDERED in 14810 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 14811 ``.amdhsa_forward_progress`` 0 GFX10 Controls FWD_PROGRESS in 14812 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 14813 ``.amdhsa_shared_vgpr_count`` 0 GFX10 Controls SHARED_VGPR_COUNT in 14814 :ref:`amdgpu-amdhsa-compute_pgm_rsrc3-gfx10-table`. 14815 ``.amdhsa_exception_fp_ieee_invalid_op`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_INVALID_OPERATION in 14816 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 14817 ``.amdhsa_exception_fp_denorm_src`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_FP_DENORMAL_SOURCE in 14818 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 14819 ``.amdhsa_exception_fp_ieee_div_zero`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_DIVISION_BY_ZERO in 14820 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 14821 ``.amdhsa_exception_fp_ieee_overflow`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_OVERFLOW in 14822 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 14823 ``.amdhsa_exception_fp_ieee_underflow`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_UNDERFLOW in 14824 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 14825 ``.amdhsa_exception_fp_ieee_inexact`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_INEXACT in 14826 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 14827 ``.amdhsa_exception_int_div_zero`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_INT_DIVIDE_BY_ZERO in 14828 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 14829 ======================================================== =================== ============ =================== 14830 14831.amdgpu_metadata 14832++++++++++++++++ 14833 14834Optional directive which declares the contents of the ``NT_AMDGPU_METADATA`` 14835note record (see :ref:`amdgpu-elf-note-records-table-v3-onwards`). 14836 14837The contents must be in the [YAML]_ markup format, with the same structure and 14838semantics described in :ref:`amdgpu-amdhsa-code-object-metadata-v3`, 14839:ref:`amdgpu-amdhsa-code-object-metadata-v4` or 14840:ref:`amdgpu-amdhsa-code-object-metadata-v5`. 14841 14842This directive is terminated by an ``.end_amdgpu_metadata`` directive. 14843 14844.. _amdgpu-amdhsa-assembler-example-v3-onwards: 14845 14846Code Object V3 and Above Example Source Code 14847~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 14848 14849Here is an example of a minimal assembly source file, defining one HSA kernel: 14850 14851.. code:: 14852 :number-lines: 14853 14854 .amdgcn_target "amdgcn-amd-amdhsa--gfx900+xnack" // optional 14855 14856 .text 14857 .globl hello_world 14858 .p2align 8 14859 .type hello_world,@function 14860 hello_world: 14861 s_load_dwordx2 s[0:1], s[0:1] 0x0 14862 v_mov_b32 v0, 3.14159 14863 s_waitcnt lgkmcnt(0) 14864 v_mov_b32 v1, s0 14865 v_mov_b32 v2, s1 14866 flat_store_dword v[1:2], v0 14867 s_endpgm 14868 .Lfunc_end0: 14869 .size hello_world, .Lfunc_end0-hello_world 14870 14871 .rodata 14872 .p2align 6 14873 .amdhsa_kernel hello_world 14874 .amdhsa_user_sgpr_kernarg_segment_ptr 1 14875 .amdhsa_next_free_vgpr .amdgcn.next_free_vgpr 14876 .amdhsa_next_free_sgpr .amdgcn.next_free_sgpr 14877 .end_amdhsa_kernel 14878 14879 .amdgpu_metadata 14880 --- 14881 amdhsa.version: 14882 - 1 14883 - 0 14884 amdhsa.kernels: 14885 - .name: hello_world 14886 .symbol: hello_world.kd 14887 .kernarg_segment_size: 48 14888 .group_segment_fixed_size: 0 14889 .private_segment_fixed_size: 0 14890 .kernarg_segment_align: 4 14891 .wavefront_size: 64 14892 .sgpr_count: 2 14893 .vgpr_count: 3 14894 .max_flat_workgroup_size: 256 14895 .args: 14896 - .size: 8 14897 .offset: 0 14898 .value_kind: global_buffer 14899 .address_space: global 14900 .actual_access: write_only 14901 //... 14902 .end_amdgpu_metadata 14903 14904This kernel is equivalent to the following HIP program: 14905 14906.. code:: 14907 :number-lines: 14908 14909 __global__ void hello_world(float *p) { 14910 *p = 3.14159f; 14911 } 14912 14913If an assembly source file contains multiple kernels and/or functions, the 14914:ref:`amdgpu-amdhsa-assembler-symbol-next_free_vgpr` and 14915:ref:`amdgpu-amdhsa-assembler-symbol-next_free_sgpr` symbols may be reset using 14916the ``.set <symbol>, <expression>`` directive. For example, in the case of two 14917kernels, where ``function1`` is only called from ``kernel1`` it is sufficient 14918to group the function with the kernel that calls it and reset the symbols 14919between the two connected components: 14920 14921.. code:: 14922 :number-lines: 14923 14924 .amdgcn_target "amdgcn-amd-amdhsa--gfx900+xnack" // optional 14925 14926 // gpr tracking symbols are implicitly set to zero 14927 14928 .text 14929 .globl kern0 14930 .p2align 8 14931 .type kern0,@function 14932 kern0: 14933 // ... 14934 s_endpgm 14935 .Lkern0_end: 14936 .size kern0, .Lkern0_end-kern0 14937 14938 .rodata 14939 .p2align 6 14940 .amdhsa_kernel kern0 14941 // ... 14942 .amdhsa_next_free_vgpr .amdgcn.next_free_vgpr 14943 .amdhsa_next_free_sgpr .amdgcn.next_free_sgpr 14944 .end_amdhsa_kernel 14945 14946 // reset symbols to begin tracking usage in func1 and kern1 14947 .set .amdgcn.next_free_vgpr, 0 14948 .set .amdgcn.next_free_sgpr, 0 14949 14950 .text 14951 .hidden func1 14952 .global func1 14953 .p2align 2 14954 .type func1,@function 14955 func1: 14956 // ... 14957 s_setpc_b64 s[30:31] 14958 .Lfunc1_end: 14959 .size func1, .Lfunc1_end-func1 14960 14961 .globl kern1 14962 .p2align 8 14963 .type kern1,@function 14964 kern1: 14965 // ... 14966 s_getpc_b64 s[4:5] 14967 s_add_u32 s4, s4, func1@rel32@lo+4 14968 s_addc_u32 s5, s5, func1@rel32@lo+4 14969 s_swappc_b64 s[30:31], s[4:5] 14970 // ... 14971 s_endpgm 14972 .Lkern1_end: 14973 .size kern1, .Lkern1_end-kern1 14974 14975 .rodata 14976 .p2align 6 14977 .amdhsa_kernel kern1 14978 // ... 14979 .amdhsa_next_free_vgpr .amdgcn.next_free_vgpr 14980 .amdhsa_next_free_sgpr .amdgcn.next_free_sgpr 14981 .end_amdhsa_kernel 14982 14983These symbols cannot identify connected components in order to automatically 14984track the usage for each kernel. However, in some cases careful organization of 14985the kernels and functions in the source file means there is minimal additional 14986effort required to accurately calculate GPR usage. 14987 14988Additional Documentation 14989======================== 14990 14991.. [AMD-GCN-GFX6] `AMD Southern Islands Series ISA <http://developer.amd.com/wordpress/media/2012/12/AMD_Southern_Islands_Instruction_Set_Architecture.pdf>`__ 14992.. [AMD-GCN-GFX7] `AMD Sea Islands Series ISA <http://developer.amd.com/wordpress/media/2013/07/AMD_Sea_Islands_Instruction_Set_Architecture.pdf>`_ 14993.. [AMD-GCN-GFX8] `AMD GCN3 Instruction Set Architecture <http://amd-dev.wpengine.netdna-cdn.com/wordpress/media/2013/12/AMD_GCN3_Instruction_Set_Architecture_rev1.1.pdf>`__ 14994.. [AMD-GCN-GFX900-GFX904-VEGA] `AMD Vega Instruction Set Architecture <http://developer.amd.com/wordpress/media/2013/12/Vega_Shader_ISA_28July2017.pdf>`__ 14995.. [AMD-GCN-GFX906-VEGA7NM] `AMD Vega 7nm Instruction Set Architecture <https://gpuopen.com/wp-content/uploads/2019/11/Vega_7nm_Shader_ISA_26November2019.pdf>`__ 14996.. [AMD-GCN-GFX908-CDNA1] `AMD Instinct MI100 Instruction Set Architecture <https://developer.amd.com/wp-content/resources/CDNA1_Shader_ISA_14December2020.pdf>`__ 14997.. [AMD-GCN-GFX90A-CDNA2] `AMD Instinct MI200 Instruction Set Architecture <https://developer.amd.com/wp-content/resources/CDNA2_Shader_ISA_4February2022.pdf>`__ 14998.. [AMD-GCN-GFX10-RDNA1] `AMD RDNA 1.0 Instruction Set Architecture <https://gpuopen.com/wp-content/uploads/2019/08/RDNA_Shader_ISA_5August2019.pdf>`__ 14999.. [AMD-GCN-GFX10-RDNA2] `AMD RDNA 2 Instruction Set Architecture <https://developer.amd.com/wp-content/resources/RDNA2_Shader_ISA_November2020.pdf>`__ 15000.. [AMD-RADEON-HD-2000-3000] `AMD R6xx shader ISA <http://developer.amd.com/wordpress/media/2012/10/R600_Instruction_Set_Architecture.pdf>`__ 15001.. [AMD-RADEON-HD-4000] `AMD R7xx shader ISA <http://developer.amd.com/wordpress/media/2012/10/R700-Family_Instruction_Set_Architecture.pdf>`__ 15002.. [AMD-RADEON-HD-5000] `AMD Evergreen shader ISA <http://developer.amd.com/wordpress/media/2012/10/AMD_Evergreen-Family_Instruction_Set_Architecture.pdf>`__ 15003.. [AMD-RADEON-HD-6000] `AMD Cayman/Trinity shader ISA <http://developer.amd.com/wordpress/media/2012/10/AMD_HD_6900_Series_Instruction_Set_Architecture.pdf>`__ 15004.. [AMD-ROCm] `AMD ROCm™ Platform <https://rocmdocs.amd.com/>`__ 15005.. [AMD-ROCm-github] `AMD ROCm™ github <http://github.com/RadeonOpenCompute>`__ 15006.. [AMD-ROCm-Release-Notes] `AMD ROCm Release Notes <https://github.com/RadeonOpenCompute/ROCm>`__ 15007.. [CLANG-ATTR] `Attributes in Clang <https://clang.llvm.org/docs/AttributeReference.html>`__ 15008.. [DWARF] `DWARF Debugging Information Format <http://dwarfstd.org/>`__ 15009.. [ELF] `Executable and Linkable Format (ELF) <http://www.sco.com/developers/gabi/>`__ 15010.. [HRF] `Heterogeneous-race-free Memory Models <http://benedictgaster.org/wp-content/uploads/2014/01/asplos269-FINAL.pdf>`__ 15011.. [HSA] `Heterogeneous System Architecture (HSA) Foundation <http://www.hsafoundation.com/>`__ 15012.. [MsgPack] `Message Pack <http://www.msgpack.org/>`__ 15013.. [OpenCL] `The OpenCL Specification Version 2.0 <http://www.khronos.org/registry/cl/specs/opencl-2.0.pdf>`__ 15014.. [SEMVER] `Semantic Versioning <https://semver.org/>`__ 15015.. [YAML] `YAML Ain't Markup Language (YAML™) Version 1.2 <http://www.yaml.org/spec/1.2/spec.html>`__ 15016