1============================= 2User Guide for AMDGPU Backend 3============================= 4 5.. contents:: 6 :local: 7 8.. toctree:: 9 :hidden: 10 11 AMDGPU/AMDGPUAsmGFX7 12 AMDGPU/AMDGPUAsmGFX8 13 AMDGPU/AMDGPUAsmGFX9 14 AMDGPU/AMDGPUAsmGFX900 15 AMDGPU/AMDGPUAsmGFX904 16 AMDGPU/AMDGPUAsmGFX906 17 AMDGPU/AMDGPUAsmGFX908 18 AMDGPU/AMDGPUAsmGFX90a 19 AMDGPU/AMDGPUAsmGFX10 20 AMDGPU/AMDGPUAsmGFX1011 21 AMDGPU/AMDGPUAsmGFX1013 22 AMDGPU/AMDGPUAsmGFX1030 23 AMDGPUModifierSyntax 24 AMDGPUOperandSyntax 25 AMDGPUInstructionSyntax 26 AMDGPUInstructionNotation 27 AMDGPUDwarfExtensionsForHeterogeneousDebugging 28 AMDGPUDwarfExtensionAllowLocationDescriptionOnTheDwarfExpressionStack/AMDGPUDwarfExtensionAllowLocationDescriptionOnTheDwarfExpressionStack 29 30Introduction 31============ 32 33The AMDGPU backend provides ISA code generation for AMD GPUs, starting with the 34R600 family up until the current GCN families. It lives in the 35``llvm/lib/Target/AMDGPU`` directory. 36 37LLVM 38==== 39 40.. _amdgpu-target-triples: 41 42Target Triples 43-------------- 44 45Use the Clang option ``-target <Architecture>-<Vendor>-<OS>-<Environment>`` 46to specify the target triple: 47 48 .. table:: AMDGPU Architectures 49 :name: amdgpu-architecture-table 50 51 ============ ============================================================== 52 Architecture Description 53 ============ ============================================================== 54 ``r600`` AMD GPUs HD2XXX-HD6XXX for graphics and compute shaders. 55 ``amdgcn`` AMD GPUs GCN GFX6 onwards for graphics and compute shaders. 56 ============ ============================================================== 57 58 .. table:: AMDGPU Vendors 59 :name: amdgpu-vendor-table 60 61 ============ ============================================================== 62 Vendor Description 63 ============ ============================================================== 64 ``amd`` Can be used for all AMD GPU usage. 65 ``mesa3d`` Can be used if the OS is ``mesa3d``. 66 ============ ============================================================== 67 68 .. table:: AMDGPU Operating Systems 69 :name: amdgpu-os 70 71 ============== ============================================================ 72 OS Description 73 ============== ============================================================ 74 *<empty>* Defaults to the *unknown* OS. 75 ``amdhsa`` Compute kernels executed on HSA [HSA]_ compatible runtimes 76 such as: 77 78 - AMD's ROCm™ runtime [AMD-ROCm]_ using the *rocm-amdhsa* 79 loader on Linux. See *AMD ROCm Platform Release Notes* 80 [AMD-ROCm-Release-Notes]_ for supported hardware and 81 software. 82 - AMD's PAL runtime using the *pal-amdhsa* loader on 83 Windows. 84 85 ``amdpal`` Graphic shaders and compute kernels executed on AMD's PAL 86 runtime using the *pal-amdpal* loader on Windows and Linux 87 Pro. 88 ``mesa3d`` Graphic shaders and compute kernels executed on AMD's Mesa 89 3D runtime using the *mesa-mesa3d* loader on Linux. 90 ============== ============================================================ 91 92 .. table:: AMDGPU Environments 93 :name: amdgpu-environment-table 94 95 ============ ============================================================== 96 Environment Description 97 ============ ============================================================== 98 *<empty>* Default. 99 ============ ============================================================== 100 101.. _amdgpu-processors: 102 103Processors 104---------- 105 106Use the Clang options ``-mcpu=<target-id>`` or ``--offload-arch=<target-id>`` to 107specify the AMDGPU processor together with optional target features. See 108:ref:`amdgpu-target-id` and :ref:`amdgpu-target-features` for AMD GPU target 109specific information. 110 111Every processor supports every OS ABI (see :ref:`amdgpu-os`) with the following exceptions: 112 113* ``amdhsa`` is not supported in ``r600`` architecture (see :ref:`amdgpu-architecture-table`). 114 115 116 .. table:: AMDGPU Processors 117 :name: amdgpu-processor-table 118 119 =========== =============== ============ ===== ================= =============== =============== ====================== 120 Processor Alternative Target dGPU/ Target Target OS Support Example 121 Processor Triple APU Features Properties *(see* Products 122 Architecture Supported `amdgpu-os`_ 123 *and 124 corresponding 125 runtime release 126 notes for 127 current 128 information and 129 level of 130 support)* 131 =========== =============== ============ ===== ================= =============== =============== ====================== 132 **Radeon HD 2000/3000 Series (R600)** [AMD-RADEON-HD-2000-3000]_ 133 ----------------------------------------------------------------------------------------------------------------------- 134 ``r600`` ``r600`` dGPU - Does not 135 support 136 generic 137 address 138 space 139 ``r630`` ``r600`` dGPU - Does not 140 support 141 generic 142 address 143 space 144 ``rs880`` ``r600`` dGPU - Does not 145 support 146 generic 147 address 148 space 149 ``rv670`` ``r600`` dGPU - Does not 150 support 151 generic 152 address 153 space 154 **Radeon HD 4000 Series (R700)** [AMD-RADEON-HD-4000]_ 155 ----------------------------------------------------------------------------------------------------------------------- 156 ``rv710`` ``r600`` dGPU - Does not 157 support 158 generic 159 address 160 space 161 ``rv730`` ``r600`` dGPU - Does not 162 support 163 generic 164 address 165 space 166 ``rv770`` ``r600`` dGPU - Does not 167 support 168 generic 169 address 170 space 171 **Radeon HD 5000 Series (Evergreen)** [AMD-RADEON-HD-5000]_ 172 ----------------------------------------------------------------------------------------------------------------------- 173 ``cedar`` ``r600`` dGPU - Does not 174 support 175 generic 176 address 177 space 178 ``cypress`` ``r600`` dGPU - Does not 179 support 180 generic 181 address 182 space 183 ``juniper`` ``r600`` dGPU - Does not 184 support 185 generic 186 address 187 space 188 ``redwood`` ``r600`` dGPU - Does not 189 support 190 generic 191 address 192 space 193 ``sumo`` ``r600`` dGPU - Does not 194 support 195 generic 196 address 197 space 198 **Radeon HD 6000 Series (Northern Islands)** [AMD-RADEON-HD-6000]_ 199 ----------------------------------------------------------------------------------------------------------------------- 200 ``barts`` ``r600`` dGPU - Does not 201 support 202 generic 203 address 204 space 205 ``caicos`` ``r600`` dGPU - Does not 206 support 207 generic 208 address 209 space 210 ``cayman`` ``r600`` dGPU - Does not 211 support 212 generic 213 address 214 space 215 ``turks`` ``r600`` dGPU - Does not 216 support 217 generic 218 address 219 space 220 **GCN GFX6 (Southern Islands (SI))** [AMD-GCN-GFX6]_ 221 ----------------------------------------------------------------------------------------------------------------------- 222 ``gfx600`` - ``tahiti`` ``amdgcn`` dGPU - Does not - *pal-amdpal* 223 support 224 generic 225 address 226 space 227 ``gfx601`` - ``pitcairn`` ``amdgcn`` dGPU - Does not - *pal-amdpal* 228 - ``verde`` support 229 generic 230 address 231 space 232 ``gfx602`` - ``hainan`` ``amdgcn`` dGPU - Does not - *pal-amdpal* 233 - ``oland`` support 234 generic 235 address 236 space 237 **GCN GFX7 (Sea Islands (CI))** [AMD-GCN-GFX7]_ 238 ----------------------------------------------------------------------------------------------------------------------- 239 ``gfx700`` - ``kaveri`` ``amdgcn`` APU - Offset - *rocm-amdhsa* - A6-7000 240 flat - *pal-amdhsa* - A6 Pro-7050B 241 scratch - *pal-amdpal* - A8-7100 242 - A8 Pro-7150B 243 - A10-7300 244 - A10 Pro-7350B 245 - FX-7500 246 - A8-7200P 247 - A10-7400P 248 - FX-7600P 249 ``gfx701`` - ``hawaii`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - FirePro W8100 250 flat - *pal-amdhsa* - FirePro W9100 251 scratch - *pal-amdpal* - FirePro S9150 252 - FirePro S9170 253 ``gfx702`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - Radeon R9 290 254 flat - *pal-amdhsa* - Radeon R9 290x 255 scratch - *pal-amdpal* - Radeon R390 256 - Radeon R390x 257 ``gfx703`` - ``kabini`` ``amdgcn`` APU - Offset - *pal-amdhsa* - E1-2100 258 - ``mullins`` flat - *pal-amdpal* - E1-2200 259 scratch - E1-2500 260 - E2-3000 261 - E2-3800 262 - A4-5000 263 - A4-5100 264 - A6-5200 265 - A4 Pro-3340B 266 ``gfx704`` - ``bonaire`` ``amdgcn`` dGPU - Offset - *pal-amdhsa* - Radeon HD 7790 267 flat - *pal-amdpal* - Radeon HD 8770 268 scratch - R7 260 269 - R7 260X 270 ``gfx705`` ``amdgcn`` APU - Offset - *pal-amdhsa* *TBA* 271 flat - *pal-amdpal* 272 scratch .. TODO:: 273 274 Add product 275 names. 276 277 **GCN GFX8 (Volcanic Islands (VI))** [AMD-GCN-GFX8]_ 278 ----------------------------------------------------------------------------------------------------------------------- 279 ``gfx801`` - ``carrizo`` ``amdgcn`` APU - xnack - Offset - *rocm-amdhsa* - A6-8500P 280 flat - *pal-amdhsa* - Pro A6-8500B 281 scratch - *pal-amdpal* - A8-8600P 282 - Pro A8-8600B 283 - FX-8800P 284 - Pro A12-8800B 285 - A10-8700P 286 - Pro A10-8700B 287 - A10-8780P 288 - A10-9600P 289 - A10-9630P 290 - A12-9700P 291 - A12-9730P 292 - FX-9800P 293 - FX-9830P 294 - E2-9010 295 - A6-9210 296 - A9-9410 297 ``gfx802`` - ``iceland`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - Radeon R9 285 298 - ``tonga`` flat - *pal-amdhsa* - Radeon R9 380 299 scratch - *pal-amdpal* - Radeon R9 385 300 ``gfx803`` - ``fiji`` ``amdgcn`` dGPU - *rocm-amdhsa* - Radeon R9 Nano 301 - *pal-amdhsa* - Radeon R9 Fury 302 - *pal-amdpal* - Radeon R9 FuryX 303 - Radeon Pro Duo 304 - FirePro S9300x2 305 - Radeon Instinct MI8 306 \ - ``polaris10`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - Radeon RX 470 307 flat - *pal-amdhsa* - Radeon RX 480 308 scratch - *pal-amdpal* - Radeon Instinct MI6 309 \ - ``polaris11`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - Radeon RX 460 310 flat - *pal-amdhsa* 311 scratch - *pal-amdpal* 312 ``gfx805`` - ``tongapro`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - FirePro S7150 313 flat - *pal-amdhsa* - FirePro S7100 314 scratch - *pal-amdpal* - FirePro W7100 315 - Mobile FirePro 316 M7170 317 ``gfx810`` - ``stoney`` ``amdgcn`` APU - xnack - Offset - *rocm-amdhsa* *TBA* 318 flat - *pal-amdhsa* 319 scratch - *pal-amdpal* .. TODO:: 320 321 Add product 322 names. 323 324 **GCN GFX9 (Vega)** [AMD-GCN-GFX900-GFX904-VEGA]_ [AMD-GCN-GFX906-VEGA7NM]_ [AMD-GCN-GFX908-CDNA1]_ [AMD-GCN-GFX90A-CDNA2]_ 325 ----------------------------------------------------------------------------------------------------------------------- 326 ``gfx900`` ``amdgcn`` dGPU - xnack - Absolute - *rocm-amdhsa* - Radeon Vega 327 flat - *pal-amdhsa* Frontier Edition 328 scratch - *pal-amdpal* - Radeon RX Vega 56 329 - Radeon RX Vega 64 330 - Radeon RX Vega 64 331 Liquid 332 - Radeon Instinct MI25 333 ``gfx902`` ``amdgcn`` APU - xnack - Absolute - *rocm-amdhsa* - Ryzen 3 2200G 334 flat - *pal-amdhsa* - Ryzen 5 2400G 335 scratch - *pal-amdpal* 336 ``gfx904`` ``amdgcn`` dGPU - xnack - *rocm-amdhsa* *TBA* 337 - *pal-amdhsa* 338 - *pal-amdpal* .. TODO:: 339 340 Add product 341 names. 342 343 ``gfx906`` ``amdgcn`` dGPU - sramecc - Absolute - *rocm-amdhsa* - Radeon Instinct MI50 344 - xnack flat - *pal-amdhsa* - Radeon Instinct MI60 345 scratch - *pal-amdpal* - Radeon VII 346 - Radeon Pro VII 347 ``gfx908`` ``amdgcn`` dGPU - sramecc - *rocm-amdhsa* - AMD Instinct MI100 Accelerator 348 - xnack - Absolute 349 flat 350 scratch 351 ``gfx909`` ``amdgcn`` APU - xnack - Absolute - *pal-amdpal* *TBA* 352 flat 353 scratch .. TODO:: 354 355 Add product 356 names. 357 358 ``gfx90a`` ``amdgcn`` dGPU - sramecc - Absolute - *rocm-amdhsa* *TBA* 359 - tgsplit flat 360 - xnack scratch .. TODO:: 361 - Packed 362 work-item Add product 363 IDs names. 364 365 ``gfx90c`` ``amdgcn`` APU - xnack - Absolute - *pal-amdpal* - Ryzen 7 4700G 366 flat - Ryzen 7 4700GE 367 scratch - Ryzen 5 4600G 368 - Ryzen 5 4600GE 369 - Ryzen 3 4300G 370 - Ryzen 3 4300GE 371 - Ryzen Pro 4000G 372 - Ryzen 7 Pro 4700G 373 - Ryzen 7 Pro 4750GE 374 - Ryzen 5 Pro 4650G 375 - Ryzen 5 Pro 4650GE 376 - Ryzen 3 Pro 4350G 377 - Ryzen 3 Pro 4350GE 378 379 ``gfx940`` ``amdgcn`` dGPU - sramecc - Architected *TBA* 380 - tgsplit flat 381 - xnack scratch .. TODO:: 382 - Packed 383 work-item Add product 384 IDs names. 385 386 **GCN GFX10.1 (RDNA 1)** [AMD-GCN-GFX10-RDNA1]_ 387 ----------------------------------------------------------------------------------------------------------------------- 388 ``gfx1010`` ``amdgcn`` dGPU - cumode - Absolute - *rocm-amdhsa* - Radeon RX 5700 389 - wavefrontsize64 flat - *pal-amdhsa* - Radeon RX 5700 XT 390 - xnack scratch - *pal-amdpal* - Radeon Pro 5600 XT 391 - Radeon Pro 5600M 392 ``gfx1011`` ``amdgcn`` dGPU - cumode - *rocm-amdhsa* - Radeon Pro V520 393 - wavefrontsize64 - Absolute - *pal-amdhsa* 394 - xnack flat - *pal-amdpal* 395 scratch 396 ``gfx1012`` ``amdgcn`` dGPU - cumode - Absolute - *rocm-amdhsa* - Radeon RX 5500 397 - wavefrontsize64 flat - *pal-amdhsa* - Radeon RX 5500 XT 398 - xnack scratch - *pal-amdpal* 399 ``gfx1013`` ``amdgcn`` APU - cumode - Absolute - *rocm-amdhsa* *TBA* 400 - wavefrontsize64 flat - *pal-amdhsa* 401 - xnack scratch - *pal-amdpal* .. TODO:: 402 403 Add product 404 names. 405 406 **GCN GFX10.3 (RDNA 2)** [AMD-GCN-GFX10-RDNA2]_ 407 ----------------------------------------------------------------------------------------------------------------------- 408 ``gfx1030`` ``amdgcn`` dGPU - cumode - Absolute - *rocm-amdhsa* - Radeon RX 6800 409 - wavefrontsize64 flat - *pal-amdhsa* - Radeon RX 6800 XT 410 scratch - *pal-amdpal* - Radeon RX 6900 XT 411 ``gfx1031`` ``amdgcn`` dGPU - cumode - Absolute - *rocm-amdhsa* - Radeon RX 6700 XT 412 - wavefrontsize64 flat - *pal-amdhsa* 413 scratch - *pal-amdpal* 414 ``gfx1032`` ``amdgcn`` dGPU - cumode - Absolute - *rocm-amdhsa* *TBA* 415 - wavefrontsize64 flat - *pal-amdhsa* 416 scratch - *pal-amdpal* .. TODO:: 417 418 Add product 419 names. 420 421 ``gfx1033`` ``amdgcn`` APU - cumode - Absolute - *pal-amdpal* *TBA* 422 - wavefrontsize64 flat 423 scratch .. TODO:: 424 425 Add product 426 names. 427 ``gfx1034`` ``amdgcn`` dGPU - cumode - Absolute - *pal-amdpal* *TBA* 428 - wavefrontsize64 flat 429 scratch .. TODO:: 430 431 Add product 432 names. 433 434 ``gfx1035`` ``amdgcn`` APU - cumode - Absolute - *pal-amdpal* *TBA* 435 - wavefrontsize64 flat 436 scratch .. TODO:: 437 Add product 438 names. 439 440 ``gfx1036`` ``amdgcn`` APU - cumode - Absolute - *pal-amdpal* *TBA* 441 - wavefrontsize64 flat 442 scratch .. TODO:: 443 444 Add product 445 names. 446 447 **GCN GFX11** 448 ----------------------------------------------------------------------------------------------------------------------- 449 ``gfx1100`` ``amdgcn`` dGPU - cumode - Architected - *pal-amdpal* *TBA* 450 - wavefrontsize64 flat 451 scratch .. TODO:: 452 - Packed 453 work-item Add product 454 IDs names. 455 456 ``gfx1101`` ``amdgcn`` dGPU - cumode - Architected *TBA* 457 - wavefrontsize64 flat 458 scratch .. TODO:: 459 - Packed 460 work-item Add product 461 IDs names. 462 463 ``gfx1102`` ``amdgcn`` dGPU - cumode - Architected *TBA* 464 - wavefrontsize64 flat 465 scratch .. TODO:: 466 - Packed 467 work-item Add product 468 IDs names. 469 470 ``gfx1103`` ``amdgcn`` APU - cumode - Architected *TBA* 471 - wavefrontsize64 flat 472 scratch .. TODO:: 473 - Packed 474 work-item Add product 475 IDs names. 476 477 =========== =============== ============ ===== ================= =============== =============== ====================== 478 479.. _amdgpu-target-features: 480 481Target Features 482--------------- 483 484Target features control how code is generated to support certain 485processor specific features. Not all target features are supported by 486all processors. The runtime must ensure that the features supported by 487the device used to execute the code match the features enabled when 488generating the code. A mismatch of features may result in incorrect 489execution, or a reduction in performance. 490 491The target features supported by each processor is listed in 492:ref:`amdgpu-processor-table`. 493 494Target features are controlled by exactly one of the following Clang 495options: 496 497``-mcpu=<target-id>`` or ``--offload-arch=<target-id>`` 498 499 The ``-mcpu`` and ``--offload-arch`` can specify the target feature as 500 optional components of the target ID. If omitted, the target feature has the 501 ``any`` value. See :ref:`amdgpu-target-id`. 502 503``-m[no-]<target-feature>`` 504 505 Target features not specified by the target ID are specified using a 506 separate option. These target features can have an ``on`` or ``off`` 507 value. ``on`` is specified by omitting the ``no-`` prefix, and 508 ``off`` is specified by including the ``no-`` prefix. The default 509 if not specified is ``off``. 510 511For example: 512 513``-mcpu=gfx908:xnack+`` 514 Enable the ``xnack`` feature. 515``-mcpu=gfx908:xnack-`` 516 Disable the ``xnack`` feature. 517``-mcumode`` 518 Enable the ``cumode`` feature. 519``-mno-cumode`` 520 Disable the ``cumode`` feature. 521 522 .. table:: AMDGPU Target Features 523 :name: amdgpu-target-features-table 524 525 =============== ============================ ================================================== 526 Target Feature Clang Option to Control Description 527 Name 528 =============== ============================ ================================================== 529 cumode - ``-m[no-]cumode`` Control the wavefront execution mode used 530 when generating code for kernels. When disabled 531 native WGP wavefront execution mode is used, 532 when enabled CU wavefront execution mode is used 533 (see :ref:`amdgpu-amdhsa-memory-model`). 534 535 sramecc - ``-mcpu`` If specified, generate code that can only be 536 - ``--offload-arch`` loaded and executed in a process that has a 537 matching setting for SRAMECC. 538 539 If not specified for code object V2 to V3, generate 540 code that can be loaded and executed in a process 541 with SRAMECC enabled. 542 543 If not specified for code object V4 or above, generate 544 code that can be loaded and executed in a process 545 with either setting of SRAMECC. 546 547 tgsplit ``-m[no-]tgsplit`` Enable/disable generating code that assumes 548 work-groups are launched in threadgroup split mode. 549 When enabled the waves of a work-group may be 550 launched in different CUs. 551 552 wavefrontsize64 - ``-m[no-]wavefrontsize64`` Control the wavefront size used when 553 generating code for kernels. When disabled 554 native wavefront size 32 is used, when enabled 555 wavefront size 64 is used. 556 557 xnack - ``-mcpu`` If specified, generate code that can only be 558 - ``--offload-arch`` loaded and executed in a process that has a 559 matching setting for XNACK replay. 560 561 If not specified for code object V2 to V3, generate 562 code that can be loaded and executed in a process 563 with XNACK replay enabled. 564 565 If not specified for code object V4 or above, generate 566 code that can be loaded and executed in a process 567 with either setting of XNACK replay. 568 569 XNACK replay can be used for demand paging and 570 page migration. If enabled in the device, then if 571 a page fault occurs the code may execute 572 incorrectly unless generated with XNACK replay 573 enabled, or generated for code object V4 or above without 574 specifying XNACK replay. Executing code that was 575 generated with XNACK replay enabled, or generated 576 for code object V4 or above without specifying XNACK replay, 577 on a device that does not have XNACK replay 578 enabled will execute correctly but may be less 579 performant than code generated for XNACK replay 580 disabled. 581 =============== ============================ ================================================== 582 583.. _amdgpu-target-id: 584 585Target ID 586--------- 587 588AMDGPU supports target IDs. See `Clang Offload Bundler 589<https://clang.llvm.org/docs/ClangOffloadBundler.html>`_ for a general 590description. The AMDGPU target specific information is: 591 592**processor** 593 Is an AMDGPU processor or alternative processor name specified in 594 :ref:`amdgpu-processor-table`. The non-canonical form target ID allows both 595 the primary processor and alternative processor names. The canonical form 596 target ID only allow the primary processor name. 597 598**target-feature** 599 Is a target feature name specified in :ref:`amdgpu-target-features-table` that 600 is supported by the processor. The target features supported by each processor 601 is specified in :ref:`amdgpu-processor-table`. Those that can be specified in 602 a target ID are marked as being controlled by ``-mcpu`` and 603 ``--offload-arch``. Each target feature must appear at most once in a target 604 ID. The non-canonical form target ID allows the target features to be 605 specified in any order. The canonical form target ID requires the target 606 features to be specified in alphabetic order. 607 608.. _amdgpu-target-id-v2-v3: 609 610Code Object V2 to V3 Target ID 611~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 612 613The target ID syntax for code object V2 to V3 is the same as defined in `Clang 614Offload Bundler <https://clang.llvm.org/docs/ClangOffloadBundler.html>`_ except 615when used in the :ref:`amdgpu-assembler-directive-amdgcn-target` assembler 616directive and the bundle entry ID. In those cases it has the following BNF 617syntax: 618 619.. code:: 620 621 <target-id> ::== <processor> ( "+" <target-feature> )* 622 623Where a target feature is omitted if *Off* and present if *On* or *Any*. 624 625.. note:: 626 627 The code object V2 to V3 cannot represent *Any* and treats it the same as 628 *On*. 629 630.. _amdgpu-embedding-bundled-objects: 631 632Embedding Bundled Code Objects 633------------------------------ 634 635AMDGPU supports the HIP and OpenMP languages that perform code object embedding 636as described in `Clang Offload Bundler 637<https://clang.llvm.org/docs/ClangOffloadBundler.html>`_. 638 639.. note:: 640 641 The target ID syntax used for code object V2 to V3 for a bundle entry ID 642 differs from that used elsewhere. See :ref:`amdgpu-target-id-v2-v3`. 643 644.. _amdgpu-address-spaces: 645 646Address Spaces 647-------------- 648 649The AMDGPU architecture supports a number of memory address spaces. The address 650space names use the OpenCL standard names, with some additions. 651 652The AMDGPU address spaces correspond to target architecture specific LLVM 653address space numbers used in LLVM IR. 654 655The AMDGPU address spaces are described in 656:ref:`amdgpu-address-spaces-table`. Only 64-bit process address spaces are 657supported for the ``amdgcn`` target. 658 659 .. table:: AMDGPU Address Spaces 660 :name: amdgpu-address-spaces-table 661 662 ================================= =============== =========== ================ ======= ============================ 663 .. 64-Bit Process Address Space 664 --------------------------------- --------------- ----------- ---------------- ------------------------------------ 665 Address Space Name LLVM IR Address HSA Segment Hardware Address NULL Value 666 Space Number Name Name Size 667 ================================= =============== =========== ================ ======= ============================ 668 Generic 0 flat flat 64 0x0000000000000000 669 Global 1 global global 64 0x0000000000000000 670 Region 2 N/A GDS 32 *not implemented for AMDHSA* 671 Local 3 group LDS 32 0xFFFFFFFF 672 Constant 4 constant *same as global* 64 0x0000000000000000 673 Private 5 private scratch 32 0xFFFFFFFF 674 Constant 32-bit 6 *TODO* 0x00000000 675 Buffer Fat Pointer (experimental) 7 *TODO* 676 ================================= =============== =========== ================ ======= ============================ 677 678**Generic** 679 The generic address space is supported unless the *Target Properties* column 680 of :ref:`amdgpu-processor-table` specifies *Does not support generic address 681 space*. 682 683 The generic address space uses the hardware flat address support for two fixed 684 ranges of virtual addresses (the private and local apertures), that are 685 outside the range of addressable global memory, to map from a flat address to 686 a private or local address. This uses FLAT instructions that can take a flat 687 address and access global, private (scratch), and group (LDS) memory depending 688 on if the address is within one of the aperture ranges. 689 690 Flat access to scratch requires hardware aperture setup and setup in the 691 kernel prologue (see :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`). Flat 692 access to LDS requires hardware aperture setup and M0 (GFX7-GFX8) register 693 setup (see :ref:`amdgpu-amdhsa-kernel-prolog-m0`). 694 695 To convert between a private or group address space address (termed a segment 696 address) and a flat address the base address of the corresponding aperture 697 can be used. For GFX7-GFX8 these are available in the 698 :ref:`amdgpu-amdhsa-hsa-aql-queue` the address of which can be obtained with 699 Queue Ptr SGPR (see :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). For 700 GFX9-GFX10 the aperture base addresses are directly available as inline 701 constant registers ``SRC_SHARED_BASE/LIMIT`` and ``SRC_PRIVATE_BASE/LIMIT``. 702 In 64-bit address mode the aperture sizes are 2^32 bytes and the base is 703 aligned to 2^32 which makes it easier to convert from flat to segment or 704 segment to flat. 705 706 A global address space address has the same value when used as a flat address 707 so no conversion is needed. 708 709**Global and Constant** 710 The global and constant address spaces both use global virtual addresses, 711 which are the same virtual address space used by the CPU. However, some 712 virtual addresses may only be accessible to the CPU, some only accessible 713 by the GPU, and some by both. 714 715 Using the constant address space indicates that the data will not change 716 during the execution of the kernel. This allows scalar read instructions to 717 be used. As the constant address space could only be modified on the host 718 side, a generic pointer loaded from the constant address space is safe to be 719 assumed as a global pointer since only the device global memory is visible 720 and managed on the host side. The vector and scalar L1 caches are invalidated 721 of volatile data before each kernel dispatch execution to allow constant 722 memory to change values between kernel dispatches. 723 724**Region** 725 The region address space uses the hardware Global Data Store (GDS). All 726 wavefronts executing on the same device will access the same memory for any 727 given region address. However, the same region address accessed by wavefronts 728 executing on different devices will access different memory. It is higher 729 performance than global memory. It is allocated by the runtime. The data 730 store (DS) instructions can be used to access it. 731 732**Local** 733 The local address space uses the hardware Local Data Store (LDS) which is 734 automatically allocated when the hardware creates the wavefronts of a 735 work-group, and freed when all the wavefronts of a work-group have 736 terminated. All wavefronts belonging to the same work-group will access the 737 same memory for any given local address. However, the same local address 738 accessed by wavefronts belonging to different work-groups will access 739 different memory. It is higher performance than global memory. The data store 740 (DS) instructions can be used to access it. 741 742**Private** 743 The private address space uses the hardware scratch memory support which 744 automatically allocates memory when it creates a wavefront and frees it when 745 a wavefronts terminates. The memory accessed by a lane of a wavefront for any 746 given private address will be different to the memory accessed by another lane 747 of the same or different wavefront for the same private address. 748 749 If a kernel dispatch uses scratch, then the hardware allocates memory from a 750 pool of backing memory allocated by the runtime for each wavefront. The lanes 751 of the wavefront access this using dword (4 byte) interleaving. The mapping 752 used from private address to backing memory address is: 753 754 ``wavefront-scratch-base + 755 ((private-address / 4) * wavefront-size * 4) + 756 (wavefront-lane-id * 4) + (private-address % 4)`` 757 758 If each lane of a wavefront accesses the same private address, the 759 interleaving results in adjacent dwords being accessed and hence requires 760 fewer cache lines to be fetched. 761 762 There are different ways that the wavefront scratch base address is 763 determined by a wavefront (see 764 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 765 766 Scratch memory can be accessed in an interleaved manner using buffer 767 instructions with the scratch buffer descriptor and per wavefront scratch 768 offset, by the scratch instructions, or by flat instructions. Multi-dword 769 access is not supported except by flat and scratch instructions in 770 GFX9-GFX10. 771 772**Constant 32-bit** 773 *TODO* 774 775**Buffer Fat Pointer** 776 The buffer fat pointer is an experimental address space that is currently 777 unsupported in the backend. It exposes a non-integral pointer that is in 778 the future intended to support the modelling of 128-bit buffer descriptors 779 plus a 32-bit offset into the buffer (in total encapsulating a 160-bit 780 *pointer*), allowing normal LLVM load/store/atomic operations to be used to 781 model the buffer descriptors used heavily in graphics workloads targeting 782 the backend. 783 784.. _amdgpu-memory-scopes: 785 786Memory Scopes 787------------- 788 789This section provides LLVM memory synchronization scopes supported by the AMDGPU 790backend memory model when the target triple OS is ``amdhsa`` (see 791:ref:`amdgpu-amdhsa-memory-model` and :ref:`amdgpu-target-triples`). 792 793The memory model supported is based on the HSA memory model [HSA]_ which is 794based in turn on HRF-indirect with scope inclusion [HRF]_. The happens-before 795relation is transitive over the synchronizes-with relation independent of scope 796and synchronizes-with allows the memory scope instances to be inclusive (see 797table :ref:`amdgpu-amdhsa-llvm-sync-scopes-table`). 798 799This is different to the OpenCL [OpenCL]_ memory model which does not have scope 800inclusion and requires the memory scopes to exactly match. However, this 801is conservatively correct for OpenCL. 802 803 .. table:: AMDHSA LLVM Sync Scopes 804 :name: amdgpu-amdhsa-llvm-sync-scopes-table 805 806 ======================= =================================================== 807 LLVM Sync Scope Description 808 ======================= =================================================== 809 *none* The default: ``system``. 810 811 Synchronizes with, and participates in modification 812 and seq_cst total orderings with, other operations 813 (except image operations) for all address spaces 814 (except private, or generic that accesses private) 815 provided the other operation's sync scope is: 816 817 - ``system``. 818 - ``agent`` and executed by a thread on the same 819 agent. 820 - ``workgroup`` and executed by a thread in the 821 same work-group. 822 - ``wavefront`` and executed by a thread in the 823 same wavefront. 824 825 ``agent`` Synchronizes with, and participates in modification 826 and seq_cst total orderings with, other operations 827 (except image operations) for all address spaces 828 (except private, or generic that accesses private) 829 provided the other operation's sync scope is: 830 831 - ``system`` or ``agent`` and executed by a thread 832 on the same agent. 833 - ``workgroup`` and executed by a thread in the 834 same work-group. 835 - ``wavefront`` and executed by a thread in the 836 same wavefront. 837 838 ``workgroup`` Synchronizes with, and participates in modification 839 and seq_cst total orderings with, other operations 840 (except image operations) for all address spaces 841 (except private, or generic that accesses private) 842 provided the other operation's sync scope is: 843 844 - ``system``, ``agent`` or ``workgroup`` and 845 executed by a thread in the same work-group. 846 - ``wavefront`` and executed by a thread in the 847 same wavefront. 848 849 ``wavefront`` Synchronizes with, and participates in modification 850 and seq_cst total orderings with, other operations 851 (except image operations) for all address spaces 852 (except private, or generic that accesses private) 853 provided the other operation's sync scope is: 854 855 - ``system``, ``agent``, ``workgroup`` or 856 ``wavefront`` and executed by a thread in the 857 same wavefront. 858 859 ``singlethread`` Only synchronizes with and participates in 860 modification and seq_cst total orderings with, 861 other operations (except image operations) running 862 in the same thread for all address spaces (for 863 example, in signal handlers). 864 865 ``one-as`` Same as ``system`` but only synchronizes with other 866 operations within the same address space. 867 868 ``agent-one-as`` Same as ``agent`` but only synchronizes with other 869 operations within the same address space. 870 871 ``workgroup-one-as`` Same as ``workgroup`` but only synchronizes with 872 other operations within the same address space. 873 874 ``wavefront-one-as`` Same as ``wavefront`` but only synchronizes with 875 other operations within the same address space. 876 877 ``singlethread-one-as`` Same as ``singlethread`` but only synchronizes with 878 other operations within the same address space. 879 ======================= =================================================== 880 881LLVM IR Intrinsics 882------------------ 883 884The AMDGPU backend implements the following LLVM IR intrinsics. 885 886*This section is WIP.* 887 888.. TODO:: 889 890 List AMDGPU intrinsics. 891 892LLVM IR Attributes 893------------------ 894 895The AMDGPU backend supports the following LLVM IR attributes. 896 897 .. table:: AMDGPU LLVM IR Attributes 898 :name: amdgpu-llvm-ir-attributes-table 899 900 ======================================= ========================================================== 901 LLVM Attribute Description 902 ======================================= ========================================================== 903 "amdgpu-flat-work-group-size"="min,max" Specify the minimum and maximum flat work group sizes that 904 will be specified when the kernel is dispatched. Generated 905 by the ``amdgpu_flat_work_group_size`` CLANG attribute [CLANG-ATTR]_. 906 The implied default value is 1,1024. 907 908 "amdgpu-implicitarg-num-bytes"="n" Number of kernel argument bytes to add to the kernel 909 argument block size for the implicit arguments. This 910 varies by OS and language (for OpenCL see 911 :ref:`opencl-kernel-implicit-arguments-appended-for-amdhsa-os-table`). 912 "amdgpu-num-sgpr"="n" Specifies the number of SGPRs to use. Generated by 913 the ``amdgpu_num_sgpr`` CLANG attribute [CLANG-ATTR]_. 914 "amdgpu-num-vgpr"="n" Specifies the number of VGPRs to use. Generated by the 915 ``amdgpu_num_vgpr`` CLANG attribute [CLANG-ATTR]_. 916 "amdgpu-waves-per-eu"="m,n" Specify the minimum and maximum number of waves per 917 execution unit. Generated by the ``amdgpu_waves_per_eu`` 918 CLANG attribute [CLANG-ATTR]_. This is an optimization hint, 919 and the backend may not be able to satisfy the request. If 920 the specified range is incompatible with the function's 921 "amdgpu-flat-work-group-size" value, the implied occupancy 922 bounds by the workgroup size takes precedence. 923 924 "amdgpu-ieee" true/false. Specify whether the function expects the IEEE field of the 925 mode register to be set on entry. Overrides the default for 926 the calling convention. 927 "amdgpu-dx10-clamp" true/false. Specify whether the function expects the DX10_CLAMP field of 928 the mode register to be set on entry. Overrides the default 929 for the calling convention. 930 931 "amdgpu-no-workitem-id-x" Indicates the function does not depend on the value of the 932 llvm.amdgcn.workitem.id.x intrinsic. If a function is marked with this 933 attribute, or reached through a call site marked with this attribute, 934 the value returned by the intrinsic is undefined. The backend can 935 generally infer this during code generation, so typically there is no 936 benefit to frontends marking functions with this. 937 938 "amdgpu-no-workitem-id-y" The same as amdgpu-no-workitem-id-x, except for the 939 llvm.amdgcn.workitem.id.y intrinsic. 940 941 "amdgpu-no-workitem-id-z" The same as amdgpu-no-workitem-id-x, except for the 942 llvm.amdgcn.workitem.id.z intrinsic. 943 944 "amdgpu-no-workgroup-id-x" The same as amdgpu-no-workitem-id-x, except for the 945 llvm.amdgcn.workgroup.id.x intrinsic. 946 947 "amdgpu-no-workgroup-id-y" The same as amdgpu-no-workitem-id-x, except for the 948 llvm.amdgcn.workgroup.id.y intrinsic. 949 950 "amdgpu-no-workgroup-id-z" The same as amdgpu-no-workitem-id-x, except for the 951 llvm.amdgcn.workgroup.id.z intrinsic. 952 953 "amdgpu-no-dispatch-ptr" The same as amdgpu-no-workitem-id-x, except for the 954 llvm.amdgcn.dispatch.ptr intrinsic. 955 956 "amdgpu-no-implicitarg-ptr" The same as amdgpu-no-workitem-id-x, except for the 957 llvm.amdgcn.implicitarg.ptr intrinsic. 958 959 "amdgpu-no-dispatch-id" The same as amdgpu-no-workitem-id-x, except for the 960 llvm.amdgcn.dispatch.id intrinsic. 961 962 "amdgpu-no-queue-ptr" Similar to amdgpu-no-workitem-id-x, except for the 963 llvm.amdgcn.queue.ptr intrinsic. Note that unlike the other ABI hint 964 attributes, the queue pointer may be required in situations where the 965 intrinsic call does not directly appear in the program. Some subtargets 966 require the queue pointer for to handle some addrspacecasts, as well 967 as the llvm.amdgcn.is.shared, llvm.amdgcn.is.private, llvm.trap, and 968 llvm.debug intrinsics. 969 970 "amdgpu-no-hostcall-ptr" Similar to amdgpu-no-implicitarg-ptr, except specific to the implicit 971 kernel argument that holds the pointer to the hostcall buffer. If this 972 attribute is absent, then the amdgpu-no-implicitarg-ptr is also removed. 973 974 "amdgpu-no-heap-ptr" Similar to amdgpu-no-implicitarg-ptr, except specific to the implicit 975 kernel argument that holds the pointer to an initialized memory buffer 976 that conforms to the requirements of the malloc/free device library V1 977 version implementation. If this attribute is absent, then the 978 amdgpu-no-implicitarg-ptr is also removed. 979 980 "amdgpu-no-multigrid-sync-arg" Similar to amdgpu-no-implicitarg-ptr, except specific to the implicit 981 kernel argument that holds the multigrid synchronization pointer. If this 982 attribute is absent, then the amdgpu-no-implicitarg-ptr is also removed. 983 ======================================= ========================================================== 984 985.. _amdgpu-elf-code-object: 986 987ELF Code Object 988=============== 989 990The AMDGPU backend generates a standard ELF [ELF]_ relocatable code object that 991can be linked by ``lld`` to produce a standard ELF shared code object which can 992be loaded and executed on an AMDGPU target. 993 994.. _amdgpu-elf-header: 995 996Header 997------ 998 999The AMDGPU backend uses the following ELF header: 1000 1001 .. table:: AMDGPU ELF Header 1002 :name: amdgpu-elf-header-table 1003 1004 ========================== =============================== 1005 Field Value 1006 ========================== =============================== 1007 ``e_ident[EI_CLASS]`` ``ELFCLASS64`` 1008 ``e_ident[EI_DATA]`` ``ELFDATA2LSB`` 1009 ``e_ident[EI_OSABI]`` - ``ELFOSABI_NONE`` 1010 - ``ELFOSABI_AMDGPU_HSA`` 1011 - ``ELFOSABI_AMDGPU_PAL`` 1012 - ``ELFOSABI_AMDGPU_MESA3D`` 1013 ``e_ident[EI_ABIVERSION]`` - ``ELFABIVERSION_AMDGPU_HSA_V2`` 1014 - ``ELFABIVERSION_AMDGPU_HSA_V3`` 1015 - ``ELFABIVERSION_AMDGPU_HSA_V4`` 1016 - ``ELFABIVERSION_AMDGPU_HSA_V5`` 1017 - ``ELFABIVERSION_AMDGPU_PAL`` 1018 - ``ELFABIVERSION_AMDGPU_MESA3D`` 1019 ``e_type`` - ``ET_REL`` 1020 - ``ET_DYN`` 1021 ``e_machine`` ``EM_AMDGPU`` 1022 ``e_entry`` 0 1023 ``e_flags`` See :ref:`amdgpu-elf-header-e_flags-v2-table`, 1024 :ref:`amdgpu-elf-header-e_flags-table-v3`, 1025 and :ref:`amdgpu-elf-header-e_flags-table-v4-onwards` 1026 ========================== =============================== 1027 1028.. 1029 1030 .. table:: AMDGPU ELF Header Enumeration Values 1031 :name: amdgpu-elf-header-enumeration-values-table 1032 1033 =============================== ===== 1034 Name Value 1035 =============================== ===== 1036 ``EM_AMDGPU`` 224 1037 ``ELFOSABI_NONE`` 0 1038 ``ELFOSABI_AMDGPU_HSA`` 64 1039 ``ELFOSABI_AMDGPU_PAL`` 65 1040 ``ELFOSABI_AMDGPU_MESA3D`` 66 1041 ``ELFABIVERSION_AMDGPU_HSA_V2`` 0 1042 ``ELFABIVERSION_AMDGPU_HSA_V3`` 1 1043 ``ELFABIVERSION_AMDGPU_HSA_V4`` 2 1044 ``ELFABIVERSION_AMDGPU_HSA_V5`` 3 1045 ``ELFABIVERSION_AMDGPU_PAL`` 0 1046 ``ELFABIVERSION_AMDGPU_MESA3D`` 0 1047 =============================== ===== 1048 1049``e_ident[EI_CLASS]`` 1050 The ELF class is: 1051 1052 * ``ELFCLASS32`` for ``r600`` architecture. 1053 1054 * ``ELFCLASS64`` for ``amdgcn`` architecture which only supports 64-bit 1055 process address space applications. 1056 1057``e_ident[EI_DATA]`` 1058 All AMDGPU targets use ``ELFDATA2LSB`` for little-endian byte ordering. 1059 1060``e_ident[EI_OSABI]`` 1061 One of the following AMDGPU target architecture specific OS ABIs 1062 (see :ref:`amdgpu-os`): 1063 1064 * ``ELFOSABI_NONE`` for *unknown* OS. 1065 1066 * ``ELFOSABI_AMDGPU_HSA`` for ``amdhsa`` OS. 1067 1068 * ``ELFOSABI_AMDGPU_PAL`` for ``amdpal`` OS. 1069 1070 * ``ELFOSABI_AMDGPU_MESA3D`` for ``mesa3D`` OS. 1071 1072``e_ident[EI_ABIVERSION]`` 1073 The ABI version of the AMDGPU target architecture specific OS ABI to which the code 1074 object conforms: 1075 1076 * ``ELFABIVERSION_AMDGPU_HSA_V2`` is used to specify the version of AMD HSA 1077 runtime ABI for code object V2. Specify using the Clang option 1078 ``-mcode-object-version=2``. 1079 1080 * ``ELFABIVERSION_AMDGPU_HSA_V3`` is used to specify the version of AMD HSA 1081 runtime ABI for code object V3. Specify using the Clang option 1082 ``-mcode-object-version=3``. 1083 1084 * ``ELFABIVERSION_AMDGPU_HSA_V4`` is used to specify the version of AMD HSA 1085 runtime ABI for code object V4. Specify using the Clang option 1086 ``-mcode-object-version=4``. This is the default code object 1087 version if not specified. 1088 1089 * ``ELFABIVERSION_AMDGPU_HSA_V5`` is used to specify the version of AMD HSA 1090 runtime ABI for code object V5. Specify using the Clang option 1091 ``-mcode-object-version=5``. 1092 1093 * ``ELFABIVERSION_AMDGPU_PAL`` is used to specify the version of AMD PAL 1094 runtime ABI. 1095 1096 * ``ELFABIVERSION_AMDGPU_MESA3D`` is used to specify the version of AMD MESA 1097 3D runtime ABI. 1098 1099``e_type`` 1100 Can be one of the following values: 1101 1102 1103 ``ET_REL`` 1104 The type produced by the AMDGPU backend compiler as it is relocatable code 1105 object. 1106 1107 ``ET_DYN`` 1108 The type produced by the linker as it is a shared code object. 1109 1110 The AMD HSA runtime loader requires a ``ET_DYN`` code object. 1111 1112``e_machine`` 1113 The value ``EM_AMDGPU`` is used for the machine for all processors supported 1114 by the ``r600`` and ``amdgcn`` architectures (see 1115 :ref:`amdgpu-processor-table`). The specific processor is specified in the 1116 ``NT_AMD_HSA_ISA_VERSION`` note record for code object V2 (see 1117 :ref:`amdgpu-note-records-v2`) and in the ``EF_AMDGPU_MACH`` bit field of the 1118 ``e_flags`` for code object V3 and above (see 1119 :ref:`amdgpu-elf-header-e_flags-table-v3` and 1120 :ref:`amdgpu-elf-header-e_flags-table-v4-onwards`). 1121 1122``e_entry`` 1123 The entry point is 0 as the entry points for individual kernels must be 1124 selected in order to invoke them through AQL packets. 1125 1126``e_flags`` 1127 The AMDGPU backend uses the following ELF header flags: 1128 1129 .. table:: AMDGPU ELF Header ``e_flags`` for Code Object V2 1130 :name: amdgpu-elf-header-e_flags-v2-table 1131 1132 ===================================== ===== ============================= 1133 Name Value Description 1134 ===================================== ===== ============================= 1135 ``EF_AMDGPU_FEATURE_XNACK_V2`` 0x01 Indicates if the ``xnack`` 1136 target feature is 1137 enabled for all code 1138 contained in the code object. 1139 If the processor 1140 does not support the 1141 ``xnack`` target 1142 feature then must 1143 be 0. 1144 See 1145 :ref:`amdgpu-target-features`. 1146 ``EF_AMDGPU_FEATURE_TRAP_HANDLER_V2`` 0x02 Indicates if the trap 1147 handler is enabled for all 1148 code contained in the code 1149 object. If the processor 1150 does not support a trap 1151 handler then must be 0. 1152 See 1153 :ref:`amdgpu-target-features`. 1154 ===================================== ===== ============================= 1155 1156 .. table:: AMDGPU ELF Header ``e_flags`` for Code Object V3 1157 :name: amdgpu-elf-header-e_flags-table-v3 1158 1159 ================================= ===== ============================= 1160 Name Value Description 1161 ================================= ===== ============================= 1162 ``EF_AMDGPU_MACH`` 0x0ff AMDGPU processor selection 1163 mask for 1164 ``EF_AMDGPU_MACH_xxx`` values 1165 defined in 1166 :ref:`amdgpu-ef-amdgpu-mach-table`. 1167 ``EF_AMDGPU_FEATURE_XNACK_V3`` 0x100 Indicates if the ``xnack`` 1168 target feature is 1169 enabled for all code 1170 contained in the code object. 1171 If the processor 1172 does not support the 1173 ``xnack`` target 1174 feature then must 1175 be 0. 1176 See 1177 :ref:`amdgpu-target-features`. 1178 ``EF_AMDGPU_FEATURE_SRAMECC_V3`` 0x200 Indicates if the ``sramecc`` 1179 target feature is 1180 enabled for all code 1181 contained in the code object. 1182 If the processor 1183 does not support the 1184 ``sramecc`` target 1185 feature then must 1186 be 0. 1187 See 1188 :ref:`amdgpu-target-features`. 1189 ================================= ===== ============================= 1190 1191 .. table:: AMDGPU ELF Header ``e_flags`` for Code Object V4 and After 1192 :name: amdgpu-elf-header-e_flags-table-v4-onwards 1193 1194 ============================================ ===== =================================== 1195 Name Value Description 1196 ============================================ ===== =================================== 1197 ``EF_AMDGPU_MACH`` 0x0ff AMDGPU processor selection 1198 mask for 1199 ``EF_AMDGPU_MACH_xxx`` values 1200 defined in 1201 :ref:`amdgpu-ef-amdgpu-mach-table`. 1202 ``EF_AMDGPU_FEATURE_XNACK_V4`` 0x300 XNACK selection mask for 1203 ``EF_AMDGPU_FEATURE_XNACK_*_V4`` 1204 values. 1205 ``EF_AMDGPU_FEATURE_XNACK_UNSUPPORTED_V4`` 0x000 XNACK unsuppored. 1206 ``EF_AMDGPU_FEATURE_XNACK_ANY_V4`` 0x100 XNACK can have any value. 1207 ``EF_AMDGPU_FEATURE_XNACK_OFF_V4`` 0x200 XNACK disabled. 1208 ``EF_AMDGPU_FEATURE_XNACK_ON_V4`` 0x300 XNACK enabled. 1209 ``EF_AMDGPU_FEATURE_SRAMECC_V4`` 0xc00 SRAMECC selection mask for 1210 ``EF_AMDGPU_FEATURE_SRAMECC_*_V4`` 1211 values. 1212 ``EF_AMDGPU_FEATURE_SRAMECC_UNSUPPORTED_V4`` 0x000 SRAMECC unsuppored. 1213 ``EF_AMDGPU_FEATURE_SRAMECC_ANY_V4`` 0x400 SRAMECC can have any value. 1214 ``EF_AMDGPU_FEATURE_SRAMECC_OFF_V4`` 0x800 SRAMECC disabled, 1215 ``EF_AMDGPU_FEATURE_SRAMECC_ON_V4`` 0xc00 SRAMECC enabled. 1216 ============================================ ===== =================================== 1217 1218 .. table:: AMDGPU ``EF_AMDGPU_MACH`` Values 1219 :name: amdgpu-ef-amdgpu-mach-table 1220 1221 ==================================== ========== ============================= 1222 Name Value Description (see 1223 :ref:`amdgpu-processor-table`) 1224 ==================================== ========== ============================= 1225 ``EF_AMDGPU_MACH_NONE`` 0x000 *not specified* 1226 ``EF_AMDGPU_MACH_R600_R600`` 0x001 ``r600`` 1227 ``EF_AMDGPU_MACH_R600_R630`` 0x002 ``r630`` 1228 ``EF_AMDGPU_MACH_R600_RS880`` 0x003 ``rs880`` 1229 ``EF_AMDGPU_MACH_R600_RV670`` 0x004 ``rv670`` 1230 ``EF_AMDGPU_MACH_R600_RV710`` 0x005 ``rv710`` 1231 ``EF_AMDGPU_MACH_R600_RV730`` 0x006 ``rv730`` 1232 ``EF_AMDGPU_MACH_R600_RV770`` 0x007 ``rv770`` 1233 ``EF_AMDGPU_MACH_R600_CEDAR`` 0x008 ``cedar`` 1234 ``EF_AMDGPU_MACH_R600_CYPRESS`` 0x009 ``cypress`` 1235 ``EF_AMDGPU_MACH_R600_JUNIPER`` 0x00a ``juniper`` 1236 ``EF_AMDGPU_MACH_R600_REDWOOD`` 0x00b ``redwood`` 1237 ``EF_AMDGPU_MACH_R600_SUMO`` 0x00c ``sumo`` 1238 ``EF_AMDGPU_MACH_R600_BARTS`` 0x00d ``barts`` 1239 ``EF_AMDGPU_MACH_R600_CAICOS`` 0x00e ``caicos`` 1240 ``EF_AMDGPU_MACH_R600_CAYMAN`` 0x00f ``cayman`` 1241 ``EF_AMDGPU_MACH_R600_TURKS`` 0x010 ``turks`` 1242 *reserved* 0x011 - Reserved for ``r600`` 1243 0x01f architecture processors. 1244 ``EF_AMDGPU_MACH_AMDGCN_GFX600`` 0x020 ``gfx600`` 1245 ``EF_AMDGPU_MACH_AMDGCN_GFX601`` 0x021 ``gfx601`` 1246 ``EF_AMDGPU_MACH_AMDGCN_GFX700`` 0x022 ``gfx700`` 1247 ``EF_AMDGPU_MACH_AMDGCN_GFX701`` 0x023 ``gfx701`` 1248 ``EF_AMDGPU_MACH_AMDGCN_GFX702`` 0x024 ``gfx702`` 1249 ``EF_AMDGPU_MACH_AMDGCN_GFX703`` 0x025 ``gfx703`` 1250 ``EF_AMDGPU_MACH_AMDGCN_GFX704`` 0x026 ``gfx704`` 1251 *reserved* 0x027 Reserved. 1252 ``EF_AMDGPU_MACH_AMDGCN_GFX801`` 0x028 ``gfx801`` 1253 ``EF_AMDGPU_MACH_AMDGCN_GFX802`` 0x029 ``gfx802`` 1254 ``EF_AMDGPU_MACH_AMDGCN_GFX803`` 0x02a ``gfx803`` 1255 ``EF_AMDGPU_MACH_AMDGCN_GFX810`` 0x02b ``gfx810`` 1256 ``EF_AMDGPU_MACH_AMDGCN_GFX900`` 0x02c ``gfx900`` 1257 ``EF_AMDGPU_MACH_AMDGCN_GFX902`` 0x02d ``gfx902`` 1258 ``EF_AMDGPU_MACH_AMDGCN_GFX904`` 0x02e ``gfx904`` 1259 ``EF_AMDGPU_MACH_AMDGCN_GFX906`` 0x02f ``gfx906`` 1260 ``EF_AMDGPU_MACH_AMDGCN_GFX908`` 0x030 ``gfx908`` 1261 ``EF_AMDGPU_MACH_AMDGCN_GFX909`` 0x031 ``gfx909`` 1262 ``EF_AMDGPU_MACH_AMDGCN_GFX90C`` 0x032 ``gfx90c`` 1263 ``EF_AMDGPU_MACH_AMDGCN_GFX1010`` 0x033 ``gfx1010`` 1264 ``EF_AMDGPU_MACH_AMDGCN_GFX1011`` 0x034 ``gfx1011`` 1265 ``EF_AMDGPU_MACH_AMDGCN_GFX1012`` 0x035 ``gfx1012`` 1266 ``EF_AMDGPU_MACH_AMDGCN_GFX1030`` 0x036 ``gfx1030`` 1267 ``EF_AMDGPU_MACH_AMDGCN_GFX1031`` 0x037 ``gfx1031`` 1268 ``EF_AMDGPU_MACH_AMDGCN_GFX1032`` 0x038 ``gfx1032`` 1269 ``EF_AMDGPU_MACH_AMDGCN_GFX1033`` 0x039 ``gfx1033`` 1270 ``EF_AMDGPU_MACH_AMDGCN_GFX602`` 0x03a ``gfx602`` 1271 ``EF_AMDGPU_MACH_AMDGCN_GFX705`` 0x03b ``gfx705`` 1272 ``EF_AMDGPU_MACH_AMDGCN_GFX805`` 0x03c ``gfx805`` 1273 ``EF_AMDGPU_MACH_AMDGCN_GFX1035`` 0x03d ``gfx1035`` 1274 ``EF_AMDGPU_MACH_AMDGCN_GFX1034`` 0x03e ``gfx1034`` 1275 ``EF_AMDGPU_MACH_AMDGCN_GFX90A`` 0x03f ``gfx90a`` 1276 ``EF_AMDGPU_MACH_AMDGCN_GFX940`` 0x040 ``gfx940`` 1277 ``EF_AMDGPU_MACH_AMDGCN_GFX1100`` 0x041 ``gfx1100`` 1278 ``EF_AMDGPU_MACH_AMDGCN_GFX1013`` 0x042 ``gfx1013`` 1279 *reserved* 0x043 Reserved. 1280 ``EF_AMDGPU_MACH_AMDGCN_GFX1103`` 0x044 ``gfx1103`` 1281 ``EF_AMDGPU_MACH_AMDGCN_GFX1036`` 0x045 ``gfx1036`` 1282 ``EF_AMDGPU_MACH_AMDGCN_GFX1101`` 0x046 ``gfx1101`` 1283 ``EF_AMDGPU_MACH_AMDGCN_GFX1102`` 0x047 ``gfx1102`` 1284 ==================================== ========== ============================= 1285 1286Sections 1287-------- 1288 1289An AMDGPU target ELF code object has the standard ELF sections which include: 1290 1291 .. table:: AMDGPU ELF Sections 1292 :name: amdgpu-elf-sections-table 1293 1294 ================== ================ ================================= 1295 Name Type Attributes 1296 ================== ================ ================================= 1297 ``.bss`` ``SHT_NOBITS`` ``SHF_ALLOC`` + ``SHF_WRITE`` 1298 ``.data`` ``SHT_PROGBITS`` ``SHF_ALLOC`` + ``SHF_WRITE`` 1299 ``.debug_``\ *\** ``SHT_PROGBITS`` *none* 1300 ``.dynamic`` ``SHT_DYNAMIC`` ``SHF_ALLOC`` 1301 ``.dynstr`` ``SHT_PROGBITS`` ``SHF_ALLOC`` 1302 ``.dynsym`` ``SHT_PROGBITS`` ``SHF_ALLOC`` 1303 ``.got`` ``SHT_PROGBITS`` ``SHF_ALLOC`` + ``SHF_WRITE`` 1304 ``.hash`` ``SHT_HASH`` ``SHF_ALLOC`` 1305 ``.note`` ``SHT_NOTE`` *none* 1306 ``.rela``\ *name* ``SHT_RELA`` *none* 1307 ``.rela.dyn`` ``SHT_RELA`` *none* 1308 ``.rodata`` ``SHT_PROGBITS`` ``SHF_ALLOC`` 1309 ``.shstrtab`` ``SHT_STRTAB`` *none* 1310 ``.strtab`` ``SHT_STRTAB`` *none* 1311 ``.symtab`` ``SHT_SYMTAB`` *none* 1312 ``.text`` ``SHT_PROGBITS`` ``SHF_ALLOC`` + ``SHF_EXECINSTR`` 1313 ================== ================ ================================= 1314 1315These sections have their standard meanings (see [ELF]_) and are only generated 1316if needed. 1317 1318``.debug``\ *\** 1319 The standard DWARF sections. See :ref:`amdgpu-dwarf-debug-information` for 1320 information on the DWARF produced by the AMDGPU backend. 1321 1322``.dynamic``, ``.dynstr``, ``.dynsym``, ``.hash`` 1323 The standard sections used by a dynamic loader. 1324 1325``.note`` 1326 See :ref:`amdgpu-note-records` for the note records supported by the AMDGPU 1327 backend. 1328 1329``.rela``\ *name*, ``.rela.dyn`` 1330 For relocatable code objects, *name* is the name of the section that the 1331 relocation records apply. For example, ``.rela.text`` is the section name for 1332 relocation records associated with the ``.text`` section. 1333 1334 For linked shared code objects, ``.rela.dyn`` contains all the relocation 1335 records from each of the relocatable code object's ``.rela``\ *name* sections. 1336 1337 See :ref:`amdgpu-relocation-records` for the relocation records supported by 1338 the AMDGPU backend. 1339 1340``.text`` 1341 The executable machine code for the kernels and functions they call. Generated 1342 as position independent code. See :ref:`amdgpu-code-conventions` for 1343 information on conventions used in the isa generation. 1344 1345.. _amdgpu-note-records: 1346 1347Note Records 1348------------ 1349 1350The AMDGPU backend code object contains ELF note records in the ``.note`` 1351section. The set of generated notes and their semantics depend on the code 1352object version; see :ref:`amdgpu-note-records-v2` and 1353:ref:`amdgpu-note-records-v3-onwards`. 1354 1355As required by ``ELFCLASS32`` and ``ELFCLASS64``, minimal zero-byte padding 1356must be generated after the ``name`` field to ensure the ``desc`` field is 4 1357byte aligned. In addition, minimal zero-byte padding must be generated to 1358ensure the ``desc`` field size is a multiple of 4 bytes. The ``sh_addralign`` 1359field of the ``.note`` section must be at least 4 to indicate at least 8 byte 1360alignment. 1361 1362.. _amdgpu-note-records-v2: 1363 1364Code Object V2 Note Records 1365~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1366 1367.. warning:: 1368 Code object V2 is not the default code object version emitted by 1369 this version of LLVM. 1370 1371The AMDGPU backend code object uses the following ELF note record in the 1372``.note`` section when compiling for code object V2. 1373 1374The note record vendor field is "AMD". 1375 1376Additional note records may be present, but any which are not documented here 1377are deprecated and should not be used. 1378 1379 .. table:: AMDGPU Code Object V2 ELF Note Records 1380 :name: amdgpu-elf-note-records-v2-table 1381 1382 ===== ===================================== ====================================== 1383 Name Type Description 1384 ===== ===================================== ====================================== 1385 "AMD" ``NT_AMD_HSA_CODE_OBJECT_VERSION`` Code object version. 1386 "AMD" ``NT_AMD_HSA_HSAIL`` HSAIL properties generated by the HSAIL 1387 Finalizer and not the LLVM compiler. 1388 "AMD" ``NT_AMD_HSA_ISA_VERSION`` Target ISA version. 1389 "AMD" ``NT_AMD_HSA_METADATA`` Metadata null terminated string in 1390 YAML [YAML]_ textual format. 1391 "AMD" ``NT_AMD_HSA_ISA_NAME`` Target ISA name. 1392 ===== ===================================== ====================================== 1393 1394.. 1395 1396 .. table:: AMDGPU Code Object V2 ELF Note Record Enumeration Values 1397 :name: amdgpu-elf-note-record-enumeration-values-v2-table 1398 1399 ===================================== ===== 1400 Name Value 1401 ===================================== ===== 1402 ``NT_AMD_HSA_CODE_OBJECT_VERSION`` 1 1403 ``NT_AMD_HSA_HSAIL`` 2 1404 ``NT_AMD_HSA_ISA_VERSION`` 3 1405 *reserved* 4-9 1406 ``NT_AMD_HSA_METADATA`` 10 1407 ``NT_AMD_HSA_ISA_NAME`` 11 1408 ===================================== ===== 1409 1410``NT_AMD_HSA_CODE_OBJECT_VERSION`` 1411 Specifies the code object version number. The description field has the 1412 following layout: 1413 1414 .. code:: c 1415 1416 struct amdgpu_hsa_note_code_object_version_s { 1417 uint32_t major_version; 1418 uint32_t minor_version; 1419 }; 1420 1421 The ``major_version`` has a value less than or equal to 2. 1422 1423``NT_AMD_HSA_HSAIL`` 1424 Specifies the HSAIL properties used by the HSAIL Finalizer. The description 1425 field has the following layout: 1426 1427 .. code:: c 1428 1429 struct amdgpu_hsa_note_hsail_s { 1430 uint32_t hsail_major_version; 1431 uint32_t hsail_minor_version; 1432 uint8_t profile; 1433 uint8_t machine_model; 1434 uint8_t default_float_round; 1435 }; 1436 1437``NT_AMD_HSA_ISA_VERSION`` 1438 Specifies the target ISA version. The description field has the following layout: 1439 1440 .. code:: c 1441 1442 struct amdgpu_hsa_note_isa_s { 1443 uint16_t vendor_name_size; 1444 uint16_t architecture_name_size; 1445 uint32_t major; 1446 uint32_t minor; 1447 uint32_t stepping; 1448 char vendor_and_architecture_name[1]; 1449 }; 1450 1451 ``vendor_name_size`` and ``architecture_name_size`` are the length of the 1452 vendor and architecture names respectively, including the NUL character. 1453 1454 ``vendor_and_architecture_name`` contains the NUL terminates string for the 1455 vendor, immediately followed by the NUL terminated string for the 1456 architecture. 1457 1458 This note record is used by the HSA runtime loader. 1459 1460 Code object V2 only supports a limited number of processors and has fixed 1461 settings for target features. See 1462 :ref:`amdgpu-elf-note-record-supported_processors-v2-table` for a list of 1463 processors and the corresponding target ID. In the table the note record ISA 1464 name is a concatenation of the vendor name, architecture name, major, minor, 1465 and stepping separated by a ":". 1466 1467 The target ID column shows the processor name and fixed target features used 1468 by the LLVM compiler. The LLVM compiler does not generate a 1469 ``NT_AMD_HSA_HSAIL`` note record. 1470 1471 A code object generated by the Finalizer also uses code object V2 and always 1472 generates a ``NT_AMD_HSA_HSAIL`` note record. The processor name and 1473 ``sramecc`` target feature is as shown in 1474 :ref:`amdgpu-elf-note-record-supported_processors-v2-table` but the ``xnack`` 1475 target feature is specified by the ``EF_AMDGPU_FEATURE_XNACK_V2`` ``e_flags`` 1476 bit. 1477 1478``NT_AMD_HSA_ISA_NAME`` 1479 Specifies the target ISA name as a non-NUL terminated string. 1480 1481 This note record is not used by the HSA runtime loader. 1482 1483 See the ``NT_AMD_HSA_ISA_VERSION`` note record description of the code object 1484 V2's limited support of processors and fixed settings for target features. 1485 1486 See :ref:`amdgpu-elf-note-record-supported_processors-v2-table` for a mapping 1487 from the string to the corresponding target ID. If the ``xnack`` target 1488 feature is supported and enabled, the string produced by the LLVM compiler 1489 will may have a ``+xnack`` appended. The Finlizer did not do the appending and 1490 instead used the ``EF_AMDGPU_FEATURE_XNACK_V2`` ``e_flags`` bit. 1491 1492``NT_AMD_HSA_METADATA`` 1493 Specifies extensible metadata associated with the code objects executed on HSA 1494 [HSA]_ compatible runtimes (see :ref:`amdgpu-os`). It is required when the 1495 target triple OS is ``amdhsa`` (see :ref:`amdgpu-target-triples`). See 1496 :ref:`amdgpu-amdhsa-code-object-metadata-v2` for the syntax of the code object 1497 metadata string. 1498 1499 .. table:: AMDGPU Code Object V2 Supported Processors and Fixed Target Feature Settings 1500 :name: amdgpu-elf-note-record-supported_processors-v2-table 1501 1502 ===================== ========================== 1503 Note Record ISA Name Target ID 1504 ===================== ========================== 1505 ``AMD:AMDGPU:6:0:0`` ``gfx600`` 1506 ``AMD:AMDGPU:6:0:1`` ``gfx601`` 1507 ``AMD:AMDGPU:6:0:2`` ``gfx602`` 1508 ``AMD:AMDGPU:7:0:0`` ``gfx700`` 1509 ``AMD:AMDGPU:7:0:1`` ``gfx701`` 1510 ``AMD:AMDGPU:7:0:2`` ``gfx702`` 1511 ``AMD:AMDGPU:7:0:3`` ``gfx703`` 1512 ``AMD:AMDGPU:7:0:4`` ``gfx704`` 1513 ``AMD:AMDGPU:7:0:5`` ``gfx705`` 1514 ``AMD:AMDGPU:8:0:0`` ``gfx802`` 1515 ``AMD:AMDGPU:8:0:1`` ``gfx801:xnack+`` 1516 ``AMD:AMDGPU:8:0:2`` ``gfx802`` 1517 ``AMD:AMDGPU:8:0:3`` ``gfx803`` 1518 ``AMD:AMDGPU:8:0:4`` ``gfx803`` 1519 ``AMD:AMDGPU:8:0:5`` ``gfx805`` 1520 ``AMD:AMDGPU:8:1:0`` ``gfx810:xnack+`` 1521 ``AMD:AMDGPU:9:0:0`` ``gfx900:xnack-`` 1522 ``AMD:AMDGPU:9:0:1`` ``gfx900:xnack+`` 1523 ``AMD:AMDGPU:9:0:2`` ``gfx902:xnack-`` 1524 ``AMD:AMDGPU:9:0:3`` ``gfx902:xnack+`` 1525 ``AMD:AMDGPU:9:0:4`` ``gfx904:xnack-`` 1526 ``AMD:AMDGPU:9:0:5`` ``gfx904:xnack+`` 1527 ``AMD:AMDGPU:9:0:6`` ``gfx906:sramecc-:xnack-`` 1528 ``AMD:AMDGPU:9:0:7`` ``gfx906:sramecc-:xnack+`` 1529 ``AMD:AMDGPU:9:0:12`` ``gfx90c:xnack-`` 1530 ===================== ========================== 1531 1532.. _amdgpu-note-records-v3-onwards: 1533 1534Code Object V3 and Above Note Records 1535~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1536 1537The AMDGPU backend code object uses the following ELF note record in the 1538``.note`` section when compiling for code object V3 and above. 1539 1540The note record vendor field is "AMDGPU". 1541 1542Additional note records may be present, but any which are not documented here 1543are deprecated and should not be used. 1544 1545 .. table:: AMDGPU Code Object V3 and Above ELF Note Records 1546 :name: amdgpu-elf-note-records-table-v3-onwards 1547 1548 ======== ============================== ====================================== 1549 Name Type Description 1550 ======== ============================== ====================================== 1551 "AMDGPU" ``NT_AMDGPU_METADATA`` Metadata in Message Pack [MsgPack]_ 1552 binary format. 1553 ======== ============================== ====================================== 1554 1555.. 1556 1557 .. table:: AMDGPU Code Object V3 and Above ELF Note Record Enumeration Values 1558 :name: amdgpu-elf-note-record-enumeration-values-table-v3-onwards 1559 1560 ============================== ===== 1561 Name Value 1562 ============================== ===== 1563 *reserved* 0-31 1564 ``NT_AMDGPU_METADATA`` 32 1565 ============================== ===== 1566 1567``NT_AMDGPU_METADATA`` 1568 Specifies extensible metadata associated with an AMDGPU code object. It is 1569 encoded as a map in the Message Pack [MsgPack]_ binary data format. See 1570 :ref:`amdgpu-amdhsa-code-object-metadata-v3`, 1571 :ref:`amdgpu-amdhsa-code-object-metadata-v4` and 1572 :ref:`amdgpu-amdhsa-code-object-metadata-v5` for the map keys defined for the 1573 ``amdhsa`` OS. 1574 1575.. _amdgpu-symbols: 1576 1577Symbols 1578------- 1579 1580Symbols include the following: 1581 1582 .. table:: AMDGPU ELF Symbols 1583 :name: amdgpu-elf-symbols-table 1584 1585 ===================== ================== ================ ================== 1586 Name Type Section Description 1587 ===================== ================== ================ ================== 1588 *link-name* ``STT_OBJECT`` - ``.data`` Global variable 1589 - ``.rodata`` 1590 - ``.bss`` 1591 *link-name*\ ``.kd`` ``STT_OBJECT`` - ``.rodata`` Kernel descriptor 1592 *link-name* ``STT_FUNC`` - ``.text`` Kernel entry point 1593 *link-name* ``STT_OBJECT`` - SHN_AMDGPU_LDS Global variable in LDS 1594 ===================== ================== ================ ================== 1595 1596Global variable 1597 Global variables both used and defined by the compilation unit. 1598 1599 If the symbol is defined in the compilation unit then it is allocated in the 1600 appropriate section according to if it has initialized data or is readonly. 1601 1602 If the symbol is external then its section is ``STN_UNDEF`` and the loader 1603 will resolve relocations using the definition provided by another code object 1604 or explicitly defined by the runtime. 1605 1606 If the symbol resides in local/group memory (LDS) then its section is the 1607 special processor specific section name ``SHN_AMDGPU_LDS``, and the 1608 ``st_value`` field describes alignment requirements as it does for common 1609 symbols. 1610 1611 .. TODO:: 1612 1613 Add description of linked shared object symbols. Seems undefined symbols 1614 are marked as STT_NOTYPE. 1615 1616Kernel descriptor 1617 Every HSA kernel has an associated kernel descriptor. It is the address of the 1618 kernel descriptor that is used in the AQL dispatch packet used to invoke the 1619 kernel, not the kernel entry point. The layout of the HSA kernel descriptor is 1620 defined in :ref:`amdgpu-amdhsa-kernel-descriptor`. 1621 1622Kernel entry point 1623 Every HSA kernel also has a symbol for its machine code entry point. 1624 1625.. _amdgpu-relocation-records: 1626 1627Relocation Records 1628------------------ 1629 1630AMDGPU backend generates ``Elf64_Rela`` relocation records. Supported 1631relocatable fields are: 1632 1633``word32`` 1634 This specifies a 32-bit field occupying 4 bytes with arbitrary byte 1635 alignment. These values use the same byte order as other word values in the 1636 AMDGPU architecture. 1637 1638``word64`` 1639 This specifies a 64-bit field occupying 8 bytes with arbitrary byte 1640 alignment. These values use the same byte order as other word values in the 1641 AMDGPU architecture. 1642 1643Following notations are used for specifying relocation calculations: 1644 1645**A** 1646 Represents the addend used to compute the value of the relocatable field. 1647 1648**G** 1649 Represents the offset into the global offset table at which the relocation 1650 entry's symbol will reside during execution. 1651 1652**GOT** 1653 Represents the address of the global offset table. 1654 1655**P** 1656 Represents the place (section offset for ``et_rel`` or address for ``et_dyn``) 1657 of the storage unit being relocated (computed using ``r_offset``). 1658 1659**S** 1660 Represents the value of the symbol whose index resides in the relocation 1661 entry. Relocations not using this must specify a symbol index of 1662 ``STN_UNDEF``. 1663 1664**B** 1665 Represents the base address of a loaded executable or shared object which is 1666 the difference between the ELF address and the actual load address. 1667 Relocations using this are only valid in executable or shared objects. 1668 1669The following relocation types are supported: 1670 1671 .. table:: AMDGPU ELF Relocation Records 1672 :name: amdgpu-elf-relocation-records-table 1673 1674 ========================== ======= ===== ========== ============================== 1675 Relocation Type Kind Value Field Calculation 1676 ========================== ======= ===== ========== ============================== 1677 ``R_AMDGPU_NONE`` 0 *none* *none* 1678 ``R_AMDGPU_ABS32_LO`` Static, 1 ``word32`` (S + A) & 0xFFFFFFFF 1679 Dynamic 1680 ``R_AMDGPU_ABS32_HI`` Static, 2 ``word32`` (S + A) >> 32 1681 Dynamic 1682 ``R_AMDGPU_ABS64`` Static, 3 ``word64`` S + A 1683 Dynamic 1684 ``R_AMDGPU_REL32`` Static 4 ``word32`` S + A - P 1685 ``R_AMDGPU_REL64`` Static 5 ``word64`` S + A - P 1686 ``R_AMDGPU_ABS32`` Static, 6 ``word32`` S + A 1687 Dynamic 1688 ``R_AMDGPU_GOTPCREL`` Static 7 ``word32`` G + GOT + A - P 1689 ``R_AMDGPU_GOTPCREL32_LO`` Static 8 ``word32`` (G + GOT + A - P) & 0xFFFFFFFF 1690 ``R_AMDGPU_GOTPCREL32_HI`` Static 9 ``word32`` (G + GOT + A - P) >> 32 1691 ``R_AMDGPU_REL32_LO`` Static 10 ``word32`` (S + A - P) & 0xFFFFFFFF 1692 ``R_AMDGPU_REL32_HI`` Static 11 ``word32`` (S + A - P) >> 32 1693 *reserved* 12 1694 ``R_AMDGPU_RELATIVE64`` Dynamic 13 ``word64`` B + A 1695 ``R_AMDGPU_REL16`` Static 14 ``word16`` ((S + A - P) - 4) / 4 1696 ========================== ======= ===== ========== ============================== 1697 1698``R_AMDGPU_ABS32_LO`` and ``R_AMDGPU_ABS32_HI`` are only supported by 1699the ``mesa3d`` OS, which does not support ``R_AMDGPU_ABS64``. 1700 1701There is no current OS loader support for 32-bit programs and so 1702``R_AMDGPU_ABS32`` is not used. 1703 1704.. _amdgpu-loaded-code-object-path-uniform-resource-identifier: 1705 1706Loaded Code Object Path Uniform Resource Identifier (URI) 1707--------------------------------------------------------- 1708 1709The AMD GPU code object loader represents the path of the ELF shared object from 1710which the code object was loaded as a textual Uniform Resource Identifier (URI). 1711Note that the code object is the in memory loaded relocated form of the ELF 1712shared object. Multiple code objects may be loaded at different memory 1713addresses in the same process from the same ELF shared object. 1714 1715The loaded code object path URI syntax is defined by the following BNF syntax: 1716 1717.. code:: 1718 1719 code_object_uri ::== file_uri | memory_uri 1720 file_uri ::== "file://" file_path [ range_specifier ] 1721 memory_uri ::== "memory://" process_id range_specifier 1722 range_specifier ::== [ "#" | "?" ] "offset=" number "&" "size=" number 1723 file_path ::== URI_ENCODED_OS_FILE_PATH 1724 process_id ::== DECIMAL_NUMBER 1725 number ::== HEX_NUMBER | DECIMAL_NUMBER | OCTAL_NUMBER 1726 1727**number** 1728 Is a C integral literal where hexadecimal values are prefixed by "0x" or "0X", 1729 and octal values by "0". 1730 1731**file_path** 1732 Is the file's path specified as a URI encoded UTF-8 string. In URI encoding, 1733 every character that is not in the regular expression ``[a-zA-Z0-9/_.~-]`` is 1734 encoded as two uppercase hexadecimal digits proceeded by "%". Directories in 1735 the path are separated by "/". 1736 1737**offset** 1738 Is a 0-based byte offset to the start of the code object. For a file URI, it 1739 is from the start of the file specified by the ``file_path``, and if omitted 1740 defaults to 0. For a memory URI, it is the memory address and is required. 1741 1742**size** 1743 Is the number of bytes in the code object. For a file URI, if omitted it 1744 defaults to the size of the file. It is required for a memory URI. 1745 1746**process_id** 1747 Is the identity of the process owning the memory. For Linux it is the C 1748 unsigned integral decimal literal for the process ID (PID). 1749 1750For example: 1751 1752.. code:: 1753 1754 file:///dir1/dir2/file1 1755 file:///dir3/dir4/file2#offset=0x2000&size=3000 1756 memory://1234#offset=0x20000&size=3000 1757 1758.. _amdgpu-dwarf-debug-information: 1759 1760DWARF Debug Information 1761======================= 1762 1763.. warning:: 1764 1765 This section describes **provisional support** for AMDGPU DWARF [DWARF]_ that 1766 is not currently fully implemented and is subject to change. 1767 1768AMDGPU generates DWARF [DWARF]_ debugging information ELF sections (see 1769:ref:`amdgpu-elf-code-object`) which contain information that maps the code 1770object executable code and data to the source language constructs. It can be 1771used by tools such as debuggers and profilers. It uses features defined in 1772:doc:`AMDGPUDwarfExtensionsForHeterogeneousDebugging` that are made available in 1773DWARF Version 4 and DWARF Version 5 as an LLVM vendor extension. 1774 1775This section defines the AMDGPU target architecture specific DWARF mappings. 1776 1777.. _amdgpu-dwarf-register-identifier: 1778 1779Register Identifier 1780------------------- 1781 1782This section defines the AMDGPU target architecture register numbers used in 1783DWARF operation expressions (see DWARF Version 5 section 2.5 and 1784:ref:`amdgpu-dwarf-operation-expressions`) and Call Frame Information 1785instructions (see DWARF Version 5 section 6.4 and 1786:ref:`amdgpu-dwarf-call-frame-information`). 1787 1788A single code object can contain code for kernels that have different wavefront 1789sizes. The vector registers and some scalar registers are based on the wavefront 1790size. AMDGPU defines distinct DWARF registers for each wavefront size. This 1791simplifies the consumer of the DWARF so that each register has a fixed size, 1792rather than being dynamic according to the wavefront size mode. Similarly, 1793distinct DWARF registers are defined for those registers that vary in size 1794according to the process address size. This allows a consumer to treat a 1795specific AMDGPU processor as a single architecture regardless of how it is 1796configured at run time. The compiler explicitly specifies the DWARF registers 1797that match the mode in which the code it is generating will be executed. 1798 1799DWARF registers are encoded as numbers, which are mapped to architecture 1800registers. The mapping for AMDGPU is defined in 1801:ref:`amdgpu-dwarf-register-mapping-table`. All AMDGPU targets use the same 1802mapping. 1803 1804.. table:: AMDGPU DWARF Register Mapping 1805 :name: amdgpu-dwarf-register-mapping-table 1806 1807 ============== ================= ======== ================================== 1808 DWARF Register AMDGPU Register Bit Size Description 1809 ============== ================= ======== ================================== 1810 0 PC_32 32 Program Counter (PC) when 1811 executing in a 32-bit process 1812 address space. Used in the CFI to 1813 describe the PC of the calling 1814 frame. 1815 1 EXEC_MASK_32 32 Execution Mask Register when 1816 executing in wavefront 32 mode. 1817 2-15 *Reserved* *Reserved for highly accessed 1818 registers using DWARF shortcut.* 1819 16 PC_64 64 Program Counter (PC) when 1820 executing in a 64-bit process 1821 address space. Used in the CFI to 1822 describe the PC of the calling 1823 frame. 1824 17 EXEC_MASK_64 64 Execution Mask Register when 1825 executing in wavefront 64 mode. 1826 18-31 *Reserved* *Reserved for highly accessed 1827 registers using DWARF shortcut.* 1828 32-95 SGPR0-SGPR63 32 Scalar General Purpose 1829 Registers. 1830 96-127 *Reserved* *Reserved for frequently accessed 1831 registers using DWARF 1-byte ULEB.* 1832 128 STATUS 32 Status Register. 1833 129-511 *Reserved* *Reserved for future Scalar 1834 Architectural Registers.* 1835 512 VCC_32 32 Vector Condition Code Register 1836 when executing in wavefront 32 1837 mode. 1838 513-767 *Reserved* *Reserved for future Vector 1839 Architectural Registers when 1840 executing in wavefront 32 mode.* 1841 768 VCC_64 64 Vector Condition Code Register 1842 when executing in wavefront 64 1843 mode. 1844 769-1023 *Reserved* *Reserved for future Vector 1845 Architectural Registers when 1846 executing in wavefront 64 mode.* 1847 1024-1087 *Reserved* *Reserved for padding.* 1848 1088-1129 SGPR64-SGPR105 32 Scalar General Purpose Registers. 1849 1130-1535 *Reserved* *Reserved for future Scalar 1850 General Purpose Registers.* 1851 1536-1791 VGPR0-VGPR255 32*32 Vector General Purpose Registers 1852 when executing in wavefront 32 1853 mode. 1854 1792-2047 *Reserved* *Reserved for future Vector 1855 General Purpose Registers when 1856 executing in wavefront 32 mode.* 1857 2048-2303 AGPR0-AGPR255 32*32 Vector Accumulation Registers 1858 when executing in wavefront 32 1859 mode. 1860 2304-2559 *Reserved* *Reserved for future Vector 1861 Accumulation Registers when 1862 executing in wavefront 32 mode.* 1863 2560-2815 VGPR0-VGPR255 64*32 Vector General Purpose Registers 1864 when executing in wavefront 64 1865 mode. 1866 2816-3071 *Reserved* *Reserved for future Vector 1867 General Purpose Registers when 1868 executing in wavefront 64 mode.* 1869 3072-3327 AGPR0-AGPR255 64*32 Vector Accumulation Registers 1870 when executing in wavefront 64 1871 mode. 1872 3328-3583 *Reserved* *Reserved for future Vector 1873 Accumulation Registers when 1874 executing in wavefront 64 mode.* 1875 ============== ================= ======== ================================== 1876 1877The vector registers are represented as the full size for the wavefront. They 1878are organized as consecutive dwords (32-bits), one per lane, with the dword at 1879the least significant bit position corresponding to lane 0 and so forth. DWARF 1880location expressions involving the ``DW_OP_LLVM_offset`` and 1881``DW_OP_LLVM_push_lane`` operations are used to select the part of the vector 1882register corresponding to the lane that is executing the current thread of 1883execution in languages that are implemented using a SIMD or SIMT execution 1884model. 1885 1886If the wavefront size is 32 lanes then the wavefront 32 mode register 1887definitions are used. If the wavefront size is 64 lanes then the wavefront 64 1888mode register definitions are used. Some AMDGPU targets support executing in 1889both wavefront 32 and wavefront 64 mode. The register definitions corresponding 1890to the wavefront mode of the generated code will be used. 1891 1892If code is generated to execute in a 32-bit process address space, then the 189332-bit process address space register definitions are used. If code is generated 1894to execute in a 64-bit process address space, then the 64-bit process address 1895space register definitions are used. The ``amdgcn`` target only supports the 189664-bit process address space. 1897 1898.. _amdgpu-dwarf-address-class-identifier: 1899 1900Address Class Identifier 1901------------------------ 1902 1903The DWARF address class represents the source language memory space. See DWARF 1904Version 5 section 2.12 which is updated by the *DWARF Extensions For 1905Heterogeneous Debugging* section :ref:`amdgpu-dwarf-segment_addresses`. 1906 1907The DWARF address class mapping used for AMDGPU is defined in 1908:ref:`amdgpu-dwarf-address-class-mapping-table`. 1909 1910.. table:: AMDGPU DWARF Address Class Mapping 1911 :name: amdgpu-dwarf-address-class-mapping-table 1912 1913 ========================= ====== ================= 1914 DWARF AMDGPU 1915 -------------------------------- ----------------- 1916 Address Class Name Value Address Space 1917 ========================= ====== ================= 1918 ``DW_ADDR_none`` 0x0000 Generic (Flat) 1919 ``DW_ADDR_LLVM_global`` 0x0001 Global 1920 ``DW_ADDR_LLVM_constant`` 0x0002 Global 1921 ``DW_ADDR_LLVM_group`` 0x0003 Local (group/LDS) 1922 ``DW_ADDR_LLVM_private`` 0x0004 Private (Scratch) 1923 ``DW_ADDR_AMDGPU_region`` 0x8000 Region (GDS) 1924 ========================= ====== ================= 1925 1926The DWARF address class values defined in the *DWARF Extensions For 1927Heterogeneous Debugging* section :ref:`amdgpu-dwarf-segment_addresses` are used. 1928 1929In addition, ``DW_ADDR_AMDGPU_region`` is encoded as a vendor extension. This is 1930available for use for the AMD extension for access to the hardware GDS memory 1931which is scratchpad memory allocated per device. 1932 1933For AMDGPU if no ``DW_AT_address_class`` attribute is present, then the default 1934address class of ``DW_ADDR_none`` is used. 1935 1936See :ref:`amdgpu-dwarf-address-space-identifier` for information on the AMDGPU 1937mapping of DWARF address classes to DWARF address spaces, including address size 1938and NULL value. 1939 1940.. _amdgpu-dwarf-address-space-identifier: 1941 1942Address Space Identifier 1943------------------------ 1944 1945DWARF address spaces correspond to target architecture specific linear 1946addressable memory areas. See DWARF Version 5 section 2.12 and *DWARF Extensions 1947For Heterogeneous Debugging* section :ref:`amdgpu-dwarf-segment_addresses`. 1948 1949The DWARF address space mapping used for AMDGPU is defined in 1950:ref:`amdgpu-dwarf-address-space-mapping-table`. 1951 1952.. table:: AMDGPU DWARF Address Space Mapping 1953 :name: amdgpu-dwarf-address-space-mapping-table 1954 1955 ======================================= ===== ======= ======== ================= ======================= 1956 DWARF AMDGPU Notes 1957 --------------------------------------- ----- ---------------- ----------------- ----------------------- 1958 Address Space Name Value Address Bit Size Address Space 1959 --------------------------------------- ----- ------- -------- ----------------- ----------------------- 1960 .. 64-bit 32-bit 1961 process process 1962 address address 1963 space space 1964 ======================================= ===== ======= ======== ================= ======================= 1965 ``DW_ASPACE_none`` 0x00 64 32 Global *default address space* 1966 ``DW_ASPACE_AMDGPU_generic`` 0x01 64 32 Generic (Flat) 1967 ``DW_ASPACE_AMDGPU_region`` 0x02 32 32 Region (GDS) 1968 ``DW_ASPACE_AMDGPU_local`` 0x03 32 32 Local (group/LDS) 1969 *Reserved* 0x04 1970 ``DW_ASPACE_AMDGPU_private_lane`` 0x05 32 32 Private (Scratch) *focused lane* 1971 ``DW_ASPACE_AMDGPU_private_wave`` 0x06 32 32 Private (Scratch) *unswizzled wavefront* 1972 ======================================= ===== ======= ======== ================= ======================= 1973 1974See :ref:`amdgpu-address-spaces` for information on the AMDGPU address spaces 1975including address size and NULL value. 1976 1977The ``DW_ASPACE_none`` address space is the default target architecture address 1978space used in DWARF operations that do not specify an address space. It 1979therefore has to map to the global address space so that the ``DW_OP_addr*`` and 1980related operations can refer to addresses in the program code. 1981 1982The ``DW_ASPACE_AMDGPU_generic`` address space allows location expressions to 1983specify the flat address space. If the address corresponds to an address in the 1984local address space, then it corresponds to the wavefront that is executing the 1985focused thread of execution. If the address corresponds to an address in the 1986private address space, then it corresponds to the lane that is executing the 1987focused thread of execution for languages that are implemented using a SIMD or 1988SIMT execution model. 1989 1990.. note:: 1991 1992 CUDA-like languages such as HIP that do not have address spaces in the 1993 language type system, but do allow variables to be allocated in different 1994 address spaces, need to explicitly specify the ``DW_ASPACE_AMDGPU_generic`` 1995 address space in the DWARF expression operations as the default address space 1996 is the global address space. 1997 1998The ``DW_ASPACE_AMDGPU_local`` address space allows location expressions to 1999specify the local address space corresponding to the wavefront that is executing 2000the focused thread of execution. 2001 2002The ``DW_ASPACE_AMDGPU_private_lane`` address space allows location expressions 2003to specify the private address space corresponding to the lane that is executing 2004the focused thread of execution for languages that are implemented using a SIMD 2005or SIMT execution model. 2006 2007The ``DW_ASPACE_AMDGPU_private_wave`` address space allows location expressions 2008to specify the unswizzled private address space corresponding to the wavefront 2009that is executing the focused thread of execution. The wavefront view of private 2010memory is the per wavefront unswizzled backing memory layout defined in 2011:ref:`amdgpu-address-spaces`, such that address 0 corresponds to the first 2012location for the backing memory of the wavefront (namely the address is not 2013offset by ``wavefront-scratch-base``). The following formula can be used to 2014convert from a ``DW_ASPACE_AMDGPU_private_lane`` address to a 2015``DW_ASPACE_AMDGPU_private_wave`` address: 2016 2017:: 2018 2019 private-address-wavefront = 2020 ((private-address-lane / 4) * wavefront-size * 4) + 2021 (wavefront-lane-id * 4) + (private-address-lane % 4) 2022 2023If the ``DW_ASPACE_AMDGPU_private_lane`` address is dword aligned, and the start 2024of the dwords for each lane starting with lane 0 is required, then this 2025simplifies to: 2026 2027:: 2028 2029 private-address-wavefront = 2030 private-address-lane * wavefront-size 2031 2032A compiler can use the ``DW_ASPACE_AMDGPU_private_wave`` address space to read a 2033complete spilled vector register back into a complete vector register in the 2034CFI. The frame pointer can be a private lane address which is dword aligned, 2035which can be shifted to multiply by the wavefront size, and then used to form a 2036private wavefront address that gives a location for a contiguous set of dwords, 2037one per lane, where the vector register dwords are spilled. The compiler knows 2038the wavefront size since it generates the code. Note that the type of the 2039address may have to be converted as the size of a 2040``DW_ASPACE_AMDGPU_private_lane`` address may be smaller than the size of a 2041``DW_ASPACE_AMDGPU_private_wave`` address. 2042 2043.. _amdgpu-dwarf-lane-identifier: 2044 2045Lane identifier 2046--------------- 2047 2048DWARF lane identifies specify a target architecture lane position for hardware 2049that executes in a SIMD or SIMT manner, and on which a source language maps its 2050threads of execution onto those lanes. The DWARF lane identifier is pushed by 2051the ``DW_OP_LLVM_push_lane`` DWARF expression operation. See DWARF Version 5 2052section 2.5 which is updated by *DWARF Extensions For Heterogeneous Debugging* 2053section :ref:`amdgpu-dwarf-operation-expressions`. 2054 2055For AMDGPU, the lane identifier corresponds to the hardware lane ID of a 2056wavefront. It is numbered from 0 to the wavefront size minus 1. 2057 2058Operation Expressions 2059--------------------- 2060 2061DWARF expressions are used to compute program values and the locations of 2062program objects. See DWARF Version 5 section 2.5 and 2063:ref:`amdgpu-dwarf-operation-expressions`. 2064 2065DWARF location descriptions describe how to access storage which includes memory 2066and registers. When accessing storage on AMDGPU, bytes are ordered with least 2067significant bytes first, and bits are ordered within bytes with least 2068significant bits first. 2069 2070For AMDGPU CFI expressions, ``DW_OP_LLVM_select_bit_piece`` is used to describe 2071unwinding vector registers that are spilled under the execution mask to memory: 2072the zero-single location description is the vector register, and the one-single 2073location description is the spilled memory location description. The 2074``DW_OP_LLVM_form_aspace_address`` is used to specify the address space of the 2075memory location description. 2076 2077In AMDGPU expressions, ``DW_OP_LLVM_select_bit_piece`` is used by the 2078``DW_AT_LLVM_lane_pc`` attribute expression where divergent control flow is 2079controlled by the execution mask. An undefined location description together 2080with ``DW_OP_LLVM_extend`` is used to indicate the lane was not active on entry 2081to the subprogram. See :ref:`amdgpu-dwarf-dw-at-llvm-lane-pc` for an example. 2082 2083Debugger Information Entry Attributes 2084------------------------------------- 2085 2086This section describes how certain debugger information entry attributes are 2087used by AMDGPU. See the sections in DWARF Version 5 section 3.3.5 and 3.1.1 2088which are updated by *DWARF Extensions For Heterogeneous Debugging* section 2089:ref:`amdgpu-dwarf-low-level-information` and 2090:ref:`amdgpu-dwarf-full-and-partial-compilation-unit-entries`. 2091 2092.. _amdgpu-dwarf-dw-at-llvm-lane-pc: 2093 2094``DW_AT_LLVM_lane_pc`` 2095~~~~~~~~~~~~~~~~~~~~~~ 2096 2097For AMDGPU, the ``DW_AT_LLVM_lane_pc`` attribute is used to specify the program 2098location of the separate lanes of a SIMT thread. 2099 2100If the lane is an active lane then this will be the same as the current program 2101location. 2102 2103If the lane is inactive, but was active on entry to the subprogram, then this is 2104the program location in the subprogram at which execution of the lane is 2105conceptual positioned. 2106 2107If the lane was not active on entry to the subprogram, then this will be the 2108undefined location. A client debugger can check if the lane is part of a valid 2109work-group by checking that the lane is in the range of the associated 2110work-group within the grid, accounting for partial work-groups. If it is not, 2111then the debugger can omit any information for the lane. Otherwise, the debugger 2112may repeatedly unwind the stack and inspect the ``DW_AT_LLVM_lane_pc`` of the 2113calling subprogram until it finds a non-undefined location. Conceptually the 2114lane only has the call frames that it has a non-undefined 2115``DW_AT_LLVM_lane_pc``. 2116 2117The following example illustrates how the AMDGPU backend can generate a DWARF 2118location list expression for the nested ``IF/THEN/ELSE`` structures of the 2119following subprogram pseudo code for a target with 64 lanes per wavefront. 2120 2121.. code:: 2122 :number-lines: 2123 2124 SUBPROGRAM X 2125 BEGIN 2126 a; 2127 IF (c1) THEN 2128 b; 2129 IF (c2) THEN 2130 c; 2131 ELSE 2132 d; 2133 ENDIF 2134 e; 2135 ELSE 2136 f; 2137 ENDIF 2138 g; 2139 END 2140 2141The AMDGPU backend may generate the following pseudo LLVM MIR to manipulate the 2142execution mask (``EXEC``) to linearize the control flow. The condition is 2143evaluated to make a mask of the lanes for which the condition evaluates to true. 2144First the ``THEN`` region is executed by setting the ``EXEC`` mask to the 2145logical ``AND`` of the current ``EXEC`` mask with the condition mask. Then the 2146``ELSE`` region is executed by negating the ``EXEC`` mask and logical ``AND`` of 2147the saved ``EXEC`` mask at the start of the region. After the ``IF/THEN/ELSE`` 2148region the ``EXEC`` mask is restored to the value it had at the beginning of the 2149region. This is shown below. Other approaches are possible, but the basic 2150concept is the same. 2151 2152.. code:: 2153 :number-lines: 2154 2155 $lex_start: 2156 a; 2157 %1 = EXEC 2158 %2 = c1 2159 $lex_1_start: 2160 EXEC = %1 & %2 2161 $if_1_then: 2162 b; 2163 %3 = EXEC 2164 %4 = c2 2165 $lex_1_1_start: 2166 EXEC = %3 & %4 2167 $lex_1_1_then: 2168 c; 2169 EXEC = ~EXEC & %3 2170 $lex_1_1_else: 2171 d; 2172 EXEC = %3 2173 $lex_1_1_end: 2174 e; 2175 EXEC = ~EXEC & %1 2176 $lex_1_else: 2177 f; 2178 EXEC = %1 2179 $lex_1_end: 2180 g; 2181 $lex_end: 2182 2183To create the DWARF location list expression that defines the location 2184description of a vector of lane program locations, the LLVM MIR ``DBG_VALUE`` 2185pseudo instruction can be used to annotate the linearized control flow. This can 2186be done by defining an artificial variable for the lane PC. The DWARF location 2187list expression created for it is used as the value of the 2188``DW_AT_LLVM_lane_pc`` attribute on the subprogram's debugger information entry. 2189 2190A DWARF procedure is defined for each well nested structured control flow region 2191which provides the conceptual lane program location for a lane if it is not 2192active (namely it is divergent). The DWARF operation expression for each region 2193conceptually inherits the value of the immediately enclosing region and modifies 2194it according to the semantics of the region. 2195 2196For an ``IF/THEN/ELSE`` region the divergent program location is at the start of 2197the region for the ``THEN`` region since it is executed first. For the ``ELSE`` 2198region the divergent program location is at the end of the ``IF/THEN/ELSE`` 2199region since the ``THEN`` region has completed. 2200 2201The lane PC artificial variable is assigned at each region transition. It uses 2202the immediately enclosing region's DWARF procedure to compute the program 2203location for each lane assuming they are divergent, and then modifies the result 2204by inserting the current program location for each lane that the ``EXEC`` mask 2205indicates is active. 2206 2207By having separate DWARF procedures for each region, they can be reused to 2208define the value for any nested region. This reduces the total size of the DWARF 2209operation expressions. 2210 2211The following provides an example using pseudo LLVM MIR. 2212 2213.. code:: 2214 :number-lines: 2215 2216 $lex_start: 2217 DEFINE_DWARF %__uint_64 = DW_TAG_base_type[ 2218 DW_AT_name = "__uint64"; 2219 DW_AT_byte_size = 8; 2220 DW_AT_encoding = DW_ATE_unsigned; 2221 ]; 2222 DEFINE_DWARF %__active_lane_pc = DW_TAG_dwarf_procedure[ 2223 DW_AT_name = "__active_lane_pc"; 2224 DW_AT_location = [ 2225 DW_OP_regx PC; 2226 DW_OP_LLVM_extend 64, 64; 2227 DW_OP_regval_type EXEC, %uint_64; 2228 DW_OP_LLVM_select_bit_piece 64, 64; 2229 ]; 2230 ]; 2231 DEFINE_DWARF %__divergent_lane_pc = DW_TAG_dwarf_procedure[ 2232 DW_AT_name = "__divergent_lane_pc"; 2233 DW_AT_location = [ 2234 DW_OP_LLVM_undefined; 2235 DW_OP_LLVM_extend 64, 64; 2236 ]; 2237 ]; 2238 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 2239 DW_OP_call_ref %__divergent_lane_pc; 2240 DW_OP_call_ref %__active_lane_pc; 2241 ]; 2242 a; 2243 %1 = EXEC; 2244 DBG_VALUE %1, $noreg, %__lex_1_save_exec; 2245 %2 = c1; 2246 $lex_1_start: 2247 EXEC = %1 & %2; 2248 $lex_1_then: 2249 DEFINE_DWARF %__divergent_lane_pc_1_then = DW_TAG_dwarf_procedure[ 2250 DW_AT_name = "__divergent_lane_pc_1_then"; 2251 DW_AT_location = DIExpression[ 2252 DW_OP_call_ref %__divergent_lane_pc; 2253 DW_OP_addrx &lex_1_start; 2254 DW_OP_stack_value; 2255 DW_OP_LLVM_extend 64, 64; 2256 DW_OP_call_ref %__lex_1_save_exec; 2257 DW_OP_deref_type 64, %__uint_64; 2258 DW_OP_LLVM_select_bit_piece 64, 64; 2259 ]; 2260 ]; 2261 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 2262 DW_OP_call_ref %__divergent_lane_pc_1_then; 2263 DW_OP_call_ref %__active_lane_pc; 2264 ]; 2265 b; 2266 %3 = EXEC; 2267 DBG_VALUE %3, %__lex_1_1_save_exec; 2268 %4 = c2; 2269 $lex_1_1_start: 2270 EXEC = %3 & %4; 2271 $lex_1_1_then: 2272 DEFINE_DWARF %__divergent_lane_pc_1_1_then = DW_TAG_dwarf_procedure[ 2273 DW_AT_name = "__divergent_lane_pc_1_1_then"; 2274 DW_AT_location = DIExpression[ 2275 DW_OP_call_ref %__divergent_lane_pc_1_then; 2276 DW_OP_addrx &lex_1_1_start; 2277 DW_OP_stack_value; 2278 DW_OP_LLVM_extend 64, 64; 2279 DW_OP_call_ref %__lex_1_1_save_exec; 2280 DW_OP_deref_type 64, %__uint_64; 2281 DW_OP_LLVM_select_bit_piece 64, 64; 2282 ]; 2283 ]; 2284 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 2285 DW_OP_call_ref %__divergent_lane_pc_1_1_then; 2286 DW_OP_call_ref %__active_lane_pc; 2287 ]; 2288 c; 2289 EXEC = ~EXEC & %3; 2290 $lex_1_1_else: 2291 DEFINE_DWARF %__divergent_lane_pc_1_1_else = DW_TAG_dwarf_procedure[ 2292 DW_AT_name = "__divergent_lane_pc_1_1_else"; 2293 DW_AT_location = DIExpression[ 2294 DW_OP_call_ref %__divergent_lane_pc_1_then; 2295 DW_OP_addrx &lex_1_1_end; 2296 DW_OP_stack_value; 2297 DW_OP_LLVM_extend 64, 64; 2298 DW_OP_call_ref %__lex_1_1_save_exec; 2299 DW_OP_deref_type 64, %__uint_64; 2300 DW_OP_LLVM_select_bit_piece 64, 64; 2301 ]; 2302 ]; 2303 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 2304 DW_OP_call_ref %__divergent_lane_pc_1_1_else; 2305 DW_OP_call_ref %__active_lane_pc; 2306 ]; 2307 d; 2308 EXEC = %3; 2309 $lex_1_1_end: 2310 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 2311 DW_OP_call_ref %__divergent_lane_pc; 2312 DW_OP_call_ref %__active_lane_pc; 2313 ]; 2314 e; 2315 EXEC = ~EXEC & %1; 2316 $lex_1_else: 2317 DEFINE_DWARF %__divergent_lane_pc_1_else = DW_TAG_dwarf_procedure[ 2318 DW_AT_name = "__divergent_lane_pc_1_else"; 2319 DW_AT_location = DIExpression[ 2320 DW_OP_call_ref %__divergent_lane_pc; 2321 DW_OP_addrx &lex_1_end; 2322 DW_OP_stack_value; 2323 DW_OP_LLVM_extend 64, 64; 2324 DW_OP_call_ref %__lex_1_save_exec; 2325 DW_OP_deref_type 64, %__uint_64; 2326 DW_OP_LLVM_select_bit_piece 64, 64; 2327 ]; 2328 ]; 2329 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 2330 DW_OP_call_ref %__divergent_lane_pc_1_else; 2331 DW_OP_call_ref %__active_lane_pc; 2332 ]; 2333 f; 2334 EXEC = %1; 2335 $lex_1_end: 2336 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc DIExpression[ 2337 DW_OP_call_ref %__divergent_lane_pc; 2338 DW_OP_call_ref %__active_lane_pc; 2339 ]; 2340 g; 2341 $lex_end: 2342 2343The DWARF procedure ``%__active_lane_pc`` is used to update the lane pc elements 2344that are active, with the current program location. 2345 2346Artificial variables %__lex_1_save_exec and %__lex_1_1_save_exec are created for 2347the execution masks saved on entry to a region. Using the ``DBG_VALUE`` pseudo 2348instruction, location list entries will be created that describe where the 2349artificial variables are allocated at any given program location. The compiler 2350may allocate them to registers or spill them to memory. 2351 2352The DWARF procedures for each region use the values of the saved execution mask 2353artificial variables to only update the lanes that are active on entry to the 2354region. All other lanes retain the value of the enclosing region where they were 2355last active. If they were not active on entry to the subprogram, then will have 2356the undefined location description. 2357 2358Other structured control flow regions can be handled similarly. For example, 2359loops would set the divergent program location for the region at the end of the 2360loop. Any lanes active will be in the loop, and any lanes not active must have 2361exited the loop. 2362 2363An ``IF/THEN/ELSEIF/ELSEIF/...`` region can be treated as a nest of 2364``IF/THEN/ELSE`` regions. 2365 2366The DWARF procedures can use the active lane artificial variable described in 2367:ref:`amdgpu-dwarf-amdgpu-dw-at-llvm-active-lane` rather than the actual 2368``EXEC`` mask in order to support whole or quad wavefront mode. 2369 2370.. _amdgpu-dwarf-amdgpu-dw-at-llvm-active-lane: 2371 2372``DW_AT_LLVM_active_lane`` 2373~~~~~~~~~~~~~~~~~~~~~~~~~~ 2374 2375The ``DW_AT_LLVM_active_lane`` attribute on a subprogram debugger information 2376entry is used to specify the lanes that are conceptually active for a SIMT 2377thread. 2378 2379The execution mask may be modified to implement whole or quad wavefront mode 2380operations. For example, all lanes may need to temporarily be made active to 2381execute a whole wavefront operation. Such regions would save the ``EXEC`` mask, 2382update it to enable the necessary lanes, perform the operations, and then 2383restore the ``EXEC`` mask from the saved value. While executing the whole 2384wavefront region, the conceptual execution mask is the saved value, not the 2385``EXEC`` value. 2386 2387This is handled by defining an artificial variable for the active lane mask. The 2388active lane mask artificial variable would be the actual ``EXEC`` mask for 2389normal regions, and the saved execution mask for regions where the mask is 2390temporarily updated. The location list expression created for this artificial 2391variable is used to define the value of the ``DW_AT_LLVM_active_lane`` 2392attribute. 2393 2394``DW_AT_LLVM_augmentation`` 2395~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2396 2397For AMDGPU, the ``DW_AT_LLVM_augmentation`` attribute of a compilation unit 2398debugger information entry has the following value for the augmentation string: 2399 2400:: 2401 2402 [amdgpu:v0.0] 2403 2404The "vX.Y" specifies the major X and minor Y version number of the AMDGPU 2405extensions used in the DWARF of the compilation unit. The version number 2406conforms to [SEMVER]_. 2407 2408Call Frame Information 2409---------------------- 2410 2411DWARF Call Frame Information (CFI) describes how a consumer can virtually 2412*unwind* call frames in a running process or core dump. See DWARF Version 5 2413section 6.4 and :ref:`amdgpu-dwarf-call-frame-information`. 2414 2415For AMDGPU, the Common Information Entry (CIE) fields have the following values: 2416 24171. ``augmentation`` string contains the following null-terminated UTF-8 string: 2418 2419 :: 2420 2421 [amd:v0.0] 2422 2423 The ``vX.Y`` specifies the major X and minor Y version number of the AMDGPU 2424 extensions used in this CIE or to the FDEs that use it. The version number 2425 conforms to [SEMVER]_. 2426 24272. ``address_size`` for the ``Global`` address space is defined in 2428 :ref:`amdgpu-dwarf-address-space-identifier`. 2429 24303. ``segment_selector_size`` is 0 as AMDGPU does not use a segment selector. 2431 24324. ``code_alignment_factor`` is 4 bytes. 2433 2434 .. TODO:: 2435 2436 Add to :ref:`amdgpu-processor-table` table. 2437 24385. ``data_alignment_factor`` is 4 bytes. 2439 2440 .. TODO:: 2441 2442 Add to :ref:`amdgpu-processor-table` table. 2443 24446. ``return_address_register`` is ``PC_32`` for 32-bit processes and ``PC_64`` 2445 for 64-bit processes defined in :ref:`amdgpu-dwarf-register-identifier`. 2446 24477. ``initial_instructions`` Since a subprogram X with fewer registers can be 2448 called from subprogram Y that has more allocated, X will not change any of 2449 the extra registers as it cannot access them. Therefore, the default rule 2450 for all columns is ``same value``. 2451 2452For AMDGPU the register number follows the numbering defined in 2453:ref:`amdgpu-dwarf-register-identifier`. 2454 2455For AMDGPU the instructions are variable size. A consumer can subtract 1 from 2456the return address to get the address of a byte within the call site 2457instructions. See DWARF Version 5 section 6.4.4. 2458 2459Accelerated Access 2460------------------ 2461 2462See DWARF Version 5 section 6.1. 2463 2464Lookup By Name Section Header 2465~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2466 2467See DWARF Version 5 section 6.1.1.4.1 and :ref:`amdgpu-dwarf-lookup-by-name`. 2468 2469For AMDGPU the lookup by name section header table: 2470 2471``augmentation_string_size`` (uword) 2472 2473 Set to the length of the ``augmentation_string`` value which is always a 2474 multiple of 4. 2475 2476``augmentation_string`` (sequence of UTF-8 characters) 2477 2478 Contains the following UTF-8 string null padded to a multiple of 4 bytes: 2479 2480 :: 2481 2482 [amdgpu:v0.0] 2483 2484 The "vX.Y" specifies the major X and minor Y version number of the AMDGPU 2485 extensions used in the DWARF of this index. The version number conforms to 2486 [SEMVER]_. 2487 2488 .. note:: 2489 2490 This is different to the DWARF Version 5 definition that requires the first 2491 4 characters to be the vendor ID. But this is consistent with the other 2492 augmentation strings and does allow multiple vendor contributions. However, 2493 backwards compatibility may be more desirable. 2494 2495Lookup By Address Section Header 2496~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2497 2498See DWARF Version 5 section 6.1.2. 2499 2500For AMDGPU the lookup by address section header table: 2501 2502``address_size`` (ubyte) 2503 2504 Match the address size for the ``Global`` address space defined in 2505 :ref:`amdgpu-dwarf-address-space-identifier`. 2506 2507``segment_selector_size`` (ubyte) 2508 2509 AMDGPU does not use a segment selector so this is 0. The entries in the 2510 ``.debug_aranges`` do not have a segment selector. 2511 2512Line Number Information 2513----------------------- 2514 2515See DWARF Version 5 section 6.2 and :ref:`amdgpu-dwarf-line-number-information`. 2516 2517AMDGPU does not use the ``isa`` state machine registers and always sets it to 0. 2518The instruction set must be obtained from the ELF file header ``e_flags`` field 2519in the ``EF_AMDGPU_MACH`` bit position (see :ref:`ELF Header 2520<amdgpu-elf-header>`). See DWARF Version 5 section 6.2.2. 2521 2522.. TODO:: 2523 2524 Should the ``isa`` state machine register be used to indicate if the code is 2525 in wavefront32 or wavefront64 mode? Or used to specify the architecture ISA? 2526 2527For AMDGPU the line number program header fields have the following values (see 2528DWARF Version 5 section 6.2.4): 2529 2530``address_size`` (ubyte) 2531 Matches the address size for the ``Global`` address space defined in 2532 :ref:`amdgpu-dwarf-address-space-identifier`. 2533 2534``segment_selector_size`` (ubyte) 2535 AMDGPU does not use a segment selector so this is 0. 2536 2537``minimum_instruction_length`` (ubyte) 2538 For GFX9-GFX10 this is 4. 2539 2540``maximum_operations_per_instruction`` (ubyte) 2541 For GFX9-GFX10 this is 1. 2542 2543Source text for online-compiled programs (for example, those compiled by the 2544OpenCL language runtime) may be embedded into the DWARF Version 5 line table. 2545See DWARF Version 5 section 6.2.4.1 which is updated by *DWARF Extensions For 2546Heterogeneous Debugging* section :ref:`DW_LNCT_LLVM_source 2547<amdgpu-dwarf-line-number-information-dw-lnct-llvm-source>`. 2548 2549The Clang option used to control source embedding in AMDGPU is defined in 2550:ref:`amdgpu-clang-debug-options-table`. 2551 2552 .. table:: AMDGPU Clang Debug Options 2553 :name: amdgpu-clang-debug-options-table 2554 2555 ==================== ================================================== 2556 Debug Flag Description 2557 ==================== ================================================== 2558 -g[no-]embed-source Enable/disable embedding source text in DWARF 2559 debug sections. Useful for environments where 2560 source cannot be written to disk, such as 2561 when performing online compilation. 2562 ==================== ================================================== 2563 2564For example: 2565 2566``-gembed-source`` 2567 Enable the embedded source. 2568 2569``-gno-embed-source`` 2570 Disable the embedded source. 2571 257232-Bit and 64-Bit DWARF Formats 2573------------------------------- 2574 2575See DWARF Version 5 section 7.4 and 2576:ref:`amdgpu-dwarf-32-bit-and-64-bit-dwarf-formats`. 2577 2578For AMDGPU: 2579 2580* For the ``amdgcn`` target architecture only the 64-bit process address space 2581 is supported. 2582 2583* The producer can generate either 32-bit or 64-bit DWARF format. LLVM generates 2584 the 32-bit DWARF format. 2585 2586Unit Headers 2587------------ 2588 2589For AMDGPU the following values apply for each of the unit headers described in 2590DWARF Version 5 sections 7.5.1.1, 7.5.1.2, and 7.5.1.3: 2591 2592``address_size`` (ubyte) 2593 Matches the address size for the ``Global`` address space defined in 2594 :ref:`amdgpu-dwarf-address-space-identifier`. 2595 2596.. _amdgpu-code-conventions: 2597 2598Code Conventions 2599================ 2600 2601This section provides code conventions used for each supported target triple OS 2602(see :ref:`amdgpu-target-triples`). 2603 2604AMDHSA 2605------ 2606 2607This section provides code conventions used when the target triple OS is 2608``amdhsa`` (see :ref:`amdgpu-target-triples`). 2609 2610.. _amdgpu-amdhsa-code-object-metadata: 2611 2612Code Object Metadata 2613~~~~~~~~~~~~~~~~~~~~ 2614 2615The code object metadata specifies extensible metadata associated with the code 2616objects executed on HSA [HSA]_ compatible runtimes (see :ref:`amdgpu-os`). The 2617encoding and semantics of this metadata depends on the code object version; see 2618:ref:`amdgpu-amdhsa-code-object-metadata-v2`, 2619:ref:`amdgpu-amdhsa-code-object-metadata-v3`, 2620:ref:`amdgpu-amdhsa-code-object-metadata-v4` and 2621:ref:`amdgpu-amdhsa-code-object-metadata-v5`. 2622 2623Code object metadata is specified in a note record (see 2624:ref:`amdgpu-note-records`) and is required when the target triple OS is 2625``amdhsa`` (see :ref:`amdgpu-target-triples`). It must contain the minimum 2626information necessary to support the HSA compatible runtime kernel queries. For 2627example, the segment sizes needed in a dispatch packet. In addition, a 2628high-level language runtime may require other information to be included. For 2629example, the AMD OpenCL runtime records kernel argument information. 2630 2631.. _amdgpu-amdhsa-code-object-metadata-v2: 2632 2633Code Object V2 Metadata 2634+++++++++++++++++++++++ 2635 2636.. warning:: 2637 Code object V2 is not the default code object version emitted by this version 2638 of LLVM. 2639 2640Code object V2 metadata is specified by the ``NT_AMD_HSA_METADATA`` note record 2641(see :ref:`amdgpu-note-records-v2`). 2642 2643The metadata is specified as a YAML formatted string (see [YAML]_ and 2644:doc:`YamlIO`). 2645 2646.. TODO:: 2647 2648 Is the string null terminated? It probably should not if YAML allows it to 2649 contain null characters, otherwise it should be. 2650 2651The metadata is represented as a single YAML document comprised of the mapping 2652defined in table :ref:`amdgpu-amdhsa-code-object-metadata-map-v2-table` and 2653referenced tables. 2654 2655For boolean values, the string values of ``false`` and ``true`` are used for 2656false and true respectively. 2657 2658Additional information can be added to the mappings. To avoid conflicts, any 2659non-AMD key names should be prefixed by "*vendor-name*.". 2660 2661 .. table:: AMDHSA Code Object V2 Metadata Map 2662 :name: amdgpu-amdhsa-code-object-metadata-map-v2-table 2663 2664 ========== ============== ========= ======================================= 2665 String Key Value Type Required? Description 2666 ========== ============== ========= ======================================= 2667 "Version" sequence of Required - The first integer is the major 2668 2 integers version. Currently 1. 2669 - The second integer is the minor 2670 version. Currently 0. 2671 "Printf" sequence of Each string is encoded information 2672 strings about a printf function call. The 2673 encoded information is organized as 2674 fields separated by colon (':'): 2675 2676 ``ID:N:S[0]:S[1]:...:S[N-1]:FormatString`` 2677 2678 where: 2679 2680 ``ID`` 2681 A 32-bit integer as a unique id for 2682 each printf function call 2683 2684 ``N`` 2685 A 32-bit integer equal to the number 2686 of arguments of printf function call 2687 minus 1 2688 2689 ``S[i]`` (where i = 0, 1, ... , N-1) 2690 32-bit integers for the size in bytes 2691 of the i-th FormatString argument of 2692 the printf function call 2693 2694 FormatString 2695 The format string passed to the 2696 printf function call. 2697 "Kernels" sequence of Required Sequence of the mappings for each 2698 mapping kernel in the code object. See 2699 :ref:`amdgpu-amdhsa-code-object-kernel-metadata-map-v2-table` 2700 for the definition of the mapping. 2701 ========== ============== ========= ======================================= 2702 2703.. 2704 2705 .. table:: AMDHSA Code Object V2 Kernel Metadata Map 2706 :name: amdgpu-amdhsa-code-object-kernel-metadata-map-v2-table 2707 2708 ================= ============== ========= ================================ 2709 String Key Value Type Required? Description 2710 ================= ============== ========= ================================ 2711 "Name" string Required Source name of the kernel. 2712 "SymbolName" string Required Name of the kernel 2713 descriptor ELF symbol. 2714 "Language" string Source language of the kernel. 2715 Values include: 2716 2717 - "OpenCL C" 2718 - "OpenCL C++" 2719 - "HCC" 2720 - "OpenMP" 2721 2722 "LanguageVersion" sequence of - The first integer is the major 2723 2 integers version. 2724 - The second integer is the 2725 minor version. 2726 "Attrs" mapping Mapping of kernel attributes. 2727 See 2728 :ref:`amdgpu-amdhsa-code-object-kernel-attribute-metadata-map-v2-table` 2729 for the mapping definition. 2730 "Args" sequence of Sequence of mappings of the 2731 mapping kernel arguments. See 2732 :ref:`amdgpu-amdhsa-code-object-kernel-argument-metadata-map-v2-table` 2733 for the definition of the mapping. 2734 "CodeProps" mapping Mapping of properties related to 2735 the kernel code. See 2736 :ref:`amdgpu-amdhsa-code-object-kernel-code-properties-metadata-map-v2-table` 2737 for the mapping definition. 2738 ================= ============== ========= ================================ 2739 2740.. 2741 2742 .. table:: AMDHSA Code Object V2 Kernel Attribute Metadata Map 2743 :name: amdgpu-amdhsa-code-object-kernel-attribute-metadata-map-v2-table 2744 2745 =================== ============== ========= ============================== 2746 String Key Value Type Required? Description 2747 =================== ============== ========= ============================== 2748 "ReqdWorkGroupSize" sequence of If not 0, 0, 0 then all values 2749 3 integers must be >=1 and the dispatch 2750 work-group size X, Y, Z must 2751 correspond to the specified 2752 values. Defaults to 0, 0, 0. 2753 2754 Corresponds to the OpenCL 2755 ``reqd_work_group_size`` 2756 attribute. 2757 "WorkGroupSizeHint" sequence of The dispatch work-group size 2758 3 integers X, Y, Z is likely to be the 2759 specified values. 2760 2761 Corresponds to the OpenCL 2762 ``work_group_size_hint`` 2763 attribute. 2764 "VecTypeHint" string The name of a scalar or vector 2765 type. 2766 2767 Corresponds to the OpenCL 2768 ``vec_type_hint`` attribute. 2769 2770 "RuntimeHandle" string The external symbol name 2771 associated with a kernel. 2772 OpenCL runtime allocates a 2773 global buffer for the symbol 2774 and saves the kernel's address 2775 to it, which is used for 2776 device side enqueueing. Only 2777 available for device side 2778 enqueued kernels. 2779 =================== ============== ========= ============================== 2780 2781.. 2782 2783 .. table:: AMDHSA Code Object V2 Kernel Argument Metadata Map 2784 :name: amdgpu-amdhsa-code-object-kernel-argument-metadata-map-v2-table 2785 2786 ================= ============== ========= ================================ 2787 String Key Value Type Required? Description 2788 ================= ============== ========= ================================ 2789 "Name" string Kernel argument name. 2790 "TypeName" string Kernel argument type name. 2791 "Size" integer Required Kernel argument size in bytes. 2792 "Align" integer Required Kernel argument alignment in 2793 bytes. Must be a power of two. 2794 "ValueKind" string Required Kernel argument kind that 2795 specifies how to set up the 2796 corresponding argument. 2797 Values include: 2798 2799 "ByValue" 2800 The argument is copied 2801 directly into the kernarg. 2802 2803 "GlobalBuffer" 2804 A global address space pointer 2805 to the buffer data is passed 2806 in the kernarg. 2807 2808 "DynamicSharedPointer" 2809 A group address space pointer 2810 to dynamically allocated LDS 2811 is passed in the kernarg. 2812 2813 "Sampler" 2814 A global address space 2815 pointer to a S# is passed in 2816 the kernarg. 2817 2818 "Image" 2819 A global address space 2820 pointer to a T# is passed in 2821 the kernarg. 2822 2823 "Pipe" 2824 A global address space pointer 2825 to an OpenCL pipe is passed in 2826 the kernarg. 2827 2828 "Queue" 2829 A global address space pointer 2830 to an OpenCL device enqueue 2831 queue is passed in the 2832 kernarg. 2833 2834 "HiddenGlobalOffsetX" 2835 The OpenCL grid dispatch 2836 global offset for the X 2837 dimension is passed in the 2838 kernarg. 2839 2840 "HiddenGlobalOffsetY" 2841 The OpenCL grid dispatch 2842 global offset for the Y 2843 dimension is passed in the 2844 kernarg. 2845 2846 "HiddenGlobalOffsetZ" 2847 The OpenCL grid dispatch 2848 global offset for the Z 2849 dimension is passed in the 2850 kernarg. 2851 2852 "HiddenNone" 2853 An argument that is not used 2854 by the kernel. Space needs to 2855 be left for it, but it does 2856 not need to be set up. 2857 2858 "HiddenPrintfBuffer" 2859 A global address space pointer 2860 to the runtime printf buffer 2861 is passed in kernarg. Mutually 2862 exclusive with 2863 "HiddenHostcallBuffer". 2864 2865 "HiddenHostcallBuffer" 2866 A global address space pointer 2867 to the runtime hostcall buffer 2868 is passed in kernarg. Mutually 2869 exclusive with 2870 "HiddenPrintfBuffer". 2871 2872 "HiddenDefaultQueue" 2873 A global address space pointer 2874 to the OpenCL device enqueue 2875 queue that should be used by 2876 the kernel by default is 2877 passed in the kernarg. 2878 2879 "HiddenCompletionAction" 2880 A global address space pointer 2881 to help link enqueued kernels into 2882 the ancestor tree for determining 2883 when the parent kernel has finished. 2884 2885 "HiddenMultiGridSyncArg" 2886 A global address space pointer for 2887 multi-grid synchronization is 2888 passed in the kernarg. 2889 2890 "ValueType" string Unused and deprecated. This should no longer 2891 be emitted, but is accepted for compatibility. 2892 2893 2894 "PointeeAlign" integer Alignment in bytes of pointee 2895 type for pointer type kernel 2896 argument. Must be a power 2897 of 2. Only present if 2898 "ValueKind" is 2899 "DynamicSharedPointer". 2900 "AddrSpaceQual" string Kernel argument address space 2901 qualifier. Only present if 2902 "ValueKind" is "GlobalBuffer" or 2903 "DynamicSharedPointer". Values 2904 are: 2905 2906 - "Private" 2907 - "Global" 2908 - "Constant" 2909 - "Local" 2910 - "Generic" 2911 - "Region" 2912 2913 .. TODO:: 2914 2915 Is GlobalBuffer only Global 2916 or Constant? Is 2917 DynamicSharedPointer always 2918 Local? Can HCC allow Generic? 2919 How can Private or Region 2920 ever happen? 2921 2922 "AccQual" string Kernel argument access 2923 qualifier. Only present if 2924 "ValueKind" is "Image" or 2925 "Pipe". Values 2926 are: 2927 2928 - "ReadOnly" 2929 - "WriteOnly" 2930 - "ReadWrite" 2931 2932 .. TODO:: 2933 2934 Does this apply to 2935 GlobalBuffer? 2936 2937 "ActualAccQual" string The actual memory accesses 2938 performed by the kernel on the 2939 kernel argument. Only present if 2940 "ValueKind" is "GlobalBuffer", 2941 "Image", or "Pipe". This may be 2942 more restrictive than indicated 2943 by "AccQual" to reflect what the 2944 kernel actual does. If not 2945 present then the runtime must 2946 assume what is implied by 2947 "AccQual" and "IsConst". Values 2948 are: 2949 2950 - "ReadOnly" 2951 - "WriteOnly" 2952 - "ReadWrite" 2953 2954 "IsConst" boolean Indicates if the kernel argument 2955 is const qualified. Only present 2956 if "ValueKind" is 2957 "GlobalBuffer". 2958 2959 "IsRestrict" boolean Indicates if the kernel argument 2960 is restrict qualified. Only 2961 present if "ValueKind" is 2962 "GlobalBuffer". 2963 2964 "IsVolatile" boolean Indicates if the kernel argument 2965 is volatile qualified. Only 2966 present if "ValueKind" is 2967 "GlobalBuffer". 2968 2969 "IsPipe" boolean Indicates if the kernel argument 2970 is pipe qualified. Only present 2971 if "ValueKind" is "Pipe". 2972 2973 .. TODO:: 2974 2975 Can GlobalBuffer be pipe 2976 qualified? 2977 2978 ================= ============== ========= ================================ 2979 2980.. 2981 2982 .. table:: AMDHSA Code Object V2 Kernel Code Properties Metadata Map 2983 :name: amdgpu-amdhsa-code-object-kernel-code-properties-metadata-map-v2-table 2984 2985 ============================ ============== ========= ===================== 2986 String Key Value Type Required? Description 2987 ============================ ============== ========= ===================== 2988 "KernargSegmentSize" integer Required The size in bytes of 2989 the kernarg segment 2990 that holds the values 2991 of the arguments to 2992 the kernel. 2993 "GroupSegmentFixedSize" integer Required The amount of group 2994 segment memory 2995 required by a 2996 work-group in 2997 bytes. This does not 2998 include any 2999 dynamically allocated 3000 group segment memory 3001 that may be added 3002 when the kernel is 3003 dispatched. 3004 "PrivateSegmentFixedSize" integer Required The amount of fixed 3005 private address space 3006 memory required for a 3007 work-item in 3008 bytes. If the kernel 3009 uses a dynamic call 3010 stack then additional 3011 space must be added 3012 to this value for the 3013 call stack. 3014 "KernargSegmentAlign" integer Required The maximum byte 3015 alignment of 3016 arguments in the 3017 kernarg segment. Must 3018 be a power of 2. 3019 "WavefrontSize" integer Required Wavefront size. Must 3020 be a power of 2. 3021 "NumSGPRs" integer Required Number of scalar 3022 registers used by a 3023 wavefront for 3024 GFX6-GFX10. This 3025 includes the special 3026 SGPRs for VCC, Flat 3027 Scratch (GFX7-GFX10) 3028 and XNACK (for 3029 GFX8-GFX10). It does 3030 not include the 16 3031 SGPR added if a trap 3032 handler is 3033 enabled. It is not 3034 rounded up to the 3035 allocation 3036 granularity. 3037 "NumVGPRs" integer Required Number of vector 3038 registers used by 3039 each work-item for 3040 GFX6-GFX10 3041 "MaxFlatWorkGroupSize" integer Required Maximum flat 3042 work-group size 3043 supported by the 3044 kernel in work-items. 3045 Must be >=1 and 3046 consistent with 3047 ReqdWorkGroupSize if 3048 not 0, 0, 0. 3049 "NumSpilledSGPRs" integer Number of stores from 3050 a scalar register to 3051 a register allocator 3052 created spill 3053 location. 3054 "NumSpilledVGPRs" integer Number of stores from 3055 a vector register to 3056 a register allocator 3057 created spill 3058 location. 3059 ============================ ============== ========= ===================== 3060 3061.. _amdgpu-amdhsa-code-object-metadata-v3: 3062 3063Code Object V3 Metadata 3064+++++++++++++++++++++++ 3065 3066.. warning:: 3067 Code object V3 is not the default code object version emitted by this version 3068 of LLVM. 3069 3070Code object V3 and above metadata is specified by the ``NT_AMDGPU_METADATA`` note 3071record (see :ref:`amdgpu-note-records-v3-onwards`). 3072 3073The metadata is represented as Message Pack formatted binary data (see 3074[MsgPack]_). The top level is a Message Pack map that includes the 3075keys defined in table 3076:ref:`amdgpu-amdhsa-code-object-metadata-map-table-v3` and referenced 3077tables. 3078 3079Additional information can be added to the maps. To avoid conflicts, 3080any key names should be prefixed by "*vendor-name*." where 3081``vendor-name`` can be the name of the vendor and specific vendor 3082tool that generates the information. The prefix is abbreviated to 3083simply "." when it appears within a map that has been added by the 3084same *vendor-name*. 3085 3086 .. table:: AMDHSA Code Object V3 Metadata Map 3087 :name: amdgpu-amdhsa-code-object-metadata-map-table-v3 3088 3089 ================= ============== ========= ======================================= 3090 String Key Value Type Required? Description 3091 ================= ============== ========= ======================================= 3092 "amdhsa.version" sequence of Required - The first integer is the major 3093 2 integers version. Currently 1. 3094 - The second integer is the minor 3095 version. Currently 0. 3096 "amdhsa.printf" sequence of Each string is encoded information 3097 strings about a printf function call. The 3098 encoded information is organized as 3099 fields separated by colon (':'): 3100 3101 ``ID:N:S[0]:S[1]:...:S[N-1]:FormatString`` 3102 3103 where: 3104 3105 ``ID`` 3106 A 32-bit integer as a unique id for 3107 each printf function call 3108 3109 ``N`` 3110 A 32-bit integer equal to the number 3111 of arguments of printf function call 3112 minus 1 3113 3114 ``S[i]`` (where i = 0, 1, ... , N-1) 3115 32-bit integers for the size in bytes 3116 of the i-th FormatString argument of 3117 the printf function call 3118 3119 FormatString 3120 The format string passed to the 3121 printf function call. 3122 "amdhsa.kernels" sequence of Required Sequence of the maps for each 3123 map kernel in the code object. See 3124 :ref:`amdgpu-amdhsa-code-object-kernel-metadata-map-table-v3` 3125 for the definition of the keys included 3126 in that map. 3127 ================= ============== ========= ======================================= 3128 3129.. 3130 3131 .. table:: AMDHSA Code Object V3 Kernel Metadata Map 3132 :name: amdgpu-amdhsa-code-object-kernel-metadata-map-table-v3 3133 3134 =================================== ============== ========= ================================ 3135 String Key Value Type Required? Description 3136 =================================== ============== ========= ================================ 3137 ".name" string Required Source name of the kernel. 3138 ".symbol" string Required Name of the kernel 3139 descriptor ELF symbol. 3140 ".language" string Source language of the kernel. 3141 Values include: 3142 3143 - "OpenCL C" 3144 - "OpenCL C++" 3145 - "HCC" 3146 - "HIP" 3147 - "OpenMP" 3148 - "Assembler" 3149 3150 ".language_version" sequence of - The first integer is the major 3151 2 integers version. 3152 - The second integer is the 3153 minor version. 3154 ".args" sequence of Sequence of maps of the 3155 map kernel arguments. See 3156 :ref:`amdgpu-amdhsa-code-object-kernel-argument-metadata-map-table-v3` 3157 for the definition of the keys 3158 included in that map. 3159 ".reqd_workgroup_size" sequence of If not 0, 0, 0 then all values 3160 3 integers must be >=1 and the dispatch 3161 work-group size X, Y, Z must 3162 correspond to the specified 3163 values. Defaults to 0, 0, 0. 3164 3165 Corresponds to the OpenCL 3166 ``reqd_work_group_size`` 3167 attribute. 3168 ".workgroup_size_hint" sequence of The dispatch work-group size 3169 3 integers X, Y, Z is likely to be the 3170 specified values. 3171 3172 Corresponds to the OpenCL 3173 ``work_group_size_hint`` 3174 attribute. 3175 ".vec_type_hint" string The name of a scalar or vector 3176 type. 3177 3178 Corresponds to the OpenCL 3179 ``vec_type_hint`` attribute. 3180 3181 ".device_enqueue_symbol" string The external symbol name 3182 associated with a kernel. 3183 OpenCL runtime allocates a 3184 global buffer for the symbol 3185 and saves the kernel's address 3186 to it, which is used for 3187 device side enqueueing. Only 3188 available for device side 3189 enqueued kernels. 3190 ".kernarg_segment_size" integer Required The size in bytes of 3191 the kernarg segment 3192 that holds the values 3193 of the arguments to 3194 the kernel. 3195 ".group_segment_fixed_size" integer Required The amount of group 3196 segment memory 3197 required by a 3198 work-group in 3199 bytes. This does not 3200 include any 3201 dynamically allocated 3202 group segment memory 3203 that may be added 3204 when the kernel is 3205 dispatched. 3206 ".private_segment_fixed_size" integer Required The amount of fixed 3207 private address space 3208 memory required for a 3209 work-item in 3210 bytes. If the kernel 3211 uses a dynamic call 3212 stack then additional 3213 space must be added 3214 to this value for the 3215 call stack. 3216 ".kernarg_segment_align" integer Required The maximum byte 3217 alignment of 3218 arguments in the 3219 kernarg segment. Must 3220 be a power of 2. 3221 ".wavefront_size" integer Required Wavefront size. Must 3222 be a power of 2. 3223 ".sgpr_count" integer Required Number of scalar 3224 registers required by a 3225 wavefront for 3226 GFX6-GFX9. A register 3227 is required if it is 3228 used explicitly, or 3229 if a higher numbered 3230 register is used 3231 explicitly. This 3232 includes the special 3233 SGPRs for VCC, Flat 3234 Scratch (GFX7-GFX9) 3235 and XNACK (for 3236 GFX8-GFX9). It does 3237 not include the 16 3238 SGPR added if a trap 3239 handler is 3240 enabled. It is not 3241 rounded up to the 3242 allocation 3243 granularity. 3244 ".vgpr_count" integer Required Number of vector 3245 registers required by 3246 each work-item for 3247 GFX6-GFX9. A register 3248 is required if it is 3249 used explicitly, or 3250 if a higher numbered 3251 register is used 3252 explicitly. 3253 ".agpr_count" integer Required Number of accumulator 3254 registers required by 3255 each work-item for 3256 GFX90A, GFX908. 3257 ".max_flat_workgroup_size" integer Required Maximum flat 3258 work-group size 3259 supported by the 3260 kernel in work-items. 3261 Must be >=1 and 3262 consistent with 3263 ReqdWorkGroupSize if 3264 not 0, 0, 0. 3265 ".sgpr_spill_count" integer Number of stores from 3266 a scalar register to 3267 a register allocator 3268 created spill 3269 location. 3270 ".vgpr_spill_count" integer Number of stores from 3271 a vector register to 3272 a register allocator 3273 created spill 3274 location. 3275 ".kind" string The kind of the kernel 3276 with the following 3277 values: 3278 3279 "normal" 3280 Regular kernels. 3281 3282 "init" 3283 These kernels must be 3284 invoked after loading 3285 the containing code 3286 object and must 3287 complete before any 3288 normal and fini 3289 kernels in the same 3290 code object are 3291 invoked. 3292 3293 "fini" 3294 These kernels must be 3295 invoked before 3296 unloading the 3297 containing code object 3298 and after all init and 3299 normal kernels in the 3300 same code object have 3301 been invoked and 3302 completed. 3303 3304 If omitted, "normal" is 3305 assumed. 3306 =================================== ============== ========= ================================ 3307 3308.. 3309 3310 .. table:: AMDHSA Code Object V3 Kernel Argument Metadata Map 3311 :name: amdgpu-amdhsa-code-object-kernel-argument-metadata-map-table-v3 3312 3313 ====================== ============== ========= ================================ 3314 String Key Value Type Required? Description 3315 ====================== ============== ========= ================================ 3316 ".name" string Kernel argument name. 3317 ".type_name" string Kernel argument type name. 3318 ".size" integer Required Kernel argument size in bytes. 3319 ".offset" integer Required Kernel argument offset in 3320 bytes. The offset must be a 3321 multiple of the alignment 3322 required by the argument. 3323 ".value_kind" string Required Kernel argument kind that 3324 specifies how to set up the 3325 corresponding argument. 3326 Values include: 3327 3328 "by_value" 3329 The argument is copied 3330 directly into the kernarg. 3331 3332 "global_buffer" 3333 A global address space pointer 3334 to the buffer data is passed 3335 in the kernarg. 3336 3337 "dynamic_shared_pointer" 3338 A group address space pointer 3339 to dynamically allocated LDS 3340 is passed in the kernarg. 3341 3342 "sampler" 3343 A global address space 3344 pointer to a S# is passed in 3345 the kernarg. 3346 3347 "image" 3348 A global address space 3349 pointer to a T# is passed in 3350 the kernarg. 3351 3352 "pipe" 3353 A global address space pointer 3354 to an OpenCL pipe is passed in 3355 the kernarg. 3356 3357 "queue" 3358 A global address space pointer 3359 to an OpenCL device enqueue 3360 queue is passed in the 3361 kernarg. 3362 3363 "hidden_global_offset_x" 3364 The OpenCL grid dispatch 3365 global offset for the X 3366 dimension is passed in the 3367 kernarg. 3368 3369 "hidden_global_offset_y" 3370 The OpenCL grid dispatch 3371 global offset for the Y 3372 dimension is passed in the 3373 kernarg. 3374 3375 "hidden_global_offset_z" 3376 The OpenCL grid dispatch 3377 global offset for the Z 3378 dimension is passed in the 3379 kernarg. 3380 3381 "hidden_none" 3382 An argument that is not used 3383 by the kernel. Space needs to 3384 be left for it, but it does 3385 not need to be set up. 3386 3387 "hidden_printf_buffer" 3388 A global address space pointer 3389 to the runtime printf buffer 3390 is passed in kernarg. Mutually 3391 exclusive with 3392 "hidden_hostcall_buffer" 3393 before Code Object V5. 3394 3395 "hidden_hostcall_buffer" 3396 A global address space pointer 3397 to the runtime hostcall buffer 3398 is passed in kernarg. Mutually 3399 exclusive with 3400 "hidden_printf_buffer" 3401 before Code Object V5. 3402 3403 "hidden_default_queue" 3404 A global address space pointer 3405 to the OpenCL device enqueue 3406 queue that should be used by 3407 the kernel by default is 3408 passed in the kernarg. 3409 3410 "hidden_completion_action" 3411 A global address space pointer 3412 to help link enqueued kernels into 3413 the ancestor tree for determining 3414 when the parent kernel has finished. 3415 3416 "hidden_multigrid_sync_arg" 3417 A global address space pointer for 3418 multi-grid synchronization is 3419 passed in the kernarg. 3420 3421 ".value_type" string Unused and deprecated. This should no longer 3422 be emitted, but is accepted for compatibility. 3423 3424 ".pointee_align" integer Alignment in bytes of pointee 3425 type for pointer type kernel 3426 argument. Must be a power 3427 of 2. Only present if 3428 ".value_kind" is 3429 "dynamic_shared_pointer". 3430 ".address_space" string Kernel argument address space 3431 qualifier. Only present if 3432 ".value_kind" is "global_buffer" or 3433 "dynamic_shared_pointer". Values 3434 are: 3435 3436 - "private" 3437 - "global" 3438 - "constant" 3439 - "local" 3440 - "generic" 3441 - "region" 3442 3443 .. TODO:: 3444 3445 Is "global_buffer" only "global" 3446 or "constant"? Is 3447 "dynamic_shared_pointer" always 3448 "local"? Can HCC allow "generic"? 3449 How can "private" or "region" 3450 ever happen? 3451 3452 ".access" string Kernel argument access 3453 qualifier. Only present if 3454 ".value_kind" is "image" or 3455 "pipe". Values 3456 are: 3457 3458 - "read_only" 3459 - "write_only" 3460 - "read_write" 3461 3462 .. TODO:: 3463 3464 Does this apply to 3465 "global_buffer"? 3466 3467 ".actual_access" string The actual memory accesses 3468 performed by the kernel on the 3469 kernel argument. Only present if 3470 ".value_kind" is "global_buffer", 3471 "image", or "pipe". This may be 3472 more restrictive than indicated 3473 by ".access" to reflect what the 3474 kernel actual does. If not 3475 present then the runtime must 3476 assume what is implied by 3477 ".access" and ".is_const" . Values 3478 are: 3479 3480 - "read_only" 3481 - "write_only" 3482 - "read_write" 3483 3484 ".is_const" boolean Indicates if the kernel argument 3485 is const qualified. Only present 3486 if ".value_kind" is 3487 "global_buffer". 3488 3489 ".is_restrict" boolean Indicates if the kernel argument 3490 is restrict qualified. Only 3491 present if ".value_kind" is 3492 "global_buffer". 3493 3494 ".is_volatile" boolean Indicates if the kernel argument 3495 is volatile qualified. Only 3496 present if ".value_kind" is 3497 "global_buffer". 3498 3499 ".is_pipe" boolean Indicates if the kernel argument 3500 is pipe qualified. Only present 3501 if ".value_kind" is "pipe". 3502 3503 .. TODO:: 3504 3505 Can "global_buffer" be pipe 3506 qualified? 3507 3508 ====================== ============== ========= ================================ 3509 3510.. _amdgpu-amdhsa-code-object-metadata-v4: 3511 3512Code Object V4 Metadata 3513+++++++++++++++++++++++ 3514 3515Code object V4 metadata is the same as 3516:ref:`amdgpu-amdhsa-code-object-metadata-v3` with the changes and additions 3517defined in table :ref:`amdgpu-amdhsa-code-object-metadata-map-table-v4`. 3518 3519 .. table:: AMDHSA Code Object V4 Metadata Map Changes 3520 :name: amdgpu-amdhsa-code-object-metadata-map-table-v4 3521 3522 ================= ============== ========= ======================================= 3523 String Key Value Type Required? Description 3524 ================= ============== ========= ======================================= 3525 "amdhsa.version" sequence of Required - The first integer is the major 3526 2 integers version. Currently 1. 3527 - The second integer is the minor 3528 version. Currently 1. 3529 "amdhsa.target" string Required The target name of the code using the syntax: 3530 3531 .. code:: 3532 3533 <target-triple> [ "-" <target-id> ] 3534 3535 A canonical target ID must be 3536 used. See :ref:`amdgpu-target-triples` 3537 and :ref:`amdgpu-target-id`. 3538 ================= ============== ========= ======================================= 3539 3540.. _amdgpu-amdhsa-code-object-metadata-v5: 3541 3542Code Object V5 Metadata 3543+++++++++++++++++++++++ 3544 3545.. warning:: 3546 Code object V5 is not the default code object version emitted by this version 3547 of LLVM. 3548 3549 3550Code object V5 metadata is the same as 3551:ref:`amdgpu-amdhsa-code-object-metadata-v4` with the changes defined in table 3552:ref:`amdgpu-amdhsa-code-object-metadata-map-table-v5` and table 3553:ref:`amdgpu-amdhsa-code-object-kernel-argument-metadata-map-table-v5`. 3554 3555 .. table:: AMDHSA Code Object V5 Metadata Map Changes 3556 :name: amdgpu-amdhsa-code-object-metadata-map-table-v5 3557 3558 ================= ============== ========= ======================================= 3559 String Key Value Type Required? Description 3560 ================= ============== ========= ======================================= 3561 "amdhsa.version" sequence of Required - The first integer is the major 3562 2 integers version. Currently 1. 3563 - The second integer is the minor 3564 version. Currently 2. 3565 ================= ============== ========= ======================================= 3566 3567.. 3568 3569 .. table:: AMDHSA Code Object V5 Kernel Argument Metadata Map Additions and Changes 3570 :name: amdgpu-amdhsa-code-object-kernel-argument-metadata-map-table-v5 3571 3572 ====================== ============== ========= ================================ 3573 String Key Value Type Required? Description 3574 ====================== ============== ========= ================================ 3575 ".value_kind" string Required Kernel argument kind that 3576 specifies how to set up the 3577 corresponding argument. 3578 Values include: 3579 the same as code object V3 metadata 3580 (see :ref:`amdgpu-amdhsa-code-object-kernel-argument-metadata-map-table-v3`) 3581 with the following additions: 3582 3583 "hidden_block_count_x" 3584 The grid dispatch work-group count for the X dimension 3585 is passed in the kernarg. Some languages, such as OpenCL, 3586 support a last work-group in each dimension being partial. 3587 This count only includes the non-partial work-group count. 3588 This is not the same as the value in the AQL dispatch packet, 3589 which has the grid size in work-items. 3590 3591 "hidden_block_count_y" 3592 The grid dispatch work-group count for the Y dimension 3593 is passed in the kernarg. Some languages, such as OpenCL, 3594 support a last work-group in each dimension being partial. 3595 This count only includes the non-partial work-group count. 3596 This is not the same as the value in the AQL dispatch packet, 3597 which has the grid size in work-items. If the grid dimensionality 3598 is 1, then must be 1. 3599 3600 "hidden_block_count_z" 3601 The grid dispatch work-group count for the Z dimension 3602 is passed in the kernarg. Some languages, such as OpenCL, 3603 support a last work-group in each dimension being partial. 3604 This count only includes the non-partial work-group count. 3605 This is not the same as the value in the AQL dispatch packet, 3606 which has the grid size in work-items. If the grid dimensionality 3607 is 1 or 2, then must be 1. 3608 3609 "hidden_group_size_x" 3610 The grid dispatch work-group size for the X dimension is 3611 passed in the kernarg. This size only applies to the 3612 non-partial work-groups. This is the same value as the AQL 3613 dispatch packet work-group size. 3614 3615 "hidden_group_size_y" 3616 The grid dispatch work-group size for the Y dimension is 3617 passed in the kernarg. This size only applies to the 3618 non-partial work-groups. This is the same value as the AQL 3619 dispatch packet work-group size. If the grid dimensionality 3620 is 1, then must be 1. 3621 3622 "hidden_group_size_z" 3623 The grid dispatch work-group size for the Z dimension is 3624 passed in the kernarg. This size only applies to the 3625 non-partial work-groups. This is the same value as the AQL 3626 dispatch packet work-group size. If the grid dimensionality 3627 is 1 or 2, then must be 1. 3628 3629 "hidden_remainder_x" 3630 The grid dispatch work group size of the partial work group 3631 of the X dimension, if it exists. Must be zero if a partial 3632 work group does not exist in the X dimension. 3633 3634 "hidden_remainder_y" 3635 The grid dispatch work group size of the partial work group 3636 of the Y dimension, if it exists. Must be zero if a partial 3637 work group does not exist in the Y dimension. 3638 3639 "hidden_remainder_z" 3640 The grid dispatch work group size of the partial work group 3641 of the Z dimension, if it exists. Must be zero if a partial 3642 work group does not exist in the Z dimension. 3643 3644 "hidden_grid_dims" 3645 The grid dispatch dimensionality. This is the same value 3646 as the AQL dispatch packet dimensionality. Must be a value 3647 between 1 and 3. 3648 3649 "hidden_heap_v1" 3650 A global address space pointer to an initialized memory 3651 buffer that conforms to the requirements of the malloc/free 3652 device library V1 version implementation. 3653 3654 "hidden_private_base" 3655 The high 32 bits of the flat addressing private aperture base. 3656 Only used by GFX8 to allow conversion between private segment 3657 and flat addresses. See :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`. 3658 3659 "hidden_shared_base" 3660 The high 32 bits of the flat addressing shared aperture base. 3661 Only used by GFX8 to allow conversion between shared segment 3662 and flat addresses. See :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`. 3663 3664 "hidden_queue_ptr" 3665 A global memory address space pointer to the ROCm runtime 3666 ``struct amd_queue_t`` structure for the HSA queue of the 3667 associated dispatch AQL packet. It is only required for pre-GFX9 3668 devices for the trap handler ABI (see :ref:`amdgpu-amdhsa-trap-handler-abi`). 3669 3670 ====================== ============== ========= ================================ 3671 3672.. 3673 3674Kernel Dispatch 3675~~~~~~~~~~~~~~~ 3676 3677The HSA architected queuing language (AQL) defines a user space memory interface 3678that can be used to control the dispatch of kernels, in an agent independent 3679way. An agent can have zero or more AQL queues created for it using an HSA 3680compatible runtime (see :ref:`amdgpu-os`), in which AQL packets (all of which 3681are 64 bytes) can be placed. See the *HSA Platform System Architecture 3682Specification* [HSA]_ for the AQL queue mechanics and packet layouts. 3683 3684The packet processor of a kernel agent is responsible for detecting and 3685dispatching HSA kernels from the AQL queues associated with it. For AMD GPUs the 3686packet processor is implemented by the hardware command processor (CP), 3687asynchronous dispatch controller (ADC) and shader processor input controller 3688(SPI). 3689 3690An HSA compatible runtime can be used to allocate an AQL queue object. It uses 3691the kernel mode driver to initialize and register the AQL queue with CP. 3692 3693To dispatch a kernel the following actions are performed. This can occur in the 3694CPU host program, or from an HSA kernel executing on a GPU. 3695 36961. A pointer to an AQL queue for the kernel agent on which the kernel is to be 3697 executed is obtained. 36982. A pointer to the kernel descriptor (see 3699 :ref:`amdgpu-amdhsa-kernel-descriptor`) of the kernel to execute is obtained. 3700 It must be for a kernel that is contained in a code object that was loaded 3701 by an HSA compatible runtime on the kernel agent with which the AQL queue is 3702 associated. 37033. Space is allocated for the kernel arguments using the HSA compatible runtime 3704 allocator for a memory region with the kernarg property for the kernel agent 3705 that will execute the kernel. It must be at least 16-byte aligned. 37064. Kernel argument values are assigned to the kernel argument memory 3707 allocation. The layout is defined in the *HSA Programmer's Language 3708 Reference* [HSA]_. For AMDGPU the kernel execution directly accesses the 3709 kernel argument memory in the same way constant memory is accessed. (Note 3710 that the HSA specification allows an implementation to copy the kernel 3711 argument contents to another location that is accessed by the kernel.) 37125. An AQL kernel dispatch packet is created on the AQL queue. The HSA compatible 3713 runtime api uses 64-bit atomic operations to reserve space in the AQL queue 3714 for the packet. The packet must be set up, and the final write must use an 3715 atomic store release to set the packet kind to ensure the packet contents are 3716 visible to the kernel agent. AQL defines a doorbell signal mechanism to 3717 notify the kernel agent that the AQL queue has been updated. These rules, and 3718 the layout of the AQL queue and kernel dispatch packet is defined in the *HSA 3719 System Architecture Specification* [HSA]_. 37206. A kernel dispatch packet includes information about the actual dispatch, 3721 such as grid and work-group size, together with information from the code 3722 object about the kernel, such as segment sizes. The HSA compatible runtime 3723 queries on the kernel symbol can be used to obtain the code object values 3724 which are recorded in the :ref:`amdgpu-amdhsa-code-object-metadata`. 37257. CP executes micro-code and is responsible for detecting and setting up the 3726 GPU to execute the wavefronts of a kernel dispatch. 37278. CP ensures that when the a wavefront starts executing the kernel machine 3728 code, the scalar general purpose registers (SGPR) and vector general purpose 3729 registers (VGPR) are set up as required by the machine code. The required 3730 setup is defined in the :ref:`amdgpu-amdhsa-kernel-descriptor`. The initial 3731 register state is defined in 3732 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`. 37339. The prolog of the kernel machine code (see 3734 :ref:`amdgpu-amdhsa-kernel-prolog`) sets up the machine state as necessary 3735 before continuing executing the machine code that corresponds to the kernel. 373610. When the kernel dispatch has completed execution, CP signals the completion 3737 signal specified in the kernel dispatch packet if not 0. 3738 3739.. _amdgpu-amdhsa-memory-spaces: 3740 3741Memory Spaces 3742~~~~~~~~~~~~~ 3743 3744The memory space properties are: 3745 3746 .. table:: AMDHSA Memory Spaces 3747 :name: amdgpu-amdhsa-memory-spaces-table 3748 3749 ================= =========== ======== ======= ================== 3750 Memory Space Name HSA Segment Hardware Address NULL Value 3751 Name Name Size 3752 ================= =========== ======== ======= ================== 3753 Private private scratch 32 0x00000000 3754 Local group LDS 32 0xFFFFFFFF 3755 Global global global 64 0x0000000000000000 3756 Constant constant *same as 64 0x0000000000000000 3757 global* 3758 Generic flat flat 64 0x0000000000000000 3759 Region N/A GDS 32 *not implemented 3760 for AMDHSA* 3761 ================= =========== ======== ======= ================== 3762 3763The global and constant memory spaces both use global virtual addresses, which 3764are the same virtual address space used by the CPU. However, some virtual 3765addresses may only be accessible to the CPU, some only accessible by the GPU, 3766and some by both. 3767 3768Using the constant memory space indicates that the data will not change during 3769the execution of the kernel. This allows scalar read instructions to be 3770used. The vector and scalar L1 caches are invalidated of volatile data before 3771each kernel dispatch execution to allow constant memory to change values between 3772kernel dispatches. 3773 3774The local memory space uses the hardware Local Data Store (LDS) which is 3775automatically allocated when the hardware creates work-groups of wavefronts, and 3776freed when all the wavefronts of a work-group have terminated. The data store 3777(DS) instructions can be used to access it. 3778 3779The private memory space uses the hardware scratch memory support. If the kernel 3780uses scratch, then the hardware allocates memory that is accessed using 3781wavefront lane dword (4 byte) interleaving. The mapping used from private 3782address to physical address is: 3783 3784 ``wavefront-scratch-base + 3785 (private-address * wavefront-size * 4) + 3786 (wavefront-lane-id * 4)`` 3787 3788There are different ways that the wavefront scratch base address is determined 3789by a wavefront (see :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). This 3790memory can be accessed in an interleaved manner using buffer instruction with 3791the scratch buffer descriptor and per wavefront scratch offset, by the scratch 3792instructions, or by flat instructions. If each lane of a wavefront accesses the 3793same private address, the interleaving results in adjacent dwords being accessed 3794and hence requires fewer cache lines to be fetched. Multi-dword access is not 3795supported except by flat and scratch instructions in GFX9-GFX10. 3796 3797The generic address space uses the hardware flat address support available in 3798GFX7-GFX10. This uses two fixed ranges of virtual addresses (the private and 3799local apertures), that are outside the range of addressible global memory, to 3800map from a flat address to a private or local address. 3801 3802FLAT instructions can take a flat address and access global, private (scratch) 3803and group (LDS) memory depending on if the address is within one of the 3804aperture ranges. Flat access to scratch requires hardware aperture setup and 3805setup in the kernel prologue (see 3806:ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`). Flat access to LDS requires 3807hardware aperture setup and M0 (GFX7-GFX8) register setup (see 3808:ref:`amdgpu-amdhsa-kernel-prolog-m0`). 3809 3810To convert between a segment address and a flat address the base address of the 3811apertures address can be used. For GFX7-GFX8 these are available in the 3812:ref:`amdgpu-amdhsa-hsa-aql-queue` the address of which can be obtained with 3813Queue Ptr SGPR (see :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). For 3814GFX9-GFX10 the aperture base addresses are directly available as inline constant 3815registers ``SRC_SHARED_BASE/LIMIT`` and ``SRC_PRIVATE_BASE/LIMIT``. In 64 bit 3816address mode the aperture sizes are 2^32 bytes and the base is aligned to 2^32 3817which makes it easier to convert from flat to segment or segment to flat. 3818 3819Image and Samplers 3820~~~~~~~~~~~~~~~~~~ 3821 3822Image and sample handles created by an HSA compatible runtime (see 3823:ref:`amdgpu-os`) are 64-bit addresses of a hardware 32-byte V# and 48 byte S# 3824object respectively. In order to support the HSA ``query_sampler`` operations 3825two extra dwords are used to store the HSA BRIG enumeration values for the 3826queries that are not trivially deducible from the S# representation. 3827 3828HSA Signals 3829~~~~~~~~~~~ 3830 3831HSA signal handles created by an HSA compatible runtime (see :ref:`amdgpu-os`) 3832are 64-bit addresses of a structure allocated in memory accessible from both the 3833CPU and GPU. The structure is defined by the runtime and subject to change 3834between releases. For example, see [AMD-ROCm-github]_. 3835 3836.. _amdgpu-amdhsa-hsa-aql-queue: 3837 3838HSA AQL Queue 3839~~~~~~~~~~~~~ 3840 3841The HSA AQL queue structure is defined by an HSA compatible runtime (see 3842:ref:`amdgpu-os`) and subject to change between releases. For example, see 3843[AMD-ROCm-github]_. For some processors it contains fields needed to implement 3844certain language features such as the flat address aperture bases. It also 3845contains fields used by CP such as managing the allocation of scratch memory. 3846 3847.. _amdgpu-amdhsa-kernel-descriptor: 3848 3849Kernel Descriptor 3850~~~~~~~~~~~~~~~~~ 3851 3852A kernel descriptor consists of the information needed by CP to initiate the 3853execution of a kernel, including the entry point address of the machine code 3854that implements the kernel. 3855 3856Code Object V3 Kernel Descriptor 3857++++++++++++++++++++++++++++++++ 3858 3859CP microcode requires the Kernel descriptor to be allocated on 64-byte 3860alignment. 3861 3862The fields used by CP for code objects before V3 also match those specified in 3863:ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 3864 3865 .. table:: Code Object V3 Kernel Descriptor 3866 :name: amdgpu-amdhsa-kernel-descriptor-v3-table 3867 3868 ======= ======= =============================== ============================ 3869 Bits Size Field Name Description 3870 ======= ======= =============================== ============================ 3871 31:0 4 bytes GROUP_SEGMENT_FIXED_SIZE The amount of fixed local 3872 address space memory 3873 required for a work-group 3874 in bytes. This does not 3875 include any dynamically 3876 allocated local address 3877 space memory that may be 3878 added when the kernel is 3879 dispatched. 3880 63:32 4 bytes PRIVATE_SEGMENT_FIXED_SIZE The amount of fixed 3881 private address space 3882 memory required for a 3883 work-item in bytes. 3884 Additional space may need to 3885 be added to this value if 3886 the call stack has 3887 non-inlined function calls. 3888 95:64 4 bytes KERNARG_SIZE The size of the kernarg 3889 memory pointed to by the 3890 AQL dispatch packet. The 3891 kernarg memory is used to 3892 pass arguments to the 3893 kernel. 3894 3895 * If the kernarg pointer in 3896 the dispatch packet is NULL 3897 then there are no kernel 3898 arguments. 3899 * If the kernarg pointer in 3900 the dispatch packet is 3901 not NULL and this value 3902 is 0 then the kernarg 3903 memory size is 3904 unspecified. 3905 * If the kernarg pointer in 3906 the dispatch packet is 3907 not NULL and this value 3908 is not 0 then the value 3909 specifies the kernarg 3910 memory size in bytes. It 3911 is recommended to provide 3912 a value as it may be used 3913 by CP to optimize making 3914 the kernarg memory 3915 visible to the kernel 3916 code. 3917 3918 127:96 4 bytes Reserved, must be 0. 3919 191:128 8 bytes KERNEL_CODE_ENTRY_BYTE_OFFSET Byte offset (possibly 3920 negative) from base 3921 address of kernel 3922 descriptor to kernel's 3923 entry point instruction 3924 which must be 256 byte 3925 aligned. 3926 351:272 20 Reserved, must be 0. 3927 bytes 3928 383:352 4 bytes COMPUTE_PGM_RSRC3 GFX6-GFX9 3929 Reserved, must be 0. 3930 GFX90A, GFX940 3931 Compute Shader (CS) 3932 program settings used by 3933 CP to set up 3934 ``COMPUTE_PGM_RSRC3`` 3935 configuration 3936 register. See 3937 :ref:`amdgpu-amdhsa-compute_pgm_rsrc3-gfx90a-table`. 3938 GFX10 3939 Compute Shader (CS) 3940 program settings used by 3941 CP to set up 3942 ``COMPUTE_PGM_RSRC3`` 3943 configuration 3944 register. See 3945 :ref:`amdgpu-amdhsa-compute_pgm_rsrc3-gfx10-table`. 3946 415:384 4 bytes COMPUTE_PGM_RSRC1 Compute Shader (CS) 3947 program settings used by 3948 CP to set up 3949 ``COMPUTE_PGM_RSRC1`` 3950 configuration 3951 register. See 3952 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 3953 447:416 4 bytes COMPUTE_PGM_RSRC2 Compute Shader (CS) 3954 program settings used by 3955 CP to set up 3956 ``COMPUTE_PGM_RSRC2`` 3957 configuration 3958 register. See 3959 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 3960 458:448 7 bits *See separate bits below.* Enable the setup of the 3961 SGPR user data registers 3962 (see 3963 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 3964 3965 The total number of SGPR 3966 user data registers 3967 requested must not exceed 3968 16 and match value in 3969 ``compute_pgm_rsrc2.user_sgpr.user_sgpr_count``. 3970 Any requests beyond 16 3971 will be ignored. 3972 >448 1 bit ENABLE_SGPR_PRIVATE_SEGMENT If the *Target Properties* 3973 _BUFFER column of 3974 :ref:`amdgpu-processor-table` 3975 specifies *Architected flat 3976 scratch* then not supported 3977 and must be 0, 3978 >449 1 bit ENABLE_SGPR_DISPATCH_PTR 3979 >450 1 bit ENABLE_SGPR_QUEUE_PTR 3980 >451 1 bit ENABLE_SGPR_KERNARG_SEGMENT_PTR 3981 >452 1 bit ENABLE_SGPR_DISPATCH_ID 3982 >453 1 bit ENABLE_SGPR_FLAT_SCRATCH_INIT If the *Target Properties* 3983 column of 3984 :ref:`amdgpu-processor-table` 3985 specifies *Architected flat 3986 scratch* then not supported 3987 and must be 0, 3988 >454 1 bit ENABLE_SGPR_PRIVATE_SEGMENT 3989 _SIZE 3990 457:455 3 bits Reserved, must be 0. 3991 458 1 bit ENABLE_WAVEFRONT_SIZE32 GFX6-GFX9 3992 Reserved, must be 0. 3993 GFX10 3994 - If 0 execute in 3995 wavefront size 64 mode. 3996 - If 1 execute in 3997 native wavefront size 3998 32 mode. 3999 463:459 1 bit Reserved, must be 0. 4000 464 1 bit RESERVED_464 Deprecated, must be 0. 4001 467:465 3 bits Reserved, must be 0. 4002 468 1 bit RESERVED_468 Deprecated, must be 0. 4003 469:471 3 bits Reserved, must be 0. 4004 511:472 5 bytes Reserved, must be 0. 4005 512 **Total size 64 bytes.** 4006 ======= ==================================================================== 4007 4008.. 4009 4010 .. table:: compute_pgm_rsrc1 for GFX6-GFX10 4011 :name: amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table 4012 4013 ======= ======= =============================== =========================================================================== 4014 Bits Size Field Name Description 4015 ======= ======= =============================== =========================================================================== 4016 5:0 6 bits GRANULATED_WORKITEM_VGPR_COUNT Number of vector register 4017 blocks used by each work-item; 4018 granularity is device 4019 specific: 4020 4021 GFX6-GFX9 4022 - vgprs_used 0..256 4023 - max(0, ceil(vgprs_used / 4) - 1) 4024 GFX90A, GFX940 4025 - vgprs_used 0..512 4026 - vgprs_used = align(arch_vgprs, 4) 4027 + acc_vgprs 4028 - max(0, ceil(vgprs_used / 8) - 1) 4029 GFX10 (wavefront size 64) 4030 - max_vgpr 1..256 4031 - max(0, ceil(vgprs_used / 4) - 1) 4032 GFX10 (wavefront size 32) 4033 - max_vgpr 1..256 4034 - max(0, ceil(vgprs_used / 8) - 1) 4035 4036 Where vgprs_used is defined 4037 as the highest VGPR number 4038 explicitly referenced plus 4039 one. 4040 4041 Used by CP to set up 4042 ``COMPUTE_PGM_RSRC1.VGPRS``. 4043 4044 The 4045 :ref:`amdgpu-assembler` 4046 calculates this 4047 automatically for the 4048 selected processor from 4049 values provided to the 4050 `.amdhsa_kernel` directive 4051 by the 4052 `.amdhsa_next_free_vgpr` 4053 nested directive (see 4054 :ref:`amdhsa-kernel-directives-table`). 4055 9:6 4 bits GRANULATED_WAVEFRONT_SGPR_COUNT Number of scalar register 4056 blocks used by a wavefront; 4057 granularity is device 4058 specific: 4059 4060 GFX6-GFX8 4061 - sgprs_used 0..112 4062 - max(0, ceil(sgprs_used / 8) - 1) 4063 GFX9 4064 - sgprs_used 0..112 4065 - 2 * max(0, ceil(sgprs_used / 16) - 1) 4066 GFX10 4067 Reserved, must be 0. 4068 (128 SGPRs always 4069 allocated.) 4070 4071 Where sgprs_used is 4072 defined as the highest 4073 SGPR number explicitly 4074 referenced plus one, plus 4075 a target specific number 4076 of additional special 4077 SGPRs for VCC, 4078 FLAT_SCRATCH (GFX7+) and 4079 XNACK_MASK (GFX8+), and 4080 any additional 4081 target specific 4082 limitations. It does not 4083 include the 16 SGPRs added 4084 if a trap handler is 4085 enabled. 4086 4087 The target specific 4088 limitations and special 4089 SGPR layout are defined in 4090 the hardware 4091 documentation, which can 4092 be found in the 4093 :ref:`amdgpu-processors` 4094 table. 4095 4096 Used by CP to set up 4097 ``COMPUTE_PGM_RSRC1.SGPRS``. 4098 4099 The 4100 :ref:`amdgpu-assembler` 4101 calculates this 4102 automatically for the 4103 selected processor from 4104 values provided to the 4105 `.amdhsa_kernel` directive 4106 by the 4107 `.amdhsa_next_free_sgpr` 4108 and `.amdhsa_reserve_*` 4109 nested directives (see 4110 :ref:`amdhsa-kernel-directives-table`). 4111 11:10 2 bits PRIORITY Must be 0. 4112 4113 Start executing wavefront 4114 at the specified priority. 4115 4116 CP is responsible for 4117 filling in 4118 ``COMPUTE_PGM_RSRC1.PRIORITY``. 4119 13:12 2 bits FLOAT_ROUND_MODE_32 Wavefront starts execution 4120 with specified rounding 4121 mode for single (32 4122 bit) floating point 4123 precision floating point 4124 operations. 4125 4126 Floating point rounding 4127 mode values are defined in 4128 :ref:`amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table`. 4129 4130 Used by CP to set up 4131 ``COMPUTE_PGM_RSRC1.FLOAT_MODE``. 4132 15:14 2 bits FLOAT_ROUND_MODE_16_64 Wavefront starts execution 4133 with specified rounding 4134 denorm mode for half/double (16 4135 and 64-bit) floating point 4136 precision floating point 4137 operations. 4138 4139 Floating point rounding 4140 mode values are defined in 4141 :ref:`amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table`. 4142 4143 Used by CP to set up 4144 ``COMPUTE_PGM_RSRC1.FLOAT_MODE``. 4145 17:16 2 bits FLOAT_DENORM_MODE_32 Wavefront starts execution 4146 with specified denorm mode 4147 for single (32 4148 bit) floating point 4149 precision floating point 4150 operations. 4151 4152 Floating point denorm mode 4153 values are defined in 4154 :ref:`amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table`. 4155 4156 Used by CP to set up 4157 ``COMPUTE_PGM_RSRC1.FLOAT_MODE``. 4158 19:18 2 bits FLOAT_DENORM_MODE_16_64 Wavefront starts execution 4159 with specified denorm mode 4160 for half/double (16 4161 and 64-bit) floating point 4162 precision floating point 4163 operations. 4164 4165 Floating point denorm mode 4166 values are defined in 4167 :ref:`amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table`. 4168 4169 Used by CP to set up 4170 ``COMPUTE_PGM_RSRC1.FLOAT_MODE``. 4171 20 1 bit PRIV Must be 0. 4172 4173 Start executing wavefront 4174 in privilege trap handler 4175 mode. 4176 4177 CP is responsible for 4178 filling in 4179 ``COMPUTE_PGM_RSRC1.PRIV``. 4180 21 1 bit ENABLE_DX10_CLAMP Wavefront starts execution 4181 with DX10 clamp mode 4182 enabled. Used by the vector 4183 ALU to force DX10 style 4184 treatment of NaN's (when 4185 set, clamp NaN to zero, 4186 otherwise pass NaN 4187 through). 4188 4189 Used by CP to set up 4190 ``COMPUTE_PGM_RSRC1.DX10_CLAMP``. 4191 22 1 bit DEBUG_MODE Must be 0. 4192 4193 Start executing wavefront 4194 in single step mode. 4195 4196 CP is responsible for 4197 filling in 4198 ``COMPUTE_PGM_RSRC1.DEBUG_MODE``. 4199 23 1 bit ENABLE_IEEE_MODE Wavefront starts execution 4200 with IEEE mode 4201 enabled. Floating point 4202 opcodes that support 4203 exception flag gathering 4204 will quiet and propagate 4205 signaling-NaN inputs per 4206 IEEE 754-2008. Min_dx10 and 4207 max_dx10 become IEEE 4208 754-2008 compliant due to 4209 signaling-NaN propagation 4210 and quieting. 4211 4212 Used by CP to set up 4213 ``COMPUTE_PGM_RSRC1.IEEE_MODE``. 4214 24 1 bit BULKY Must be 0. 4215 4216 Only one work-group allowed 4217 to execute on a compute 4218 unit. 4219 4220 CP is responsible for 4221 filling in 4222 ``COMPUTE_PGM_RSRC1.BULKY``. 4223 25 1 bit CDBG_USER Must be 0. 4224 4225 Flag that can be used to 4226 control debugging code. 4227 4228 CP is responsible for 4229 filling in 4230 ``COMPUTE_PGM_RSRC1.CDBG_USER``. 4231 26 1 bit FP16_OVFL GFX6-GFX8 4232 Reserved, must be 0. 4233 GFX9-GFX10 4234 Wavefront starts execution 4235 with specified fp16 overflow 4236 mode. 4237 4238 - If 0, fp16 overflow generates 4239 +/-INF values. 4240 - If 1, fp16 overflow that is the 4241 result of an +/-INF input value 4242 or divide by 0 produces a +/-INF, 4243 otherwise clamps computed 4244 overflow to +/-MAX_FP16 as 4245 appropriate. 4246 4247 Used by CP to set up 4248 ``COMPUTE_PGM_RSRC1.FP16_OVFL``. 4249 28:27 2 bits Reserved, must be 0. 4250 29 1 bit WGP_MODE GFX6-GFX9 4251 Reserved, must be 0. 4252 GFX10 4253 - If 0 execute work-groups in 4254 CU wavefront execution mode. 4255 - If 1 execute work-groups on 4256 in WGP wavefront execution mode. 4257 4258 See :ref:`amdgpu-amdhsa-memory-model`. 4259 4260 Used by CP to set up 4261 ``COMPUTE_PGM_RSRC1.WGP_MODE``. 4262 30 1 bit MEM_ORDERED GFX6-GFX9 4263 Reserved, must be 0. 4264 GFX10 4265 Controls the behavior of the 4266 s_waitcnt's vmcnt and vscnt 4267 counters. 4268 4269 - If 0 vmcnt reports completion 4270 of load and atomic with return 4271 out of order with sample 4272 instructions, and the vscnt 4273 reports the completion of 4274 store and atomic without 4275 return in order. 4276 - If 1 vmcnt reports completion 4277 of load, atomic with return 4278 and sample instructions in 4279 order, and the vscnt reports 4280 the completion of store and 4281 atomic without return in order. 4282 4283 Used by CP to set up 4284 ``COMPUTE_PGM_RSRC1.MEM_ORDERED``. 4285 31 1 bit FWD_PROGRESS GFX6-GFX9 4286 Reserved, must be 0. 4287 GFX10 4288 - If 0 execute SIMD wavefronts 4289 using oldest first policy. 4290 - If 1 execute SIMD wavefronts to 4291 ensure wavefronts will make some 4292 forward progress. 4293 4294 Used by CP to set up 4295 ``COMPUTE_PGM_RSRC1.FWD_PROGRESS``. 4296 32 **Total size 4 bytes** 4297 ======= =================================================================================================================== 4298 4299.. 4300 4301 .. table:: compute_pgm_rsrc2 for GFX6-GFX10 4302 :name: amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table 4303 4304 ======= ======= =============================== =========================================================================== 4305 Bits Size Field Name Description 4306 ======= ======= =============================== =========================================================================== 4307 0 1 bit ENABLE_PRIVATE_SEGMENT * Enable the setup of the 4308 private segment. 4309 * If the *Target Properties* 4310 column of 4311 :ref:`amdgpu-processor-table` 4312 does not specify 4313 *Architected flat 4314 scratch* then enable the 4315 setup of the SGPR 4316 wavefront scratch offset 4317 system register (see 4318 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4319 * If the *Target Properties* 4320 column of 4321 :ref:`amdgpu-processor-table` 4322 specifies *Architected 4323 flat scratch* then enable 4324 the setup of the 4325 FLAT_SCRATCH register 4326 pair (see 4327 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4328 4329 Used by CP to set up 4330 ``COMPUTE_PGM_RSRC2.SCRATCH_EN``. 4331 5:1 5 bits USER_SGPR_COUNT The total number of SGPR 4332 user data 4333 registers requested. This 4334 number must be greater than 4335 or equal to the number of user 4336 data registers enabled. 4337 4338 Used by CP to set up 4339 ``COMPUTE_PGM_RSRC2.USER_SGPR``. 4340 6 1 bit ENABLE_TRAP_HANDLER Must be 0. 4341 4342 This bit represents 4343 ``COMPUTE_PGM_RSRC2.TRAP_PRESENT``, 4344 which is set by the CP if 4345 the runtime has installed a 4346 trap handler. 4347 7 1 bit ENABLE_SGPR_WORKGROUP_ID_X Enable the setup of the 4348 system SGPR register for 4349 the work-group id in the X 4350 dimension (see 4351 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4352 4353 Used by CP to set up 4354 ``COMPUTE_PGM_RSRC2.TGID_X_EN``. 4355 8 1 bit ENABLE_SGPR_WORKGROUP_ID_Y Enable the setup of the 4356 system SGPR register for 4357 the work-group id in the Y 4358 dimension (see 4359 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4360 4361 Used by CP to set up 4362 ``COMPUTE_PGM_RSRC2.TGID_Y_EN``. 4363 9 1 bit ENABLE_SGPR_WORKGROUP_ID_Z Enable the setup of the 4364 system SGPR register for 4365 the work-group id in the Z 4366 dimension (see 4367 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4368 4369 Used by CP to set up 4370 ``COMPUTE_PGM_RSRC2.TGID_Z_EN``. 4371 10 1 bit ENABLE_SGPR_WORKGROUP_INFO Enable the setup of the 4372 system SGPR register for 4373 work-group information (see 4374 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4375 4376 Used by CP to set up 4377 ``COMPUTE_PGM_RSRC2.TGID_SIZE_EN``. 4378 12:11 2 bits ENABLE_VGPR_WORKITEM_ID Enable the setup of the 4379 VGPR system registers used 4380 for the work-item ID. 4381 :ref:`amdgpu-amdhsa-system-vgpr-work-item-id-enumeration-values-table` 4382 defines the values. 4383 4384 Used by CP to set up 4385 ``COMPUTE_PGM_RSRC2.TIDIG_CMP_CNT``. 4386 13 1 bit ENABLE_EXCEPTION_ADDRESS_WATCH Must be 0. 4387 4388 Wavefront starts execution 4389 with address watch 4390 exceptions enabled which 4391 are generated when L1 has 4392 witnessed a thread access 4393 an *address of 4394 interest*. 4395 4396 CP is responsible for 4397 filling in the address 4398 watch bit in 4399 ``COMPUTE_PGM_RSRC2.EXCP_EN_MSB`` 4400 according to what the 4401 runtime requests. 4402 14 1 bit ENABLE_EXCEPTION_MEMORY Must be 0. 4403 4404 Wavefront starts execution 4405 with memory violation 4406 exceptions exceptions 4407 enabled which are generated 4408 when a memory violation has 4409 occurred for this wavefront from 4410 L1 or LDS 4411 (write-to-read-only-memory, 4412 mis-aligned atomic, LDS 4413 address out of range, 4414 illegal address, etc.). 4415 4416 CP sets the memory 4417 violation bit in 4418 ``COMPUTE_PGM_RSRC2.EXCP_EN_MSB`` 4419 according to what the 4420 runtime requests. 4421 23:15 9 bits GRANULATED_LDS_SIZE Must be 0. 4422 4423 CP uses the rounded value 4424 from the dispatch packet, 4425 not this value, as the 4426 dispatch may contain 4427 dynamically allocated group 4428 segment memory. CP writes 4429 directly to 4430 ``COMPUTE_PGM_RSRC2.LDS_SIZE``. 4431 4432 Amount of group segment 4433 (LDS) to allocate for each 4434 work-group. Granularity is 4435 device specific: 4436 4437 GFX6 4438 roundup(lds-size / (64 * 4)) 4439 GFX7-GFX10 4440 roundup(lds-size / (128 * 4)) 4441 4442 24 1 bit ENABLE_EXCEPTION_IEEE_754_FP Wavefront starts execution 4443 _INVALID_OPERATION with specified exceptions 4444 enabled. 4445 4446 Used by CP to set up 4447 ``COMPUTE_PGM_RSRC2.EXCP_EN`` 4448 (set from bits 0..6). 4449 4450 IEEE 754 FP Invalid 4451 Operation 4452 25 1 bit ENABLE_EXCEPTION_FP_DENORMAL FP Denormal one or more 4453 _SOURCE input operands is a 4454 denormal number 4455 26 1 bit ENABLE_EXCEPTION_IEEE_754_FP IEEE 754 FP Division by 4456 _DIVISION_BY_ZERO Zero 4457 27 1 bit ENABLE_EXCEPTION_IEEE_754_FP IEEE 754 FP FP Overflow 4458 _OVERFLOW 4459 28 1 bit ENABLE_EXCEPTION_IEEE_754_FP IEEE 754 FP Underflow 4460 _UNDERFLOW 4461 29 1 bit ENABLE_EXCEPTION_IEEE_754_FP IEEE 754 FP Inexact 4462 _INEXACT 4463 30 1 bit ENABLE_EXCEPTION_INT_DIVIDE_BY Integer Division by Zero 4464 _ZERO (rcp_iflag_f32 instruction 4465 only) 4466 31 1 bit Reserved, must be 0. 4467 32 **Total size 4 bytes.** 4468 ======= =================================================================================================================== 4469 4470.. 4471 4472 .. table:: compute_pgm_rsrc3 for GFX90A, GFX940 4473 :name: amdgpu-amdhsa-compute_pgm_rsrc3-gfx90a-table 4474 4475 ======= ======= =============================== =========================================================================== 4476 Bits Size Field Name Description 4477 ======= ======= =============================== =========================================================================== 4478 5:0 6 bits ACCUM_OFFSET Offset of a first AccVGPR in the unified register file. Granularity 4. 4479 Value 0-63. 0 - accum-offset = 4, 1 - accum-offset = 8, ..., 4480 63 - accum-offset = 256. 4481 6:15 10 Reserved, must be 0. 4482 bits 4483 16 1 bit TG_SPLIT - If 0 the waves of a work-group are 4484 launched in the same CU. 4485 - If 1 the waves of a work-group can be 4486 launched in different CUs. The waves 4487 cannot use S_BARRIER or LDS. 4488 17:31 15 Reserved, must be 0. 4489 bits 4490 32 **Total size 4 bytes.** 4491 ======= =================================================================================================================== 4492 4493.. 4494 4495 .. table:: compute_pgm_rsrc3 for GFX10 4496 :name: amdgpu-amdhsa-compute_pgm_rsrc3-gfx10-table 4497 4498 ======= ======= =============================== =========================================================================== 4499 Bits Size Field Name Description 4500 ======= ======= =============================== =========================================================================== 4501 3:0 4 bits SHARED_VGPR_COUNT Number of shared VGPR blocks when executing in subvector mode. For 4502 wavefront size 64 the value is 0-15, representing 0-120 VGPRs (granularity 4503 of 8), such that (compute_pgm_rsrc1.vgprs +1)*4 + shared_vgpr_count*8 does 4504 not exceed 256. For wavefront size 32 shared_vgpr_count must be 0. 4505 31:4 28 Reserved, must be 0. 4506 bits 4507 32 **Total size 4 bytes.** 4508 ======= =================================================================================================================== 4509 4510.. 4511 4512 .. table:: Floating Point Rounding Mode Enumeration Values 4513 :name: amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table 4514 4515 ====================================== ===== ============================== 4516 Enumeration Name Value Description 4517 ====================================== ===== ============================== 4518 FLOAT_ROUND_MODE_NEAR_EVEN 0 Round Ties To Even 4519 FLOAT_ROUND_MODE_PLUS_INFINITY 1 Round Toward +infinity 4520 FLOAT_ROUND_MODE_MINUS_INFINITY 2 Round Toward -infinity 4521 FLOAT_ROUND_MODE_ZERO 3 Round Toward 0 4522 ====================================== ===== ============================== 4523 4524.. 4525 4526 .. table:: Floating Point Denorm Mode Enumeration Values 4527 :name: amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table 4528 4529 ====================================== ===== ============================== 4530 Enumeration Name Value Description 4531 ====================================== ===== ============================== 4532 FLOAT_DENORM_MODE_FLUSH_SRC_DST 0 Flush Source and Destination 4533 Denorms 4534 FLOAT_DENORM_MODE_FLUSH_DST 1 Flush Output Denorms 4535 FLOAT_DENORM_MODE_FLUSH_SRC 2 Flush Source Denorms 4536 FLOAT_DENORM_MODE_FLUSH_NONE 3 No Flush 4537 ====================================== ===== ============================== 4538 4539.. 4540 4541 .. table:: System VGPR Work-Item ID Enumeration Values 4542 :name: amdgpu-amdhsa-system-vgpr-work-item-id-enumeration-values-table 4543 4544 ======================================== ===== ============================ 4545 Enumeration Name Value Description 4546 ======================================== ===== ============================ 4547 SYSTEM_VGPR_WORKITEM_ID_X 0 Set work-item X dimension 4548 ID. 4549 SYSTEM_VGPR_WORKITEM_ID_X_Y 1 Set work-item X and Y 4550 dimensions ID. 4551 SYSTEM_VGPR_WORKITEM_ID_X_Y_Z 2 Set work-item X, Y and Z 4552 dimensions ID. 4553 SYSTEM_VGPR_WORKITEM_ID_UNDEFINED 3 Undefined. 4554 ======================================== ===== ============================ 4555 4556.. _amdgpu-amdhsa-initial-kernel-execution-state: 4557 4558Initial Kernel Execution State 4559~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 4560 4561This section defines the register state that will be set up by the packet 4562processor prior to the start of execution of every wavefront. This is limited by 4563the constraints of the hardware controllers of CP/ADC/SPI. 4564 4565The order of the SGPR registers is defined, but the compiler can specify which 4566ones are actually setup in the kernel descriptor using the ``enable_sgpr_*`` bit 4567fields (see :ref:`amdgpu-amdhsa-kernel-descriptor`). The register numbers used 4568for enabled registers are dense starting at SGPR0: the first enabled register is 4569SGPR0, the next enabled register is SGPR1 etc.; disabled registers do not have 4570an SGPR number. 4571 4572The initial SGPRs comprise up to 16 User SRGPs that are set by CP and apply to 4573all wavefronts of the grid. It is possible to specify more than 16 User SGPRs 4574using the ``enable_sgpr_*`` bit fields, in which case only the first 16 are 4575actually initialized. These are then immediately followed by the System SGPRs 4576that are set up by ADC/SPI and can have different values for each wavefront of 4577the grid dispatch. 4578 4579SGPR register initial state is defined in 4580:ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 4581 4582 .. table:: SGPR Register Set Up Order 4583 :name: amdgpu-amdhsa-sgpr-register-set-up-order-table 4584 4585 ========== ========================== ====== ============================== 4586 SGPR Order Name Number Description 4587 (kernel descriptor enable of 4588 field) SGPRs 4589 ========== ========================== ====== ============================== 4590 First Private Segment Buffer 4 See 4591 (enable_sgpr_private :ref:`amdgpu-amdhsa-kernel-prolog-private-segment-buffer`. 4592 _segment_buffer) 4593 then Dispatch Ptr 2 64-bit address of AQL dispatch 4594 (enable_sgpr_dispatch_ptr) packet for kernel dispatch 4595 actually executing. 4596 then Queue Ptr 2 64-bit address of amd_queue_t 4597 (enable_sgpr_queue_ptr) object for AQL queue on which 4598 the dispatch packet was 4599 queued. 4600 then Kernarg Segment Ptr 2 64-bit address of Kernarg 4601 (enable_sgpr_kernarg segment. This is directly 4602 _segment_ptr) copied from the 4603 kernarg_address in the kernel 4604 dispatch packet. 4605 4606 Having CP load it once avoids 4607 loading it at the beginning of 4608 every wavefront. 4609 then Dispatch Id 2 64-bit Dispatch ID of the 4610 (enable_sgpr_dispatch_id) dispatch packet being 4611 executed. 4612 then Flat Scratch Init 2 See 4613 (enable_sgpr_flat_scratch :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`. 4614 _init) 4615 then Private Segment Size 1 The 32-bit byte size of a 4616 (enable_sgpr_private single work-item's memory 4617 _segment_size) allocation. This is the 4618 value from the kernel 4619 dispatch packet Private 4620 Segment Byte Size rounded up 4621 by CP to a multiple of 4622 DWORD. 4623 4624 Having CP load it once avoids 4625 loading it at the beginning of 4626 every wavefront. 4627 4628 This is not used for 4629 GFX7-GFX8 since it is the same 4630 value as the second SGPR of 4631 Flat Scratch Init. However, it 4632 may be needed for GFX9-GFX10 which 4633 changes the meaning of the 4634 Flat Scratch Init value. 4635 then Work-Group Id X 1 32-bit work-group id in X 4636 (enable_sgpr_workgroup_id dimension of grid for 4637 _X) wavefront. 4638 then Work-Group Id Y 1 32-bit work-group id in Y 4639 (enable_sgpr_workgroup_id dimension of grid for 4640 _Y) wavefront. 4641 then Work-Group Id Z 1 32-bit work-group id in Z 4642 (enable_sgpr_workgroup_id dimension of grid for 4643 _Z) wavefront. 4644 then Work-Group Info 1 {first_wavefront, 14'b0000, 4645 (enable_sgpr_workgroup ordered_append_term[10:0], 4646 _info) threadgroup_size_in_wavefronts[5:0]} 4647 then Scratch Wavefront Offset 1 See 4648 (enable_sgpr_private :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`. 4649 _segment_wavefront_offset) and 4650 :ref:`amdgpu-amdhsa-kernel-prolog-private-segment-buffer`. 4651 ========== ========================== ====== ============================== 4652 4653The order of the VGPR registers is defined, but the compiler can specify which 4654ones are actually setup in the kernel descriptor using the ``enable_vgpr*`` bit 4655fields (see :ref:`amdgpu-amdhsa-kernel-descriptor`). The register numbers used 4656for enabled registers are dense starting at VGPR0: the first enabled register is 4657VGPR0, the next enabled register is VGPR1 etc.; disabled registers do not have a 4658VGPR number. 4659 4660There are different methods used for the VGPR initial state: 4661 4662* Unless the *Target Properties* column of :ref:`amdgpu-processor-table` 4663 specifies otherwise, a separate VGPR register is used per work-item ID. The 4664 VGPR register initial state for this method is defined in 4665 :ref:`amdgpu-amdhsa-vgpr-register-set-up-order-for-unpacked-work-item-id-method-table`. 4666* If *Target Properties* column of :ref:`amdgpu-processor-table` 4667 specifies *Packed work-item IDs*, the initial value of VGPR0 register is used 4668 for all work-item IDs. The register layout for this method is defined in 4669 :ref:`amdgpu-amdhsa-register-layout-for-packed-work-item-id-method-table`. 4670 4671 .. table:: VGPR Register Set Up Order for Unpacked Work-Item ID Method 4672 :name: amdgpu-amdhsa-vgpr-register-set-up-order-for-unpacked-work-item-id-method-table 4673 4674 ========== ========================== ====== ============================== 4675 VGPR Order Name Number Description 4676 (kernel descriptor enable of 4677 field) VGPRs 4678 ========== ========================== ====== ============================== 4679 First Work-Item Id X 1 32-bit work-item id in X 4680 (Always initialized) dimension of work-group for 4681 wavefront lane. 4682 then Work-Item Id Y 1 32-bit work-item id in Y 4683 (enable_vgpr_workitem_id dimension of work-group for 4684 > 0) wavefront lane. 4685 then Work-Item Id Z 1 32-bit work-item id in Z 4686 (enable_vgpr_workitem_id dimension of work-group for 4687 > 1) wavefront lane. 4688 ========== ========================== ====== ============================== 4689 4690.. 4691 4692 .. table:: Register Layout for Packed Work-Item ID Method 4693 :name: amdgpu-amdhsa-register-layout-for-packed-work-item-id-method-table 4694 4695 ======= ======= ================ ========================================= 4696 Bits Size Field Name Description 4697 ======= ======= ================ ========================================= 4698 0:9 10 bits Work-Item Id X Work-item id in X 4699 dimension of work-group for 4700 wavefront lane. 4701 4702 Always initialized. 4703 4704 10:19 10 bits Work-Item Id Y Work-item id in Y 4705 dimension of work-group for 4706 wavefront lane. 4707 4708 Initialized if enable_vgpr_workitem_id > 4709 0, otherwise set to 0. 4710 20:29 10 bits Work-Item Id Z Work-item id in Z 4711 dimension of work-group for 4712 wavefront lane. 4713 4714 Initialized if enable_vgpr_workitem_id > 4715 1, otherwise set to 0. 4716 30:31 2 bits Reserved, set to 0. 4717 ======= ======= ================ ========================================= 4718 4719The setting of registers is done by GPU CP/ADC/SPI hardware as follows: 4720 47211. SGPRs before the Work-Group Ids are set by CP using the 16 User Data 4722 registers. 47232. Work-group Id registers X, Y, Z are set by ADC which supports any 4724 combination including none. 47253. Scratch Wavefront Offset is set by SPI in a per wavefront basis which is why 4726 its value cannot be included with the flat scratch init value which is per 4727 queue (see :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`). 47284. The VGPRs are set by SPI which only supports specifying either (X), (X, Y) 4729 or (X, Y, Z). 47305. Flat Scratch register pair initialization is described in 4731 :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`. 4732 4733The global segment can be accessed either using buffer instructions (GFX6 which 4734has V# 64-bit address support), flat instructions (GFX7-GFX10), or global 4735instructions (GFX9-GFX10). 4736 4737If buffer operations are used, then the compiler can generate a V# with the 4738following properties: 4739 4740* base address of 0 4741* no swizzle 4742* ATC: 1 if IOMMU present (such as APU) 4743* ptr64: 1 4744* MTYPE set to support memory coherence that matches the runtime (such as CC for 4745 APU and NC for dGPU). 4746 4747.. _amdgpu-amdhsa-kernel-prolog: 4748 4749Kernel Prolog 4750~~~~~~~~~~~~~ 4751 4752The compiler performs initialization in the kernel prologue depending on the 4753target and information about things like stack usage in the kernel and called 4754functions. Some of this initialization requires the compiler to request certain 4755User and System SGPRs be present in the 4756:ref:`amdgpu-amdhsa-initial-kernel-execution-state` via the 4757:ref:`amdgpu-amdhsa-kernel-descriptor`. 4758 4759.. _amdgpu-amdhsa-kernel-prolog-cfi: 4760 4761CFI 4762+++ 4763 47641. The CFI return address is undefined. 4765 47662. The CFI CFA is defined using an expression which evaluates to a location 4767 description that comprises one memory location description for the 4768 ``DW_ASPACE_AMDGPU_private_lane`` address space address ``0``. 4769 4770.. _amdgpu-amdhsa-kernel-prolog-m0: 4771 4772M0 4773++ 4774 4775GFX6-GFX8 4776 The M0 register must be initialized with a value at least the total LDS size 4777 if the kernel may access LDS via DS or flat operations. Total LDS size is 4778 available in dispatch packet. For M0, it is also possible to use maximum 4779 possible value of LDS for given target (0x7FFF for GFX6 and 0xFFFF for 4780 GFX7-GFX8). 4781GFX9-GFX10 4782 The M0 register is not used for range checking LDS accesses and so does not 4783 need to be initialized in the prolog. 4784 4785.. _amdgpu-amdhsa-kernel-prolog-stack-pointer: 4786 4787Stack Pointer 4788+++++++++++++ 4789 4790If the kernel has function calls it must set up the ABI stack pointer described 4791in :ref:`amdgpu-amdhsa-function-call-convention-non-kernel-functions` by setting 4792SGPR32 to the unswizzled scratch offset of the address past the last local 4793allocation. 4794 4795.. _amdgpu-amdhsa-kernel-prolog-frame-pointer: 4796 4797Frame Pointer 4798+++++++++++++ 4799 4800If the kernel needs a frame pointer for the reasons defined in 4801``SIFrameLowering`` then SGPR33 is used and is always set to ``0`` in the 4802kernel prolog. If a frame pointer is not required then all uses of the frame 4803pointer are replaced with immediate ``0`` offsets. 4804 4805.. _amdgpu-amdhsa-kernel-prolog-flat-scratch: 4806 4807Flat Scratch 4808++++++++++++ 4809 4810There are different methods used for initializing flat scratch: 4811 4812* If the *Target Properties* column of :ref:`amdgpu-processor-table` 4813 specifies *Does not support generic address space*: 4814 4815 Flat scratch is not supported and there is no flat scratch register pair. 4816 4817* If the *Target Properties* column of :ref:`amdgpu-processor-table` 4818 specifies *Offset flat scratch*: 4819 4820 If the kernel or any function it calls may use flat operations to access 4821 scratch memory, the prolog code must set up the FLAT_SCRATCH register pair 4822 (FLAT_SCRATCH_LO/FLAT_SCRATCH_HI). Initialization uses Flat Scratch Init and 4823 Scratch Wavefront Offset SGPR registers (see 4824 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`): 4825 4826 1. The low word of Flat Scratch Init is the 32-bit byte offset from 4827 ``SH_HIDDEN_PRIVATE_BASE_VIMID`` to the base of scratch backing memory 4828 being managed by SPI for the queue executing the kernel dispatch. This is 4829 the same value used in the Scratch Segment Buffer V# base address. 4830 4831 CP obtains this from the runtime. (The Scratch Segment Buffer base address 4832 is ``SH_HIDDEN_PRIVATE_BASE_VIMID`` plus this offset.) 4833 4834 The prolog must add the value of Scratch Wavefront Offset to get the 4835 wavefront's byte scratch backing memory offset from 4836 ``SH_HIDDEN_PRIVATE_BASE_VIMID``. 4837 4838 The Scratch Wavefront Offset must also be used as an offset with Private 4839 segment address when using the Scratch Segment Buffer. 4840 4841 Since FLAT_SCRATCH_LO is in units of 256 bytes, the offset must be right 4842 shifted by 8 before moving into FLAT_SCRATCH_HI. 4843 4844 FLAT_SCRATCH_HI corresponds to SGPRn-4 on GFX7, and SGPRn-6 on GFX8 (where 4845 SGPRn is the highest numbered SGPR allocated to the wavefront). 4846 FLAT_SCRATCH_HI is multiplied by 256 (as it is in units of 256 bytes) and 4847 added to ``SH_HIDDEN_PRIVATE_BASE_VIMID`` to calculate the per wavefront 4848 FLAT SCRATCH BASE in flat memory instructions that access the scratch 4849 aperture. 4850 2. The second word of Flat Scratch Init is 32-bit byte size of a single 4851 work-items scratch memory usage. 4852 4853 CP obtains this from the runtime, and it is always a multiple of DWORD. CP 4854 checks that the value in the kernel dispatch packet Private Segment Byte 4855 Size is not larger and requests the runtime to increase the queue's scratch 4856 size if necessary. 4857 4858 CP directly loads from the kernel dispatch packet Private Segment Byte Size 4859 field and rounds up to a multiple of DWORD. Having CP load it once avoids 4860 loading it at the beginning of every wavefront. 4861 4862 The kernel prolog code must move it to FLAT_SCRATCH_LO which is SGPRn-3 on 4863 GFX7 and SGPRn-5 on GFX8. FLAT_SCRATCH_LO is used as the FLAT SCRATCH SIZE 4864 in flat memory instructions. 4865 4866* If the *Target Properties* column of :ref:`amdgpu-processor-table` 4867 specifies *Absolute flat scratch*: 4868 4869 If the kernel or any function it calls may use flat operations to access 4870 scratch memory, the prolog code must set up the FLAT_SCRATCH register pair 4871 (FLAT_SCRATCH_LO/FLAT_SCRATCH_HI which are in SGPRn-4/SGPRn-3). Initialization 4872 uses Flat Scratch Init and Scratch Wavefront Offset SGPR registers (see 4873 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`): 4874 4875 The Flat Scratch Init is the 64-bit address of the base of scratch backing 4876 memory being managed by SPI for the queue executing the kernel dispatch. 4877 4878 CP obtains this from the runtime. 4879 4880 The kernel prolog must add the value of the wave's Scratch Wavefront Offset 4881 and move the result as a 64-bit value to the FLAT_SCRATCH SGPR register pair 4882 which is SGPRn-6 and SGPRn-5. It is used as the FLAT SCRATCH BASE in flat 4883 memory instructions. 4884 4885 The Scratch Wavefront Offset must also be used as an offset with Private 4886 segment address when using the Scratch Segment Buffer (see 4887 :ref:`amdgpu-amdhsa-kernel-prolog-private-segment-buffer`). 4888 4889* If the *Target Properties* column of :ref:`amdgpu-processor-table` 4890 specifies *Architected flat scratch*: 4891 4892 If ENABLE_PRIVATE_SEGMENT is enabled in 4893 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table` then the FLAT_SCRATCH 4894 register pair will be initialized to the 64-bit address of the base of scratch 4895 backing memory being managed by SPI for the queue executing the kernel 4896 dispatch plus the value of the wave's Scratch Wavefront Offset for use as the 4897 flat scratch base in flat memory instructions. 4898 4899.. _amdgpu-amdhsa-kernel-prolog-private-segment-buffer: 4900 4901Private Segment Buffer 4902++++++++++++++++++++++ 4903 4904If the *Target Properties* column of :ref:`amdgpu-processor-table` specifies 4905*Architected flat scratch* then a Private Segment Buffer is not supported. 4906Instead the flat SCRATCH instructions are used. 4907 4908Otherwise, Private Segment Buffer SGPR register is used to initialize 4 SGPRs 4909that are used as a V# to access scratch. CP uses the value provided by the 4910runtime. It is used, together with Scratch Wavefront Offset as an offset, to 4911access the private memory space using a segment address. See 4912:ref:`amdgpu-amdhsa-initial-kernel-execution-state`. 4913 4914The scratch V# is a four-aligned SGPR and always selected for the kernel as 4915follows: 4916 4917 - If it is known during instruction selection that there is stack usage, 4918 SGPR0-3 is reserved for use as the scratch V#. Stack usage is assumed if 4919 optimizations are disabled (``-O0``), if stack objects already exist (for 4920 locals, etc.), or if there are any function calls. 4921 4922 - Otherwise, four high numbered SGPRs beginning at a four-aligned SGPR index 4923 are reserved for the tentative scratch V#. These will be used if it is 4924 determined that spilling is needed. 4925 4926 - If no use is made of the tentative scratch V#, then it is unreserved, 4927 and the register count is determined ignoring it. 4928 - If use is made of the tentative scratch V#, then its register numbers 4929 are shifted to the first four-aligned SGPR index after the highest one 4930 allocated by the register allocator, and all uses are updated. The 4931 register count includes them in the shifted location. 4932 - In either case, if the processor has the SGPR allocation bug, the 4933 tentative allocation is not shifted or unreserved in order to ensure 4934 the register count is higher to workaround the bug. 4935 4936 .. note:: 4937 4938 This approach of using a tentative scratch V# and shifting the register 4939 numbers if used avoids having to perform register allocation a second 4940 time if the tentative V# is eliminated. This is more efficient and 4941 avoids the problem that the second register allocation may perform 4942 spilling which will fail as there is no longer a scratch V#. 4943 4944When the kernel prolog code is being emitted it is known whether the scratch V# 4945described above is actually used. If it is, the prolog code must set it up by 4946copying the Private Segment Buffer to the scratch V# registers and then adding 4947the Private Segment Wavefront Offset to the queue base address in the V#. The 4948result is a V# with a base address pointing to the beginning of the wavefront 4949scratch backing memory. 4950 4951The Private Segment Buffer is always requested, but the Private Segment 4952Wavefront Offset is only requested if it is used (see 4953:ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4954 4955.. _amdgpu-amdhsa-memory-model: 4956 4957Memory Model 4958~~~~~~~~~~~~ 4959 4960This section describes the mapping of the LLVM memory model onto AMDGPU machine 4961code (see :ref:`memmodel`). 4962 4963The AMDGPU backend supports the memory synchronization scopes specified in 4964:ref:`amdgpu-memory-scopes`. 4965 4966The code sequences used to implement the memory model specify the order of 4967instructions that a single thread must execute. The ``s_waitcnt`` and cache 4968management instructions such as ``buffer_wbinvl1_vol`` are defined with respect 4969to other memory instructions executed by the same thread. This allows them to be 4970moved earlier or later which can allow them to be combined with other instances 4971of the same instruction, or hoisted/sunk out of loops to improve performance. 4972Only the instructions related to the memory model are given; additional 4973``s_waitcnt`` instructions are required to ensure registers are defined before 4974being used. These may be able to be combined with the memory model ``s_waitcnt`` 4975instructions as described above. 4976 4977The AMDGPU backend supports the following memory models: 4978 4979 HSA Memory Model [HSA]_ 4980 The HSA memory model uses a single happens-before relation for all address 4981 spaces (see :ref:`amdgpu-address-spaces`). 4982 OpenCL Memory Model [OpenCL]_ 4983 The OpenCL memory model which has separate happens-before relations for the 4984 global and local address spaces. Only a fence specifying both global and 4985 local address space, and seq_cst instructions join the relationships. Since 4986 the LLVM ``memfence`` instruction does not allow an address space to be 4987 specified the OpenCL fence has to conservatively assume both local and 4988 global address space was specified. However, optimizations can often be 4989 done to eliminate the additional ``s_waitcnt`` instructions when there are 4990 no intervening memory instructions which access the corresponding address 4991 space. The code sequences in the table indicate what can be omitted for the 4992 OpenCL memory. The target triple environment is used to determine if the 4993 source language is OpenCL (see :ref:`amdgpu-opencl`). 4994 4995``ds/flat_load/store/atomic`` instructions to local memory are termed LDS 4996operations. 4997 4998``buffer/global/flat_load/store/atomic`` instructions to global memory are 4999termed vector memory operations. 5000 5001Private address space uses ``buffer_load/store`` using the scratch V# 5002(GFX6-GFX8), or ``scratch_load/store`` (GFX9-GFX10). Since only a single thread 5003is accessing the memory, atomic memory orderings are not meaningful, and all 5004accesses are treated as non-atomic. 5005 5006Constant address space uses ``buffer/global_load`` instructions (or equivalent 5007scalar memory instructions). Since the constant address space contents do not 5008change during the execution of a kernel dispatch it is not legal to perform 5009stores, and atomic memory orderings are not meaningful, and all accesses are 5010treated as non-atomic. 5011 5012A memory synchronization scope wider than work-group is not meaningful for the 5013group (LDS) address space and is treated as work-group. 5014 5015The memory model does not support the region address space which is treated as 5016non-atomic. 5017 5018Acquire memory ordering is not meaningful on store atomic instructions and is 5019treated as non-atomic. 5020 5021Release memory ordering is not meaningful on load atomic instructions and is 5022treated a non-atomic. 5023 5024Acquire-release memory ordering is not meaningful on load or store atomic 5025instructions and is treated as acquire and release respectively. 5026 5027The memory order also adds the single thread optimization constraints defined in 5028table 5029:ref:`amdgpu-amdhsa-memory-model-single-thread-optimization-constraints-table`. 5030 5031 .. table:: AMDHSA Memory Model Single Thread Optimization Constraints 5032 :name: amdgpu-amdhsa-memory-model-single-thread-optimization-constraints-table 5033 5034 ============ ============================================================== 5035 LLVM Memory Optimization Constraints 5036 Ordering 5037 ============ ============================================================== 5038 unordered *none* 5039 monotonic *none* 5040 acquire - If a load atomic/atomicrmw then no following load/load 5041 atomic/store/store atomic/atomicrmw/fence instruction can be 5042 moved before the acquire. 5043 - If a fence then same as load atomic, plus no preceding 5044 associated fence-paired-atomic can be moved after the fence. 5045 release - If a store atomic/atomicrmw then no preceding load/load 5046 atomic/store/store atomic/atomicrmw/fence instruction can be 5047 moved after the release. 5048 - If a fence then same as store atomic, plus no following 5049 associated fence-paired-atomic can be moved before the 5050 fence. 5051 acq_rel Same constraints as both acquire and release. 5052 seq_cst - If a load atomic then same constraints as acquire, plus no 5053 preceding sequentially consistent load atomic/store 5054 atomic/atomicrmw/fence instruction can be moved after the 5055 seq_cst. 5056 - If a store atomic then the same constraints as release, plus 5057 no following sequentially consistent load atomic/store 5058 atomic/atomicrmw/fence instruction can be moved before the 5059 seq_cst. 5060 - If an atomicrmw/fence then same constraints as acq_rel. 5061 ============ ============================================================== 5062 5063The code sequences used to implement the memory model are defined in the 5064following sections: 5065 5066* :ref:`amdgpu-amdhsa-memory-model-gfx6-gfx9` 5067* :ref:`amdgpu-amdhsa-memory-model-gfx90a` 5068* :ref:`amdgpu-amdhsa-memory-model-gfx940` 5069* :ref:`amdgpu-amdhsa-memory-model-gfx10` 5070 5071.. _amdgpu-amdhsa-memory-model-gfx6-gfx9: 5072 5073Memory Model GFX6-GFX9 5074++++++++++++++++++++++ 5075 5076For GFX6-GFX9: 5077 5078* Each agent has multiple shader arrays (SA). 5079* Each SA has multiple compute units (CU). 5080* Each CU has multiple SIMDs that execute wavefronts. 5081* The wavefronts for a single work-group are executed in the same CU but may be 5082 executed by different SIMDs. 5083* Each CU has a single LDS memory shared by the wavefronts of the work-groups 5084 executing on it. 5085* All LDS operations of a CU are performed as wavefront wide operations in a 5086 global order and involve no caching. Completion is reported to a wavefront in 5087 execution order. 5088* The LDS memory has multiple request queues shared by the SIMDs of a 5089 CU. Therefore, the LDS operations performed by different wavefronts of a 5090 work-group can be reordered relative to each other, which can result in 5091 reordering the visibility of vector memory operations with respect to LDS 5092 operations of other wavefronts in the same work-group. A ``s_waitcnt 5093 lgkmcnt(0)`` is required to ensure synchronization between LDS operations and 5094 vector memory operations between wavefronts of a work-group, but not between 5095 operations performed by the same wavefront. 5096* The vector memory operations are performed as wavefront wide operations and 5097 completion is reported to a wavefront in execution order. The exception is 5098 that for GFX7-GFX9 ``flat_load/store/atomic`` instructions can report out of 5099 vector memory order if they access LDS memory, and out of LDS operation order 5100 if they access global memory. 5101* The vector memory operations access a single vector L1 cache shared by all 5102 SIMDs a CU. Therefore, no special action is required for coherence between the 5103 lanes of a single wavefront, or for coherence between wavefronts in the same 5104 work-group. A ``buffer_wbinvl1_vol`` is required for coherence between 5105 wavefronts executing in different work-groups as they may be executing on 5106 different CUs. 5107* The scalar memory operations access a scalar L1 cache shared by all wavefronts 5108 on a group of CUs. The scalar and vector L1 caches are not coherent. However, 5109 scalar operations are used in a restricted way so do not impact the memory 5110 model. See :ref:`amdgpu-amdhsa-memory-spaces`. 5111* The vector and scalar memory operations use an L2 cache shared by all CUs on 5112 the same agent. 5113* The L2 cache has independent channels to service disjoint ranges of virtual 5114 addresses. 5115* Each CU has a separate request queue per channel. Therefore, the vector and 5116 scalar memory operations performed by wavefronts executing in different 5117 work-groups (which may be executing on different CUs) of an agent can be 5118 reordered relative to each other. A ``s_waitcnt vmcnt(0)`` is required to 5119 ensure synchronization between vector memory operations of different CUs. It 5120 ensures a previous vector memory operation has completed before executing a 5121 subsequent vector memory or LDS operation and so can be used to meet the 5122 requirements of acquire and release. 5123* The L2 cache can be kept coherent with other agents on some targets, or ranges 5124 of virtual addresses can be set up to bypass it to ensure system coherence. 5125 5126Scalar memory operations are only used to access memory that is proven to not 5127change during the execution of the kernel dispatch. This includes constant 5128address space and global address space for program scope ``const`` variables. 5129Therefore, the kernel machine code does not have to maintain the scalar cache to 5130ensure it is coherent with the vector caches. The scalar and vector caches are 5131invalidated between kernel dispatches by CP since constant address space data 5132may change between kernel dispatch executions. See 5133:ref:`amdgpu-amdhsa-memory-spaces`. 5134 5135The one exception is if scalar writes are used to spill SGPR registers. In this 5136case the AMDGPU backend ensures the memory location used to spill is never 5137accessed by vector memory operations at the same time. If scalar writes are used 5138then a ``s_dcache_wb`` is inserted before the ``s_endpgm`` and before a function 5139return since the locations may be used for vector memory instructions by a 5140future wavefront that uses the same scratch area, or a function call that 5141creates a frame at the same address, respectively. There is no need for a 5142``s_dcache_inv`` as all scalar writes are write-before-read in the same thread. 5143 5144For kernarg backing memory: 5145 5146* CP invalidates the L1 cache at the start of each kernel dispatch. 5147* On dGPU the kernarg backing memory is allocated in host memory accessed as 5148 MTYPE UC (uncached) to avoid needing to invalidate the L2 cache. This also 5149 causes it to be treated as non-volatile and so is not invalidated by 5150 ``*_vol``. 5151* On APU the kernarg backing memory it is accessed as MTYPE CC (cache coherent) 5152 and so the L2 cache will be coherent with the CPU and other agents. 5153 5154Scratch backing memory (which is used for the private address space) is accessed 5155with MTYPE NC_NV (non-coherent non-volatile). Since the private address space is 5156only accessed by a single thread, and is always write-before-read, there is 5157never a need to invalidate these entries from the L1 cache. Hence all cache 5158invalidates are done as ``*_vol`` to only invalidate the volatile cache lines. 5159 5160The code sequences used to implement the memory model for GFX6-GFX9 are defined 5161in table :ref:`amdgpu-amdhsa-memory-model-code-sequences-gfx6-gfx9-table`. 5162 5163 .. table:: AMDHSA Memory Model Code Sequences GFX6-GFX9 5164 :name: amdgpu-amdhsa-memory-model-code-sequences-gfx6-gfx9-table 5165 5166 ============ ============ ============== ========== ================================ 5167 LLVM Instr LLVM Memory LLVM Memory AMDGPU AMDGPU Machine Code 5168 Ordering Sync Scope Address GFX6-GFX9 5169 Space 5170 ============ ============ ============== ========== ================================ 5171 **Non-Atomic** 5172 ------------------------------------------------------------------------------------ 5173 load *none* *none* - global - !volatile & !nontemporal 5174 - generic 5175 - private 1. buffer/global/flat_load 5176 - constant 5177 - !volatile & nontemporal 5178 5179 1. buffer/global/flat_load 5180 glc=1 slc=1 5181 5182 - volatile 5183 5184 1. buffer/global/flat_load 5185 glc=1 5186 2. s_waitcnt vmcnt(0) 5187 5188 - Must happen before 5189 any following volatile 5190 global/generic 5191 load/store. 5192 - Ensures that 5193 volatile 5194 operations to 5195 different 5196 addresses will not 5197 be reordered by 5198 hardware. 5199 5200 load *none* *none* - local 1. ds_load 5201 store *none* *none* - global - !volatile & !nontemporal 5202 - generic 5203 - private 1. buffer/global/flat_store 5204 - constant 5205 - !volatile & nontemporal 5206 5207 1. buffer/global/flat_store 5208 glc=1 slc=1 5209 5210 - volatile 5211 5212 1. buffer/global/flat_store 5213 2. s_waitcnt vmcnt(0) 5214 5215 - Must happen before 5216 any following volatile 5217 global/generic 5218 load/store. 5219 - Ensures that 5220 volatile 5221 operations to 5222 different 5223 addresses will not 5224 be reordered by 5225 hardware. 5226 5227 store *none* *none* - local 1. ds_store 5228 **Unordered Atomic** 5229 ------------------------------------------------------------------------------------ 5230 load atomic unordered *any* *any* *Same as non-atomic*. 5231 store atomic unordered *any* *any* *Same as non-atomic*. 5232 atomicrmw unordered *any* *any* *Same as monotonic atomic*. 5233 **Monotonic Atomic** 5234 ------------------------------------------------------------------------------------ 5235 load atomic monotonic - singlethread - global 1. buffer/global/ds/flat_load 5236 - wavefront - local 5237 - workgroup - generic 5238 load atomic monotonic - agent - global 1. buffer/global/flat_load 5239 - system - generic glc=1 5240 store atomic monotonic - singlethread - global 1. buffer/global/flat_store 5241 - wavefront - generic 5242 - workgroup 5243 - agent 5244 - system 5245 store atomic monotonic - singlethread - local 1. ds_store 5246 - wavefront 5247 - workgroup 5248 atomicrmw monotonic - singlethread - global 1. buffer/global/flat_atomic 5249 - wavefront - generic 5250 - workgroup 5251 - agent 5252 - system 5253 atomicrmw monotonic - singlethread - local 1. ds_atomic 5254 - wavefront 5255 - workgroup 5256 **Acquire Atomic** 5257 ------------------------------------------------------------------------------------ 5258 load atomic acquire - singlethread - global 1. buffer/global/ds/flat_load 5259 - wavefront - local 5260 - generic 5261 load atomic acquire - workgroup - global 1. buffer/global_load 5262 load atomic acquire - workgroup - local 1. ds/flat_load 5263 - generic 2. s_waitcnt lgkmcnt(0) 5264 5265 - If OpenCL, omit. 5266 - Must happen before 5267 any following 5268 global/generic 5269 load/load 5270 atomic/store/store 5271 atomic/atomicrmw. 5272 - Ensures any 5273 following global 5274 data read is no 5275 older than a local load 5276 atomic value being 5277 acquired. 5278 5279 load atomic acquire - agent - global 1. buffer/global_load 5280 - system glc=1 5281 2. s_waitcnt vmcnt(0) 5282 5283 - Must happen before 5284 following 5285 buffer_wbinvl1_vol. 5286 - Ensures the load 5287 has completed 5288 before invalidating 5289 the cache. 5290 5291 3. buffer_wbinvl1_vol 5292 5293 - Must happen before 5294 any following 5295 global/generic 5296 load/load 5297 atomic/atomicrmw. 5298 - Ensures that 5299 following 5300 loads will not see 5301 stale global data. 5302 5303 load atomic acquire - agent - generic 1. flat_load glc=1 5304 - system 2. s_waitcnt vmcnt(0) & 5305 lgkmcnt(0) 5306 5307 - If OpenCL omit 5308 lgkmcnt(0). 5309 - Must happen before 5310 following 5311 buffer_wbinvl1_vol. 5312 - Ensures the flat_load 5313 has completed 5314 before invalidating 5315 the cache. 5316 5317 3. buffer_wbinvl1_vol 5318 5319 - Must happen before 5320 any following 5321 global/generic 5322 load/load 5323 atomic/atomicrmw. 5324 - Ensures that 5325 following loads 5326 will not see stale 5327 global data. 5328 5329 atomicrmw acquire - singlethread - global 1. buffer/global/ds/flat_atomic 5330 - wavefront - local 5331 - generic 5332 atomicrmw acquire - workgroup - global 1. buffer/global_atomic 5333 atomicrmw acquire - workgroup - local 1. ds/flat_atomic 5334 - generic 2. s_waitcnt lgkmcnt(0) 5335 5336 - If OpenCL, omit. 5337 - Must happen before 5338 any following 5339 global/generic 5340 load/load 5341 atomic/store/store 5342 atomic/atomicrmw. 5343 - Ensures any 5344 following global 5345 data read is no 5346 older than a local 5347 atomicrmw value 5348 being acquired. 5349 5350 atomicrmw acquire - agent - global 1. buffer/global_atomic 5351 - system 2. s_waitcnt vmcnt(0) 5352 5353 - Must happen before 5354 following 5355 buffer_wbinvl1_vol. 5356 - Ensures the 5357 atomicrmw has 5358 completed before 5359 invalidating the 5360 cache. 5361 5362 3. buffer_wbinvl1_vol 5363 5364 - Must happen before 5365 any following 5366 global/generic 5367 load/load 5368 atomic/atomicrmw. 5369 - Ensures that 5370 following loads 5371 will not see stale 5372 global data. 5373 5374 atomicrmw acquire - agent - generic 1. flat_atomic 5375 - system 2. s_waitcnt vmcnt(0) & 5376 lgkmcnt(0) 5377 5378 - If OpenCL, omit 5379 lgkmcnt(0). 5380 - Must happen before 5381 following 5382 buffer_wbinvl1_vol. 5383 - Ensures the 5384 atomicrmw has 5385 completed before 5386 invalidating the 5387 cache. 5388 5389 3. buffer_wbinvl1_vol 5390 5391 - Must happen before 5392 any following 5393 global/generic 5394 load/load 5395 atomic/atomicrmw. 5396 - Ensures that 5397 following loads 5398 will not see stale 5399 global data. 5400 5401 fence acquire - singlethread *none* *none* 5402 - wavefront 5403 fence acquire - workgroup *none* 1. s_waitcnt lgkmcnt(0) 5404 5405 - If OpenCL and 5406 address space is 5407 not generic, omit. 5408 - However, since LLVM 5409 currently has no 5410 address space on 5411 the fence need to 5412 conservatively 5413 always generate. If 5414 fence had an 5415 address space then 5416 set to address 5417 space of OpenCL 5418 fence flag, or to 5419 generic if both 5420 local and global 5421 flags are 5422 specified. 5423 - Must happen after 5424 any preceding 5425 local/generic load 5426 atomic/atomicrmw 5427 with an equal or 5428 wider sync scope 5429 and memory ordering 5430 stronger than 5431 unordered (this is 5432 termed the 5433 fence-paired-atomic). 5434 - Must happen before 5435 any following 5436 global/generic 5437 load/load 5438 atomic/store/store 5439 atomic/atomicrmw. 5440 - Ensures any 5441 following global 5442 data read is no 5443 older than the 5444 value read by the 5445 fence-paired-atomic. 5446 5447 fence acquire - agent *none* 1. s_waitcnt lgkmcnt(0) & 5448 - system vmcnt(0) 5449 5450 - If OpenCL and 5451 address space is 5452 not generic, omit 5453 lgkmcnt(0). 5454 - However, since LLVM 5455 currently has no 5456 address space on 5457 the fence need to 5458 conservatively 5459 always generate 5460 (see comment for 5461 previous fence). 5462 - Could be split into 5463 separate s_waitcnt 5464 vmcnt(0) and 5465 s_waitcnt 5466 lgkmcnt(0) to allow 5467 them to be 5468 independently moved 5469 according to the 5470 following rules. 5471 - s_waitcnt vmcnt(0) 5472 must happen after 5473 any preceding 5474 global/generic load 5475 atomic/atomicrmw 5476 with an equal or 5477 wider sync scope 5478 and memory ordering 5479 stronger than 5480 unordered (this is 5481 termed the 5482 fence-paired-atomic). 5483 - s_waitcnt lgkmcnt(0) 5484 must happen after 5485 any preceding 5486 local/generic load 5487 atomic/atomicrmw 5488 with an equal or 5489 wider sync scope 5490 and memory ordering 5491 stronger than 5492 unordered (this is 5493 termed the 5494 fence-paired-atomic). 5495 - Must happen before 5496 the following 5497 buffer_wbinvl1_vol. 5498 - Ensures that the 5499 fence-paired atomic 5500 has completed 5501 before invalidating 5502 the 5503 cache. Therefore 5504 any following 5505 locations read must 5506 be no older than 5507 the value read by 5508 the 5509 fence-paired-atomic. 5510 5511 2. buffer_wbinvl1_vol 5512 5513 - Must happen before any 5514 following global/generic 5515 load/load 5516 atomic/store/store 5517 atomic/atomicrmw. 5518 - Ensures that 5519 following loads 5520 will not see stale 5521 global data. 5522 5523 **Release Atomic** 5524 ------------------------------------------------------------------------------------ 5525 store atomic release - singlethread - global 1. buffer/global/ds/flat_store 5526 - wavefront - local 5527 - generic 5528 store atomic release - workgroup - global 1. s_waitcnt lgkmcnt(0) 5529 - generic 5530 - If OpenCL, omit. 5531 - Must happen after 5532 any preceding 5533 local/generic 5534 load/store/load 5535 atomic/store 5536 atomic/atomicrmw. 5537 - Must happen before 5538 the following 5539 store. 5540 - Ensures that all 5541 memory operations 5542 to local have 5543 completed before 5544 performing the 5545 store that is being 5546 released. 5547 5548 2. buffer/global/flat_store 5549 store atomic release - workgroup - local 1. ds_store 5550 store atomic release - agent - global 1. s_waitcnt lgkmcnt(0) & 5551 - system - generic vmcnt(0) 5552 5553 - If OpenCL and 5554 address space is 5555 not generic, omit 5556 lgkmcnt(0). 5557 - Could be split into 5558 separate s_waitcnt 5559 vmcnt(0) and 5560 s_waitcnt 5561 lgkmcnt(0) to allow 5562 them to be 5563 independently moved 5564 according to the 5565 following rules. 5566 - s_waitcnt vmcnt(0) 5567 must happen after 5568 any preceding 5569 global/generic 5570 load/store/load 5571 atomic/store 5572 atomic/atomicrmw. 5573 - s_waitcnt lgkmcnt(0) 5574 must happen after 5575 any preceding 5576 local/generic 5577 load/store/load 5578 atomic/store 5579 atomic/atomicrmw. 5580 - Must happen before 5581 the following 5582 store. 5583 - Ensures that all 5584 memory operations 5585 to memory have 5586 completed before 5587 performing the 5588 store that is being 5589 released. 5590 5591 2. buffer/global/flat_store 5592 atomicrmw release - singlethread - global 1. buffer/global/ds/flat_atomic 5593 - wavefront - local 5594 - generic 5595 atomicrmw release - workgroup - global 1. s_waitcnt lgkmcnt(0) 5596 - generic 5597 - If OpenCL, omit. 5598 - Must happen after 5599 any preceding 5600 local/generic 5601 load/store/load 5602 atomic/store 5603 atomic/atomicrmw. 5604 - Must happen before 5605 the following 5606 atomicrmw. 5607 - Ensures that all 5608 memory operations 5609 to local have 5610 completed before 5611 performing the 5612 atomicrmw that is 5613 being released. 5614 5615 2. buffer/global/flat_atomic 5616 atomicrmw release - workgroup - local 1. ds_atomic 5617 atomicrmw release - agent - global 1. s_waitcnt lgkmcnt(0) & 5618 - system - generic vmcnt(0) 5619 5620 - If OpenCL, omit 5621 lgkmcnt(0). 5622 - Could be split into 5623 separate s_waitcnt 5624 vmcnt(0) and 5625 s_waitcnt 5626 lgkmcnt(0) to allow 5627 them to be 5628 independently moved 5629 according to the 5630 following rules. 5631 - s_waitcnt vmcnt(0) 5632 must happen after 5633 any preceding 5634 global/generic 5635 load/store/load 5636 atomic/store 5637 atomic/atomicrmw. 5638 - s_waitcnt lgkmcnt(0) 5639 must happen after 5640 any preceding 5641 local/generic 5642 load/store/load 5643 atomic/store 5644 atomic/atomicrmw. 5645 - Must happen before 5646 the following 5647 atomicrmw. 5648 - Ensures that all 5649 memory operations 5650 to global and local 5651 have completed 5652 before performing 5653 the atomicrmw that 5654 is being released. 5655 5656 2. buffer/global/flat_atomic 5657 fence release - singlethread *none* *none* 5658 - wavefront 5659 fence release - workgroup *none* 1. s_waitcnt lgkmcnt(0) 5660 5661 - If OpenCL and 5662 address space is 5663 not generic, omit. 5664 - However, since LLVM 5665 currently has no 5666 address space on 5667 the fence need to 5668 conservatively 5669 always generate. If 5670 fence had an 5671 address space then 5672 set to address 5673 space of OpenCL 5674 fence flag, or to 5675 generic if both 5676 local and global 5677 flags are 5678 specified. 5679 - Must happen after 5680 any preceding 5681 local/generic 5682 load/load 5683 atomic/store/store 5684 atomic/atomicrmw. 5685 - Must happen before 5686 any following store 5687 atomic/atomicrmw 5688 with an equal or 5689 wider sync scope 5690 and memory ordering 5691 stronger than 5692 unordered (this is 5693 termed the 5694 fence-paired-atomic). 5695 - Ensures that all 5696 memory operations 5697 to local have 5698 completed before 5699 performing the 5700 following 5701 fence-paired-atomic. 5702 5703 fence release - agent *none* 1. s_waitcnt lgkmcnt(0) & 5704 - system vmcnt(0) 5705 5706 - If OpenCL and 5707 address space is 5708 not generic, omit 5709 lgkmcnt(0). 5710 - If OpenCL and 5711 address space is 5712 local, omit 5713 vmcnt(0). 5714 - However, since LLVM 5715 currently has no 5716 address space on 5717 the fence need to 5718 conservatively 5719 always generate. If 5720 fence had an 5721 address space then 5722 set to address 5723 space of OpenCL 5724 fence flag, or to 5725 generic if both 5726 local and global 5727 flags are 5728 specified. 5729 - Could be split into 5730 separate s_waitcnt 5731 vmcnt(0) and 5732 s_waitcnt 5733 lgkmcnt(0) to allow 5734 them to be 5735 independently moved 5736 according to the 5737 following rules. 5738 - s_waitcnt vmcnt(0) 5739 must happen after 5740 any preceding 5741 global/generic 5742 load/store/load 5743 atomic/store 5744 atomic/atomicrmw. 5745 - s_waitcnt lgkmcnt(0) 5746 must happen after 5747 any preceding 5748 local/generic 5749 load/store/load 5750 atomic/store 5751 atomic/atomicrmw. 5752 - Must happen before 5753 any following store 5754 atomic/atomicrmw 5755 with an equal or 5756 wider sync scope 5757 and memory ordering 5758 stronger than 5759 unordered (this is 5760 termed the 5761 fence-paired-atomic). 5762 - Ensures that all 5763 memory operations 5764 have 5765 completed before 5766 performing the 5767 following 5768 fence-paired-atomic. 5769 5770 **Acquire-Release Atomic** 5771 ------------------------------------------------------------------------------------ 5772 atomicrmw acq_rel - singlethread - global 1. buffer/global/ds/flat_atomic 5773 - wavefront - local 5774 - generic 5775 atomicrmw acq_rel - workgroup - global 1. s_waitcnt lgkmcnt(0) 5776 5777 - If OpenCL, omit. 5778 - Must happen after 5779 any preceding 5780 local/generic 5781 load/store/load 5782 atomic/store 5783 atomic/atomicrmw. 5784 - Must happen before 5785 the following 5786 atomicrmw. 5787 - Ensures that all 5788 memory operations 5789 to local have 5790 completed before 5791 performing the 5792 atomicrmw that is 5793 being released. 5794 5795 2. buffer/global_atomic 5796 5797 atomicrmw acq_rel - workgroup - local 1. ds_atomic 5798 2. s_waitcnt lgkmcnt(0) 5799 5800 - If OpenCL, omit. 5801 - Must happen before 5802 any following 5803 global/generic 5804 load/load 5805 atomic/store/store 5806 atomic/atomicrmw. 5807 - Ensures any 5808 following global 5809 data read is no 5810 older than the local load 5811 atomic value being 5812 acquired. 5813 5814 atomicrmw acq_rel - workgroup - generic 1. s_waitcnt lgkmcnt(0) 5815 5816 - If OpenCL, omit. 5817 - Must happen after 5818 any preceding 5819 local/generic 5820 load/store/load 5821 atomic/store 5822 atomic/atomicrmw. 5823 - Must happen before 5824 the following 5825 atomicrmw. 5826 - Ensures that all 5827 memory operations 5828 to local have 5829 completed before 5830 performing the 5831 atomicrmw that is 5832 being released. 5833 5834 2. flat_atomic 5835 3. s_waitcnt lgkmcnt(0) 5836 5837 - If OpenCL, omit. 5838 - Must happen before 5839 any following 5840 global/generic 5841 load/load 5842 atomic/store/store 5843 atomic/atomicrmw. 5844 - Ensures any 5845 following global 5846 data read is no 5847 older than a local load 5848 atomic value being 5849 acquired. 5850 5851 atomicrmw acq_rel - agent - global 1. s_waitcnt lgkmcnt(0) & 5852 - system vmcnt(0) 5853 5854 - If OpenCL, omit 5855 lgkmcnt(0). 5856 - Could be split into 5857 separate s_waitcnt 5858 vmcnt(0) and 5859 s_waitcnt 5860 lgkmcnt(0) to allow 5861 them to be 5862 independently moved 5863 according to the 5864 following rules. 5865 - s_waitcnt vmcnt(0) 5866 must happen after 5867 any preceding 5868 global/generic 5869 load/store/load 5870 atomic/store 5871 atomic/atomicrmw. 5872 - s_waitcnt lgkmcnt(0) 5873 must happen after 5874 any preceding 5875 local/generic 5876 load/store/load 5877 atomic/store 5878 atomic/atomicrmw. 5879 - Must happen before 5880 the following 5881 atomicrmw. 5882 - Ensures that all 5883 memory operations 5884 to global have 5885 completed before 5886 performing the 5887 atomicrmw that is 5888 being released. 5889 5890 2. buffer/global_atomic 5891 3. s_waitcnt vmcnt(0) 5892 5893 - Must happen before 5894 following 5895 buffer_wbinvl1_vol. 5896 - Ensures the 5897 atomicrmw has 5898 completed before 5899 invalidating the 5900 cache. 5901 5902 4. buffer_wbinvl1_vol 5903 5904 - Must happen before 5905 any following 5906 global/generic 5907 load/load 5908 atomic/atomicrmw. 5909 - Ensures that 5910 following loads 5911 will not see stale 5912 global data. 5913 5914 atomicrmw acq_rel - agent - generic 1. s_waitcnt lgkmcnt(0) & 5915 - system vmcnt(0) 5916 5917 - If OpenCL, omit 5918 lgkmcnt(0). 5919 - Could be split into 5920 separate s_waitcnt 5921 vmcnt(0) and 5922 s_waitcnt 5923 lgkmcnt(0) to allow 5924 them to be 5925 independently moved 5926 according to the 5927 following rules. 5928 - s_waitcnt vmcnt(0) 5929 must happen after 5930 any preceding 5931 global/generic 5932 load/store/load 5933 atomic/store 5934 atomic/atomicrmw. 5935 - s_waitcnt lgkmcnt(0) 5936 must happen after 5937 any preceding 5938 local/generic 5939 load/store/load 5940 atomic/store 5941 atomic/atomicrmw. 5942 - Must happen before 5943 the following 5944 atomicrmw. 5945 - Ensures that all 5946 memory operations 5947 to global have 5948 completed before 5949 performing the 5950 atomicrmw that is 5951 being released. 5952 5953 2. flat_atomic 5954 3. s_waitcnt vmcnt(0) & 5955 lgkmcnt(0) 5956 5957 - If OpenCL, omit 5958 lgkmcnt(0). 5959 - Must happen before 5960 following 5961 buffer_wbinvl1_vol. 5962 - Ensures the 5963 atomicrmw has 5964 completed before 5965 invalidating the 5966 cache. 5967 5968 4. buffer_wbinvl1_vol 5969 5970 - Must happen before 5971 any following 5972 global/generic 5973 load/load 5974 atomic/atomicrmw. 5975 - Ensures that 5976 following loads 5977 will not see stale 5978 global data. 5979 5980 fence acq_rel - singlethread *none* *none* 5981 - wavefront 5982 fence acq_rel - workgroup *none* 1. s_waitcnt lgkmcnt(0) 5983 5984 - If OpenCL and 5985 address space is 5986 not generic, omit. 5987 - However, 5988 since LLVM 5989 currently has no 5990 address space on 5991 the fence need to 5992 conservatively 5993 always generate 5994 (see comment for 5995 previous fence). 5996 - Must happen after 5997 any preceding 5998 local/generic 5999 load/load 6000 atomic/store/store 6001 atomic/atomicrmw. 6002 - Must happen before 6003 any following 6004 global/generic 6005 load/load 6006 atomic/store/store 6007 atomic/atomicrmw. 6008 - Ensures that all 6009 memory operations 6010 to local have 6011 completed before 6012 performing any 6013 following global 6014 memory operations. 6015 - Ensures that the 6016 preceding 6017 local/generic load 6018 atomic/atomicrmw 6019 with an equal or 6020 wider sync scope 6021 and memory ordering 6022 stronger than 6023 unordered (this is 6024 termed the 6025 acquire-fence-paired-atomic) 6026 has completed 6027 before following 6028 global memory 6029 operations. This 6030 satisfies the 6031 requirements of 6032 acquire. 6033 - Ensures that all 6034 previous memory 6035 operations have 6036 completed before a 6037 following 6038 local/generic store 6039 atomic/atomicrmw 6040 with an equal or 6041 wider sync scope 6042 and memory ordering 6043 stronger than 6044 unordered (this is 6045 termed the 6046 release-fence-paired-atomic). 6047 This satisfies the 6048 requirements of 6049 release. 6050 6051 fence acq_rel - agent *none* 1. s_waitcnt lgkmcnt(0) & 6052 - system vmcnt(0) 6053 6054 - If OpenCL and 6055 address space is 6056 not generic, omit 6057 lgkmcnt(0). 6058 - However, since LLVM 6059 currently has no 6060 address space on 6061 the fence need to 6062 conservatively 6063 always generate 6064 (see comment for 6065 previous fence). 6066 - Could be split into 6067 separate s_waitcnt 6068 vmcnt(0) and 6069 s_waitcnt 6070 lgkmcnt(0) to allow 6071 them to be 6072 independently moved 6073 according to the 6074 following rules. 6075 - s_waitcnt vmcnt(0) 6076 must happen after 6077 any preceding 6078 global/generic 6079 load/store/load 6080 atomic/store 6081 atomic/atomicrmw. 6082 - s_waitcnt lgkmcnt(0) 6083 must happen after 6084 any preceding 6085 local/generic 6086 load/store/load 6087 atomic/store 6088 atomic/atomicrmw. 6089 - Must happen before 6090 the following 6091 buffer_wbinvl1_vol. 6092 - Ensures that the 6093 preceding 6094 global/local/generic 6095 load 6096 atomic/atomicrmw 6097 with an equal or 6098 wider sync scope 6099 and memory ordering 6100 stronger than 6101 unordered (this is 6102 termed the 6103 acquire-fence-paired-atomic) 6104 has completed 6105 before invalidating 6106 the cache. This 6107 satisfies the 6108 requirements of 6109 acquire. 6110 - Ensures that all 6111 previous memory 6112 operations have 6113 completed before a 6114 following 6115 global/local/generic 6116 store 6117 atomic/atomicrmw 6118 with an equal or 6119 wider sync scope 6120 and memory ordering 6121 stronger than 6122 unordered (this is 6123 termed the 6124 release-fence-paired-atomic). 6125 This satisfies the 6126 requirements of 6127 release. 6128 6129 2. buffer_wbinvl1_vol 6130 6131 - Must happen before 6132 any following 6133 global/generic 6134 load/load 6135 atomic/store/store 6136 atomic/atomicrmw. 6137 - Ensures that 6138 following loads 6139 will not see stale 6140 global data. This 6141 satisfies the 6142 requirements of 6143 acquire. 6144 6145 **Sequential Consistent Atomic** 6146 ------------------------------------------------------------------------------------ 6147 load atomic seq_cst - singlethread - global *Same as corresponding 6148 - wavefront - local load atomic acquire, 6149 - generic except must generate 6150 all instructions even 6151 for OpenCL.* 6152 load atomic seq_cst - workgroup - global 1. s_waitcnt lgkmcnt(0) 6153 - generic 6154 6155 - Must 6156 happen after 6157 preceding 6158 local/generic load 6159 atomic/store 6160 atomic/atomicrmw 6161 with memory 6162 ordering of seq_cst 6163 and with equal or 6164 wider sync scope. 6165 (Note that seq_cst 6166 fences have their 6167 own s_waitcnt 6168 lgkmcnt(0) and so do 6169 not need to be 6170 considered.) 6171 - Ensures any 6172 preceding 6173 sequential 6174 consistent local 6175 memory instructions 6176 have completed 6177 before executing 6178 this sequentially 6179 consistent 6180 instruction. This 6181 prevents reordering 6182 a seq_cst store 6183 followed by a 6184 seq_cst load. (Note 6185 that seq_cst is 6186 stronger than 6187 acquire/release as 6188 the reordering of 6189 load acquire 6190 followed by a store 6191 release is 6192 prevented by the 6193 s_waitcnt of 6194 the release, but 6195 there is nothing 6196 preventing a store 6197 release followed by 6198 load acquire from 6199 completing out of 6200 order. The s_waitcnt 6201 could be placed after 6202 seq_store or before 6203 the seq_load. We 6204 choose the load to 6205 make the s_waitcnt be 6206 as late as possible 6207 so that the store 6208 may have already 6209 completed.) 6210 6211 2. *Following 6212 instructions same as 6213 corresponding load 6214 atomic acquire, 6215 except must generate 6216 all instructions even 6217 for OpenCL.* 6218 load atomic seq_cst - workgroup - local *Same as corresponding 6219 load atomic acquire, 6220 except must generate 6221 all instructions even 6222 for OpenCL.* 6223 6224 load atomic seq_cst - agent - global 1. s_waitcnt lgkmcnt(0) & 6225 - system - generic vmcnt(0) 6226 6227 - Could be split into 6228 separate s_waitcnt 6229 vmcnt(0) 6230 and s_waitcnt 6231 lgkmcnt(0) to allow 6232 them to be 6233 independently moved 6234 according to the 6235 following rules. 6236 - s_waitcnt lgkmcnt(0) 6237 must happen after 6238 preceding 6239 global/generic load 6240 atomic/store 6241 atomic/atomicrmw 6242 with memory 6243 ordering of seq_cst 6244 and with equal or 6245 wider sync scope. 6246 (Note that seq_cst 6247 fences have their 6248 own s_waitcnt 6249 lgkmcnt(0) and so do 6250 not need to be 6251 considered.) 6252 - s_waitcnt vmcnt(0) 6253 must happen after 6254 preceding 6255 global/generic load 6256 atomic/store 6257 atomic/atomicrmw 6258 with memory 6259 ordering of seq_cst 6260 and with equal or 6261 wider sync scope. 6262 (Note that seq_cst 6263 fences have their 6264 own s_waitcnt 6265 vmcnt(0) and so do 6266 not need to be 6267 considered.) 6268 - Ensures any 6269 preceding 6270 sequential 6271 consistent global 6272 memory instructions 6273 have completed 6274 before executing 6275 this sequentially 6276 consistent 6277 instruction. This 6278 prevents reordering 6279 a seq_cst store 6280 followed by a 6281 seq_cst load. (Note 6282 that seq_cst is 6283 stronger than 6284 acquire/release as 6285 the reordering of 6286 load acquire 6287 followed by a store 6288 release is 6289 prevented by the 6290 s_waitcnt of 6291 the release, but 6292 there is nothing 6293 preventing a store 6294 release followed by 6295 load acquire from 6296 completing out of 6297 order. The s_waitcnt 6298 could be placed after 6299 seq_store or before 6300 the seq_load. We 6301 choose the load to 6302 make the s_waitcnt be 6303 as late as possible 6304 so that the store 6305 may have already 6306 completed.) 6307 6308 2. *Following 6309 instructions same as 6310 corresponding load 6311 atomic acquire, 6312 except must generate 6313 all instructions even 6314 for OpenCL.* 6315 store atomic seq_cst - singlethread - global *Same as corresponding 6316 - wavefront - local store atomic release, 6317 - workgroup - generic except must generate 6318 - agent all instructions even 6319 - system for OpenCL.* 6320 atomicrmw seq_cst - singlethread - global *Same as corresponding 6321 - wavefront - local atomicrmw acq_rel, 6322 - workgroup - generic except must generate 6323 - agent all instructions even 6324 - system for OpenCL.* 6325 fence seq_cst - singlethread *none* *Same as corresponding 6326 - wavefront fence acq_rel, 6327 - workgroup except must generate 6328 - agent all instructions even 6329 - system for OpenCL.* 6330 ============ ============ ============== ========== ================================ 6331 6332.. _amdgpu-amdhsa-memory-model-gfx90a: 6333 6334Memory Model GFX90A 6335+++++++++++++++++++ 6336 6337For GFX90A: 6338 6339* Each agent has multiple shader arrays (SA). 6340* Each SA has multiple compute units (CU). 6341* Each CU has multiple SIMDs that execute wavefronts. 6342* The wavefronts for a single work-group are executed in the same CU but may be 6343 executed by different SIMDs. The exception is when in tgsplit execution mode 6344 when the wavefronts may be executed by different SIMDs in different CUs. 6345* Each CU has a single LDS memory shared by the wavefronts of the work-groups 6346 executing on it. The exception is when in tgsplit execution mode when no LDS 6347 is allocated as wavefronts of the same work-group can be in different CUs. 6348* All LDS operations of a CU are performed as wavefront wide operations in a 6349 global order and involve no caching. Completion is reported to a wavefront in 6350 execution order. 6351* The LDS memory has multiple request queues shared by the SIMDs of a 6352 CU. Therefore, the LDS operations performed by different wavefronts of a 6353 work-group can be reordered relative to each other, which can result in 6354 reordering the visibility of vector memory operations with respect to LDS 6355 operations of other wavefronts in the same work-group. A ``s_waitcnt 6356 lgkmcnt(0)`` is required to ensure synchronization between LDS operations and 6357 vector memory operations between wavefronts of a work-group, but not between 6358 operations performed by the same wavefront. 6359* The vector memory operations are performed as wavefront wide operations and 6360 completion is reported to a wavefront in execution order. The exception is 6361 that ``flat_load/store/atomic`` instructions can report out of vector memory 6362 order if they access LDS memory, and out of LDS operation order if they access 6363 global memory. 6364* The vector memory operations access a single vector L1 cache shared by all 6365 SIMDs a CU. Therefore: 6366 6367 * No special action is required for coherence between the lanes of a single 6368 wavefront. 6369 6370 * No special action is required for coherence between wavefronts in the same 6371 work-group since they execute on the same CU. The exception is when in 6372 tgsplit execution mode as wavefronts of the same work-group can be in 6373 different CUs and so a ``buffer_wbinvl1_vol`` is required as described in 6374 the following item. 6375 6376 * A ``buffer_wbinvl1_vol`` is required for coherence between wavefronts 6377 executing in different work-groups as they may be executing on different 6378 CUs. 6379 6380* The scalar memory operations access a scalar L1 cache shared by all wavefronts 6381 on a group of CUs. The scalar and vector L1 caches are not coherent. However, 6382 scalar operations are used in a restricted way so do not impact the memory 6383 model. See :ref:`amdgpu-amdhsa-memory-spaces`. 6384* The vector and scalar memory operations use an L2 cache shared by all CUs on 6385 the same agent. 6386 6387 * The L2 cache has independent channels to service disjoint ranges of virtual 6388 addresses. 6389 * Each CU has a separate request queue per channel. Therefore, the vector and 6390 scalar memory operations performed by wavefronts executing in different 6391 work-groups (which may be executing on different CUs), or the same 6392 work-group if executing in tgsplit mode, of an agent can be reordered 6393 relative to each other. A ``s_waitcnt vmcnt(0)`` is required to ensure 6394 synchronization between vector memory operations of different CUs. It 6395 ensures a previous vector memory operation has completed before executing a 6396 subsequent vector memory or LDS operation and so can be used to meet the 6397 requirements of acquire and release. 6398 * The L2 cache of one agent can be kept coherent with other agents by: 6399 using the MTYPE RW (read-write) or MTYPE CC (cache-coherent) with the PTE 6400 C-bit for memory local to the L2; and using the MTYPE NC (non-coherent) with 6401 the PTE C-bit set or MTYPE UC (uncached) for memory not local to the L2. 6402 6403 * Any local memory cache lines will be automatically invalidated by writes 6404 from CUs associated with other L2 caches, or writes from the CPU, due to 6405 the cache probe caused by coherent requests. Coherent requests are caused 6406 by GPU accesses to pages with the PTE C-bit set, by CPU accesses over 6407 XGMI, and by PCIe requests that are configured to be coherent requests. 6408 * XGMI accesses from the CPU to local memory may be cached on the CPU. 6409 Subsequent access from the GPU will automatically invalidate or writeback 6410 the CPU cache due to the L2 probe filter and and the PTE C-bit being set. 6411 * Since all work-groups on the same agent share the same L2, no L2 6412 invalidation or writeback is required for coherence. 6413 * To ensure coherence of local and remote memory writes of work-groups in 6414 different agents a ``buffer_wbl2`` is required. It will writeback dirty L2 6415 cache lines of MTYPE RW (used for local coarse grain memory) and MTYPE NC 6416 ()used for remote coarse grain memory). Note that MTYPE CC (used for local 6417 fine grain memory) causes write through to DRAM, and MTYPE UC (used for 6418 remote fine grain memory) bypasses the L2, so both will never result in 6419 dirty L2 cache lines. 6420 * To ensure coherence of local and remote memory reads of work-groups in 6421 different agents a ``buffer_invl2`` is required. It will invalidate L2 6422 cache lines with MTYPE NC (used for remote coarse grain memory). Note that 6423 MTYPE CC (used for local fine grain memory) and MTYPE RW (used for local 6424 coarse memory) cause local reads to be invalidated by remote writes with 6425 with the PTE C-bit so these cache lines are not invalidated. Note that 6426 MTYPE UC (used for remote fine grain memory) bypasses the L2, so will 6427 never result in L2 cache lines that need to be invalidated. 6428 6429 * PCIe access from the GPU to the CPU memory is kept coherent by using the 6430 MTYPE UC (uncached) which bypasses the L2. 6431 6432Scalar memory operations are only used to access memory that is proven to not 6433change during the execution of the kernel dispatch. This includes constant 6434address space and global address space for program scope ``const`` variables. 6435Therefore, the kernel machine code does not have to maintain the scalar cache to 6436ensure it is coherent with the vector caches. The scalar and vector caches are 6437invalidated between kernel dispatches by CP since constant address space data 6438may change between kernel dispatch executions. See 6439:ref:`amdgpu-amdhsa-memory-spaces`. 6440 6441The one exception is if scalar writes are used to spill SGPR registers. In this 6442case the AMDGPU backend ensures the memory location used to spill is never 6443accessed by vector memory operations at the same time. If scalar writes are used 6444then a ``s_dcache_wb`` is inserted before the ``s_endpgm`` and before a function 6445return since the locations may be used for vector memory instructions by a 6446future wavefront that uses the same scratch area, or a function call that 6447creates a frame at the same address, respectively. There is no need for a 6448``s_dcache_inv`` as all scalar writes are write-before-read in the same thread. 6449 6450For kernarg backing memory: 6451 6452* CP invalidates the L1 cache at the start of each kernel dispatch. 6453* On dGPU over XGMI or PCIe the kernarg backing memory is allocated in host 6454 memory accessed as MTYPE UC (uncached) to avoid needing to invalidate the L2 6455 cache. This also causes it to be treated as non-volatile and so is not 6456 invalidated by ``*_vol``. 6457* On APU the kernarg backing memory is accessed as MTYPE CC (cache coherent) and 6458 so the L2 cache will be coherent with the CPU and other agents. 6459 6460Scratch backing memory (which is used for the private address space) is accessed 6461with MTYPE NC_NV (non-coherent non-volatile). Since the private address space is 6462only accessed by a single thread, and is always write-before-read, there is 6463never a need to invalidate these entries from the L1 cache. Hence all cache 6464invalidates are done as ``*_vol`` to only invalidate the volatile cache lines. 6465 6466The code sequences used to implement the memory model for GFX90A are defined 6467in table :ref:`amdgpu-amdhsa-memory-model-code-sequences-gfx90a-table`. 6468 6469 .. table:: AMDHSA Memory Model Code Sequences GFX90A 6470 :name: amdgpu-amdhsa-memory-model-code-sequences-gfx90a-table 6471 6472 ============ ============ ============== ========== ================================ 6473 LLVM Instr LLVM Memory LLVM Memory AMDGPU AMDGPU Machine Code 6474 Ordering Sync Scope Address GFX90A 6475 Space 6476 ============ ============ ============== ========== ================================ 6477 **Non-Atomic** 6478 ------------------------------------------------------------------------------------ 6479 load *none* *none* - global - !volatile & !nontemporal 6480 - generic 6481 - private 1. buffer/global/flat_load 6482 - constant 6483 - !volatile & nontemporal 6484 6485 1. buffer/global/flat_load 6486 glc=1 slc=1 6487 6488 - volatile 6489 6490 1. buffer/global/flat_load 6491 glc=1 6492 2. s_waitcnt vmcnt(0) 6493 6494 - Must happen before 6495 any following volatile 6496 global/generic 6497 load/store. 6498 - Ensures that 6499 volatile 6500 operations to 6501 different 6502 addresses will not 6503 be reordered by 6504 hardware. 6505 6506 load *none* *none* - local 1. ds_load 6507 store *none* *none* - global - !volatile & !nontemporal 6508 - generic 6509 - private 1. buffer/global/flat_store 6510 - constant 6511 - !volatile & nontemporal 6512 6513 1. buffer/global/flat_store 6514 glc=1 slc=1 6515 6516 - volatile 6517 6518 1. buffer/global/flat_store 6519 2. s_waitcnt vmcnt(0) 6520 6521 - Must happen before 6522 any following volatile 6523 global/generic 6524 load/store. 6525 - Ensures that 6526 volatile 6527 operations to 6528 different 6529 addresses will not 6530 be reordered by 6531 hardware. 6532 6533 store *none* *none* - local 1. ds_store 6534 **Unordered Atomic** 6535 ------------------------------------------------------------------------------------ 6536 load atomic unordered *any* *any* *Same as non-atomic*. 6537 store atomic unordered *any* *any* *Same as non-atomic*. 6538 atomicrmw unordered *any* *any* *Same as monotonic atomic*. 6539 **Monotonic Atomic** 6540 ------------------------------------------------------------------------------------ 6541 load atomic monotonic - singlethread - global 1. buffer/global/flat_load 6542 - wavefront - generic 6543 load atomic monotonic - workgroup - global 1. buffer/global/flat_load 6544 - generic glc=1 6545 6546 - If not TgSplit execution 6547 mode, omit glc=1. 6548 6549 load atomic monotonic - singlethread - local *If TgSplit execution mode, 6550 - wavefront local address space cannot 6551 - workgroup be used.* 6552 6553 1. ds_load 6554 load atomic monotonic - agent - global 1. buffer/global/flat_load 6555 - generic glc=1 6556 load atomic monotonic - system - global 1. buffer/global/flat_load 6557 - generic glc=1 6558 store atomic monotonic - singlethread - global 1. buffer/global/flat_store 6559 - wavefront - generic 6560 - workgroup 6561 - agent 6562 store atomic monotonic - system - global 1. buffer/global/flat_store 6563 - generic 6564 store atomic monotonic - singlethread - local *If TgSplit execution mode, 6565 - wavefront local address space cannot 6566 - workgroup be used.* 6567 6568 1. ds_store 6569 atomicrmw monotonic - singlethread - global 1. buffer/global/flat_atomic 6570 - wavefront - generic 6571 - workgroup 6572 - agent 6573 atomicrmw monotonic - system - global 1. buffer/global/flat_atomic 6574 - generic 6575 atomicrmw monotonic - singlethread - local *If TgSplit execution mode, 6576 - wavefront local address space cannot 6577 - workgroup be used.* 6578 6579 1. ds_atomic 6580 **Acquire Atomic** 6581 ------------------------------------------------------------------------------------ 6582 load atomic acquire - singlethread - global 1. buffer/global/ds/flat_load 6583 - wavefront - local 6584 - generic 6585 load atomic acquire - workgroup - global 1. buffer/global_load glc=1 6586 6587 - If not TgSplit execution 6588 mode, omit glc=1. 6589 6590 2. s_waitcnt vmcnt(0) 6591 6592 - If not TgSplit execution 6593 mode, omit. 6594 - Must happen before the 6595 following buffer_wbinvl1_vol. 6596 6597 3. buffer_wbinvl1_vol 6598 6599 - If not TgSplit execution 6600 mode, omit. 6601 - Must happen before 6602 any following 6603 global/generic 6604 load/load 6605 atomic/store/store 6606 atomic/atomicrmw. 6607 - Ensures that 6608 following 6609 loads will not see 6610 stale data. 6611 6612 load atomic acquire - workgroup - local *If TgSplit execution mode, 6613 local address space cannot 6614 be used.* 6615 6616 1. ds_load 6617 2. s_waitcnt lgkmcnt(0) 6618 6619 - If OpenCL, omit. 6620 - Must happen before 6621 any following 6622 global/generic 6623 load/load 6624 atomic/store/store 6625 atomic/atomicrmw. 6626 - Ensures any 6627 following global 6628 data read is no 6629 older than the local load 6630 atomic value being 6631 acquired. 6632 6633 load atomic acquire - workgroup - generic 1. flat_load glc=1 6634 6635 - If not TgSplit execution 6636 mode, omit glc=1. 6637 6638 2. s_waitcnt lgkm/vmcnt(0) 6639 6640 - Use lgkmcnt(0) if not 6641 TgSplit execution mode 6642 and vmcnt(0) if TgSplit 6643 execution mode. 6644 - If OpenCL, omit lgkmcnt(0). 6645 - Must happen before 6646 the following 6647 buffer_wbinvl1_vol and any 6648 following global/generic 6649 load/load 6650 atomic/store/store 6651 atomic/atomicrmw. 6652 - Ensures any 6653 following global 6654 data read is no 6655 older than a local load 6656 atomic value being 6657 acquired. 6658 6659 3. buffer_wbinvl1_vol 6660 6661 - If not TgSplit execution 6662 mode, omit. 6663 - Ensures that 6664 following 6665 loads will not see 6666 stale data. 6667 6668 load atomic acquire - agent - global 1. buffer/global_load 6669 glc=1 6670 2. s_waitcnt vmcnt(0) 6671 6672 - Must happen before 6673 following 6674 buffer_wbinvl1_vol. 6675 - Ensures the load 6676 has completed 6677 before invalidating 6678 the cache. 6679 6680 3. buffer_wbinvl1_vol 6681 6682 - Must happen before 6683 any following 6684 global/generic 6685 load/load 6686 atomic/atomicrmw. 6687 - Ensures that 6688 following 6689 loads will not see 6690 stale global data. 6691 6692 load atomic acquire - system - global 1. buffer/global/flat_load 6693 glc=1 6694 2. s_waitcnt vmcnt(0) 6695 6696 - Must happen before 6697 following buffer_invl2 and 6698 buffer_wbinvl1_vol. 6699 - Ensures the load 6700 has completed 6701 before invalidating 6702 the cache. 6703 6704 3. buffer_invl2; 6705 buffer_wbinvl1_vol 6706 6707 - Must happen before 6708 any following 6709 global/generic 6710 load/load 6711 atomic/atomicrmw. 6712 - Ensures that 6713 following 6714 loads will not see 6715 stale L1 global data, 6716 nor see stale L2 MTYPE 6717 NC global data. 6718 MTYPE RW and CC memory will 6719 never be stale in L2 due to 6720 the memory probes. 6721 6722 load atomic acquire - agent - generic 1. flat_load glc=1 6723 2. s_waitcnt vmcnt(0) & 6724 lgkmcnt(0) 6725 6726 - If TgSplit execution mode, 6727 omit lgkmcnt(0). 6728 - If OpenCL omit 6729 lgkmcnt(0). 6730 - Must happen before 6731 following 6732 buffer_wbinvl1_vol. 6733 - Ensures the flat_load 6734 has completed 6735 before invalidating 6736 the cache. 6737 6738 3. buffer_wbinvl1_vol 6739 6740 - Must happen before 6741 any following 6742 global/generic 6743 load/load 6744 atomic/atomicrmw. 6745 - Ensures that 6746 following loads 6747 will not see stale 6748 global data. 6749 6750 load atomic acquire - system - generic 1. flat_load glc=1 6751 2. s_waitcnt vmcnt(0) & 6752 lgkmcnt(0) 6753 6754 - If TgSplit execution mode, 6755 omit lgkmcnt(0). 6756 - If OpenCL omit 6757 lgkmcnt(0). 6758 - Must happen before 6759 following 6760 buffer_invl2 and 6761 buffer_wbinvl1_vol. 6762 - Ensures the flat_load 6763 has completed 6764 before invalidating 6765 the caches. 6766 6767 3. buffer_invl2; 6768 buffer_wbinvl1_vol 6769 6770 - Must happen before 6771 any following 6772 global/generic 6773 load/load 6774 atomic/atomicrmw. 6775 - Ensures that 6776 following 6777 loads will not see 6778 stale L1 global data, 6779 nor see stale L2 MTYPE 6780 NC global data. 6781 MTYPE RW and CC memory will 6782 never be stale in L2 due to 6783 the memory probes. 6784 6785 atomicrmw acquire - singlethread - global 1. buffer/global/flat_atomic 6786 - wavefront - generic 6787 atomicrmw acquire - singlethread - local *If TgSplit execution mode, 6788 - wavefront local address space cannot 6789 be used.* 6790 6791 1. ds_atomic 6792 atomicrmw acquire - workgroup - global 1. buffer/global_atomic 6793 2. s_waitcnt vmcnt(0) 6794 6795 - If not TgSplit execution 6796 mode, omit. 6797 - Must happen before the 6798 following buffer_wbinvl1_vol. 6799 - Ensures the atomicrmw 6800 has completed 6801 before invalidating 6802 the cache. 6803 6804 3. buffer_wbinvl1_vol 6805 6806 - If not TgSplit execution 6807 mode, omit. 6808 - Must happen before 6809 any following 6810 global/generic 6811 load/load 6812 atomic/atomicrmw. 6813 - Ensures that 6814 following loads 6815 will not see stale 6816 global data. 6817 6818 atomicrmw acquire - workgroup - local *If TgSplit execution mode, 6819 local address space cannot 6820 be used.* 6821 6822 1. ds_atomic 6823 2. s_waitcnt lgkmcnt(0) 6824 6825 - If OpenCL, omit. 6826 - Must happen before 6827 any following 6828 global/generic 6829 load/load 6830 atomic/store/store 6831 atomic/atomicrmw. 6832 - Ensures any 6833 following global 6834 data read is no 6835 older than the local 6836 atomicrmw value 6837 being acquired. 6838 6839 atomicrmw acquire - workgroup - generic 1. flat_atomic 6840 2. s_waitcnt lgkm/vmcnt(0) 6841 6842 - Use lgkmcnt(0) if not 6843 TgSplit execution mode 6844 and vmcnt(0) if TgSplit 6845 execution mode. 6846 - If OpenCL, omit lgkmcnt(0). 6847 - Must happen before 6848 the following 6849 buffer_wbinvl1_vol and 6850 any following 6851 global/generic 6852 load/load 6853 atomic/store/store 6854 atomic/atomicrmw. 6855 - Ensures any 6856 following global 6857 data read is no 6858 older than a local 6859 atomicrmw value 6860 being acquired. 6861 6862 3. buffer_wbinvl1_vol 6863 6864 - If not TgSplit execution 6865 mode, omit. 6866 - Ensures that 6867 following 6868 loads will not see 6869 stale data. 6870 6871 atomicrmw acquire - agent - global 1. buffer/global_atomic 6872 2. s_waitcnt vmcnt(0) 6873 6874 - Must happen before 6875 following 6876 buffer_wbinvl1_vol. 6877 - Ensures the 6878 atomicrmw has 6879 completed before 6880 invalidating the 6881 cache. 6882 6883 3. buffer_wbinvl1_vol 6884 6885 - Must happen before 6886 any following 6887 global/generic 6888 load/load 6889 atomic/atomicrmw. 6890 - Ensures that 6891 following loads 6892 will not see stale 6893 global data. 6894 6895 atomicrmw acquire - system - global 1. buffer/global_atomic 6896 2. s_waitcnt vmcnt(0) 6897 6898 - Must happen before 6899 following buffer_invl2 and 6900 buffer_wbinvl1_vol. 6901 - Ensures the 6902 atomicrmw has 6903 completed before 6904 invalidating the 6905 caches. 6906 6907 3. buffer_invl2; 6908 buffer_wbinvl1_vol 6909 6910 - Must happen before 6911 any following 6912 global/generic 6913 load/load 6914 atomic/atomicrmw. 6915 - Ensures that 6916 following 6917 loads will not see 6918 stale L1 global data, 6919 nor see stale L2 MTYPE 6920 NC global data. 6921 MTYPE RW and CC memory will 6922 never be stale in L2 due to 6923 the memory probes. 6924 6925 atomicrmw acquire - agent - generic 1. flat_atomic 6926 2. s_waitcnt vmcnt(0) & 6927 lgkmcnt(0) 6928 6929 - If TgSplit execution mode, 6930 omit lgkmcnt(0). 6931 - If OpenCL, omit 6932 lgkmcnt(0). 6933 - Must happen before 6934 following 6935 buffer_wbinvl1_vol. 6936 - Ensures the 6937 atomicrmw has 6938 completed before 6939 invalidating the 6940 cache. 6941 6942 3. buffer_wbinvl1_vol 6943 6944 - Must happen before 6945 any following 6946 global/generic 6947 load/load 6948 atomic/atomicrmw. 6949 - Ensures that 6950 following loads 6951 will not see stale 6952 global data. 6953 6954 atomicrmw acquire - system - generic 1. flat_atomic 6955 2. s_waitcnt vmcnt(0) & 6956 lgkmcnt(0) 6957 6958 - If TgSplit execution mode, 6959 omit lgkmcnt(0). 6960 - If OpenCL, omit 6961 lgkmcnt(0). 6962 - Must happen before 6963 following 6964 buffer_invl2 and 6965 buffer_wbinvl1_vol. 6966 - Ensures the 6967 atomicrmw has 6968 completed before 6969 invalidating the 6970 caches. 6971 6972 3. buffer_invl2; 6973 buffer_wbinvl1_vol 6974 6975 - Must happen before 6976 any following 6977 global/generic 6978 load/load 6979 atomic/atomicrmw. 6980 - Ensures that 6981 following 6982 loads will not see 6983 stale L1 global data, 6984 nor see stale L2 MTYPE 6985 NC global data. 6986 MTYPE RW and CC memory will 6987 never be stale in L2 due to 6988 the memory probes. 6989 6990 fence acquire - singlethread *none* *none* 6991 - wavefront 6992 fence acquire - workgroup *none* 1. s_waitcnt lgkm/vmcnt(0) 6993 6994 - Use lgkmcnt(0) if not 6995 TgSplit execution mode 6996 and vmcnt(0) if TgSplit 6997 execution mode. 6998 - If OpenCL and 6999 address space is 7000 not generic, omit 7001 lgkmcnt(0). 7002 - If OpenCL and 7003 address space is 7004 local, omit 7005 vmcnt(0). 7006 - However, since LLVM 7007 currently has no 7008 address space on 7009 the fence need to 7010 conservatively 7011 always generate. If 7012 fence had an 7013 address space then 7014 set to address 7015 space of OpenCL 7016 fence flag, or to 7017 generic if both 7018 local and global 7019 flags are 7020 specified. 7021 - s_waitcnt vmcnt(0) 7022 must happen after 7023 any preceding 7024 global/generic load 7025 atomic/ 7026 atomicrmw 7027 with an equal or 7028 wider sync scope 7029 and memory ordering 7030 stronger than 7031 unordered (this is 7032 termed the 7033 fence-paired-atomic). 7034 - s_waitcnt lgkmcnt(0) 7035 must happen after 7036 any preceding 7037 local/generic load 7038 atomic/atomicrmw 7039 with an equal or 7040 wider sync scope 7041 and memory ordering 7042 stronger than 7043 unordered (this is 7044 termed the 7045 fence-paired-atomic). 7046 - Must happen before 7047 the following 7048 buffer_wbinvl1_vol and 7049 any following 7050 global/generic 7051 load/load 7052 atomic/store/store 7053 atomic/atomicrmw. 7054 - Ensures any 7055 following global 7056 data read is no 7057 older than the 7058 value read by the 7059 fence-paired-atomic. 7060 7061 2. buffer_wbinvl1_vol 7062 7063 - If not TgSplit execution 7064 mode, omit. 7065 - Ensures that 7066 following 7067 loads will not see 7068 stale data. 7069 7070 fence acquire - agent *none* 1. s_waitcnt lgkmcnt(0) & 7071 vmcnt(0) 7072 7073 - If TgSplit execution mode, 7074 omit lgkmcnt(0). 7075 - If OpenCL and 7076 address space is 7077 not generic, omit 7078 lgkmcnt(0). 7079 - However, since LLVM 7080 currently has no 7081 address space on 7082 the fence need to 7083 conservatively 7084 always generate 7085 (see comment for 7086 previous fence). 7087 - Could be split into 7088 separate s_waitcnt 7089 vmcnt(0) and 7090 s_waitcnt 7091 lgkmcnt(0) to allow 7092 them to be 7093 independently moved 7094 according to the 7095 following rules. 7096 - s_waitcnt vmcnt(0) 7097 must happen after 7098 any preceding 7099 global/generic load 7100 atomic/atomicrmw 7101 with an equal or 7102 wider sync scope 7103 and memory ordering 7104 stronger than 7105 unordered (this is 7106 termed the 7107 fence-paired-atomic). 7108 - s_waitcnt lgkmcnt(0) 7109 must happen after 7110 any preceding 7111 local/generic load 7112 atomic/atomicrmw 7113 with an equal or 7114 wider sync scope 7115 and memory ordering 7116 stronger than 7117 unordered (this is 7118 termed the 7119 fence-paired-atomic). 7120 - Must happen before 7121 the following 7122 buffer_wbinvl1_vol. 7123 - Ensures that the 7124 fence-paired atomic 7125 has completed 7126 before invalidating 7127 the 7128 cache. Therefore 7129 any following 7130 locations read must 7131 be no older than 7132 the value read by 7133 the 7134 fence-paired-atomic. 7135 7136 2. buffer_wbinvl1_vol 7137 7138 - Must happen before any 7139 following global/generic 7140 load/load 7141 atomic/store/store 7142 atomic/atomicrmw. 7143 - Ensures that 7144 following loads 7145 will not see stale 7146 global data. 7147 7148 fence acquire - system *none* 1. s_waitcnt lgkmcnt(0) & 7149 vmcnt(0) 7150 7151 - If TgSplit execution mode, 7152 omit lgkmcnt(0). 7153 - If OpenCL and 7154 address space is 7155 not generic, omit 7156 lgkmcnt(0). 7157 - However, since LLVM 7158 currently has no 7159 address space on 7160 the fence need to 7161 conservatively 7162 always generate 7163 (see comment for 7164 previous fence). 7165 - Could be split into 7166 separate s_waitcnt 7167 vmcnt(0) and 7168 s_waitcnt 7169 lgkmcnt(0) to allow 7170 them to be 7171 independently moved 7172 according to the 7173 following rules. 7174 - s_waitcnt vmcnt(0) 7175 must happen after 7176 any preceding 7177 global/generic load 7178 atomic/atomicrmw 7179 with an equal or 7180 wider sync scope 7181 and memory ordering 7182 stronger than 7183 unordered (this is 7184 termed the 7185 fence-paired-atomic). 7186 - s_waitcnt lgkmcnt(0) 7187 must happen after 7188 any preceding 7189 local/generic load 7190 atomic/atomicrmw 7191 with an equal or 7192 wider sync scope 7193 and memory ordering 7194 stronger than 7195 unordered (this is 7196 termed the 7197 fence-paired-atomic). 7198 - Must happen before 7199 the following buffer_invl2 and 7200 buffer_wbinvl1_vol. 7201 - Ensures that the 7202 fence-paired atomic 7203 has completed 7204 before invalidating 7205 the 7206 cache. Therefore 7207 any following 7208 locations read must 7209 be no older than 7210 the value read by 7211 the 7212 fence-paired-atomic. 7213 7214 2. buffer_invl2; 7215 buffer_wbinvl1_vol 7216 7217 - Must happen before any 7218 following global/generic 7219 load/load 7220 atomic/store/store 7221 atomic/atomicrmw. 7222 - Ensures that 7223 following 7224 loads will not see 7225 stale L1 global data, 7226 nor see stale L2 MTYPE 7227 NC global data. 7228 MTYPE RW and CC memory will 7229 never be stale in L2 due to 7230 the memory probes. 7231 **Release Atomic** 7232 ------------------------------------------------------------------------------------ 7233 store atomic release - singlethread - global 1. buffer/global/flat_store 7234 - wavefront - generic 7235 store atomic release - singlethread - local *If TgSplit execution mode, 7236 - wavefront local address space cannot 7237 be used.* 7238 7239 1. ds_store 7240 store atomic release - workgroup - global 1. s_waitcnt lgkm/vmcnt(0) 7241 - generic 7242 - Use lgkmcnt(0) if not 7243 TgSplit execution mode 7244 and vmcnt(0) if TgSplit 7245 execution mode. 7246 - If OpenCL, omit lgkmcnt(0). 7247 - s_waitcnt vmcnt(0) 7248 must happen after 7249 any preceding 7250 global/generic load/store/ 7251 load atomic/store atomic/ 7252 atomicrmw. 7253 - s_waitcnt lgkmcnt(0) 7254 must happen after 7255 any preceding 7256 local/generic 7257 load/store/load 7258 atomic/store 7259 atomic/atomicrmw. 7260 - Must happen before 7261 the following 7262 store. 7263 - Ensures that all 7264 memory operations 7265 have 7266 completed before 7267 performing the 7268 store that is being 7269 released. 7270 7271 2. buffer/global/flat_store 7272 store atomic release - workgroup - local *If TgSplit execution mode, 7273 local address space cannot 7274 be used.* 7275 7276 1. ds_store 7277 store atomic release - agent - global 1. s_waitcnt lgkmcnt(0) & 7278 - generic vmcnt(0) 7279 7280 - If TgSplit execution mode, 7281 omit lgkmcnt(0). 7282 - If OpenCL and 7283 address space is 7284 not generic, omit 7285 lgkmcnt(0). 7286 - Could be split into 7287 separate s_waitcnt 7288 vmcnt(0) and 7289 s_waitcnt 7290 lgkmcnt(0) to allow 7291 them to be 7292 independently moved 7293 according to the 7294 following rules. 7295 - s_waitcnt vmcnt(0) 7296 must happen after 7297 any preceding 7298 global/generic 7299 load/store/load 7300 atomic/store 7301 atomic/atomicrmw. 7302 - s_waitcnt lgkmcnt(0) 7303 must happen after 7304 any preceding 7305 local/generic 7306 load/store/load 7307 atomic/store 7308 atomic/atomicrmw. 7309 - Must happen before 7310 the following 7311 store. 7312 - Ensures that all 7313 memory operations 7314 to memory have 7315 completed before 7316 performing the 7317 store that is being 7318 released. 7319 7320 2. buffer/global/flat_store 7321 store atomic release - system - global 1. buffer_wbl2 7322 - generic 7323 - Must happen before 7324 following s_waitcnt. 7325 - Performs L2 writeback to 7326 ensure previous 7327 global/generic 7328 store/atomicrmw are 7329 visible at system scope. 7330 7331 2. s_waitcnt lgkmcnt(0) & 7332 vmcnt(0) 7333 7334 - If TgSplit execution mode, 7335 omit lgkmcnt(0). 7336 - If OpenCL and 7337 address space is 7338 not generic, omit 7339 lgkmcnt(0). 7340 - Could be split into 7341 separate s_waitcnt 7342 vmcnt(0) and 7343 s_waitcnt 7344 lgkmcnt(0) to allow 7345 them to be 7346 independently moved 7347 according to the 7348 following rules. 7349 - s_waitcnt vmcnt(0) 7350 must happen after any 7351 preceding 7352 global/generic 7353 load/store/load 7354 atomic/store 7355 atomic/atomicrmw. 7356 - s_waitcnt lgkmcnt(0) 7357 must happen after any 7358 preceding 7359 local/generic 7360 load/store/load 7361 atomic/store 7362 atomic/atomicrmw. 7363 - Must happen before 7364 the following 7365 store. 7366 - Ensures that all 7367 memory operations 7368 to memory and the L2 7369 writeback have 7370 completed before 7371 performing the 7372 store that is being 7373 released. 7374 7375 3. buffer/global/flat_store 7376 atomicrmw release - singlethread - global 1. buffer/global/flat_atomic 7377 - wavefront - generic 7378 atomicrmw release - singlethread - local *If TgSplit execution mode, 7379 - wavefront local address space cannot 7380 be used.* 7381 7382 1. ds_atomic 7383 atomicrmw release - workgroup - global 1. s_waitcnt lgkm/vmcnt(0) 7384 - generic 7385 - Use lgkmcnt(0) if not 7386 TgSplit execution mode 7387 and vmcnt(0) if TgSplit 7388 execution mode. 7389 - If OpenCL, omit 7390 lgkmcnt(0). 7391 - s_waitcnt vmcnt(0) 7392 must happen after 7393 any preceding 7394 global/generic load/store/ 7395 load atomic/store atomic/ 7396 atomicrmw. 7397 - s_waitcnt lgkmcnt(0) 7398 must happen after 7399 any preceding 7400 local/generic 7401 load/store/load 7402 atomic/store 7403 atomic/atomicrmw. 7404 - Must happen before 7405 the following 7406 atomicrmw. 7407 - Ensures that all 7408 memory operations 7409 have 7410 completed before 7411 performing the 7412 atomicrmw that is 7413 being released. 7414 7415 2. buffer/global/flat_atomic 7416 atomicrmw release - workgroup - local *If TgSplit execution mode, 7417 local address space cannot 7418 be used.* 7419 7420 1. ds_atomic 7421 atomicrmw release - agent - global 1. s_waitcnt lgkmcnt(0) & 7422 - generic vmcnt(0) 7423 7424 - If TgSplit execution mode, 7425 omit lgkmcnt(0). 7426 - If OpenCL, omit 7427 lgkmcnt(0). 7428 - Could be split into 7429 separate s_waitcnt 7430 vmcnt(0) and 7431 s_waitcnt 7432 lgkmcnt(0) to allow 7433 them to be 7434 independently moved 7435 according to the 7436 following rules. 7437 - s_waitcnt vmcnt(0) 7438 must happen after 7439 any preceding 7440 global/generic 7441 load/store/load 7442 atomic/store 7443 atomic/atomicrmw. 7444 - s_waitcnt lgkmcnt(0) 7445 must happen after 7446 any preceding 7447 local/generic 7448 load/store/load 7449 atomic/store 7450 atomic/atomicrmw. 7451 - Must happen before 7452 the following 7453 atomicrmw. 7454 - Ensures that all 7455 memory operations 7456 to global and local 7457 have completed 7458 before performing 7459 the atomicrmw that 7460 is being released. 7461 7462 2. buffer/global/flat_atomic 7463 atomicrmw release - system - global 1. buffer_wbl2 7464 - generic 7465 - Must happen before 7466 following s_waitcnt. 7467 - Performs L2 writeback to 7468 ensure previous 7469 global/generic 7470 store/atomicrmw are 7471 visible at system scope. 7472 7473 2. s_waitcnt lgkmcnt(0) & 7474 vmcnt(0) 7475 7476 - If TgSplit execution mode, 7477 omit lgkmcnt(0). 7478 - If OpenCL, omit 7479 lgkmcnt(0). 7480 - Could be split into 7481 separate s_waitcnt 7482 vmcnt(0) and 7483 s_waitcnt 7484 lgkmcnt(0) to allow 7485 them to be 7486 independently moved 7487 according to the 7488 following rules. 7489 - s_waitcnt vmcnt(0) 7490 must happen after 7491 any preceding 7492 global/generic 7493 load/store/load 7494 atomic/store 7495 atomic/atomicrmw. 7496 - s_waitcnt lgkmcnt(0) 7497 must happen after 7498 any preceding 7499 local/generic 7500 load/store/load 7501 atomic/store 7502 atomic/atomicrmw. 7503 - Must happen before 7504 the following 7505 atomicrmw. 7506 - Ensures that all 7507 memory operations 7508 to memory and the L2 7509 writeback have 7510 completed before 7511 performing the 7512 store that is being 7513 released. 7514 7515 3. buffer/global/flat_atomic 7516 fence release - singlethread *none* *none* 7517 - wavefront 7518 fence release - workgroup *none* 1. s_waitcnt lgkm/vmcnt(0) 7519 7520 - Use lgkmcnt(0) if not 7521 TgSplit execution mode 7522 and vmcnt(0) if TgSplit 7523 execution mode. 7524 - If OpenCL and 7525 address space is 7526 not generic, omit 7527 lgkmcnt(0). 7528 - If OpenCL and 7529 address space is 7530 local, omit 7531 vmcnt(0). 7532 - However, since LLVM 7533 currently has no 7534 address space on 7535 the fence need to 7536 conservatively 7537 always generate. If 7538 fence had an 7539 address space then 7540 set to address 7541 space of OpenCL 7542 fence flag, or to 7543 generic if both 7544 local and global 7545 flags are 7546 specified. 7547 - s_waitcnt vmcnt(0) 7548 must happen after 7549 any preceding 7550 global/generic 7551 load/store/ 7552 load atomic/store atomic/ 7553 atomicrmw. 7554 - s_waitcnt lgkmcnt(0) 7555 must happen after 7556 any preceding 7557 local/generic 7558 load/load 7559 atomic/store/store 7560 atomic/atomicrmw. 7561 - Must happen before 7562 any following store 7563 atomic/atomicrmw 7564 with an equal or 7565 wider sync scope 7566 and memory ordering 7567 stronger than 7568 unordered (this is 7569 termed the 7570 fence-paired-atomic). 7571 - Ensures that all 7572 memory operations 7573 have 7574 completed before 7575 performing the 7576 following 7577 fence-paired-atomic. 7578 7579 fence release - agent *none* 1. s_waitcnt lgkmcnt(0) & 7580 vmcnt(0) 7581 7582 - If TgSplit execution mode, 7583 omit lgkmcnt(0). 7584 - If OpenCL and 7585 address space is 7586 not generic, omit 7587 lgkmcnt(0). 7588 - If OpenCL and 7589 address space is 7590 local, omit 7591 vmcnt(0). 7592 - However, since LLVM 7593 currently has no 7594 address space on 7595 the fence need to 7596 conservatively 7597 always generate. If 7598 fence had an 7599 address space then 7600 set to address 7601 space of OpenCL 7602 fence flag, or to 7603 generic if both 7604 local and global 7605 flags are 7606 specified. 7607 - Could be split into 7608 separate s_waitcnt 7609 vmcnt(0) and 7610 s_waitcnt 7611 lgkmcnt(0) to allow 7612 them to be 7613 independently moved 7614 according to the 7615 following rules. 7616 - s_waitcnt vmcnt(0) 7617 must happen after 7618 any preceding 7619 global/generic 7620 load/store/load 7621 atomic/store 7622 atomic/atomicrmw. 7623 - s_waitcnt lgkmcnt(0) 7624 must happen after 7625 any preceding 7626 local/generic 7627 load/store/load 7628 atomic/store 7629 atomic/atomicrmw. 7630 - Must happen before 7631 any following store 7632 atomic/atomicrmw 7633 with an equal or 7634 wider sync scope 7635 and memory ordering 7636 stronger than 7637 unordered (this is 7638 termed the 7639 fence-paired-atomic). 7640 - Ensures that all 7641 memory operations 7642 have 7643 completed before 7644 performing the 7645 following 7646 fence-paired-atomic. 7647 7648 fence release - system *none* 1. buffer_wbl2 7649 7650 - If OpenCL and 7651 address space is 7652 local, omit. 7653 - Must happen before 7654 following s_waitcnt. 7655 - Performs L2 writeback to 7656 ensure previous 7657 global/generic 7658 store/atomicrmw are 7659 visible at system scope. 7660 7661 2. s_waitcnt lgkmcnt(0) & 7662 vmcnt(0) 7663 7664 - If TgSplit execution mode, 7665 omit lgkmcnt(0). 7666 - If OpenCL and 7667 address space is 7668 not generic, omit 7669 lgkmcnt(0). 7670 - If OpenCL and 7671 address space is 7672 local, omit 7673 vmcnt(0). 7674 - However, since LLVM 7675 currently has no 7676 address space on 7677 the fence need to 7678 conservatively 7679 always generate. If 7680 fence had an 7681 address space then 7682 set to address 7683 space of OpenCL 7684 fence flag, or to 7685 generic if both 7686 local and global 7687 flags are 7688 specified. 7689 - Could be split into 7690 separate s_waitcnt 7691 vmcnt(0) and 7692 s_waitcnt 7693 lgkmcnt(0) to allow 7694 them to be 7695 independently moved 7696 according to the 7697 following rules. 7698 - s_waitcnt vmcnt(0) 7699 must happen after 7700 any preceding 7701 global/generic 7702 load/store/load 7703 atomic/store 7704 atomic/atomicrmw. 7705 - s_waitcnt lgkmcnt(0) 7706 must happen after 7707 any preceding 7708 local/generic 7709 load/store/load 7710 atomic/store 7711 atomic/atomicrmw. 7712 - Must happen before 7713 any following store 7714 atomic/atomicrmw 7715 with an equal or 7716 wider sync scope 7717 and memory ordering 7718 stronger than 7719 unordered (this is 7720 termed the 7721 fence-paired-atomic). 7722 - Ensures that all 7723 memory operations 7724 have 7725 completed before 7726 performing the 7727 following 7728 fence-paired-atomic. 7729 7730 **Acquire-Release Atomic** 7731 ------------------------------------------------------------------------------------ 7732 atomicrmw acq_rel - singlethread - global 1. buffer/global/flat_atomic 7733 - wavefront - generic 7734 atomicrmw acq_rel - singlethread - local *If TgSplit execution mode, 7735 - wavefront local address space cannot 7736 be used.* 7737 7738 1. ds_atomic 7739 atomicrmw acq_rel - workgroup - global 1. s_waitcnt lgkm/vmcnt(0) 7740 7741 - Use lgkmcnt(0) if not 7742 TgSplit execution mode 7743 and vmcnt(0) if TgSplit 7744 execution mode. 7745 - If OpenCL, omit 7746 lgkmcnt(0). 7747 - Must happen after 7748 any preceding 7749 local/generic 7750 load/store/load 7751 atomic/store 7752 atomic/atomicrmw. 7753 - s_waitcnt vmcnt(0) 7754 must happen after 7755 any preceding 7756 global/generic load/store/ 7757 load atomic/store atomic/ 7758 atomicrmw. 7759 - s_waitcnt lgkmcnt(0) 7760 must happen after 7761 any preceding 7762 local/generic 7763 load/store/load 7764 atomic/store 7765 atomic/atomicrmw. 7766 - Must happen before 7767 the following 7768 atomicrmw. 7769 - Ensures that all 7770 memory operations 7771 have 7772 completed before 7773 performing the 7774 atomicrmw that is 7775 being released. 7776 7777 2. buffer/global_atomic 7778 3. s_waitcnt vmcnt(0) 7779 7780 - If not TgSplit execution 7781 mode, omit. 7782 - Must happen before 7783 the following 7784 buffer_wbinvl1_vol. 7785 - Ensures any 7786 following global 7787 data read is no 7788 older than the 7789 atomicrmw value 7790 being acquired. 7791 7792 4. buffer_wbinvl1_vol 7793 7794 - If not TgSplit execution 7795 mode, omit. 7796 - Ensures that 7797 following 7798 loads will not see 7799 stale data. 7800 7801 atomicrmw acq_rel - workgroup - local *If TgSplit execution mode, 7802 local address space cannot 7803 be used.* 7804 7805 1. ds_atomic 7806 2. s_waitcnt lgkmcnt(0) 7807 7808 - If OpenCL, omit. 7809 - Must happen before 7810 any following 7811 global/generic 7812 load/load 7813 atomic/store/store 7814 atomic/atomicrmw. 7815 - Ensures any 7816 following global 7817 data read is no 7818 older than the local load 7819 atomic value being 7820 acquired. 7821 7822 atomicrmw acq_rel - workgroup - generic 1. s_waitcnt lgkm/vmcnt(0) 7823 7824 - Use lgkmcnt(0) if not 7825 TgSplit execution mode 7826 and vmcnt(0) if TgSplit 7827 execution mode. 7828 - If OpenCL, omit 7829 lgkmcnt(0). 7830 - s_waitcnt vmcnt(0) 7831 must happen after 7832 any preceding 7833 global/generic load/store/ 7834 load atomic/store atomic/ 7835 atomicrmw. 7836 - s_waitcnt lgkmcnt(0) 7837 must happen after 7838 any preceding 7839 local/generic 7840 load/store/load 7841 atomic/store 7842 atomic/atomicrmw. 7843 - Must happen before 7844 the following 7845 atomicrmw. 7846 - Ensures that all 7847 memory operations 7848 have 7849 completed before 7850 performing the 7851 atomicrmw that is 7852 being released. 7853 7854 2. flat_atomic 7855 3. s_waitcnt lgkmcnt(0) & 7856 vmcnt(0) 7857 7858 - If not TgSplit execution 7859 mode, omit vmcnt(0). 7860 - If OpenCL, omit 7861 lgkmcnt(0). 7862 - Must happen before 7863 the following 7864 buffer_wbinvl1_vol and 7865 any following 7866 global/generic 7867 load/load 7868 atomic/store/store 7869 atomic/atomicrmw. 7870 - Ensures any 7871 following global 7872 data read is no 7873 older than a local load 7874 atomic value being 7875 acquired. 7876 7877 3. buffer_wbinvl1_vol 7878 7879 - If not TgSplit execution 7880 mode, omit. 7881 - Ensures that 7882 following 7883 loads will not see 7884 stale data. 7885 7886 atomicrmw acq_rel - agent - global 1. s_waitcnt lgkmcnt(0) & 7887 vmcnt(0) 7888 7889 - If TgSplit execution mode, 7890 omit lgkmcnt(0). 7891 - If OpenCL, omit 7892 lgkmcnt(0). 7893 - Could be split into 7894 separate s_waitcnt 7895 vmcnt(0) and 7896 s_waitcnt 7897 lgkmcnt(0) to allow 7898 them to be 7899 independently moved 7900 according to the 7901 following rules. 7902 - s_waitcnt vmcnt(0) 7903 must happen after 7904 any preceding 7905 global/generic 7906 load/store/load 7907 atomic/store 7908 atomic/atomicrmw. 7909 - s_waitcnt lgkmcnt(0) 7910 must happen after 7911 any preceding 7912 local/generic 7913 load/store/load 7914 atomic/store 7915 atomic/atomicrmw. 7916 - Must happen before 7917 the following 7918 atomicrmw. 7919 - Ensures that all 7920 memory operations 7921 to global have 7922 completed before 7923 performing the 7924 atomicrmw that is 7925 being released. 7926 7927 2. buffer/global_atomic 7928 3. s_waitcnt vmcnt(0) 7929 7930 - Must happen before 7931 following 7932 buffer_wbinvl1_vol. 7933 - Ensures the 7934 atomicrmw has 7935 completed before 7936 invalidating the 7937 cache. 7938 7939 4. buffer_wbinvl1_vol 7940 7941 - Must happen before 7942 any following 7943 global/generic 7944 load/load 7945 atomic/atomicrmw. 7946 - Ensures that 7947 following loads 7948 will not see stale 7949 global data. 7950 7951 atomicrmw acq_rel - system - global 1. buffer_wbl2 7952 7953 - Must happen before 7954 following s_waitcnt. 7955 - Performs L2 writeback to 7956 ensure previous 7957 global/generic 7958 store/atomicrmw are 7959 visible at system scope. 7960 7961 2. s_waitcnt lgkmcnt(0) & 7962 vmcnt(0) 7963 7964 - If TgSplit execution mode, 7965 omit lgkmcnt(0). 7966 - If OpenCL, omit 7967 lgkmcnt(0). 7968 - Could be split into 7969 separate s_waitcnt 7970 vmcnt(0) and 7971 s_waitcnt 7972 lgkmcnt(0) to allow 7973 them to be 7974 independently moved 7975 according to the 7976 following rules. 7977 - s_waitcnt vmcnt(0) 7978 must happen after 7979 any preceding 7980 global/generic 7981 load/store/load 7982 atomic/store 7983 atomic/atomicrmw. 7984 - s_waitcnt lgkmcnt(0) 7985 must happen after 7986 any preceding 7987 local/generic 7988 load/store/load 7989 atomic/store 7990 atomic/atomicrmw. 7991 - Must happen before 7992 the following 7993 atomicrmw. 7994 - Ensures that all 7995 memory operations 7996 to global and L2 writeback 7997 have completed before 7998 performing the 7999 atomicrmw that is 8000 being released. 8001 8002 3. buffer/global_atomic 8003 4. s_waitcnt vmcnt(0) 8004 8005 - Must happen before 8006 following buffer_invl2 and 8007 buffer_wbinvl1_vol. 8008 - Ensures the 8009 atomicrmw has 8010 completed before 8011 invalidating the 8012 caches. 8013 8014 5. buffer_invl2; 8015 buffer_wbinvl1_vol 8016 8017 - Must happen before 8018 any following 8019 global/generic 8020 load/load 8021 atomic/atomicrmw. 8022 - Ensures that 8023 following 8024 loads will not see 8025 stale L1 global data, 8026 nor see stale L2 MTYPE 8027 NC global data. 8028 MTYPE RW and CC memory will 8029 never be stale in L2 due to 8030 the memory probes. 8031 8032 atomicrmw acq_rel - agent - generic 1. s_waitcnt lgkmcnt(0) & 8033 vmcnt(0) 8034 8035 - If TgSplit execution mode, 8036 omit lgkmcnt(0). 8037 - If OpenCL, omit 8038 lgkmcnt(0). 8039 - Could be split into 8040 separate s_waitcnt 8041 vmcnt(0) and 8042 s_waitcnt 8043 lgkmcnt(0) to allow 8044 them to be 8045 independently moved 8046 according to the 8047 following rules. 8048 - s_waitcnt vmcnt(0) 8049 must happen after 8050 any preceding 8051 global/generic 8052 load/store/load 8053 atomic/store 8054 atomic/atomicrmw. 8055 - s_waitcnt lgkmcnt(0) 8056 must happen after 8057 any preceding 8058 local/generic 8059 load/store/load 8060 atomic/store 8061 atomic/atomicrmw. 8062 - Must happen before 8063 the following 8064 atomicrmw. 8065 - Ensures that all 8066 memory operations 8067 to global have 8068 completed before 8069 performing the 8070 atomicrmw that is 8071 being released. 8072 8073 2. flat_atomic 8074 3. s_waitcnt vmcnt(0) & 8075 lgkmcnt(0) 8076 8077 - If TgSplit execution mode, 8078 omit lgkmcnt(0). 8079 - If OpenCL, omit 8080 lgkmcnt(0). 8081 - Must happen before 8082 following 8083 buffer_wbinvl1_vol. 8084 - Ensures the 8085 atomicrmw has 8086 completed before 8087 invalidating the 8088 cache. 8089 8090 4. buffer_wbinvl1_vol 8091 8092 - Must happen before 8093 any following 8094 global/generic 8095 load/load 8096 atomic/atomicrmw. 8097 - Ensures that 8098 following loads 8099 will not see stale 8100 global data. 8101 8102 atomicrmw acq_rel - system - generic 1. buffer_wbl2 8103 8104 - Must happen before 8105 following s_waitcnt. 8106 - Performs L2 writeback to 8107 ensure previous 8108 global/generic 8109 store/atomicrmw are 8110 visible at system scope. 8111 8112 2. s_waitcnt lgkmcnt(0) & 8113 vmcnt(0) 8114 8115 - If TgSplit execution mode, 8116 omit lgkmcnt(0). 8117 - If OpenCL, omit 8118 lgkmcnt(0). 8119 - Could be split into 8120 separate s_waitcnt 8121 vmcnt(0) and 8122 s_waitcnt 8123 lgkmcnt(0) to allow 8124 them to be 8125 independently moved 8126 according to the 8127 following rules. 8128 - s_waitcnt vmcnt(0) 8129 must happen after 8130 any preceding 8131 global/generic 8132 load/store/load 8133 atomic/store 8134 atomic/atomicrmw. 8135 - s_waitcnt lgkmcnt(0) 8136 must happen after 8137 any preceding 8138 local/generic 8139 load/store/load 8140 atomic/store 8141 atomic/atomicrmw. 8142 - Must happen before 8143 the following 8144 atomicrmw. 8145 - Ensures that all 8146 memory operations 8147 to global and L2 writeback 8148 have completed before 8149 performing the 8150 atomicrmw that is 8151 being released. 8152 8153 3. flat_atomic 8154 4. s_waitcnt vmcnt(0) & 8155 lgkmcnt(0) 8156 8157 - If TgSplit execution mode, 8158 omit lgkmcnt(0). 8159 - If OpenCL, omit 8160 lgkmcnt(0). 8161 - Must happen before 8162 following buffer_invl2 and 8163 buffer_wbinvl1_vol. 8164 - Ensures the 8165 atomicrmw has 8166 completed before 8167 invalidating the 8168 caches. 8169 8170 5. buffer_invl2; 8171 buffer_wbinvl1_vol 8172 8173 - Must happen before 8174 any following 8175 global/generic 8176 load/load 8177 atomic/atomicrmw. 8178 - Ensures that 8179 following 8180 loads will not see 8181 stale L1 global data, 8182 nor see stale L2 MTYPE 8183 NC global data. 8184 MTYPE RW and CC memory will 8185 never be stale in L2 due to 8186 the memory probes. 8187 8188 fence acq_rel - singlethread *none* *none* 8189 - wavefront 8190 fence acq_rel - workgroup *none* 1. s_waitcnt lgkm/vmcnt(0) 8191 8192 - Use lgkmcnt(0) if not 8193 TgSplit execution mode 8194 and vmcnt(0) if TgSplit 8195 execution mode. 8196 - If OpenCL and 8197 address space is 8198 not generic, omit 8199 lgkmcnt(0). 8200 - If OpenCL and 8201 address space is 8202 local, omit 8203 vmcnt(0). 8204 - However, 8205 since LLVM 8206 currently has no 8207 address space on 8208 the fence need to 8209 conservatively 8210 always generate 8211 (see comment for 8212 previous fence). 8213 - s_waitcnt vmcnt(0) 8214 must happen after 8215 any preceding 8216 global/generic 8217 load/store/ 8218 load atomic/store atomic/ 8219 atomicrmw. 8220 - s_waitcnt lgkmcnt(0) 8221 must happen after 8222 any preceding 8223 local/generic 8224 load/load 8225 atomic/store/store 8226 atomic/atomicrmw. 8227 - Must happen before 8228 any following 8229 global/generic 8230 load/load 8231 atomic/store/store 8232 atomic/atomicrmw. 8233 - Ensures that all 8234 memory operations 8235 have 8236 completed before 8237 performing any 8238 following global 8239 memory operations. 8240 - Ensures that the 8241 preceding 8242 local/generic load 8243 atomic/atomicrmw 8244 with an equal or 8245 wider sync scope 8246 and memory ordering 8247 stronger than 8248 unordered (this is 8249 termed the 8250 acquire-fence-paired-atomic) 8251 has completed 8252 before following 8253 global memory 8254 operations. This 8255 satisfies the 8256 requirements of 8257 acquire. 8258 - Ensures that all 8259 previous memory 8260 operations have 8261 completed before a 8262 following 8263 local/generic store 8264 atomic/atomicrmw 8265 with an equal or 8266 wider sync scope 8267 and memory ordering 8268 stronger than 8269 unordered (this is 8270 termed the 8271 release-fence-paired-atomic). 8272 This satisfies the 8273 requirements of 8274 release. 8275 - Must happen before 8276 the following 8277 buffer_wbinvl1_vol. 8278 - Ensures that the 8279 acquire-fence-paired 8280 atomic has completed 8281 before invalidating 8282 the 8283 cache. Therefore 8284 any following 8285 locations read must 8286 be no older than 8287 the value read by 8288 the 8289 acquire-fence-paired-atomic. 8290 8291 2. buffer_wbinvl1_vol 8292 8293 - If not TgSplit execution 8294 mode, omit. 8295 - Ensures that 8296 following 8297 loads will not see 8298 stale data. 8299 8300 fence acq_rel - agent *none* 1. s_waitcnt lgkmcnt(0) & 8301 vmcnt(0) 8302 8303 - If TgSplit execution mode, 8304 omit lgkmcnt(0). 8305 - If OpenCL and 8306 address space is 8307 not generic, omit 8308 lgkmcnt(0). 8309 - However, since LLVM 8310 currently has no 8311 address space on 8312 the fence need to 8313 conservatively 8314 always generate 8315 (see comment for 8316 previous fence). 8317 - Could be split into 8318 separate s_waitcnt 8319 vmcnt(0) and 8320 s_waitcnt 8321 lgkmcnt(0) to allow 8322 them to be 8323 independently moved 8324 according to the 8325 following rules. 8326 - s_waitcnt vmcnt(0) 8327 must happen after 8328 any preceding 8329 global/generic 8330 load/store/load 8331 atomic/store 8332 atomic/atomicrmw. 8333 - s_waitcnt lgkmcnt(0) 8334 must happen after 8335 any preceding 8336 local/generic 8337 load/store/load 8338 atomic/store 8339 atomic/atomicrmw. 8340 - Must happen before 8341 the following 8342 buffer_wbinvl1_vol. 8343 - Ensures that the 8344 preceding 8345 global/local/generic 8346 load 8347 atomic/atomicrmw 8348 with an equal or 8349 wider sync scope 8350 and memory ordering 8351 stronger than 8352 unordered (this is 8353 termed the 8354 acquire-fence-paired-atomic) 8355 has completed 8356 before invalidating 8357 the cache. This 8358 satisfies the 8359 requirements of 8360 acquire. 8361 - Ensures that all 8362 previous memory 8363 operations have 8364 completed before a 8365 following 8366 global/local/generic 8367 store 8368 atomic/atomicrmw 8369 with an equal or 8370 wider sync scope 8371 and memory ordering 8372 stronger than 8373 unordered (this is 8374 termed the 8375 release-fence-paired-atomic). 8376 This satisfies the 8377 requirements of 8378 release. 8379 8380 2. buffer_wbinvl1_vol 8381 8382 - Must happen before 8383 any following 8384 global/generic 8385 load/load 8386 atomic/store/store 8387 atomic/atomicrmw. 8388 - Ensures that 8389 following loads 8390 will not see stale 8391 global data. This 8392 satisfies the 8393 requirements of 8394 acquire. 8395 8396 fence acq_rel - system *none* 1. buffer_wbl2 8397 8398 - If OpenCL and 8399 address space is 8400 local, omit. 8401 - Must happen before 8402 following s_waitcnt. 8403 - Performs L2 writeback to 8404 ensure previous 8405 global/generic 8406 store/atomicrmw are 8407 visible at system scope. 8408 8409 2. s_waitcnt lgkmcnt(0) & 8410 vmcnt(0) 8411 8412 - If TgSplit execution mode, 8413 omit lgkmcnt(0). 8414 - If OpenCL and 8415 address space is 8416 not generic, omit 8417 lgkmcnt(0). 8418 - However, since LLVM 8419 currently has no 8420 address space on 8421 the fence need to 8422 conservatively 8423 always generate 8424 (see comment for 8425 previous fence). 8426 - Could be split into 8427 separate s_waitcnt 8428 vmcnt(0) and 8429 s_waitcnt 8430 lgkmcnt(0) to allow 8431 them to be 8432 independently moved 8433 according to the 8434 following rules. 8435 - s_waitcnt vmcnt(0) 8436 must happen after 8437 any preceding 8438 global/generic 8439 load/store/load 8440 atomic/store 8441 atomic/atomicrmw. 8442 - s_waitcnt lgkmcnt(0) 8443 must happen after 8444 any preceding 8445 local/generic 8446 load/store/load 8447 atomic/store 8448 atomic/atomicrmw. 8449 - Must happen before 8450 the following buffer_invl2 and 8451 buffer_wbinvl1_vol. 8452 - Ensures that the 8453 preceding 8454 global/local/generic 8455 load 8456 atomic/atomicrmw 8457 with an equal or 8458 wider sync scope 8459 and memory ordering 8460 stronger than 8461 unordered (this is 8462 termed the 8463 acquire-fence-paired-atomic) 8464 has completed 8465 before invalidating 8466 the cache. This 8467 satisfies the 8468 requirements of 8469 acquire. 8470 - Ensures that all 8471 previous memory 8472 operations have 8473 completed before a 8474 following 8475 global/local/generic 8476 store 8477 atomic/atomicrmw 8478 with an equal or 8479 wider sync scope 8480 and memory ordering 8481 stronger than 8482 unordered (this is 8483 termed the 8484 release-fence-paired-atomic). 8485 This satisfies the 8486 requirements of 8487 release. 8488 8489 3. buffer_invl2; 8490 buffer_wbinvl1_vol 8491 8492 - Must happen before 8493 any following 8494 global/generic 8495 load/load 8496 atomic/store/store 8497 atomic/atomicrmw. 8498 - Ensures that 8499 following 8500 loads will not see 8501 stale L1 global data, 8502 nor see stale L2 MTYPE 8503 NC global data. 8504 MTYPE RW and CC memory will 8505 never be stale in L2 due to 8506 the memory probes. 8507 8508 **Sequential Consistent Atomic** 8509 ------------------------------------------------------------------------------------ 8510 load atomic seq_cst - singlethread - global *Same as corresponding 8511 - wavefront - local load atomic acquire, 8512 - generic except must generate 8513 all instructions even 8514 for OpenCL.* 8515 load atomic seq_cst - workgroup - global 1. s_waitcnt lgkm/vmcnt(0) 8516 - generic 8517 - Use lgkmcnt(0) if not 8518 TgSplit execution mode 8519 and vmcnt(0) if TgSplit 8520 execution mode. 8521 - s_waitcnt lgkmcnt(0) must 8522 happen after 8523 preceding 8524 local/generic load 8525 atomic/store 8526 atomic/atomicrmw 8527 with memory 8528 ordering of seq_cst 8529 and with equal or 8530 wider sync scope. 8531 (Note that seq_cst 8532 fences have their 8533 own s_waitcnt 8534 lgkmcnt(0) and so do 8535 not need to be 8536 considered.) 8537 - s_waitcnt vmcnt(0) 8538 must happen after 8539 preceding 8540 global/generic load 8541 atomic/store 8542 atomic/atomicrmw 8543 with memory 8544 ordering of seq_cst 8545 and with equal or 8546 wider sync scope. 8547 (Note that seq_cst 8548 fences have their 8549 own s_waitcnt 8550 vmcnt(0) and so do 8551 not need to be 8552 considered.) 8553 - Ensures any 8554 preceding 8555 sequential 8556 consistent global/local 8557 memory instructions 8558 have completed 8559 before executing 8560 this sequentially 8561 consistent 8562 instruction. This 8563 prevents reordering 8564 a seq_cst store 8565 followed by a 8566 seq_cst load. (Note 8567 that seq_cst is 8568 stronger than 8569 acquire/release as 8570 the reordering of 8571 load acquire 8572 followed by a store 8573 release is 8574 prevented by the 8575 s_waitcnt of 8576 the release, but 8577 there is nothing 8578 preventing a store 8579 release followed by 8580 load acquire from 8581 completing out of 8582 order. The s_waitcnt 8583 could be placed after 8584 seq_store or before 8585 the seq_load. We 8586 choose the load to 8587 make the s_waitcnt be 8588 as late as possible 8589 so that the store 8590 may have already 8591 completed.) 8592 8593 2. *Following 8594 instructions same as 8595 corresponding load 8596 atomic acquire, 8597 except must generate 8598 all instructions even 8599 for OpenCL.* 8600 load atomic seq_cst - workgroup - local *If TgSplit execution mode, 8601 local address space cannot 8602 be used.* 8603 8604 *Same as corresponding 8605 load atomic acquire, 8606 except must generate 8607 all instructions even 8608 for OpenCL.* 8609 8610 load atomic seq_cst - agent - global 1. s_waitcnt lgkmcnt(0) & 8611 - system - generic vmcnt(0) 8612 8613 - If TgSplit execution mode, 8614 omit lgkmcnt(0). 8615 - Could be split into 8616 separate s_waitcnt 8617 vmcnt(0) 8618 and s_waitcnt 8619 lgkmcnt(0) to allow 8620 them to be 8621 independently moved 8622 according to the 8623 following rules. 8624 - s_waitcnt lgkmcnt(0) 8625 must happen after 8626 preceding 8627 global/generic load 8628 atomic/store 8629 atomic/atomicrmw 8630 with memory 8631 ordering of seq_cst 8632 and with equal or 8633 wider sync scope. 8634 (Note that seq_cst 8635 fences have their 8636 own s_waitcnt 8637 lgkmcnt(0) and so do 8638 not need to be 8639 considered.) 8640 - s_waitcnt vmcnt(0) 8641 must happen after 8642 preceding 8643 global/generic load 8644 atomic/store 8645 atomic/atomicrmw 8646 with memory 8647 ordering of seq_cst 8648 and with equal or 8649 wider sync scope. 8650 (Note that seq_cst 8651 fences have their 8652 own s_waitcnt 8653 vmcnt(0) and so do 8654 not need to be 8655 considered.) 8656 - Ensures any 8657 preceding 8658 sequential 8659 consistent global 8660 memory instructions 8661 have completed 8662 before executing 8663 this sequentially 8664 consistent 8665 instruction. This 8666 prevents reordering 8667 a seq_cst store 8668 followed by a 8669 seq_cst load. (Note 8670 that seq_cst is 8671 stronger than 8672 acquire/release as 8673 the reordering of 8674 load acquire 8675 followed by a store 8676 release is 8677 prevented by the 8678 s_waitcnt of 8679 the release, but 8680 there is nothing 8681 preventing a store 8682 release followed by 8683 load acquire from 8684 completing out of 8685 order. The s_waitcnt 8686 could be placed after 8687 seq_store or before 8688 the seq_load. We 8689 choose the load to 8690 make the s_waitcnt be 8691 as late as possible 8692 so that the store 8693 may have already 8694 completed.) 8695 8696 2. *Following 8697 instructions same as 8698 corresponding load 8699 atomic acquire, 8700 except must generate 8701 all instructions even 8702 for OpenCL.* 8703 store atomic seq_cst - singlethread - global *Same as corresponding 8704 - wavefront - local store atomic release, 8705 - workgroup - generic except must generate 8706 - agent all instructions even 8707 - system for OpenCL.* 8708 atomicrmw seq_cst - singlethread - global *Same as corresponding 8709 - wavefront - local atomicrmw acq_rel, 8710 - workgroup - generic except must generate 8711 - agent all instructions even 8712 - system for OpenCL.* 8713 fence seq_cst - singlethread *none* *Same as corresponding 8714 - wavefront fence acq_rel, 8715 - workgroup except must generate 8716 - agent all instructions even 8717 - system for OpenCL.* 8718 ============ ============ ============== ========== ================================ 8719 8720.. _amdgpu-amdhsa-memory-model-gfx940: 8721 8722Memory Model GFX940 8723+++++++++++++++++++ 8724 8725For GFX940: 8726 8727* Each agent has multiple shader arrays (SA). 8728* Each SA has multiple compute units (CU). 8729* Each CU has multiple SIMDs that execute wavefronts. 8730* The wavefronts for a single work-group are executed in the same CU but may be 8731 executed by different SIMDs. The exception is when in tgsplit execution mode 8732 when the wavefronts may be executed by different SIMDs in different CUs. 8733* Each CU has a single LDS memory shared by the wavefronts of the work-groups 8734 executing on it. The exception is when in tgsplit execution mode when no LDS 8735 is allocated as wavefronts of the same work-group can be in different CUs. 8736* All LDS operations of a CU are performed as wavefront wide operations in a 8737 global order and involve no caching. Completion is reported to a wavefront in 8738 execution order. 8739* The LDS memory has multiple request queues shared by the SIMDs of a 8740 CU. Therefore, the LDS operations performed by different wavefronts of a 8741 work-group can be reordered relative to each other, which can result in 8742 reordering the visibility of vector memory operations with respect to LDS 8743 operations of other wavefronts in the same work-group. A ``s_waitcnt 8744 lgkmcnt(0)`` is required to ensure synchronization between LDS operations and 8745 vector memory operations between wavefronts of a work-group, but not between 8746 operations performed by the same wavefront. 8747* The vector memory operations are performed as wavefront wide operations and 8748 completion is reported to a wavefront in execution order. The exception is 8749 that ``flat_load/store/atomic`` instructions can report out of vector memory 8750 order if they access LDS memory, and out of LDS operation order if they access 8751 global memory. 8752* The vector memory operations access a single vector L1 cache shared by all 8753 SIMDs a CU. Therefore: 8754 8755 * No special action is required for coherence between the lanes of a single 8756 wavefront. 8757 8758 * No special action is required for coherence between wavefronts in the same 8759 work-group since they execute on the same CU. The exception is when in 8760 tgsplit execution mode as wavefronts of the same work-group can be in 8761 different CUs and so a ``buffer_inv sc0`` is required which will invalidate 8762 the L1 cache. 8763 8764 * A ``buffer_inv sc0`` is required to invalidate the L1 cache for coherence 8765 between wavefronts executing in different work-groups as they may be 8766 executing on different CUs. 8767 8768 * Atomic read-modify-write instructions implicitly bypass the L1 cache. 8769 Therefore, they do not use the sc0 bit for coherence and instead use it to 8770 indicate if the instruction returns the original value being updated. They 8771 do use sc1 to indicate system or agent scope coherence. 8772 8773* The scalar memory operations access a scalar L1 cache shared by all wavefronts 8774 on a group of CUs. The scalar and vector L1 caches are not coherent. However, 8775 scalar operations are used in a restricted way so do not impact the memory 8776 model. See :ref:`amdgpu-amdhsa-memory-spaces`. 8777* The vector and scalar memory operations use an L2 cache. 8778 8779 * The gfx940 can be configured as a number of smaller agents with each having 8780 a single L2 shared by all CUs on the same agent, or as fewer (possibly one) 8781 larger agents with groups of CUs on each agent each sharing separate L2 8782 caches. 8783 * The L2 cache has independent channels to service disjoint ranges of virtual 8784 addresses. 8785 * Each CU has a separate request queue per channel for its associated L2. 8786 Therefore, the vector and scalar memory operations performed by wavefronts 8787 executing with different L1 caches and the same L2 cache can be reordered 8788 relative to each other. 8789 * A ``s_waitcnt vmcnt(0)`` is required to ensure synchronization between 8790 vector memory operations of different CUs. It ensures a previous vector 8791 memory operation has completed before executing a subsequent vector memory 8792 or LDS operation and so can be used to meet the requirements of acquire and 8793 release. 8794 * An L2 cache can be kept coherent with other L2 caches by using the MTYPE RW 8795 (read-write) for memory local to the L2, and MTYPE NC (non-coherent) with 8796 the PTE C-bit set for memory not local to the L2. 8797 8798 * Any local memory cache lines will be automatically invalidated by writes 8799 from CUs associated with other L2 caches, or writes from the CPU, due to 8800 the cache probe caused by the PTE C-bit. 8801 * XGMI accesses from the CPU to local memory may be cached on the CPU. 8802 Subsequent access from the GPU will automatically invalidate or writeback 8803 the CPU cache due to the L2 probe filter. 8804 * To ensure coherence of local memory writes of CUs with different L1 caches 8805 in the same agent a ``buffer_wbl2`` is required. It does nothing if the 8806 agent is configured to have a single L2, or will writeback dirty L2 cache 8807 lines if configured to have multiple L2 caches. 8808 * To ensure coherence of local memory writes of CUs in different agents a 8809 ``buffer_wbl2 sc1`` is required. It will writeback dirty L2 cache lines. 8810 * To ensure coherence of local memory reads of CUs with different L1 caches 8811 in the same agent a ``buffer_inv sc1`` is required. It does nothing if the 8812 agent is configured to have a single L2, or will invalidate non-local L2 8813 cache lines if configured to have multiple L2 caches. 8814 * To ensure coherence of local memory reads of CUs in different agents a 8815 ``buffer_inv sc0 sc1`` is required. It will invalidate non-local L2 cache 8816 lines if configured to have multiple L2 caches. 8817 8818 * PCIe access from the GPU to the CPU can be kept coherent by using the MTYPE 8819 UC (uncached) which bypasses the L2. 8820 8821Scalar memory operations are only used to access memory that is proven to not 8822change during the execution of the kernel dispatch. This includes constant 8823address space and global address space for program scope ``const`` variables. 8824Therefore, the kernel machine code does not have to maintain the scalar cache to 8825ensure it is coherent with the vector caches. The scalar and vector caches are 8826invalidated between kernel dispatches by CP since constant address space data 8827may change between kernel dispatch executions. See 8828:ref:`amdgpu-amdhsa-memory-spaces`. 8829 8830The one exception is if scalar writes are used to spill SGPR registers. In this 8831case the AMDGPU backend ensures the memory location used to spill is never 8832accessed by vector memory operations at the same time. If scalar writes are used 8833then a ``s_dcache_wb`` is inserted before the ``s_endpgm`` and before a function 8834return since the locations may be used for vector memory instructions by a 8835future wavefront that uses the same scratch area, or a function call that 8836creates a frame at the same address, respectively. There is no need for a 8837``s_dcache_inv`` as all scalar writes are write-before-read in the same thread. 8838 8839For kernarg backing memory: 8840 8841* CP invalidates the L1 cache at the start of each kernel dispatch. 8842* On dGPU over XGMI or PCIe the kernarg backing memory is allocated in host 8843 memory accessed as MTYPE UC (uncached) to avoid needing to invalidate the L2 8844 cache. This also causes it to be treated as non-volatile and so is not 8845 invalidated by ``*_vol``. 8846* On APU the kernarg backing memory is accessed as MTYPE CC (cache coherent) and 8847 so the L2 cache will be coherent with the CPU and other agents. 8848 8849Scratch backing memory (which is used for the private address space) is accessed 8850with MTYPE NC_NV (non-coherent non-volatile). Since the private address space is 8851only accessed by a single thread, and is always write-before-read, there is 8852never a need to invalidate these entries from the L1 cache. Hence all cache 8853invalidates are done as ``*_vol`` to only invalidate the volatile cache lines. 8854 8855The code sequences used to implement the memory model for GFX940 are defined 8856in table :ref:`amdgpu-amdhsa-memory-model-code-sequences-gfx940-table`. 8857 8858 .. table:: AMDHSA Memory Model Code Sequences GFX940 8859 :name: amdgpu-amdhsa-memory-model-code-sequences-gfx940-table 8860 8861 ============ ============ ============== ========== ================================ 8862 LLVM Instr LLVM Memory LLVM Memory AMDGPU AMDGPU Machine Code 8863 Ordering Sync Scope Address GFX940 8864 Space 8865 ============ ============ ============== ========== ================================ 8866 **Non-Atomic** 8867 ------------------------------------------------------------------------------------ 8868 load *none* *none* - global - !volatile & !nontemporal 8869 - generic 8870 - private 1. buffer/global/flat_load 8871 - constant 8872 - !volatile & nontemporal 8873 8874 1. buffer/global/flat_load 8875 nt=1 8876 8877 - volatile 8878 8879 1. buffer/global/flat_load 8880 sc0=1 sc1=1 8881 2. s_waitcnt vmcnt(0) 8882 8883 - Must happen before 8884 any following volatile 8885 global/generic 8886 load/store. 8887 - Ensures that 8888 volatile 8889 operations to 8890 different 8891 addresses will not 8892 be reordered by 8893 hardware. 8894 8895 load *none* *none* - local 1. ds_load 8896 store *none* *none* - global - !volatile & !nontemporal 8897 - generic 8898 - private 1. buffer/global/flat_store 8899 - constant 8900 - !volatile & nontemporal 8901 8902 1. buffer/global/flat_store 8903 nt=1 8904 8905 - volatile 8906 8907 1. buffer/global/flat_store 8908 sc0=1 sc1=1 8909 2. s_waitcnt vmcnt(0) 8910 8911 - Must happen before 8912 any following volatile 8913 global/generic 8914 load/store. 8915 - Ensures that 8916 volatile 8917 operations to 8918 different 8919 addresses will not 8920 be reordered by 8921 hardware. 8922 8923 store *none* *none* - local 1. ds_store 8924 **Unordered Atomic** 8925 ------------------------------------------------------------------------------------ 8926 load atomic unordered *any* *any* *Same as non-atomic*. 8927 store atomic unordered *any* *any* *Same as non-atomic*. 8928 atomicrmw unordered *any* *any* *Same as monotonic atomic*. 8929 **Monotonic Atomic** 8930 ------------------------------------------------------------------------------------ 8931 load atomic monotonic - singlethread - global 1. buffer/global/flat_load 8932 - wavefront - generic 8933 load atomic monotonic - workgroup - global 1. buffer/global/flat_load 8934 - generic sc0=1 8935 load atomic monotonic - singlethread - local *If TgSplit execution mode, 8936 - wavefront local address space cannot 8937 - workgroup be used.* 8938 8939 1. ds_load 8940 load atomic monotonic - agent - global 1. buffer/global/flat_load 8941 - generic sc1=1 8942 load atomic monotonic - system - global 1. buffer/global/flat_load 8943 - generic sc0=1 sc1=1 8944 store atomic monotonic - singlethread - global 1. buffer/global/flat_store 8945 - wavefront - generic 8946 store atomic monotonic - workgroup - global 1. buffer/global/flat_store 8947 - generic sc0=1 8948 store atomic monotonic - agent - global 1. buffer/global/flat_store 8949 - generic sc1=1 8950 store atomic monotonic - system - global 1. buffer/global/flat_store 8951 - generic sc0=1 sc1=1 8952 store atomic monotonic - singlethread - local *If TgSplit execution mode, 8953 - wavefront local address space cannot 8954 - workgroup be used.* 8955 8956 1. ds_store 8957 atomicrmw monotonic - singlethread - global 1. buffer/global/flat_atomic 8958 - wavefront - generic 8959 - workgroup 8960 - agent 8961 atomicrmw monotonic - system - global 1. buffer/global/flat_atomic 8962 - generic sc1=1 8963 atomicrmw monotonic - singlethread - local *If TgSplit execution mode, 8964 - wavefront local address space cannot 8965 - workgroup be used.* 8966 8967 1. ds_atomic 8968 **Acquire Atomic** 8969 ------------------------------------------------------------------------------------ 8970 load atomic acquire - singlethread - global 1. buffer/global/ds/flat_load 8971 - wavefront - local 8972 - generic 8973 load atomic acquire - workgroup - global 1. buffer/global_load sc0=1 8974 2. s_waitcnt vmcnt(0) 8975 8976 - If not TgSplit execution 8977 mode, omit. 8978 - Must happen before the 8979 following buffer_inv. 8980 8981 3. buffer_inv sc0=1 8982 8983 - If not TgSplit execution 8984 mode, omit. 8985 - Must happen before 8986 any following 8987 global/generic 8988 load/load 8989 atomic/store/store 8990 atomic/atomicrmw. 8991 - Ensures that 8992 following 8993 loads will not see 8994 stale data. 8995 8996 load atomic acquire - workgroup - local *If TgSplit execution mode, 8997 local address space cannot 8998 be used.* 8999 9000 1. ds_load 9001 2. s_waitcnt lgkmcnt(0) 9002 9003 - If OpenCL, omit. 9004 - Must happen before 9005 any following 9006 global/generic 9007 load/load 9008 atomic/store/store 9009 atomic/atomicrmw. 9010 - Ensures any 9011 following global 9012 data read is no 9013 older than the local load 9014 atomic value being 9015 acquired. 9016 9017 load atomic acquire - workgroup - generic 1. flat_load sc0=1 9018 2. s_waitcnt lgkm/vmcnt(0) 9019 9020 - Use lgkmcnt(0) if not 9021 TgSplit execution mode 9022 and vmcnt(0) if TgSplit 9023 execution mode. 9024 - If OpenCL, omit lgkmcnt(0). 9025 - Must happen before 9026 the following 9027 buffer_inv and any 9028 following global/generic 9029 load/load 9030 atomic/store/store 9031 atomic/atomicrmw. 9032 - Ensures any 9033 following global 9034 data read is no 9035 older than a local load 9036 atomic value being 9037 acquired. 9038 9039 3. buffer_inv sc0=1 9040 9041 - If not TgSplit execution 9042 mode, omit. 9043 - Ensures that 9044 following 9045 loads will not see 9046 stale data. 9047 9048 load atomic acquire - agent - global 1. buffer/global_load 9049 sc1=1 9050 2. s_waitcnt vmcnt(0) 9051 9052 - Must happen before 9053 following 9054 buffer_inv. 9055 - Ensures the load 9056 has completed 9057 before invalidating 9058 the cache. 9059 9060 3. buffer_inv sc1=1 9061 9062 - Must happen before 9063 any following 9064 global/generic 9065 load/load 9066 atomic/atomicrmw. 9067 - Ensures that 9068 following 9069 loads will not see 9070 stale global data. 9071 9072 load atomic acquire - system - global 1. buffer/global/flat_load 9073 sc0=1 sc1=1 9074 2. s_waitcnt vmcnt(0) 9075 9076 - Must happen before 9077 following 9078 buffer_inv. 9079 - Ensures the load 9080 has completed 9081 before invalidating 9082 the cache. 9083 9084 3. buffer_inv sc0=1 sc1=1 9085 9086 - Must happen before 9087 any following 9088 global/generic 9089 load/load 9090 atomic/atomicrmw. 9091 - Ensures that 9092 following 9093 loads will not see 9094 stale MTYPE NC global data. 9095 MTYPE RW and CC memory will 9096 never be stale due to the 9097 memory probes. 9098 9099 load atomic acquire - agent - generic 1. flat_load sc1=1 9100 2. s_waitcnt vmcnt(0) & 9101 lgkmcnt(0) 9102 9103 - If TgSplit execution mode, 9104 omit lgkmcnt(0). 9105 - If OpenCL omit 9106 lgkmcnt(0). 9107 - Must happen before 9108 following 9109 buffer_inv. 9110 - Ensures the flat_load 9111 has completed 9112 before invalidating 9113 the cache. 9114 9115 3. buffer_inv sc1=1 9116 9117 - Must happen before 9118 any following 9119 global/generic 9120 load/load 9121 atomic/atomicrmw. 9122 - Ensures that 9123 following loads 9124 will not see stale 9125 global data. 9126 9127 load atomic acquire - system - generic 1. flat_load sc0=1 sc1=1 9128 2. s_waitcnt vmcnt(0) & 9129 lgkmcnt(0) 9130 9131 - If TgSplit execution mode, 9132 omit lgkmcnt(0). 9133 - If OpenCL omit 9134 lgkmcnt(0). 9135 - Must happen before 9136 the following 9137 buffer_inv. 9138 - Ensures the flat_load 9139 has completed 9140 before invalidating 9141 the caches. 9142 9143 3. buffer_inv sc0=1 sc1=1 9144 9145 - Must happen before 9146 any following 9147 global/generic 9148 load/load 9149 atomic/atomicrmw. 9150 - Ensures that 9151 following 9152 loads will not see 9153 stale MTYPE NC global data. 9154 MTYPE RW and CC memory will 9155 never be stale due to the 9156 memory probes. 9157 9158 atomicrmw acquire - singlethread - global 1. buffer/global/flat_atomic 9159 - wavefront - generic 9160 atomicrmw acquire - singlethread - local *If TgSplit execution mode, 9161 - wavefront local address space cannot 9162 be used.* 9163 9164 1. ds_atomic 9165 atomicrmw acquire - workgroup - global 1. buffer/global_atomic 9166 2. s_waitcnt vmcnt(0) 9167 9168 - If not TgSplit execution 9169 mode, omit. 9170 - Must happen before the 9171 following buffer_inv. 9172 - Ensures the atomicrmw 9173 has completed 9174 before invalidating 9175 the cache. 9176 9177 3. buffer_inv sc0=1 9178 9179 - If not TgSplit execution 9180 mode, omit. 9181 - Must happen before 9182 any following 9183 global/generic 9184 load/load 9185 atomic/atomicrmw. 9186 - Ensures that 9187 following loads 9188 will not see stale 9189 global data. 9190 9191 atomicrmw acquire - workgroup - local *If TgSplit execution mode, 9192 local address space cannot 9193 be used.* 9194 9195 1. ds_atomic 9196 2. s_waitcnt lgkmcnt(0) 9197 9198 - If OpenCL, omit. 9199 - Must happen before 9200 any following 9201 global/generic 9202 load/load 9203 atomic/store/store 9204 atomic/atomicrmw. 9205 - Ensures any 9206 following global 9207 data read is no 9208 older than the local 9209 atomicrmw value 9210 being acquired. 9211 9212 atomicrmw acquire - workgroup - generic 1. flat_atomic 9213 2. s_waitcnt lgkm/vmcnt(0) 9214 9215 - Use lgkmcnt(0) if not 9216 TgSplit execution mode 9217 and vmcnt(0) if TgSplit 9218 execution mode. 9219 - If OpenCL, omit lgkmcnt(0). 9220 - Must happen before 9221 the following 9222 buffer_inv and 9223 any following 9224 global/generic 9225 load/load 9226 atomic/store/store 9227 atomic/atomicrmw. 9228 - Ensures any 9229 following global 9230 data read is no 9231 older than a local 9232 atomicrmw value 9233 being acquired. 9234 9235 3. buffer_inv sc0=1 9236 9237 - If not TgSplit execution 9238 mode, omit. 9239 - Ensures that 9240 following 9241 loads will not see 9242 stale data. 9243 9244 atomicrmw acquire - agent - global 1. buffer/global_atomic 9245 2. s_waitcnt vmcnt(0) 9246 9247 - Must happen before 9248 following 9249 buffer_inv. 9250 - Ensures the 9251 atomicrmw has 9252 completed before 9253 invalidating the 9254 cache. 9255 9256 3. buffer_inv sc1=1 9257 9258 - Must happen before 9259 any following 9260 global/generic 9261 load/load 9262 atomic/atomicrmw. 9263 - Ensures that 9264 following loads 9265 will not see stale 9266 global data. 9267 9268 atomicrmw acquire - system - global 1. buffer/global_atomic 9269 sc1=1 9270 2. s_waitcnt vmcnt(0) 9271 9272 - Must happen before 9273 following 9274 buffer_inv. 9275 - Ensures the 9276 atomicrmw has 9277 completed before 9278 invalidating the 9279 caches. 9280 9281 3. buffer_inv sc0=1 sc1=1 9282 9283 - Must happen before 9284 any following 9285 global/generic 9286 load/load 9287 atomic/atomicrmw. 9288 - Ensures that 9289 following 9290 loads will not see 9291 stale MTYPE NC global data. 9292 MTYPE RW and CC memory will 9293 never be stale due to the 9294 memory probes. 9295 9296 atomicrmw acquire - agent - generic 1. flat_atomic 9297 2. s_waitcnt vmcnt(0) & 9298 lgkmcnt(0) 9299 9300 - If TgSplit execution mode, 9301 omit lgkmcnt(0). 9302 - If OpenCL, omit 9303 lgkmcnt(0). 9304 - Must happen before 9305 following 9306 buffer_inv. 9307 - Ensures the 9308 atomicrmw has 9309 completed before 9310 invalidating the 9311 cache. 9312 9313 3. buffer_inv sc1=1 9314 9315 - Must happen before 9316 any following 9317 global/generic 9318 load/load 9319 atomic/atomicrmw. 9320 - Ensures that 9321 following loads 9322 will not see stale 9323 global data. 9324 9325 atomicrmw acquire - system - generic 1. flat_atomic sc1=1 9326 2. s_waitcnt vmcnt(0) & 9327 lgkmcnt(0) 9328 9329 - If TgSplit execution mode, 9330 omit lgkmcnt(0). 9331 - If OpenCL, omit 9332 lgkmcnt(0). 9333 - Must happen before 9334 following 9335 buffer_inv. 9336 - Ensures the 9337 atomicrmw has 9338 completed before 9339 invalidating the 9340 caches. 9341 9342 3. buffer_inv sc0=1 sc1=1 9343 9344 - Must happen before 9345 any following 9346 global/generic 9347 load/load 9348 atomic/atomicrmw. 9349 - Ensures that 9350 following 9351 loads will not see 9352 stale MTYPE NC global data. 9353 MTYPE RW and CC memory will 9354 never be stale due to the 9355 memory probes. 9356 9357 fence acquire - singlethread *none* *none* 9358 - wavefront 9359 fence acquire - workgroup *none* 1. s_waitcnt lgkm/vmcnt(0) 9360 9361 - Use lgkmcnt(0) if not 9362 TgSplit execution mode 9363 and vmcnt(0) if TgSplit 9364 execution mode. 9365 - If OpenCL and 9366 address space is 9367 not generic, omit 9368 lgkmcnt(0). 9369 - If OpenCL and 9370 address space is 9371 local, omit 9372 vmcnt(0). 9373 - However, since LLVM 9374 currently has no 9375 address space on 9376 the fence need to 9377 conservatively 9378 always generate. If 9379 fence had an 9380 address space then 9381 set to address 9382 space of OpenCL 9383 fence flag, or to 9384 generic if both 9385 local and global 9386 flags are 9387 specified. 9388 - s_waitcnt vmcnt(0) 9389 must happen after 9390 any preceding 9391 global/generic load 9392 atomic/ 9393 atomicrmw 9394 with an equal or 9395 wider sync scope 9396 and memory ordering 9397 stronger than 9398 unordered (this is 9399 termed the 9400 fence-paired-atomic). 9401 - s_waitcnt lgkmcnt(0) 9402 must happen after 9403 any preceding 9404 local/generic load 9405 atomic/atomicrmw 9406 with an equal or 9407 wider sync scope 9408 and memory ordering 9409 stronger than 9410 unordered (this is 9411 termed the 9412 fence-paired-atomic). 9413 - Must happen before 9414 the following 9415 buffer_inv and 9416 any following 9417 global/generic 9418 load/load 9419 atomic/store/store 9420 atomic/atomicrmw. 9421 - Ensures any 9422 following global 9423 data read is no 9424 older than the 9425 value read by the 9426 fence-paired-atomic. 9427 9428 3. buffer_inv sc0=1 9429 9430 - If not TgSplit execution 9431 mode, omit. 9432 - Ensures that 9433 following 9434 loads will not see 9435 stale data. 9436 9437 fence acquire - agent *none* 1. s_waitcnt lgkmcnt(0) & 9438 vmcnt(0) 9439 9440 - If TgSplit execution mode, 9441 omit lgkmcnt(0). 9442 - If OpenCL and 9443 address space is 9444 not generic, omit 9445 lgkmcnt(0). 9446 - However, since LLVM 9447 currently has no 9448 address space on 9449 the fence need to 9450 conservatively 9451 always generate 9452 (see comment for 9453 previous fence). 9454 - Could be split into 9455 separate s_waitcnt 9456 vmcnt(0) and 9457 s_waitcnt 9458 lgkmcnt(0) to allow 9459 them to be 9460 independently moved 9461 according to the 9462 following rules. 9463 - s_waitcnt vmcnt(0) 9464 must happen after 9465 any preceding 9466 global/generic load 9467 atomic/atomicrmw 9468 with an equal or 9469 wider sync scope 9470 and memory ordering 9471 stronger than 9472 unordered (this is 9473 termed the 9474 fence-paired-atomic). 9475 - s_waitcnt lgkmcnt(0) 9476 must happen after 9477 any preceding 9478 local/generic load 9479 atomic/atomicrmw 9480 with an equal or 9481 wider sync scope 9482 and memory ordering 9483 stronger than 9484 unordered (this is 9485 termed the 9486 fence-paired-atomic). 9487 - Must happen before 9488 the following 9489 buffer_inv. 9490 - Ensures that the 9491 fence-paired atomic 9492 has completed 9493 before invalidating 9494 the 9495 cache. Therefore 9496 any following 9497 locations read must 9498 be no older than 9499 the value read by 9500 the 9501 fence-paired-atomic. 9502 9503 2. buffer_inv sc1=1 9504 9505 - Must happen before any 9506 following global/generic 9507 load/load 9508 atomic/store/store 9509 atomic/atomicrmw. 9510 - Ensures that 9511 following loads 9512 will not see stale 9513 global data. 9514 9515 fence acquire - system *none* 1. s_waitcnt lgkmcnt(0) & 9516 vmcnt(0) 9517 9518 - If TgSplit execution mode, 9519 omit lgkmcnt(0). 9520 - If OpenCL and 9521 address space is 9522 not generic, omit 9523 lgkmcnt(0). 9524 - However, since LLVM 9525 currently has no 9526 address space on 9527 the fence need to 9528 conservatively 9529 always generate 9530 (see comment for 9531 previous fence). 9532 - Could be split into 9533 separate s_waitcnt 9534 vmcnt(0) and 9535 s_waitcnt 9536 lgkmcnt(0) to allow 9537 them to be 9538 independently moved 9539 according to the 9540 following rules. 9541 - s_waitcnt vmcnt(0) 9542 must happen after 9543 any preceding 9544 global/generic load 9545 atomic/atomicrmw 9546 with an equal or 9547 wider sync scope 9548 and memory ordering 9549 stronger than 9550 unordered (this is 9551 termed the 9552 fence-paired-atomic). 9553 - s_waitcnt lgkmcnt(0) 9554 must happen after 9555 any preceding 9556 local/generic load 9557 atomic/atomicrmw 9558 with an equal or 9559 wider sync scope 9560 and memory ordering 9561 stronger than 9562 unordered (this is 9563 termed the 9564 fence-paired-atomic). 9565 - Must happen before 9566 the following 9567 buffer_inv. 9568 - Ensures that the 9569 fence-paired atomic 9570 has completed 9571 before invalidating 9572 the 9573 cache. Therefore 9574 any following 9575 locations read must 9576 be no older than 9577 the value read by 9578 the 9579 fence-paired-atomic. 9580 9581 2. buffer_inv sc0=1 sc1=1 9582 9583 - Must happen before any 9584 following global/generic 9585 load/load 9586 atomic/store/store 9587 atomic/atomicrmw. 9588 - Ensures that 9589 following loads 9590 will not see stale 9591 global data. 9592 9593 **Release Atomic** 9594 ------------------------------------------------------------------------------------ 9595 store atomic release - singlethread - global 1. buffer/global/flat_store 9596 - wavefront - generic 9597 store atomic release - singlethread - local *If TgSplit execution mode, 9598 - wavefront local address space cannot 9599 be used.* 9600 9601 1. ds_store 9602 store atomic release - workgroup - global 1. s_waitcnt lgkm/vmcnt(0) 9603 - generic 9604 - Use lgkmcnt(0) if not 9605 TgSplit execution mode 9606 and vmcnt(0) if TgSplit 9607 execution mode. 9608 - If OpenCL, omit lgkmcnt(0). 9609 - s_waitcnt vmcnt(0) 9610 must happen after 9611 any preceding 9612 global/generic load/store/ 9613 load atomic/store atomic/ 9614 atomicrmw. 9615 - s_waitcnt lgkmcnt(0) 9616 must happen after 9617 any preceding 9618 local/generic 9619 load/store/load 9620 atomic/store 9621 atomic/atomicrmw. 9622 - Must happen before 9623 the following 9624 store. 9625 - Ensures that all 9626 memory operations 9627 have 9628 completed before 9629 performing the 9630 store that is being 9631 released. 9632 9633 2. buffer/global/flat_store sc0=1 9634 store atomic release - workgroup - local *If TgSplit execution mode, 9635 local address space cannot 9636 be used.* 9637 9638 1. ds_store 9639 store atomic release - agent - global 1. buffer_wbl2 sc1=1 9640 - generic 9641 - Must happen before 9642 following s_waitcnt. 9643 - Performs L2 writeback to 9644 ensure previous 9645 global/generic 9646 store/atomicrmw are 9647 visible at agent scope. 9648 9649 2. s_waitcnt lgkmcnt(0) & 9650 vmcnt(0) 9651 9652 - If TgSplit execution mode, 9653 omit lgkmcnt(0). 9654 - If OpenCL and 9655 address space is 9656 not generic, omit 9657 lgkmcnt(0). 9658 - Could be split into 9659 separate s_waitcnt 9660 vmcnt(0) and 9661 s_waitcnt 9662 lgkmcnt(0) to allow 9663 them to be 9664 independently moved 9665 according to the 9666 following rules. 9667 - s_waitcnt vmcnt(0) 9668 must happen after 9669 any preceding 9670 global/generic 9671 load/store/load 9672 atomic/store 9673 atomic/atomicrmw. 9674 - s_waitcnt lgkmcnt(0) 9675 must happen after 9676 any preceding 9677 local/generic 9678 load/store/load 9679 atomic/store 9680 atomic/atomicrmw. 9681 - Must happen before 9682 the following 9683 store. 9684 - Ensures that all 9685 memory operations 9686 to memory have 9687 completed before 9688 performing the 9689 store that is being 9690 released. 9691 9692 3. buffer/global/flat_store sc1=1 9693 store atomic release - system - global 1. buffer_wbl2 sc0=1 sc1=1 9694 - generic 9695 - Must happen before 9696 following s_waitcnt. 9697 - Performs L2 writeback to 9698 ensure previous 9699 global/generic 9700 store/atomicrmw are 9701 visible at system scope. 9702 9703 2. s_waitcnt lgkmcnt(0) & 9704 vmcnt(0) 9705 9706 - If TgSplit execution mode, 9707 omit lgkmcnt(0). 9708 - If OpenCL and 9709 address space is 9710 not generic, omit 9711 lgkmcnt(0). 9712 - Could be split into 9713 separate s_waitcnt 9714 vmcnt(0) and 9715 s_waitcnt 9716 lgkmcnt(0) to allow 9717 them to be 9718 independently moved 9719 according to the 9720 following rules. 9721 - s_waitcnt vmcnt(0) 9722 must happen after any 9723 preceding 9724 global/generic 9725 load/store/load 9726 atomic/store 9727 atomic/atomicrmw. 9728 - s_waitcnt lgkmcnt(0) 9729 must happen after any 9730 preceding 9731 local/generic 9732 load/store/load 9733 atomic/store 9734 atomic/atomicrmw. 9735 - Must happen before 9736 the following 9737 store. 9738 - Ensures that all 9739 memory operations 9740 to memory and the L2 9741 writeback have 9742 completed before 9743 performing the 9744 store that is being 9745 released. 9746 9747 3. buffer/global/flat_store 9748 sc0=1 sc1=1 9749 atomicrmw release - singlethread - global 1. buffer/global/flat_atomic 9750 - wavefront - generic 9751 atomicrmw release - singlethread - local *If TgSplit execution mode, 9752 - wavefront local address space cannot 9753 be used.* 9754 9755 1. ds_atomic 9756 atomicrmw release - workgroup - global 1. s_waitcnt lgkm/vmcnt(0) 9757 - generic 9758 - Use lgkmcnt(0) if not 9759 TgSplit execution mode 9760 and vmcnt(0) if TgSplit 9761 execution mode. 9762 - If OpenCL, omit 9763 lgkmcnt(0). 9764 - s_waitcnt vmcnt(0) 9765 must happen after 9766 any preceding 9767 global/generic load/store/ 9768 load atomic/store atomic/ 9769 atomicrmw. 9770 - s_waitcnt lgkmcnt(0) 9771 must happen after 9772 any preceding 9773 local/generic 9774 load/store/load 9775 atomic/store 9776 atomic/atomicrmw. 9777 - Must happen before 9778 the following 9779 atomicrmw. 9780 - Ensures that all 9781 memory operations 9782 have 9783 completed before 9784 performing the 9785 atomicrmw that is 9786 being released. 9787 9788 2. buffer/global/flat_atomic sc0=1 9789 atomicrmw release - workgroup - local *If TgSplit execution mode, 9790 local address space cannot 9791 be used.* 9792 9793 1. ds_atomic 9794 atomicrmw release - agent - global 1. buffer_wbl2 sc1=1 9795 - generic 9796 - Must happen before 9797 following s_waitcnt. 9798 - Performs L2 writeback to 9799 ensure previous 9800 global/generic 9801 store/atomicrmw are 9802 visible at agent scope. 9803 9804 2. s_waitcnt lgkmcnt(0) & 9805 vmcnt(0) 9806 9807 - If TgSplit execution mode, 9808 omit lgkmcnt(0). 9809 - If OpenCL, omit 9810 lgkmcnt(0). 9811 - Could be split into 9812 separate s_waitcnt 9813 vmcnt(0) and 9814 s_waitcnt 9815 lgkmcnt(0) to allow 9816 them to be 9817 independently moved 9818 according to the 9819 following rules. 9820 - s_waitcnt vmcnt(0) 9821 must happen after 9822 any preceding 9823 global/generic 9824 load/store/load 9825 atomic/store 9826 atomic/atomicrmw. 9827 - s_waitcnt lgkmcnt(0) 9828 must happen after 9829 any preceding 9830 local/generic 9831 load/store/load 9832 atomic/store 9833 atomic/atomicrmw. 9834 - Must happen before 9835 the following 9836 atomicrmw. 9837 - Ensures that all 9838 memory operations 9839 to global and local 9840 have completed 9841 before performing 9842 the atomicrmw that 9843 is being released. 9844 9845 3. buffer/global/flat_atomic sc1=1 9846 atomicrmw release - system - global 1. buffer_wbl2 sc0=1 sc1=1 9847 - generic 9848 - Must happen before 9849 following s_waitcnt. 9850 - Performs L2 writeback to 9851 ensure previous 9852 global/generic 9853 store/atomicrmw are 9854 visible at system scope. 9855 9856 2. s_waitcnt lgkmcnt(0) & 9857 vmcnt(0) 9858 9859 - If TgSplit execution mode, 9860 omit lgkmcnt(0). 9861 - If OpenCL, omit 9862 lgkmcnt(0). 9863 - Could be split into 9864 separate s_waitcnt 9865 vmcnt(0) and 9866 s_waitcnt 9867 lgkmcnt(0) to allow 9868 them to be 9869 independently moved 9870 according to the 9871 following rules. 9872 - s_waitcnt vmcnt(0) 9873 must happen after 9874 any preceding 9875 global/generic 9876 load/store/load 9877 atomic/store 9878 atomic/atomicrmw. 9879 - s_waitcnt lgkmcnt(0) 9880 must happen after 9881 any preceding 9882 local/generic 9883 load/store/load 9884 atomic/store 9885 atomic/atomicrmw. 9886 - Must happen before 9887 the following 9888 atomicrmw. 9889 - Ensures that all 9890 memory operations 9891 to memory and the L2 9892 writeback have 9893 completed before 9894 performing the 9895 store that is being 9896 released. 9897 9898 3. buffer/global/flat_atomic 9899 sc0=1 sc1=1 9900 fence release - singlethread *none* *none* 9901 - wavefront 9902 fence release - workgroup *none* 1. s_waitcnt lgkm/vmcnt(0) 9903 9904 - Use lgkmcnt(0) if not 9905 TgSplit execution mode 9906 and vmcnt(0) if TgSplit 9907 execution mode. 9908 - If OpenCL and 9909 address space is 9910 not generic, omit 9911 lgkmcnt(0). 9912 - If OpenCL and 9913 address space is 9914 local, omit 9915 vmcnt(0). 9916 - However, since LLVM 9917 currently has no 9918 address space on 9919 the fence need to 9920 conservatively 9921 always generate. If 9922 fence had an 9923 address space then 9924 set to address 9925 space of OpenCL 9926 fence flag, or to 9927 generic if both 9928 local and global 9929 flags are 9930 specified. 9931 - s_waitcnt vmcnt(0) 9932 must happen after 9933 any preceding 9934 global/generic 9935 load/store/ 9936 load atomic/store atomic/ 9937 atomicrmw. 9938 - s_waitcnt lgkmcnt(0) 9939 must happen after 9940 any preceding 9941 local/generic 9942 load/load 9943 atomic/store/store 9944 atomic/atomicrmw. 9945 - Must happen before 9946 any following store 9947 atomic/atomicrmw 9948 with an equal or 9949 wider sync scope 9950 and memory ordering 9951 stronger than 9952 unordered (this is 9953 termed the 9954 fence-paired-atomic). 9955 - Ensures that all 9956 memory operations 9957 have 9958 completed before 9959 performing the 9960 following 9961 fence-paired-atomic. 9962 9963 fence release - agent *none* 1. buffer_wbl2 sc1=1 9964 9965 - If OpenCL and 9966 address space is 9967 local, omit. 9968 - Must happen before 9969 following s_waitcnt. 9970 - Performs L2 writeback to 9971 ensure previous 9972 global/generic 9973 store/atomicrmw are 9974 visible at agent scope. 9975 9976 2. s_waitcnt lgkmcnt(0) & 9977 vmcnt(0) 9978 9979 - If TgSplit execution mode, 9980 omit lgkmcnt(0). 9981 - If OpenCL and 9982 address space is 9983 not generic, omit 9984 lgkmcnt(0). 9985 - If OpenCL and 9986 address space is 9987 local, omit 9988 vmcnt(0). 9989 - However, since LLVM 9990 currently has no 9991 address space on 9992 the fence need to 9993 conservatively 9994 always generate. If 9995 fence had an 9996 address space then 9997 set to address 9998 space of OpenCL 9999 fence flag, or to 10000 generic if both 10001 local and global 10002 flags are 10003 specified. 10004 - Could be split into 10005 separate s_waitcnt 10006 vmcnt(0) and 10007 s_waitcnt 10008 lgkmcnt(0) to allow 10009 them to be 10010 independently moved 10011 according to the 10012 following rules. 10013 - s_waitcnt vmcnt(0) 10014 must happen after 10015 any preceding 10016 global/generic 10017 load/store/load 10018 atomic/store 10019 atomic/atomicrmw. 10020 - s_waitcnt lgkmcnt(0) 10021 must happen after 10022 any preceding 10023 local/generic 10024 load/store/load 10025 atomic/store 10026 atomic/atomicrmw. 10027 - Must happen before 10028 any following store 10029 atomic/atomicrmw 10030 with an equal or 10031 wider sync scope 10032 and memory ordering 10033 stronger than 10034 unordered (this is 10035 termed the 10036 fence-paired-atomic). 10037 - Ensures that all 10038 memory operations 10039 have 10040 completed before 10041 performing the 10042 following 10043 fence-paired-atomic. 10044 10045 fence release - system *none* 1. buffer_wbl2 sc0=1 sc1=1 10046 10047 - Must happen before 10048 following s_waitcnt. 10049 - Performs L2 writeback to 10050 ensure previous 10051 global/generic 10052 store/atomicrmw are 10053 visible at system scope. 10054 10055 2. s_waitcnt lgkmcnt(0) & 10056 vmcnt(0) 10057 10058 - If TgSplit execution mode, 10059 omit lgkmcnt(0). 10060 - If OpenCL and 10061 address space is 10062 not generic, omit 10063 lgkmcnt(0). 10064 - If OpenCL and 10065 address space is 10066 local, omit 10067 vmcnt(0). 10068 - However, since LLVM 10069 currently has no 10070 address space on 10071 the fence need to 10072 conservatively 10073 always generate. If 10074 fence had an 10075 address space then 10076 set to address 10077 space of OpenCL 10078 fence flag, or to 10079 generic if both 10080 local and global 10081 flags are 10082 specified. 10083 - Could be split into 10084 separate s_waitcnt 10085 vmcnt(0) and 10086 s_waitcnt 10087 lgkmcnt(0) to allow 10088 them to be 10089 independently moved 10090 according to the 10091 following rules. 10092 - s_waitcnt vmcnt(0) 10093 must happen after 10094 any preceding 10095 global/generic 10096 load/store/load 10097 atomic/store 10098 atomic/atomicrmw. 10099 - s_waitcnt lgkmcnt(0) 10100 must happen after 10101 any preceding 10102 local/generic 10103 load/store/load 10104 atomic/store 10105 atomic/atomicrmw. 10106 - Must happen before 10107 any following store 10108 atomic/atomicrmw 10109 with an equal or 10110 wider sync scope 10111 and memory ordering 10112 stronger than 10113 unordered (this is 10114 termed the 10115 fence-paired-atomic). 10116 - Ensures that all 10117 memory operations 10118 have 10119 completed before 10120 performing the 10121 following 10122 fence-paired-atomic. 10123 10124 **Acquire-Release Atomic** 10125 ------------------------------------------------------------------------------------ 10126 atomicrmw acq_rel - singlethread - global 1. buffer/global/flat_atomic 10127 - wavefront - generic 10128 atomicrmw acq_rel - singlethread - local *If TgSplit execution mode, 10129 - wavefront local address space cannot 10130 be used.* 10131 10132 1. ds_atomic 10133 atomicrmw acq_rel - workgroup - global 1. s_waitcnt lgkm/vmcnt(0) 10134 10135 - Use lgkmcnt(0) if not 10136 TgSplit execution mode 10137 and vmcnt(0) if TgSplit 10138 execution mode. 10139 - If OpenCL, omit 10140 lgkmcnt(0). 10141 - Must happen after 10142 any preceding 10143 local/generic 10144 load/store/load 10145 atomic/store 10146 atomic/atomicrmw. 10147 - s_waitcnt vmcnt(0) 10148 must happen after 10149 any preceding 10150 global/generic load/store/ 10151 load atomic/store atomic/ 10152 atomicrmw. 10153 - s_waitcnt lgkmcnt(0) 10154 must happen after 10155 any preceding 10156 local/generic 10157 load/store/load 10158 atomic/store 10159 atomic/atomicrmw. 10160 - Must happen before 10161 the following 10162 atomicrmw. 10163 - Ensures that all 10164 memory operations 10165 have 10166 completed before 10167 performing the 10168 atomicrmw that is 10169 being released. 10170 10171 2. buffer/global_atomic 10172 3. s_waitcnt vmcnt(0) 10173 10174 - If not TgSplit execution 10175 mode, omit. 10176 - Must happen before 10177 the following 10178 buffer_inv. 10179 - Ensures any 10180 following global 10181 data read is no 10182 older than the 10183 atomicrmw value 10184 being acquired. 10185 10186 4. buffer_inv sc0=1 10187 10188 - If not TgSplit execution 10189 mode, omit. 10190 - Ensures that 10191 following 10192 loads will not see 10193 stale data. 10194 10195 atomicrmw acq_rel - workgroup - local *If TgSplit execution mode, 10196 local address space cannot 10197 be used.* 10198 10199 1. ds_atomic 10200 2. s_waitcnt lgkmcnt(0) 10201 10202 - If OpenCL, omit. 10203 - Must happen before 10204 any following 10205 global/generic 10206 load/load 10207 atomic/store/store 10208 atomic/atomicrmw. 10209 - Ensures any 10210 following global 10211 data read is no 10212 older than the local load 10213 atomic value being 10214 acquired. 10215 10216 atomicrmw acq_rel - workgroup - generic 1. s_waitcnt lgkm/vmcnt(0) 10217 10218 - Use lgkmcnt(0) if not 10219 TgSplit execution mode 10220 and vmcnt(0) if TgSplit 10221 execution mode. 10222 - If OpenCL, omit 10223 lgkmcnt(0). 10224 - s_waitcnt vmcnt(0) 10225 must happen after 10226 any preceding 10227 global/generic load/store/ 10228 load atomic/store atomic/ 10229 atomicrmw. 10230 - s_waitcnt lgkmcnt(0) 10231 must happen after 10232 any preceding 10233 local/generic 10234 load/store/load 10235 atomic/store 10236 atomic/atomicrmw. 10237 - Must happen before 10238 the following 10239 atomicrmw. 10240 - Ensures that all 10241 memory operations 10242 have 10243 completed before 10244 performing the 10245 atomicrmw that is 10246 being released. 10247 10248 2. flat_atomic 10249 3. s_waitcnt lgkmcnt(0) & 10250 vmcnt(0) 10251 10252 - If not TgSplit execution 10253 mode, omit vmcnt(0). 10254 - If OpenCL, omit 10255 lgkmcnt(0). 10256 - Must happen before 10257 the following 10258 buffer_inv and 10259 any following 10260 global/generic 10261 load/load 10262 atomic/store/store 10263 atomic/atomicrmw. 10264 - Ensures any 10265 following global 10266 data read is no 10267 older than a local load 10268 atomic value being 10269 acquired. 10270 10271 3. buffer_inv sc0=1 10272 10273 - If not TgSplit execution 10274 mode, omit. 10275 - Ensures that 10276 following 10277 loads will not see 10278 stale data. 10279 10280 atomicrmw acq_rel - agent - global 1. buffer_wbl2 sc1=1 10281 10282 - Must happen before 10283 following s_waitcnt. 10284 - Performs L2 writeback to 10285 ensure previous 10286 global/generic 10287 store/atomicrmw are 10288 visible at agent scope. 10289 10290 2. s_waitcnt lgkmcnt(0) & 10291 vmcnt(0) 10292 10293 - If TgSplit execution mode, 10294 omit lgkmcnt(0). 10295 - If OpenCL, omit 10296 lgkmcnt(0). 10297 - Could be split into 10298 separate s_waitcnt 10299 vmcnt(0) and 10300 s_waitcnt 10301 lgkmcnt(0) to allow 10302 them to be 10303 independently moved 10304 according to the 10305 following rules. 10306 - s_waitcnt vmcnt(0) 10307 must happen after 10308 any preceding 10309 global/generic 10310 load/store/load 10311 atomic/store 10312 atomic/atomicrmw. 10313 - s_waitcnt lgkmcnt(0) 10314 must happen after 10315 any preceding 10316 local/generic 10317 load/store/load 10318 atomic/store 10319 atomic/atomicrmw. 10320 - Must happen before 10321 the following 10322 atomicrmw. 10323 - Ensures that all 10324 memory operations 10325 to global have 10326 completed before 10327 performing the 10328 atomicrmw that is 10329 being released. 10330 10331 3. buffer/global_atomic 10332 4. s_waitcnt vmcnt(0) 10333 10334 - Must happen before 10335 following 10336 buffer_inv. 10337 - Ensures the 10338 atomicrmw has 10339 completed before 10340 invalidating the 10341 cache. 10342 10343 5. buffer_inv sc1=1 10344 10345 - Must happen before 10346 any following 10347 global/generic 10348 load/load 10349 atomic/atomicrmw. 10350 - Ensures that 10351 following loads 10352 will not see stale 10353 global data. 10354 10355 atomicrmw acq_rel - system - global 1. buffer_wbl2 sc0=1 sc1=1 10356 10357 - Must happen before 10358 following s_waitcnt. 10359 - Performs L2 writeback to 10360 ensure previous 10361 global/generic 10362 store/atomicrmw are 10363 visible at system scope. 10364 10365 2. s_waitcnt lgkmcnt(0) & 10366 vmcnt(0) 10367 10368 - If TgSplit execution mode, 10369 omit lgkmcnt(0). 10370 - If OpenCL, omit 10371 lgkmcnt(0). 10372 - Could be split into 10373 separate s_waitcnt 10374 vmcnt(0) and 10375 s_waitcnt 10376 lgkmcnt(0) to allow 10377 them to be 10378 independently moved 10379 according to the 10380 following rules. 10381 - s_waitcnt vmcnt(0) 10382 must happen after 10383 any preceding 10384 global/generic 10385 load/store/load 10386 atomic/store 10387 atomic/atomicrmw. 10388 - s_waitcnt lgkmcnt(0) 10389 must happen after 10390 any preceding 10391 local/generic 10392 load/store/load 10393 atomic/store 10394 atomic/atomicrmw. 10395 - Must happen before 10396 the following 10397 atomicrmw. 10398 - Ensures that all 10399 memory operations 10400 to global and L2 writeback 10401 have completed before 10402 performing the 10403 atomicrmw that is 10404 being released. 10405 10406 3. buffer/global_atomic 10407 sc1=1 10408 4. s_waitcnt vmcnt(0) 10409 10410 - Must happen before 10411 following 10412 buffer_inv. 10413 - Ensures the 10414 atomicrmw has 10415 completed before 10416 invalidating the 10417 caches. 10418 10419 5. buffer_inv sc0=1 sc1=1 10420 10421 - Must happen before 10422 any following 10423 global/generic 10424 load/load 10425 atomic/atomicrmw. 10426 - Ensures that 10427 following loads 10428 will not see stale 10429 MTYPE NC global data. 10430 MTYPE RW and CC memory will 10431 never be stale due to the 10432 memory probes. 10433 10434 atomicrmw acq_rel - agent - generic 1. buffer_wbl2 sc1=1 10435 10436 - Must happen before 10437 following s_waitcnt. 10438 - Performs L2 writeback to 10439 ensure previous 10440 global/generic 10441 store/atomicrmw are 10442 visible at agent scope. 10443 10444 2. s_waitcnt lgkmcnt(0) & 10445 vmcnt(0) 10446 10447 - If TgSplit execution mode, 10448 omit lgkmcnt(0). 10449 - If OpenCL, omit 10450 lgkmcnt(0). 10451 - Could be split into 10452 separate s_waitcnt 10453 vmcnt(0) and 10454 s_waitcnt 10455 lgkmcnt(0) to allow 10456 them to be 10457 independently moved 10458 according to the 10459 following rules. 10460 - s_waitcnt vmcnt(0) 10461 must happen after 10462 any preceding 10463 global/generic 10464 load/store/load 10465 atomic/store 10466 atomic/atomicrmw. 10467 - s_waitcnt lgkmcnt(0) 10468 must happen after 10469 any preceding 10470 local/generic 10471 load/store/load 10472 atomic/store 10473 atomic/atomicrmw. 10474 - Must happen before 10475 the following 10476 atomicrmw. 10477 - Ensures that all 10478 memory operations 10479 to global have 10480 completed before 10481 performing the 10482 atomicrmw that is 10483 being released. 10484 10485 3. flat_atomic 10486 4. s_waitcnt vmcnt(0) & 10487 lgkmcnt(0) 10488 10489 - If TgSplit execution mode, 10490 omit lgkmcnt(0). 10491 - If OpenCL, omit 10492 lgkmcnt(0). 10493 - Must happen before 10494 following 10495 buffer_inv. 10496 - Ensures the 10497 atomicrmw has 10498 completed before 10499 invalidating the 10500 cache. 10501 10502 5. buffer_inv sc1=1 10503 10504 - Must happen before 10505 any following 10506 global/generic 10507 load/load 10508 atomic/atomicrmw. 10509 - Ensures that 10510 following loads 10511 will not see stale 10512 global data. 10513 10514 atomicrmw acq_rel - system - generic 1. buffer_wbl2 sc0=1 sc1=1 10515 10516 - Must happen before 10517 following s_waitcnt. 10518 - Performs L2 writeback to 10519 ensure previous 10520 global/generic 10521 store/atomicrmw are 10522 visible at system scope. 10523 10524 2. s_waitcnt lgkmcnt(0) & 10525 vmcnt(0) 10526 10527 - If TgSplit execution mode, 10528 omit lgkmcnt(0). 10529 - If OpenCL, omit 10530 lgkmcnt(0). 10531 - Could be split into 10532 separate s_waitcnt 10533 vmcnt(0) and 10534 s_waitcnt 10535 lgkmcnt(0) to allow 10536 them to be 10537 independently moved 10538 according to the 10539 following rules. 10540 - s_waitcnt vmcnt(0) 10541 must happen after 10542 any preceding 10543 global/generic 10544 load/store/load 10545 atomic/store 10546 atomic/atomicrmw. 10547 - s_waitcnt lgkmcnt(0) 10548 must happen after 10549 any preceding 10550 local/generic 10551 load/store/load 10552 atomic/store 10553 atomic/atomicrmw. 10554 - Must happen before 10555 the following 10556 atomicrmw. 10557 - Ensures that all 10558 memory operations 10559 to global and L2 writeback 10560 have completed before 10561 performing the 10562 atomicrmw that is 10563 being released. 10564 10565 3. flat_atomic sc1=1 10566 4. s_waitcnt vmcnt(0) & 10567 lgkmcnt(0) 10568 10569 - If TgSplit execution mode, 10570 omit lgkmcnt(0). 10571 - If OpenCL, omit 10572 lgkmcnt(0). 10573 - Must happen before 10574 following 10575 buffer_inv. 10576 - Ensures the 10577 atomicrmw has 10578 completed before 10579 invalidating the 10580 caches. 10581 10582 5. buffer_inv sc0=1 sc1=1 10583 10584 - Must happen before 10585 any following 10586 global/generic 10587 load/load 10588 atomic/atomicrmw. 10589 - Ensures that 10590 following loads 10591 will not see stale 10592 MTYPE NC global data. 10593 MTYPE RW and CC memory will 10594 never be stale due to the 10595 memory probes. 10596 10597 fence acq_rel - singlethread *none* *none* 10598 - wavefront 10599 fence acq_rel - workgroup *none* 1. s_waitcnt lgkm/vmcnt(0) 10600 10601 - Use lgkmcnt(0) if not 10602 TgSplit execution mode 10603 and vmcnt(0) if TgSplit 10604 execution mode. 10605 - If OpenCL and 10606 address space is 10607 not generic, omit 10608 lgkmcnt(0). 10609 - If OpenCL and 10610 address space is 10611 local, omit 10612 vmcnt(0). 10613 - However, 10614 since LLVM 10615 currently has no 10616 address space on 10617 the fence need to 10618 conservatively 10619 always generate 10620 (see comment for 10621 previous fence). 10622 - s_waitcnt vmcnt(0) 10623 must happen after 10624 any preceding 10625 global/generic 10626 load/store/ 10627 load atomic/store atomic/ 10628 atomicrmw. 10629 - s_waitcnt lgkmcnt(0) 10630 must happen after 10631 any preceding 10632 local/generic 10633 load/load 10634 atomic/store/store 10635 atomic/atomicrmw. 10636 - Must happen before 10637 any following 10638 global/generic 10639 load/load 10640 atomic/store/store 10641 atomic/atomicrmw. 10642 - Ensures that all 10643 memory operations 10644 have 10645 completed before 10646 performing any 10647 following global 10648 memory operations. 10649 - Ensures that the 10650 preceding 10651 local/generic load 10652 atomic/atomicrmw 10653 with an equal or 10654 wider sync scope 10655 and memory ordering 10656 stronger than 10657 unordered (this is 10658 termed the 10659 acquire-fence-paired-atomic) 10660 has completed 10661 before following 10662 global memory 10663 operations. This 10664 satisfies the 10665 requirements of 10666 acquire. 10667 - Ensures that all 10668 previous memory 10669 operations have 10670 completed before a 10671 following 10672 local/generic store 10673 atomic/atomicrmw 10674 with an equal or 10675 wider sync scope 10676 and memory ordering 10677 stronger than 10678 unordered (this is 10679 termed the 10680 release-fence-paired-atomic). 10681 This satisfies the 10682 requirements of 10683 release. 10684 - Must happen before 10685 the following 10686 buffer_inv. 10687 - Ensures that the 10688 acquire-fence-paired 10689 atomic has completed 10690 before invalidating 10691 the 10692 cache. Therefore 10693 any following 10694 locations read must 10695 be no older than 10696 the value read by 10697 the 10698 acquire-fence-paired-atomic. 10699 10700 3. buffer_inv sc0=1 10701 10702 - If not TgSplit execution 10703 mode, omit. 10704 - Ensures that 10705 following 10706 loads will not see 10707 stale data. 10708 10709 fence acq_rel - agent *none* 1. buffer_wbl2 sc1=1 10710 10711 - If OpenCL and 10712 address space is 10713 local, omit. 10714 - Must happen before 10715 following s_waitcnt. 10716 - Performs L2 writeback to 10717 ensure previous 10718 global/generic 10719 store/atomicrmw are 10720 visible at agent scope. 10721 10722 2. s_waitcnt lgkmcnt(0) & 10723 vmcnt(0) 10724 10725 - If TgSplit execution mode, 10726 omit lgkmcnt(0). 10727 - If OpenCL and 10728 address space is 10729 not generic, omit 10730 lgkmcnt(0). 10731 - However, since LLVM 10732 currently has no 10733 address space on 10734 the fence need to 10735 conservatively 10736 always generate 10737 (see comment for 10738 previous fence). 10739 - Could be split into 10740 separate s_waitcnt 10741 vmcnt(0) and 10742 s_waitcnt 10743 lgkmcnt(0) to allow 10744 them to be 10745 independently moved 10746 according to the 10747 following rules. 10748 - s_waitcnt vmcnt(0) 10749 must happen after 10750 any preceding 10751 global/generic 10752 load/store/load 10753 atomic/store 10754 atomic/atomicrmw. 10755 - s_waitcnt lgkmcnt(0) 10756 must happen after 10757 any preceding 10758 local/generic 10759 load/store/load 10760 atomic/store 10761 atomic/atomicrmw. 10762 - Must happen before 10763 the following 10764 buffer_inv. 10765 - Ensures that the 10766 preceding 10767 global/local/generic 10768 load 10769 atomic/atomicrmw 10770 with an equal or 10771 wider sync scope 10772 and memory ordering 10773 stronger than 10774 unordered (this is 10775 termed the 10776 acquire-fence-paired-atomic) 10777 has completed 10778 before invalidating 10779 the cache. This 10780 satisfies the 10781 requirements of 10782 acquire. 10783 - Ensures that all 10784 previous memory 10785 operations have 10786 completed before a 10787 following 10788 global/local/generic 10789 store 10790 atomic/atomicrmw 10791 with an equal or 10792 wider sync scope 10793 and memory ordering 10794 stronger than 10795 unordered (this is 10796 termed the 10797 release-fence-paired-atomic). 10798 This satisfies the 10799 requirements of 10800 release. 10801 10802 3. buffer_inv sc1=1 10803 10804 - Must happen before 10805 any following 10806 global/generic 10807 load/load 10808 atomic/store/store 10809 atomic/atomicrmw. 10810 - Ensures that 10811 following loads 10812 will not see stale 10813 global data. This 10814 satisfies the 10815 requirements of 10816 acquire. 10817 10818 fence acq_rel - system *none* 1. buffer_wbl2 sc0=1 sc1=1 10819 10820 - If OpenCL and 10821 address space is 10822 local, omit. 10823 - Must happen before 10824 following s_waitcnt. 10825 - Performs L2 writeback to 10826 ensure previous 10827 global/generic 10828 store/atomicrmw are 10829 visible at system scope. 10830 10831 1. s_waitcnt lgkmcnt(0) & 10832 vmcnt(0) 10833 10834 - If TgSplit execution mode, 10835 omit lgkmcnt(0). 10836 - If OpenCL and 10837 address space is 10838 not generic, omit 10839 lgkmcnt(0). 10840 - However, since LLVM 10841 currently has no 10842 address space on 10843 the fence need to 10844 conservatively 10845 always generate 10846 (see comment for 10847 previous fence). 10848 - Could be split into 10849 separate s_waitcnt 10850 vmcnt(0) and 10851 s_waitcnt 10852 lgkmcnt(0) to allow 10853 them to be 10854 independently moved 10855 according to the 10856 following rules. 10857 - s_waitcnt vmcnt(0) 10858 must happen after 10859 any preceding 10860 global/generic 10861 load/store/load 10862 atomic/store 10863 atomic/atomicrmw. 10864 - s_waitcnt lgkmcnt(0) 10865 must happen after 10866 any preceding 10867 local/generic 10868 load/store/load 10869 atomic/store 10870 atomic/atomicrmw. 10871 - Must happen before 10872 the following 10873 buffer_inv. 10874 - Ensures that the 10875 preceding 10876 global/local/generic 10877 load 10878 atomic/atomicrmw 10879 with an equal or 10880 wider sync scope 10881 and memory ordering 10882 stronger than 10883 unordered (this is 10884 termed the 10885 acquire-fence-paired-atomic) 10886 has completed 10887 before invalidating 10888 the cache. This 10889 satisfies the 10890 requirements of 10891 acquire. 10892 - Ensures that all 10893 previous memory 10894 operations have 10895 completed before a 10896 following 10897 global/local/generic 10898 store 10899 atomic/atomicrmw 10900 with an equal or 10901 wider sync scope 10902 and memory ordering 10903 stronger than 10904 unordered (this is 10905 termed the 10906 release-fence-paired-atomic). 10907 This satisfies the 10908 requirements of 10909 release. 10910 10911 2. buffer_inv sc0=1 sc1=1 10912 10913 - Must happen before 10914 any following 10915 global/generic 10916 load/load 10917 atomic/store/store 10918 atomic/atomicrmw. 10919 - Ensures that 10920 following loads 10921 will not see stale 10922 MTYPE NC global data. 10923 MTYPE RW and CC memory will 10924 never be stale due to the 10925 memory probes. 10926 10927 **Sequential Consistent Atomic** 10928 ------------------------------------------------------------------------------------ 10929 load atomic seq_cst - singlethread - global *Same as corresponding 10930 - wavefront - local load atomic acquire, 10931 - generic except must generate 10932 all instructions even 10933 for OpenCL.* 10934 load atomic seq_cst - workgroup - global 1. s_waitcnt lgkm/vmcnt(0) 10935 - generic 10936 - Use lgkmcnt(0) if not 10937 TgSplit execution mode 10938 and vmcnt(0) if TgSplit 10939 execution mode. 10940 - s_waitcnt lgkmcnt(0) must 10941 happen after 10942 preceding 10943 local/generic load 10944 atomic/store 10945 atomic/atomicrmw 10946 with memory 10947 ordering of seq_cst 10948 and with equal or 10949 wider sync scope. 10950 (Note that seq_cst 10951 fences have their 10952 own s_waitcnt 10953 lgkmcnt(0) and so do 10954 not need to be 10955 considered.) 10956 - s_waitcnt vmcnt(0) 10957 must happen after 10958 preceding 10959 global/generic load 10960 atomic/store 10961 atomic/atomicrmw 10962 with memory 10963 ordering of seq_cst 10964 and with equal or 10965 wider sync scope. 10966 (Note that seq_cst 10967 fences have their 10968 own s_waitcnt 10969 vmcnt(0) and so do 10970 not need to be 10971 considered.) 10972 - Ensures any 10973 preceding 10974 sequential 10975 consistent global/local 10976 memory instructions 10977 have completed 10978 before executing 10979 this sequentially 10980 consistent 10981 instruction. This 10982 prevents reordering 10983 a seq_cst store 10984 followed by a 10985 seq_cst load. (Note 10986 that seq_cst is 10987 stronger than 10988 acquire/release as 10989 the reordering of 10990 load acquire 10991 followed by a store 10992 release is 10993 prevented by the 10994 s_waitcnt of 10995 the release, but 10996 there is nothing 10997 preventing a store 10998 release followed by 10999 load acquire from 11000 completing out of 11001 order. The s_waitcnt 11002 could be placed after 11003 seq_store or before 11004 the seq_load. We 11005 choose the load to 11006 make the s_waitcnt be 11007 as late as possible 11008 so that the store 11009 may have already 11010 completed.) 11011 11012 2. *Following 11013 instructions same as 11014 corresponding load 11015 atomic acquire, 11016 except must generate 11017 all instructions even 11018 for OpenCL.* 11019 load atomic seq_cst - workgroup - local *If TgSplit execution mode, 11020 local address space cannot 11021 be used.* 11022 11023 *Same as corresponding 11024 load atomic acquire, 11025 except must generate 11026 all instructions even 11027 for OpenCL.* 11028 11029 load atomic seq_cst - agent - global 1. s_waitcnt lgkmcnt(0) & 11030 - system - generic vmcnt(0) 11031 11032 - If TgSplit execution mode, 11033 omit lgkmcnt(0). 11034 - Could be split into 11035 separate s_waitcnt 11036 vmcnt(0) 11037 and s_waitcnt 11038 lgkmcnt(0) to allow 11039 them to be 11040 independently moved 11041 according to the 11042 following rules. 11043 - s_waitcnt lgkmcnt(0) 11044 must happen after 11045 preceding 11046 global/generic load 11047 atomic/store 11048 atomic/atomicrmw 11049 with memory 11050 ordering of seq_cst 11051 and with equal or 11052 wider sync scope. 11053 (Note that seq_cst 11054 fences have their 11055 own s_waitcnt 11056 lgkmcnt(0) and so do 11057 not need to be 11058 considered.) 11059 - s_waitcnt vmcnt(0) 11060 must happen after 11061 preceding 11062 global/generic load 11063 atomic/store 11064 atomic/atomicrmw 11065 with memory 11066 ordering of seq_cst 11067 and with equal or 11068 wider sync scope. 11069 (Note that seq_cst 11070 fences have their 11071 own s_waitcnt 11072 vmcnt(0) and so do 11073 not need to be 11074 considered.) 11075 - Ensures any 11076 preceding 11077 sequential 11078 consistent global 11079 memory instructions 11080 have completed 11081 before executing 11082 this sequentially 11083 consistent 11084 instruction. This 11085 prevents reordering 11086 a seq_cst store 11087 followed by a 11088 seq_cst load. (Note 11089 that seq_cst is 11090 stronger than 11091 acquire/release as 11092 the reordering of 11093 load acquire 11094 followed by a store 11095 release is 11096 prevented by the 11097 s_waitcnt of 11098 the release, but 11099 there is nothing 11100 preventing a store 11101 release followed by 11102 load acquire from 11103 completing out of 11104 order. The s_waitcnt 11105 could be placed after 11106 seq_store or before 11107 the seq_load. We 11108 choose the load to 11109 make the s_waitcnt be 11110 as late as possible 11111 so that the store 11112 may have already 11113 completed.) 11114 11115 2. *Following 11116 instructions same as 11117 corresponding load 11118 atomic acquire, 11119 except must generate 11120 all instructions even 11121 for OpenCL.* 11122 store atomic seq_cst - singlethread - global *Same as corresponding 11123 - wavefront - local store atomic release, 11124 - workgroup - generic except must generate 11125 - agent all instructions even 11126 - system for OpenCL.* 11127 atomicrmw seq_cst - singlethread - global *Same as corresponding 11128 - wavefront - local atomicrmw acq_rel, 11129 - workgroup - generic except must generate 11130 - agent all instructions even 11131 - system for OpenCL.* 11132 fence seq_cst - singlethread *none* *Same as corresponding 11133 - wavefront fence acq_rel, 11134 - workgroup except must generate 11135 - agent all instructions even 11136 - system for OpenCL.* 11137 ============ ============ ============== ========== ================================ 11138 11139.. _amdgpu-amdhsa-memory-model-gfx10: 11140 11141Memory Model GFX10 11142++++++++++++++++++ 11143 11144For GFX10: 11145 11146* Each agent has multiple shader arrays (SA). 11147* Each SA has multiple work-group processors (WGP). 11148* Each WGP has multiple compute units (CU). 11149* Each CU has multiple SIMDs that execute wavefronts. 11150* The wavefronts for a single work-group are executed in the same 11151 WGP. In CU wavefront execution mode the wavefronts may be executed by 11152 different SIMDs in the same CU. In WGP wavefront execution mode the 11153 wavefronts may be executed by different SIMDs in different CUs in the same 11154 WGP. 11155* Each WGP has a single LDS memory shared by the wavefronts of the work-groups 11156 executing on it. 11157* All LDS operations of a WGP are performed as wavefront wide operations in a 11158 global order and involve no caching. Completion is reported to a wavefront in 11159 execution order. 11160* The LDS memory has multiple request queues shared by the SIMDs of a 11161 WGP. Therefore, the LDS operations performed by different wavefronts of a 11162 work-group can be reordered relative to each other, which can result in 11163 reordering the visibility of vector memory operations with respect to LDS 11164 operations of other wavefronts in the same work-group. A ``s_waitcnt 11165 lgkmcnt(0)`` is required to ensure synchronization between LDS operations and 11166 vector memory operations between wavefronts of a work-group, but not between 11167 operations performed by the same wavefront. 11168* The vector memory operations are performed as wavefront wide operations. 11169 Completion of load/store/sample operations are reported to a wavefront in 11170 execution order of other load/store/sample operations performed by that 11171 wavefront. 11172* The vector memory operations access a vector L0 cache. There is a single L0 11173 cache per CU. Each SIMD of a CU accesses the same L0 cache. Therefore, no 11174 special action is required for coherence between the lanes of a single 11175 wavefront. However, a ``buffer_gl0_inv`` is required for coherence between 11176 wavefronts executing in the same work-group as they may be executing on SIMDs 11177 of different CUs that access different L0s. A ``buffer_gl0_inv`` is also 11178 required for coherence between wavefronts executing in different work-groups 11179 as they may be executing on different WGPs. 11180* The scalar memory operations access a scalar L0 cache shared by all wavefronts 11181 on a WGP. The scalar and vector L0 caches are not coherent. However, scalar 11182 operations are used in a restricted way so do not impact the memory model. See 11183 :ref:`amdgpu-amdhsa-memory-spaces`. 11184* The vector and scalar memory L0 caches use an L1 cache shared by all WGPs on 11185 the same SA. Therefore, no special action is required for coherence between 11186 the wavefronts of a single work-group. However, a ``buffer_gl1_inv`` is 11187 required for coherence between wavefronts executing in different work-groups 11188 as they may be executing on different SAs that access different L1s. 11189* The L1 caches have independent quadrants to service disjoint ranges of virtual 11190 addresses. 11191* Each L0 cache has a separate request queue per L1 quadrant. Therefore, the 11192 vector and scalar memory operations performed by different wavefronts, whether 11193 executing in the same or different work-groups (which may be executing on 11194 different CUs accessing different L0s), can be reordered relative to each 11195 other. A ``s_waitcnt vmcnt(0) & vscnt(0)`` is required to ensure 11196 synchronization between vector memory operations of different wavefronts. It 11197 ensures a previous vector memory operation has completed before executing a 11198 subsequent vector memory or LDS operation and so can be used to meet the 11199 requirements of acquire, release and sequential consistency. 11200* The L1 caches use an L2 cache shared by all SAs on the same agent. 11201* The L2 cache has independent channels to service disjoint ranges of virtual 11202 addresses. 11203* Each L1 quadrant of a single SA accesses a different L2 channel. Each L1 11204 quadrant has a separate request queue per L2 channel. Therefore, the vector 11205 and scalar memory operations performed by wavefronts executing in different 11206 work-groups (which may be executing on different SAs) of an agent can be 11207 reordered relative to each other. A ``s_waitcnt vmcnt(0) & vscnt(0)`` is 11208 required to ensure synchronization between vector memory operations of 11209 different SAs. It ensures a previous vector memory operation has completed 11210 before executing a subsequent vector memory and so can be used to meet the 11211 requirements of acquire, release and sequential consistency. 11212* The L2 cache can be kept coherent with other agents on some targets, or ranges 11213 of virtual addresses can be set up to bypass it to ensure system coherence. 11214* On GFX10.3 a memory attached last level (MALL) cache exists for GPU memory. 11215 The MALL cache is fully coherent with GPU memory and has no impact on system 11216 coherence. All agents (GPU and CPU) access GPU memory through the MALL cache. 11217 11218Scalar memory operations are only used to access memory that is proven to not 11219change during the execution of the kernel dispatch. This includes constant 11220address space and global address space for program scope ``const`` variables. 11221Therefore, the kernel machine code does not have to maintain the scalar cache to 11222ensure it is coherent with the vector caches. The scalar and vector caches are 11223invalidated between kernel dispatches by CP since constant address space data 11224may change between kernel dispatch executions. See 11225:ref:`amdgpu-amdhsa-memory-spaces`. 11226 11227The one exception is if scalar writes are used to spill SGPR registers. In this 11228case the AMDGPU backend ensures the memory location used to spill is never 11229accessed by vector memory operations at the same time. If scalar writes are used 11230then a ``s_dcache_wb`` is inserted before the ``s_endpgm`` and before a function 11231return since the locations may be used for vector memory instructions by a 11232future wavefront that uses the same scratch area, or a function call that 11233creates a frame at the same address, respectively. There is no need for a 11234``s_dcache_inv`` as all scalar writes are write-before-read in the same thread. 11235 11236For kernarg backing memory: 11237 11238* CP invalidates the L0 and L1 caches at the start of each kernel dispatch. 11239* On dGPU the kernarg backing memory is accessed as MTYPE UC (uncached) to avoid 11240 needing to invalidate the L2 cache. 11241* On APU the kernarg backing memory is accessed as MTYPE CC (cache coherent) and 11242 so the L2 cache will be coherent with the CPU and other agents. 11243 11244Scratch backing memory (which is used for the private address space) is accessed 11245with MTYPE NC (non-coherent). Since the private address space is only accessed 11246by a single thread, and is always write-before-read, there is never a need to 11247invalidate these entries from the L0 or L1 caches. 11248 11249Wavefronts are executed in native mode with in-order reporting of loads and 11250sample instructions. In this mode vmcnt reports completion of load, atomic with 11251return and sample instructions in order, and the vscnt reports the completion of 11252store and atomic without return in order. See ``MEM_ORDERED`` field in 11253:ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 11254 11255Wavefronts can be executed in WGP or CU wavefront execution mode: 11256 11257* In WGP wavefront execution mode the wavefronts of a work-group are executed 11258 on the SIMDs of both CUs of the WGP. Therefore, explicit management of the per 11259 CU L0 caches is required for work-group synchronization. Also accesses to L1 11260 at work-group scope need to be explicitly ordered as the accesses from 11261 different CUs are not ordered. 11262* In CU wavefront execution mode the wavefronts of a work-group are executed on 11263 the SIMDs of a single CU of the WGP. Therefore, all global memory access by 11264 the work-group access the same L0 which in turn ensures L1 accesses are 11265 ordered and so do not require explicit management of the caches for 11266 work-group synchronization. 11267 11268See ``WGP_MODE`` field in 11269:ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table` and 11270:ref:`amdgpu-target-features`. 11271 11272The code sequences used to implement the memory model for GFX10 are defined in 11273table :ref:`amdgpu-amdhsa-memory-model-code-sequences-gfx10-table`. 11274 11275 .. table:: AMDHSA Memory Model Code Sequences GFX10 11276 :name: amdgpu-amdhsa-memory-model-code-sequences-gfx10-table 11277 11278 ============ ============ ============== ========== ================================ 11279 LLVM Instr LLVM Memory LLVM Memory AMDGPU AMDGPU Machine Code 11280 Ordering Sync Scope Address GFX10 11281 Space 11282 ============ ============ ============== ========== ================================ 11283 **Non-Atomic** 11284 ------------------------------------------------------------------------------------ 11285 load *none* *none* - global - !volatile & !nontemporal 11286 - generic 11287 - private 1. buffer/global/flat_load 11288 - constant 11289 - !volatile & nontemporal 11290 11291 1. buffer/global/flat_load 11292 slc=1 11293 11294 - volatile 11295 11296 1. buffer/global/flat_load 11297 glc=1 dlc=1 11298 2. s_waitcnt vmcnt(0) 11299 11300 - Must happen before 11301 any following volatile 11302 global/generic 11303 load/store. 11304 - Ensures that 11305 volatile 11306 operations to 11307 different 11308 addresses will not 11309 be reordered by 11310 hardware. 11311 11312 load *none* *none* - local 1. ds_load 11313 store *none* *none* - global - !volatile & !nontemporal 11314 - generic 11315 - private 1. buffer/global/flat_store 11316 - constant 11317 - !volatile & nontemporal 11318 11319 1. buffer/global/flat_store 11320 glc=1 slc=1 11321 11322 - volatile 11323 11324 1. buffer/global/flat_store 11325 2. s_waitcnt vscnt(0) 11326 11327 - Must happen before 11328 any following volatile 11329 global/generic 11330 load/store. 11331 - Ensures that 11332 volatile 11333 operations to 11334 different 11335 addresses will not 11336 be reordered by 11337 hardware. 11338 11339 store *none* *none* - local 1. ds_store 11340 **Unordered Atomic** 11341 ------------------------------------------------------------------------------------ 11342 load atomic unordered *any* *any* *Same as non-atomic*. 11343 store atomic unordered *any* *any* *Same as non-atomic*. 11344 atomicrmw unordered *any* *any* *Same as monotonic atomic*. 11345 **Monotonic Atomic** 11346 ------------------------------------------------------------------------------------ 11347 load atomic monotonic - singlethread - global 1. buffer/global/flat_load 11348 - wavefront - generic 11349 load atomic monotonic - workgroup - global 1. buffer/global/flat_load 11350 - generic glc=1 11351 11352 - If CU wavefront execution 11353 mode, omit glc=1. 11354 11355 load atomic monotonic - singlethread - local 1. ds_load 11356 - wavefront 11357 - workgroup 11358 load atomic monotonic - agent - global 1. buffer/global/flat_load 11359 - system - generic glc=1 dlc=1 11360 store atomic monotonic - singlethread - global 1. buffer/global/flat_store 11361 - wavefront - generic 11362 - workgroup 11363 - agent 11364 - system 11365 store atomic monotonic - singlethread - local 1. ds_store 11366 - wavefront 11367 - workgroup 11368 atomicrmw monotonic - singlethread - global 1. buffer/global/flat_atomic 11369 - wavefront - generic 11370 - workgroup 11371 - agent 11372 - system 11373 atomicrmw monotonic - singlethread - local 1. ds_atomic 11374 - wavefront 11375 - workgroup 11376 **Acquire Atomic** 11377 ------------------------------------------------------------------------------------ 11378 load atomic acquire - singlethread - global 1. buffer/global/ds/flat_load 11379 - wavefront - local 11380 - generic 11381 load atomic acquire - workgroup - global 1. buffer/global_load glc=1 11382 11383 - If CU wavefront execution 11384 mode, omit glc=1. 11385 11386 2. s_waitcnt vmcnt(0) 11387 11388 - If CU wavefront execution 11389 mode, omit. 11390 - Must happen before 11391 the following buffer_gl0_inv 11392 and before any following 11393 global/generic 11394 load/load 11395 atomic/store/store 11396 atomic/atomicrmw. 11397 11398 3. buffer_gl0_inv 11399 11400 - If CU wavefront execution 11401 mode, omit. 11402 - Ensures that 11403 following 11404 loads will not see 11405 stale data. 11406 11407 load atomic acquire - workgroup - local 1. ds_load 11408 2. s_waitcnt lgkmcnt(0) 11409 11410 - If OpenCL, omit. 11411 - Must happen before 11412 the following buffer_gl0_inv 11413 and before any following 11414 global/generic load/load 11415 atomic/store/store 11416 atomic/atomicrmw. 11417 - Ensures any 11418 following global 11419 data read is no 11420 older than the local load 11421 atomic value being 11422 acquired. 11423 11424 3. buffer_gl0_inv 11425 11426 - If CU wavefront execution 11427 mode, omit. 11428 - If OpenCL, omit. 11429 - Ensures that 11430 following 11431 loads will not see 11432 stale data. 11433 11434 load atomic acquire - workgroup - generic 1. flat_load glc=1 11435 11436 - If CU wavefront execution 11437 mode, omit glc=1. 11438 11439 2. s_waitcnt lgkmcnt(0) & 11440 vmcnt(0) 11441 11442 - If CU wavefront execution 11443 mode, omit vmcnt(0). 11444 - If OpenCL, omit 11445 lgkmcnt(0). 11446 - Must happen before 11447 the following 11448 buffer_gl0_inv and any 11449 following global/generic 11450 load/load 11451 atomic/store/store 11452 atomic/atomicrmw. 11453 - Ensures any 11454 following global 11455 data read is no 11456 older than a local load 11457 atomic value being 11458 acquired. 11459 11460 3. buffer_gl0_inv 11461 11462 - If CU wavefront execution 11463 mode, omit. 11464 - Ensures that 11465 following 11466 loads will not see 11467 stale data. 11468 11469 load atomic acquire - agent - global 1. buffer/global_load 11470 - system glc=1 dlc=1 11471 2. s_waitcnt vmcnt(0) 11472 11473 - Must happen before 11474 following 11475 buffer_gl*_inv. 11476 - Ensures the load 11477 has completed 11478 before invalidating 11479 the caches. 11480 11481 3. buffer_gl0_inv; 11482 buffer_gl1_inv 11483 11484 - Must happen before 11485 any following 11486 global/generic 11487 load/load 11488 atomic/atomicrmw. 11489 - Ensures that 11490 following 11491 loads will not see 11492 stale global data. 11493 11494 load atomic acquire - agent - generic 1. flat_load glc=1 dlc=1 11495 - system 2. s_waitcnt vmcnt(0) & 11496 lgkmcnt(0) 11497 11498 - If OpenCL omit 11499 lgkmcnt(0). 11500 - Must happen before 11501 following 11502 buffer_gl*_invl. 11503 - Ensures the flat_load 11504 has completed 11505 before invalidating 11506 the caches. 11507 11508 3. buffer_gl0_inv; 11509 buffer_gl1_inv 11510 11511 - Must happen before 11512 any following 11513 global/generic 11514 load/load 11515 atomic/atomicrmw. 11516 - Ensures that 11517 following loads 11518 will not see stale 11519 global data. 11520 11521 atomicrmw acquire - singlethread - global 1. buffer/global/ds/flat_atomic 11522 - wavefront - local 11523 - generic 11524 atomicrmw acquire - workgroup - global 1. buffer/global_atomic 11525 2. s_waitcnt vm/vscnt(0) 11526 11527 - If CU wavefront execution 11528 mode, omit. 11529 - Use vmcnt(0) if atomic with 11530 return and vscnt(0) if 11531 atomic with no-return. 11532 - Must happen before 11533 the following buffer_gl0_inv 11534 and before any following 11535 global/generic 11536 load/load 11537 atomic/store/store 11538 atomic/atomicrmw. 11539 11540 3. buffer_gl0_inv 11541 11542 - If CU wavefront execution 11543 mode, omit. 11544 - Ensures that 11545 following 11546 loads will not see 11547 stale data. 11548 11549 atomicrmw acquire - workgroup - local 1. ds_atomic 11550 2. s_waitcnt lgkmcnt(0) 11551 11552 - If OpenCL, omit. 11553 - Must happen before 11554 the following 11555 buffer_gl0_inv. 11556 - Ensures any 11557 following global 11558 data read is no 11559 older than the local 11560 atomicrmw value 11561 being acquired. 11562 11563 3. buffer_gl0_inv 11564 11565 - If OpenCL omit. 11566 - Ensures that 11567 following 11568 loads will not see 11569 stale data. 11570 11571 atomicrmw acquire - workgroup - generic 1. flat_atomic 11572 2. s_waitcnt lgkmcnt(0) & 11573 vm/vscnt(0) 11574 11575 - If CU wavefront execution 11576 mode, omit vm/vscnt(0). 11577 - If OpenCL, omit lgkmcnt(0). 11578 - Use vmcnt(0) if atomic with 11579 return and vscnt(0) if 11580 atomic with no-return. 11581 - Must happen before 11582 the following 11583 buffer_gl0_inv. 11584 - Ensures any 11585 following global 11586 data read is no 11587 older than a local 11588 atomicrmw value 11589 being acquired. 11590 11591 3. buffer_gl0_inv 11592 11593 - If CU wavefront execution 11594 mode, omit. 11595 - Ensures that 11596 following 11597 loads will not see 11598 stale data. 11599 11600 atomicrmw acquire - agent - global 1. buffer/global_atomic 11601 - system 2. s_waitcnt vm/vscnt(0) 11602 11603 - Use vmcnt(0) if atomic with 11604 return and vscnt(0) if 11605 atomic with no-return. 11606 - Must happen before 11607 following 11608 buffer_gl*_inv. 11609 - Ensures the 11610 atomicrmw has 11611 completed before 11612 invalidating the 11613 caches. 11614 11615 3. buffer_gl0_inv; 11616 buffer_gl1_inv 11617 11618 - Must happen before 11619 any following 11620 global/generic 11621 load/load 11622 atomic/atomicrmw. 11623 - Ensures that 11624 following loads 11625 will not see stale 11626 global data. 11627 11628 atomicrmw acquire - agent - generic 1. flat_atomic 11629 - system 2. s_waitcnt vm/vscnt(0) & 11630 lgkmcnt(0) 11631 11632 - If OpenCL, omit 11633 lgkmcnt(0). 11634 - Use vmcnt(0) if atomic with 11635 return and vscnt(0) if 11636 atomic with no-return. 11637 - Must happen before 11638 following 11639 buffer_gl*_inv. 11640 - Ensures the 11641 atomicrmw has 11642 completed before 11643 invalidating the 11644 caches. 11645 11646 3. buffer_gl0_inv; 11647 buffer_gl1_inv 11648 11649 - Must happen before 11650 any following 11651 global/generic 11652 load/load 11653 atomic/atomicrmw. 11654 - Ensures that 11655 following loads 11656 will not see stale 11657 global data. 11658 11659 fence acquire - singlethread *none* *none* 11660 - wavefront 11661 fence acquire - workgroup *none* 1. s_waitcnt lgkmcnt(0) & 11662 vmcnt(0) & vscnt(0) 11663 11664 - If CU wavefront execution 11665 mode, omit vmcnt(0) and 11666 vscnt(0). 11667 - If OpenCL and 11668 address space is 11669 not generic, omit 11670 lgkmcnt(0). 11671 - If OpenCL and 11672 address space is 11673 local, omit 11674 vmcnt(0) and vscnt(0). 11675 - However, since LLVM 11676 currently has no 11677 address space on 11678 the fence need to 11679 conservatively 11680 always generate. If 11681 fence had an 11682 address space then 11683 set to address 11684 space of OpenCL 11685 fence flag, or to 11686 generic if both 11687 local and global 11688 flags are 11689 specified. 11690 - Could be split into 11691 separate s_waitcnt 11692 vmcnt(0), s_waitcnt 11693 vscnt(0) and s_waitcnt 11694 lgkmcnt(0) to allow 11695 them to be 11696 independently moved 11697 according to the 11698 following rules. 11699 - s_waitcnt vmcnt(0) 11700 must happen after 11701 any preceding 11702 global/generic load 11703 atomic/ 11704 atomicrmw-with-return-value 11705 with an equal or 11706 wider sync scope 11707 and memory ordering 11708 stronger than 11709 unordered (this is 11710 termed the 11711 fence-paired-atomic). 11712 - s_waitcnt vscnt(0) 11713 must happen after 11714 any preceding 11715 global/generic 11716 atomicrmw-no-return-value 11717 with an equal or 11718 wider sync scope 11719 and memory ordering 11720 stronger than 11721 unordered (this is 11722 termed the 11723 fence-paired-atomic). 11724 - s_waitcnt lgkmcnt(0) 11725 must happen after 11726 any preceding 11727 local/generic load 11728 atomic/atomicrmw 11729 with an equal or 11730 wider sync scope 11731 and memory ordering 11732 stronger than 11733 unordered (this is 11734 termed the 11735 fence-paired-atomic). 11736 - Must happen before 11737 the following 11738 buffer_gl0_inv. 11739 - Ensures that the 11740 fence-paired atomic 11741 has completed 11742 before invalidating 11743 the 11744 cache. Therefore 11745 any following 11746 locations read must 11747 be no older than 11748 the value read by 11749 the 11750 fence-paired-atomic. 11751 11752 3. buffer_gl0_inv 11753 11754 - If CU wavefront execution 11755 mode, omit. 11756 - Ensures that 11757 following 11758 loads will not see 11759 stale data. 11760 11761 fence acquire - agent *none* 1. s_waitcnt lgkmcnt(0) & 11762 - system vmcnt(0) & vscnt(0) 11763 11764 - If OpenCL and 11765 address space is 11766 not generic, omit 11767 lgkmcnt(0). 11768 - If OpenCL and 11769 address space is 11770 local, omit 11771 vmcnt(0) and vscnt(0). 11772 - However, since LLVM 11773 currently has no 11774 address space on 11775 the fence need to 11776 conservatively 11777 always generate 11778 (see comment for 11779 previous fence). 11780 - Could be split into 11781 separate s_waitcnt 11782 vmcnt(0), s_waitcnt 11783 vscnt(0) and s_waitcnt 11784 lgkmcnt(0) to allow 11785 them to be 11786 independently moved 11787 according to the 11788 following rules. 11789 - s_waitcnt vmcnt(0) 11790 must happen after 11791 any preceding 11792 global/generic load 11793 atomic/ 11794 atomicrmw-with-return-value 11795 with an equal or 11796 wider sync scope 11797 and memory ordering 11798 stronger than 11799 unordered (this is 11800 termed the 11801 fence-paired-atomic). 11802 - s_waitcnt vscnt(0) 11803 must happen after 11804 any preceding 11805 global/generic 11806 atomicrmw-no-return-value 11807 with an equal or 11808 wider sync scope 11809 and memory ordering 11810 stronger than 11811 unordered (this is 11812 termed the 11813 fence-paired-atomic). 11814 - s_waitcnt lgkmcnt(0) 11815 must happen after 11816 any preceding 11817 local/generic load 11818 atomic/atomicrmw 11819 with an equal or 11820 wider sync scope 11821 and memory ordering 11822 stronger than 11823 unordered (this is 11824 termed the 11825 fence-paired-atomic). 11826 - Must happen before 11827 the following 11828 buffer_gl*_inv. 11829 - Ensures that the 11830 fence-paired atomic 11831 has completed 11832 before invalidating 11833 the 11834 caches. Therefore 11835 any following 11836 locations read must 11837 be no older than 11838 the value read by 11839 the 11840 fence-paired-atomic. 11841 11842 2. buffer_gl0_inv; 11843 buffer_gl1_inv 11844 11845 - Must happen before any 11846 following global/generic 11847 load/load 11848 atomic/store/store 11849 atomic/atomicrmw. 11850 - Ensures that 11851 following loads 11852 will not see stale 11853 global data. 11854 11855 **Release Atomic** 11856 ------------------------------------------------------------------------------------ 11857 store atomic release - singlethread - global 1. buffer/global/ds/flat_store 11858 - wavefront - local 11859 - generic 11860 store atomic release - workgroup - global 1. s_waitcnt lgkmcnt(0) & 11861 - generic vmcnt(0) & vscnt(0) 11862 11863 - If CU wavefront execution 11864 mode, omit vmcnt(0) and 11865 vscnt(0). 11866 - If OpenCL, omit 11867 lgkmcnt(0). 11868 - Could be split into 11869 separate s_waitcnt 11870 vmcnt(0), s_waitcnt 11871 vscnt(0) and s_waitcnt 11872 lgkmcnt(0) to allow 11873 them to be 11874 independently moved 11875 according to the 11876 following rules. 11877 - s_waitcnt vmcnt(0) 11878 must happen after 11879 any preceding 11880 global/generic load/load 11881 atomic/ 11882 atomicrmw-with-return-value. 11883 - s_waitcnt vscnt(0) 11884 must happen after 11885 any preceding 11886 global/generic 11887 store/store 11888 atomic/ 11889 atomicrmw-no-return-value. 11890 - s_waitcnt lgkmcnt(0) 11891 must happen after 11892 any preceding 11893 local/generic 11894 load/store/load 11895 atomic/store 11896 atomic/atomicrmw. 11897 - Must happen before 11898 the following 11899 store. 11900 - Ensures that all 11901 memory operations 11902 have 11903 completed before 11904 performing the 11905 store that is being 11906 released. 11907 11908 2. buffer/global/flat_store 11909 store atomic release - workgroup - local 1. s_waitcnt vmcnt(0) & vscnt(0) 11910 11911 - If CU wavefront execution 11912 mode, omit. 11913 - If OpenCL, omit. 11914 - Could be split into 11915 separate s_waitcnt 11916 vmcnt(0) and s_waitcnt 11917 vscnt(0) to allow 11918 them to be 11919 independently moved 11920 according to the 11921 following rules. 11922 - s_waitcnt vmcnt(0) 11923 must happen after 11924 any preceding 11925 global/generic load/load 11926 atomic/ 11927 atomicrmw-with-return-value. 11928 - s_waitcnt vscnt(0) 11929 must happen after 11930 any preceding 11931 global/generic 11932 store/store atomic/ 11933 atomicrmw-no-return-value. 11934 - Must happen before 11935 the following 11936 store. 11937 - Ensures that all 11938 global memory 11939 operations have 11940 completed before 11941 performing the 11942 store that is being 11943 released. 11944 11945 2. ds_store 11946 store atomic release - agent - global 1. s_waitcnt lgkmcnt(0) & 11947 - system - generic vmcnt(0) & vscnt(0) 11948 11949 - If OpenCL and 11950 address space is 11951 not generic, omit 11952 lgkmcnt(0). 11953 - Could be split into 11954 separate s_waitcnt 11955 vmcnt(0), s_waitcnt vscnt(0) 11956 and s_waitcnt 11957 lgkmcnt(0) to allow 11958 them to be 11959 independently moved 11960 according to the 11961 following rules. 11962 - s_waitcnt vmcnt(0) 11963 must happen after 11964 any preceding 11965 global/generic 11966 load/load 11967 atomic/ 11968 atomicrmw-with-return-value. 11969 - s_waitcnt vscnt(0) 11970 must happen after 11971 any preceding 11972 global/generic 11973 store/store atomic/ 11974 atomicrmw-no-return-value. 11975 - s_waitcnt lgkmcnt(0) 11976 must happen after 11977 any preceding 11978 local/generic 11979 load/store/load 11980 atomic/store 11981 atomic/atomicrmw. 11982 - Must happen before 11983 the following 11984 store. 11985 - Ensures that all 11986 memory operations 11987 have 11988 completed before 11989 performing the 11990 store that is being 11991 released. 11992 11993 2. buffer/global/flat_store 11994 atomicrmw release - singlethread - global 1. buffer/global/ds/flat_atomic 11995 - wavefront - local 11996 - generic 11997 atomicrmw release - workgroup - global 1. s_waitcnt lgkmcnt(0) & 11998 - generic vmcnt(0) & vscnt(0) 11999 12000 - If CU wavefront execution 12001 mode, omit vmcnt(0) and 12002 vscnt(0). 12003 - If OpenCL, omit lgkmcnt(0). 12004 - Could be split into 12005 separate s_waitcnt 12006 vmcnt(0), s_waitcnt 12007 vscnt(0) and s_waitcnt 12008 lgkmcnt(0) to allow 12009 them to be 12010 independently moved 12011 according to the 12012 following rules. 12013 - s_waitcnt vmcnt(0) 12014 must happen after 12015 any preceding 12016 global/generic load/load 12017 atomic/ 12018 atomicrmw-with-return-value. 12019 - s_waitcnt vscnt(0) 12020 must happen after 12021 any preceding 12022 global/generic 12023 store/store 12024 atomic/ 12025 atomicrmw-no-return-value. 12026 - s_waitcnt lgkmcnt(0) 12027 must happen after 12028 any preceding 12029 local/generic 12030 load/store/load 12031 atomic/store 12032 atomic/atomicrmw. 12033 - Must happen before 12034 the following 12035 atomicrmw. 12036 - Ensures that all 12037 memory operations 12038 have 12039 completed before 12040 performing the 12041 atomicrmw that is 12042 being released. 12043 12044 2. buffer/global/flat_atomic 12045 atomicrmw release - workgroup - local 1. s_waitcnt vmcnt(0) & vscnt(0) 12046 12047 - If CU wavefront execution 12048 mode, omit. 12049 - If OpenCL, omit. 12050 - Could be split into 12051 separate s_waitcnt 12052 vmcnt(0) and s_waitcnt 12053 vscnt(0) to allow 12054 them to be 12055 independently moved 12056 according to the 12057 following rules. 12058 - s_waitcnt vmcnt(0) 12059 must happen after 12060 any preceding 12061 global/generic load/load 12062 atomic/ 12063 atomicrmw-with-return-value. 12064 - s_waitcnt vscnt(0) 12065 must happen after 12066 any preceding 12067 global/generic 12068 store/store atomic/ 12069 atomicrmw-no-return-value. 12070 - Must happen before 12071 the following 12072 store. 12073 - Ensures that all 12074 global memory 12075 operations have 12076 completed before 12077 performing the 12078 store that is being 12079 released. 12080 12081 2. ds_atomic 12082 atomicrmw release - agent - global 1. s_waitcnt lgkmcnt(0) & 12083 - system - generic vmcnt(0) & vscnt(0) 12084 12085 - If OpenCL, omit 12086 lgkmcnt(0). 12087 - Could be split into 12088 separate s_waitcnt 12089 vmcnt(0), s_waitcnt 12090 vscnt(0) and s_waitcnt 12091 lgkmcnt(0) to allow 12092 them to be 12093 independently moved 12094 according to the 12095 following rules. 12096 - s_waitcnt vmcnt(0) 12097 must happen after 12098 any preceding 12099 global/generic 12100 load/load atomic/ 12101 atomicrmw-with-return-value. 12102 - s_waitcnt vscnt(0) 12103 must happen after 12104 any preceding 12105 global/generic 12106 store/store atomic/ 12107 atomicrmw-no-return-value. 12108 - s_waitcnt lgkmcnt(0) 12109 must happen after 12110 any preceding 12111 local/generic 12112 load/store/load 12113 atomic/store 12114 atomic/atomicrmw. 12115 - Must happen before 12116 the following 12117 atomicrmw. 12118 - Ensures that all 12119 memory operations 12120 to global and local 12121 have completed 12122 before performing 12123 the atomicrmw that 12124 is being released. 12125 12126 2. buffer/global/flat_atomic 12127 fence release - singlethread *none* *none* 12128 - wavefront 12129 fence release - workgroup *none* 1. s_waitcnt lgkmcnt(0) & 12130 vmcnt(0) & vscnt(0) 12131 12132 - If CU wavefront execution 12133 mode, omit vmcnt(0) and 12134 vscnt(0). 12135 - If OpenCL and 12136 address space is 12137 not generic, omit 12138 lgkmcnt(0). 12139 - If OpenCL and 12140 address space is 12141 local, omit 12142 vmcnt(0) and vscnt(0). 12143 - However, since LLVM 12144 currently has no 12145 address space on 12146 the fence need to 12147 conservatively 12148 always generate. If 12149 fence had an 12150 address space then 12151 set to address 12152 space of OpenCL 12153 fence flag, or to 12154 generic if both 12155 local and global 12156 flags are 12157 specified. 12158 - Could be split into 12159 separate s_waitcnt 12160 vmcnt(0), s_waitcnt 12161 vscnt(0) and s_waitcnt 12162 lgkmcnt(0) to allow 12163 them to be 12164 independently moved 12165 according to the 12166 following rules. 12167 - s_waitcnt vmcnt(0) 12168 must happen after 12169 any preceding 12170 global/generic 12171 load/load 12172 atomic/ 12173 atomicrmw-with-return-value. 12174 - s_waitcnt vscnt(0) 12175 must happen after 12176 any preceding 12177 global/generic 12178 store/store atomic/ 12179 atomicrmw-no-return-value. 12180 - s_waitcnt lgkmcnt(0) 12181 must happen after 12182 any preceding 12183 local/generic 12184 load/store/load 12185 atomic/store atomic/ 12186 atomicrmw. 12187 - Must happen before 12188 any following store 12189 atomic/atomicrmw 12190 with an equal or 12191 wider sync scope 12192 and memory ordering 12193 stronger than 12194 unordered (this is 12195 termed the 12196 fence-paired-atomic). 12197 - Ensures that all 12198 memory operations 12199 have 12200 completed before 12201 performing the 12202 following 12203 fence-paired-atomic. 12204 12205 fence release - agent *none* 1. s_waitcnt lgkmcnt(0) & 12206 - system vmcnt(0) & vscnt(0) 12207 12208 - If OpenCL and 12209 address space is 12210 not generic, omit 12211 lgkmcnt(0). 12212 - If OpenCL and 12213 address space is 12214 local, omit 12215 vmcnt(0) and vscnt(0). 12216 - However, since LLVM 12217 currently has no 12218 address space on 12219 the fence need to 12220 conservatively 12221 always generate. If 12222 fence had an 12223 address space then 12224 set to address 12225 space of OpenCL 12226 fence flag, or to 12227 generic if both 12228 local and global 12229 flags are 12230 specified. 12231 - Could be split into 12232 separate s_waitcnt 12233 vmcnt(0), s_waitcnt 12234 vscnt(0) and s_waitcnt 12235 lgkmcnt(0) to allow 12236 them to be 12237 independently moved 12238 according to the 12239 following rules. 12240 - s_waitcnt vmcnt(0) 12241 must happen after 12242 any preceding 12243 global/generic 12244 load/load atomic/ 12245 atomicrmw-with-return-value. 12246 - s_waitcnt vscnt(0) 12247 must happen after 12248 any preceding 12249 global/generic 12250 store/store atomic/ 12251 atomicrmw-no-return-value. 12252 - s_waitcnt lgkmcnt(0) 12253 must happen after 12254 any preceding 12255 local/generic 12256 load/store/load 12257 atomic/store 12258 atomic/atomicrmw. 12259 - Must happen before 12260 any following store 12261 atomic/atomicrmw 12262 with an equal or 12263 wider sync scope 12264 and memory ordering 12265 stronger than 12266 unordered (this is 12267 termed the 12268 fence-paired-atomic). 12269 - Ensures that all 12270 memory operations 12271 have 12272 completed before 12273 performing the 12274 following 12275 fence-paired-atomic. 12276 12277 **Acquire-Release Atomic** 12278 ------------------------------------------------------------------------------------ 12279 atomicrmw acq_rel - singlethread - global 1. buffer/global/ds/flat_atomic 12280 - wavefront - local 12281 - generic 12282 atomicrmw acq_rel - workgroup - global 1. s_waitcnt lgkmcnt(0) & 12283 vmcnt(0) & vscnt(0) 12284 12285 - If CU wavefront execution 12286 mode, omit vmcnt(0) and 12287 vscnt(0). 12288 - If OpenCL, omit 12289 lgkmcnt(0). 12290 - Must happen after 12291 any preceding 12292 local/generic 12293 load/store/load 12294 atomic/store 12295 atomic/atomicrmw. 12296 - Could be split into 12297 separate s_waitcnt 12298 vmcnt(0), s_waitcnt 12299 vscnt(0), and s_waitcnt 12300 lgkmcnt(0) to allow 12301 them to be 12302 independently moved 12303 according to the 12304 following rules. 12305 - s_waitcnt vmcnt(0) 12306 must happen after 12307 any preceding 12308 global/generic load/load 12309 atomic/ 12310 atomicrmw-with-return-value. 12311 - s_waitcnt vscnt(0) 12312 must happen after 12313 any preceding 12314 global/generic 12315 store/store 12316 atomic/ 12317 atomicrmw-no-return-value. 12318 - s_waitcnt lgkmcnt(0) 12319 must happen after 12320 any preceding 12321 local/generic 12322 load/store/load 12323 atomic/store 12324 atomic/atomicrmw. 12325 - Must happen before 12326 the following 12327 atomicrmw. 12328 - Ensures that all 12329 memory operations 12330 have 12331 completed before 12332 performing the 12333 atomicrmw that is 12334 being released. 12335 12336 2. buffer/global_atomic 12337 3. s_waitcnt vm/vscnt(0) 12338 12339 - If CU wavefront execution 12340 mode, omit. 12341 - Use vmcnt(0) if atomic with 12342 return and vscnt(0) if 12343 atomic with no-return. 12344 - Must happen before 12345 the following 12346 buffer_gl0_inv. 12347 - Ensures any 12348 following global 12349 data read is no 12350 older than the 12351 atomicrmw value 12352 being acquired. 12353 12354 4. buffer_gl0_inv 12355 12356 - If CU wavefront execution 12357 mode, omit. 12358 - Ensures that 12359 following 12360 loads will not see 12361 stale data. 12362 12363 atomicrmw acq_rel - workgroup - local 1. s_waitcnt vmcnt(0) & vscnt(0) 12364 12365 - If CU wavefront execution 12366 mode, omit. 12367 - If OpenCL, omit. 12368 - Could be split into 12369 separate s_waitcnt 12370 vmcnt(0) and s_waitcnt 12371 vscnt(0) to allow 12372 them to be 12373 independently moved 12374 according to the 12375 following rules. 12376 - s_waitcnt vmcnt(0) 12377 must happen after 12378 any preceding 12379 global/generic load/load 12380 atomic/ 12381 atomicrmw-with-return-value. 12382 - s_waitcnt vscnt(0) 12383 must happen after 12384 any preceding 12385 global/generic 12386 store/store atomic/ 12387 atomicrmw-no-return-value. 12388 - Must happen before 12389 the following 12390 store. 12391 - Ensures that all 12392 global memory 12393 operations have 12394 completed before 12395 performing the 12396 store that is being 12397 released. 12398 12399 2. ds_atomic 12400 3. s_waitcnt lgkmcnt(0) 12401 12402 - If OpenCL, omit. 12403 - Must happen before 12404 the following 12405 buffer_gl0_inv. 12406 - Ensures any 12407 following global 12408 data read is no 12409 older than the local load 12410 atomic value being 12411 acquired. 12412 12413 4. buffer_gl0_inv 12414 12415 - If CU wavefront execution 12416 mode, omit. 12417 - If OpenCL omit. 12418 - Ensures that 12419 following 12420 loads will not see 12421 stale data. 12422 12423 atomicrmw acq_rel - workgroup - generic 1. s_waitcnt lgkmcnt(0) & 12424 vmcnt(0) & vscnt(0) 12425 12426 - If CU wavefront execution 12427 mode, omit vmcnt(0) and 12428 vscnt(0). 12429 - If OpenCL, omit lgkmcnt(0). 12430 - Could be split into 12431 separate s_waitcnt 12432 vmcnt(0), s_waitcnt 12433 vscnt(0) and s_waitcnt 12434 lgkmcnt(0) to allow 12435 them to be 12436 independently moved 12437 according to the 12438 following rules. 12439 - s_waitcnt vmcnt(0) 12440 must happen after 12441 any preceding 12442 global/generic load/load 12443 atomic/ 12444 atomicrmw-with-return-value. 12445 - s_waitcnt vscnt(0) 12446 must happen after 12447 any preceding 12448 global/generic 12449 store/store 12450 atomic/ 12451 atomicrmw-no-return-value. 12452 - s_waitcnt lgkmcnt(0) 12453 must happen after 12454 any preceding 12455 local/generic 12456 load/store/load 12457 atomic/store 12458 atomic/atomicrmw. 12459 - Must happen before 12460 the following 12461 atomicrmw. 12462 - Ensures that all 12463 memory operations 12464 have 12465 completed before 12466 performing the 12467 atomicrmw that is 12468 being released. 12469 12470 2. flat_atomic 12471 3. s_waitcnt lgkmcnt(0) & 12472 vmcnt(0) & vscnt(0) 12473 12474 - If CU wavefront execution 12475 mode, omit vmcnt(0) and 12476 vscnt(0). 12477 - If OpenCL, omit lgkmcnt(0). 12478 - Must happen before 12479 the following 12480 buffer_gl0_inv. 12481 - Ensures any 12482 following global 12483 data read is no 12484 older than the load 12485 atomic value being 12486 acquired. 12487 12488 3. buffer_gl0_inv 12489 12490 - If CU wavefront execution 12491 mode, omit. 12492 - Ensures that 12493 following 12494 loads will not see 12495 stale data. 12496 12497 atomicrmw acq_rel - agent - global 1. s_waitcnt lgkmcnt(0) & 12498 - system vmcnt(0) & vscnt(0) 12499 12500 - If OpenCL, omit 12501 lgkmcnt(0). 12502 - Could be split into 12503 separate s_waitcnt 12504 vmcnt(0), s_waitcnt 12505 vscnt(0) and s_waitcnt 12506 lgkmcnt(0) to allow 12507 them to be 12508 independently moved 12509 according to the 12510 following rules. 12511 - s_waitcnt vmcnt(0) 12512 must happen after 12513 any preceding 12514 global/generic 12515 load/load atomic/ 12516 atomicrmw-with-return-value. 12517 - s_waitcnt vscnt(0) 12518 must happen after 12519 any preceding 12520 global/generic 12521 store/store atomic/ 12522 atomicrmw-no-return-value. 12523 - s_waitcnt lgkmcnt(0) 12524 must happen after 12525 any preceding 12526 local/generic 12527 load/store/load 12528 atomic/store 12529 atomic/atomicrmw. 12530 - Must happen before 12531 the following 12532 atomicrmw. 12533 - Ensures that all 12534 memory operations 12535 to global have 12536 completed before 12537 performing the 12538 atomicrmw that is 12539 being released. 12540 12541 2. buffer/global_atomic 12542 3. s_waitcnt vm/vscnt(0) 12543 12544 - Use vmcnt(0) if atomic with 12545 return and vscnt(0) if 12546 atomic with no-return. 12547 - Must happen before 12548 following 12549 buffer_gl*_inv. 12550 - Ensures the 12551 atomicrmw has 12552 completed before 12553 invalidating the 12554 caches. 12555 12556 4. buffer_gl0_inv; 12557 buffer_gl1_inv 12558 12559 - Must happen before 12560 any following 12561 global/generic 12562 load/load 12563 atomic/atomicrmw. 12564 - Ensures that 12565 following loads 12566 will not see stale 12567 global data. 12568 12569 atomicrmw acq_rel - agent - generic 1. s_waitcnt lgkmcnt(0) & 12570 - system vmcnt(0) & vscnt(0) 12571 12572 - If OpenCL, omit 12573 lgkmcnt(0). 12574 - Could be split into 12575 separate s_waitcnt 12576 vmcnt(0), s_waitcnt 12577 vscnt(0), and s_waitcnt 12578 lgkmcnt(0) to allow 12579 them to be 12580 independently moved 12581 according to the 12582 following rules. 12583 - s_waitcnt vmcnt(0) 12584 must happen after 12585 any preceding 12586 global/generic 12587 load/load atomic 12588 atomicrmw-with-return-value. 12589 - s_waitcnt vscnt(0) 12590 must happen after 12591 any preceding 12592 global/generic 12593 store/store atomic/ 12594 atomicrmw-no-return-value. 12595 - s_waitcnt lgkmcnt(0) 12596 must happen after 12597 any preceding 12598 local/generic 12599 load/store/load 12600 atomic/store 12601 atomic/atomicrmw. 12602 - Must happen before 12603 the following 12604 atomicrmw. 12605 - Ensures that all 12606 memory operations 12607 have 12608 completed before 12609 performing the 12610 atomicrmw that is 12611 being released. 12612 12613 2. flat_atomic 12614 3. s_waitcnt vm/vscnt(0) & 12615 lgkmcnt(0) 12616 12617 - If OpenCL, omit 12618 lgkmcnt(0). 12619 - Use vmcnt(0) if atomic with 12620 return and vscnt(0) if 12621 atomic with no-return. 12622 - Must happen before 12623 following 12624 buffer_gl*_inv. 12625 - Ensures the 12626 atomicrmw has 12627 completed before 12628 invalidating the 12629 caches. 12630 12631 4. buffer_gl0_inv; 12632 buffer_gl1_inv 12633 12634 - Must happen before 12635 any following 12636 global/generic 12637 load/load 12638 atomic/atomicrmw. 12639 - Ensures that 12640 following loads 12641 will not see stale 12642 global data. 12643 12644 fence acq_rel - singlethread *none* *none* 12645 - wavefront 12646 fence acq_rel - workgroup *none* 1. s_waitcnt lgkmcnt(0) & 12647 vmcnt(0) & vscnt(0) 12648 12649 - If CU wavefront execution 12650 mode, omit vmcnt(0) and 12651 vscnt(0). 12652 - If OpenCL and 12653 address space is 12654 not generic, omit 12655 lgkmcnt(0). 12656 - If OpenCL and 12657 address space is 12658 local, omit 12659 vmcnt(0) and vscnt(0). 12660 - However, 12661 since LLVM 12662 currently has no 12663 address space on 12664 the fence need to 12665 conservatively 12666 always generate 12667 (see comment for 12668 previous fence). 12669 - Could be split into 12670 separate s_waitcnt 12671 vmcnt(0), s_waitcnt 12672 vscnt(0) and s_waitcnt 12673 lgkmcnt(0) to allow 12674 them to be 12675 independently moved 12676 according to the 12677 following rules. 12678 - s_waitcnt vmcnt(0) 12679 must happen after 12680 any preceding 12681 global/generic 12682 load/load 12683 atomic/ 12684 atomicrmw-with-return-value. 12685 - s_waitcnt vscnt(0) 12686 must happen after 12687 any preceding 12688 global/generic 12689 store/store atomic/ 12690 atomicrmw-no-return-value. 12691 - s_waitcnt lgkmcnt(0) 12692 must happen after 12693 any preceding 12694 local/generic 12695 load/store/load 12696 atomic/store atomic/ 12697 atomicrmw. 12698 - Must happen before 12699 any following 12700 global/generic 12701 load/load 12702 atomic/store/store 12703 atomic/atomicrmw. 12704 - Ensures that all 12705 memory operations 12706 have 12707 completed before 12708 performing any 12709 following global 12710 memory operations. 12711 - Ensures that the 12712 preceding 12713 local/generic load 12714 atomic/atomicrmw 12715 with an equal or 12716 wider sync scope 12717 and memory ordering 12718 stronger than 12719 unordered (this is 12720 termed the 12721 acquire-fence-paired-atomic) 12722 has completed 12723 before following 12724 global memory 12725 operations. This 12726 satisfies the 12727 requirements of 12728 acquire. 12729 - Ensures that all 12730 previous memory 12731 operations have 12732 completed before a 12733 following 12734 local/generic store 12735 atomic/atomicrmw 12736 with an equal or 12737 wider sync scope 12738 and memory ordering 12739 stronger than 12740 unordered (this is 12741 termed the 12742 release-fence-paired-atomic). 12743 This satisfies the 12744 requirements of 12745 release. 12746 - Must happen before 12747 the following 12748 buffer_gl0_inv. 12749 - Ensures that the 12750 acquire-fence-paired 12751 atomic has completed 12752 before invalidating 12753 the 12754 cache. Therefore 12755 any following 12756 locations read must 12757 be no older than 12758 the value read by 12759 the 12760 acquire-fence-paired-atomic. 12761 12762 3. buffer_gl0_inv 12763 12764 - If CU wavefront execution 12765 mode, omit. 12766 - Ensures that 12767 following 12768 loads will not see 12769 stale data. 12770 12771 fence acq_rel - agent *none* 1. s_waitcnt lgkmcnt(0) & 12772 - system vmcnt(0) & vscnt(0) 12773 12774 - If OpenCL and 12775 address space is 12776 not generic, omit 12777 lgkmcnt(0). 12778 - If OpenCL and 12779 address space is 12780 local, omit 12781 vmcnt(0) and vscnt(0). 12782 - However, since LLVM 12783 currently has no 12784 address space on 12785 the fence need to 12786 conservatively 12787 always generate 12788 (see comment for 12789 previous fence). 12790 - Could be split into 12791 separate s_waitcnt 12792 vmcnt(0), s_waitcnt 12793 vscnt(0) and s_waitcnt 12794 lgkmcnt(0) to allow 12795 them to be 12796 independently moved 12797 according to the 12798 following rules. 12799 - s_waitcnt vmcnt(0) 12800 must happen after 12801 any preceding 12802 global/generic 12803 load/load 12804 atomic/ 12805 atomicrmw-with-return-value. 12806 - s_waitcnt vscnt(0) 12807 must happen after 12808 any preceding 12809 global/generic 12810 store/store atomic/ 12811 atomicrmw-no-return-value. 12812 - s_waitcnt lgkmcnt(0) 12813 must happen after 12814 any preceding 12815 local/generic 12816 load/store/load 12817 atomic/store 12818 atomic/atomicrmw. 12819 - Must happen before 12820 the following 12821 buffer_gl*_inv. 12822 - Ensures that the 12823 preceding 12824 global/local/generic 12825 load 12826 atomic/atomicrmw 12827 with an equal or 12828 wider sync scope 12829 and memory ordering 12830 stronger than 12831 unordered (this is 12832 termed the 12833 acquire-fence-paired-atomic) 12834 has completed 12835 before invalidating 12836 the caches. This 12837 satisfies the 12838 requirements of 12839 acquire. 12840 - Ensures that all 12841 previous memory 12842 operations have 12843 completed before a 12844 following 12845 global/local/generic 12846 store 12847 atomic/atomicrmw 12848 with an equal or 12849 wider sync scope 12850 and memory ordering 12851 stronger than 12852 unordered (this is 12853 termed the 12854 release-fence-paired-atomic). 12855 This satisfies the 12856 requirements of 12857 release. 12858 12859 2. buffer_gl0_inv; 12860 buffer_gl1_inv 12861 12862 - Must happen before 12863 any following 12864 global/generic 12865 load/load 12866 atomic/store/store 12867 atomic/atomicrmw. 12868 - Ensures that 12869 following loads 12870 will not see stale 12871 global data. This 12872 satisfies the 12873 requirements of 12874 acquire. 12875 12876 **Sequential Consistent Atomic** 12877 ------------------------------------------------------------------------------------ 12878 load atomic seq_cst - singlethread - global *Same as corresponding 12879 - wavefront - local load atomic acquire, 12880 - generic except must generate 12881 all instructions even 12882 for OpenCL.* 12883 load atomic seq_cst - workgroup - global 1. s_waitcnt lgkmcnt(0) & 12884 - generic vmcnt(0) & vscnt(0) 12885 12886 - If CU wavefront execution 12887 mode, omit vmcnt(0) and 12888 vscnt(0). 12889 - Could be split into 12890 separate s_waitcnt 12891 vmcnt(0), s_waitcnt 12892 vscnt(0), and s_waitcnt 12893 lgkmcnt(0) to allow 12894 them to be 12895 independently moved 12896 according to the 12897 following rules. 12898 - s_waitcnt lgkmcnt(0) must 12899 happen after 12900 preceding 12901 local/generic load 12902 atomic/store 12903 atomic/atomicrmw 12904 with memory 12905 ordering of seq_cst 12906 and with equal or 12907 wider sync scope. 12908 (Note that seq_cst 12909 fences have their 12910 own s_waitcnt 12911 lgkmcnt(0) and so do 12912 not need to be 12913 considered.) 12914 - s_waitcnt vmcnt(0) 12915 must happen after 12916 preceding 12917 global/generic load 12918 atomic/ 12919 atomicrmw-with-return-value 12920 with memory 12921 ordering of seq_cst 12922 and with equal or 12923 wider sync scope. 12924 (Note that seq_cst 12925 fences have their 12926 own s_waitcnt 12927 vmcnt(0) and so do 12928 not need to be 12929 considered.) 12930 - s_waitcnt vscnt(0) 12931 Must happen after 12932 preceding 12933 global/generic store 12934 atomic/ 12935 atomicrmw-no-return-value 12936 with memory 12937 ordering of seq_cst 12938 and with equal or 12939 wider sync scope. 12940 (Note that seq_cst 12941 fences have their 12942 own s_waitcnt 12943 vscnt(0) and so do 12944 not need to be 12945 considered.) 12946 - Ensures any 12947 preceding 12948 sequential 12949 consistent global/local 12950 memory instructions 12951 have completed 12952 before executing 12953 this sequentially 12954 consistent 12955 instruction. This 12956 prevents reordering 12957 a seq_cst store 12958 followed by a 12959 seq_cst load. (Note 12960 that seq_cst is 12961 stronger than 12962 acquire/release as 12963 the reordering of 12964 load acquire 12965 followed by a store 12966 release is 12967 prevented by the 12968 s_waitcnt of 12969 the release, but 12970 there is nothing 12971 preventing a store 12972 release followed by 12973 load acquire from 12974 completing out of 12975 order. The s_waitcnt 12976 could be placed after 12977 seq_store or before 12978 the seq_load. We 12979 choose the load to 12980 make the s_waitcnt be 12981 as late as possible 12982 so that the store 12983 may have already 12984 completed.) 12985 12986 2. *Following 12987 instructions same as 12988 corresponding load 12989 atomic acquire, 12990 except must generate 12991 all instructions even 12992 for OpenCL.* 12993 load atomic seq_cst - workgroup - local 12994 12995 1. s_waitcnt vmcnt(0) & vscnt(0) 12996 12997 - If CU wavefront execution 12998 mode, omit. 12999 - Could be split into 13000 separate s_waitcnt 13001 vmcnt(0) and s_waitcnt 13002 vscnt(0) to allow 13003 them to be 13004 independently moved 13005 according to the 13006 following rules. 13007 - s_waitcnt vmcnt(0) 13008 Must happen after 13009 preceding 13010 global/generic load 13011 atomic/ 13012 atomicrmw-with-return-value 13013 with memory 13014 ordering of seq_cst 13015 and with equal or 13016 wider sync scope. 13017 (Note that seq_cst 13018 fences have their 13019 own s_waitcnt 13020 vmcnt(0) and so do 13021 not need to be 13022 considered.) 13023 - s_waitcnt vscnt(0) 13024 Must happen after 13025 preceding 13026 global/generic store 13027 atomic/ 13028 atomicrmw-no-return-value 13029 with memory 13030 ordering of seq_cst 13031 and with equal or 13032 wider sync scope. 13033 (Note that seq_cst 13034 fences have their 13035 own s_waitcnt 13036 vscnt(0) and so do 13037 not need to be 13038 considered.) 13039 - Ensures any 13040 preceding 13041 sequential 13042 consistent global 13043 memory instructions 13044 have completed 13045 before executing 13046 this sequentially 13047 consistent 13048 instruction. This 13049 prevents reordering 13050 a seq_cst store 13051 followed by a 13052 seq_cst load. (Note 13053 that seq_cst is 13054 stronger than 13055 acquire/release as 13056 the reordering of 13057 load acquire 13058 followed by a store 13059 release is 13060 prevented by the 13061 s_waitcnt of 13062 the release, but 13063 there is nothing 13064 preventing a store 13065 release followed by 13066 load acquire from 13067 completing out of 13068 order. The s_waitcnt 13069 could be placed after 13070 seq_store or before 13071 the seq_load. We 13072 choose the load to 13073 make the s_waitcnt be 13074 as late as possible 13075 so that the store 13076 may have already 13077 completed.) 13078 13079 2. *Following 13080 instructions same as 13081 corresponding load 13082 atomic acquire, 13083 except must generate 13084 all instructions even 13085 for OpenCL.* 13086 13087 load atomic seq_cst - agent - global 1. s_waitcnt lgkmcnt(0) & 13088 - system - generic vmcnt(0) & vscnt(0) 13089 13090 - Could be split into 13091 separate s_waitcnt 13092 vmcnt(0), s_waitcnt 13093 vscnt(0) and s_waitcnt 13094 lgkmcnt(0) to allow 13095 them to be 13096 independently moved 13097 according to the 13098 following rules. 13099 - s_waitcnt lgkmcnt(0) 13100 must happen after 13101 preceding 13102 local load 13103 atomic/store 13104 atomic/atomicrmw 13105 with memory 13106 ordering of seq_cst 13107 and with equal or 13108 wider sync scope. 13109 (Note that seq_cst 13110 fences have their 13111 own s_waitcnt 13112 lgkmcnt(0) and so do 13113 not need to be 13114 considered.) 13115 - s_waitcnt vmcnt(0) 13116 must happen after 13117 preceding 13118 global/generic load 13119 atomic/ 13120 atomicrmw-with-return-value 13121 with memory 13122 ordering of seq_cst 13123 and with equal or 13124 wider sync scope. 13125 (Note that seq_cst 13126 fences have their 13127 own s_waitcnt 13128 vmcnt(0) and so do 13129 not need to be 13130 considered.) 13131 - s_waitcnt vscnt(0) 13132 Must happen after 13133 preceding 13134 global/generic store 13135 atomic/ 13136 atomicrmw-no-return-value 13137 with memory 13138 ordering of seq_cst 13139 and with equal or 13140 wider sync scope. 13141 (Note that seq_cst 13142 fences have their 13143 own s_waitcnt 13144 vscnt(0) and so do 13145 not need to be 13146 considered.) 13147 - Ensures any 13148 preceding 13149 sequential 13150 consistent global 13151 memory instructions 13152 have completed 13153 before executing 13154 this sequentially 13155 consistent 13156 instruction. This 13157 prevents reordering 13158 a seq_cst store 13159 followed by a 13160 seq_cst load. (Note 13161 that seq_cst is 13162 stronger than 13163 acquire/release as 13164 the reordering of 13165 load acquire 13166 followed by a store 13167 release is 13168 prevented by the 13169 s_waitcnt of 13170 the release, but 13171 there is nothing 13172 preventing a store 13173 release followed by 13174 load acquire from 13175 completing out of 13176 order. The s_waitcnt 13177 could be placed after 13178 seq_store or before 13179 the seq_load. We 13180 choose the load to 13181 make the s_waitcnt be 13182 as late as possible 13183 so that the store 13184 may have already 13185 completed.) 13186 13187 2. *Following 13188 instructions same as 13189 corresponding load 13190 atomic acquire, 13191 except must generate 13192 all instructions even 13193 for OpenCL.* 13194 store atomic seq_cst - singlethread - global *Same as corresponding 13195 - wavefront - local store atomic release, 13196 - workgroup - generic except must generate 13197 - agent all instructions even 13198 - system for OpenCL.* 13199 atomicrmw seq_cst - singlethread - global *Same as corresponding 13200 - wavefront - local atomicrmw acq_rel, 13201 - workgroup - generic except must generate 13202 - agent all instructions even 13203 - system for OpenCL.* 13204 fence seq_cst - singlethread *none* *Same as corresponding 13205 - wavefront fence acq_rel, 13206 - workgroup except must generate 13207 - agent all instructions even 13208 - system for OpenCL.* 13209 ============ ============ ============== ========== ================================ 13210 13211.. _amdgpu-amdhsa-trap-handler-abi: 13212 13213Trap Handler ABI 13214~~~~~~~~~~~~~~~~ 13215 13216For code objects generated by the AMDGPU backend for HSA [HSA]_ compatible 13217runtimes (see :ref:`amdgpu-os`), the runtime installs a trap handler that 13218supports the ``s_trap`` instruction. For usage see: 13219 13220- :ref:`amdgpu-trap-handler-for-amdhsa-os-v2-table` 13221- :ref:`amdgpu-trap-handler-for-amdhsa-os-v3-table` 13222- :ref:`amdgpu-trap-handler-for-amdhsa-os-v4-onwards-table` 13223 13224 .. table:: AMDGPU Trap Handler for AMDHSA OS Code Object V2 13225 :name: amdgpu-trap-handler-for-amdhsa-os-v2-table 13226 13227 =================== =============== =============== ======================================= 13228 Usage Code Sequence Trap Handler Description 13229 Inputs 13230 =================== =============== =============== ======================================= 13231 reserved ``s_trap 0x00`` Reserved by hardware. 13232 ``debugtrap(arg)`` ``s_trap 0x01`` ``SGPR0-1``: Reserved for Finalizer HSA ``debugtrap`` 13233 ``queue_ptr`` intrinsic (not implemented). 13234 ``VGPR0``: 13235 ``arg`` 13236 ``llvm.trap`` ``s_trap 0x02`` ``SGPR0-1``: Causes wave to be halted with the PC at 13237 ``queue_ptr`` the trap instruction. The associated 13238 queue is signalled to put it into the 13239 error state. When the queue is put in 13240 the error state, the waves executing 13241 dispatches on the queue will be 13242 terminated. 13243 ``llvm.debugtrap`` ``s_trap 0x03`` *none* - If debugger not enabled then behaves 13244 as a no-operation. The trap handler 13245 is entered and immediately returns to 13246 continue execution of the wavefront. 13247 - If the debugger is enabled, causes 13248 the debug trap to be reported by the 13249 debugger and the wavefront is put in 13250 the halt state with the PC at the 13251 instruction. The debugger must 13252 increment the PC and resume the wave. 13253 reserved ``s_trap 0x04`` Reserved. 13254 reserved ``s_trap 0x05`` Reserved. 13255 reserved ``s_trap 0x06`` Reserved. 13256 reserved ``s_trap 0x07`` Reserved. 13257 reserved ``s_trap 0x08`` Reserved. 13258 reserved ``s_trap 0xfe`` Reserved. 13259 reserved ``s_trap 0xff`` Reserved. 13260 =================== =============== =============== ======================================= 13261 13262.. 13263 13264 .. table:: AMDGPU Trap Handler for AMDHSA OS Code Object V3 13265 :name: amdgpu-trap-handler-for-amdhsa-os-v3-table 13266 13267 =================== =============== =============== ======================================= 13268 Usage Code Sequence Trap Handler Description 13269 Inputs 13270 =================== =============== =============== ======================================= 13271 reserved ``s_trap 0x00`` Reserved by hardware. 13272 debugger breakpoint ``s_trap 0x01`` *none* Reserved for debugger to use for 13273 breakpoints. Causes wave to be halted 13274 with the PC at the trap instruction. 13275 The debugger is responsible to resume 13276 the wave, including the instruction 13277 that the breakpoint overwrote. 13278 ``llvm.trap`` ``s_trap 0x02`` ``SGPR0-1``: Causes wave to be halted with the PC at 13279 ``queue_ptr`` the trap instruction. The associated 13280 queue is signalled to put it into the 13281 error state. When the queue is put in 13282 the error state, the waves executing 13283 dispatches on the queue will be 13284 terminated. 13285 ``llvm.debugtrap`` ``s_trap 0x03`` *none* - If debugger not enabled then behaves 13286 as a no-operation. The trap handler 13287 is entered and immediately returns to 13288 continue execution of the wavefront. 13289 - If the debugger is enabled, causes 13290 the debug trap to be reported by the 13291 debugger and the wavefront is put in 13292 the halt state with the PC at the 13293 instruction. The debugger must 13294 increment the PC and resume the wave. 13295 reserved ``s_trap 0x04`` Reserved. 13296 reserved ``s_trap 0x05`` Reserved. 13297 reserved ``s_trap 0x06`` Reserved. 13298 reserved ``s_trap 0x07`` Reserved. 13299 reserved ``s_trap 0x08`` Reserved. 13300 reserved ``s_trap 0xfe`` Reserved. 13301 reserved ``s_trap 0xff`` Reserved. 13302 =================== =============== =============== ======================================= 13303 13304.. 13305 13306 .. table:: AMDGPU Trap Handler for AMDHSA OS Code Object V4 and Above 13307 :name: amdgpu-trap-handler-for-amdhsa-os-v4-onwards-table 13308 13309 =================== =============== ================ ================= ======================================= 13310 Usage Code Sequence GFX6-GFX8 Inputs GFX9-GFX10 Inputs Description 13311 =================== =============== ================ ================= ======================================= 13312 reserved ``s_trap 0x00`` Reserved by hardware. 13313 debugger breakpoint ``s_trap 0x01`` *none* *none* Reserved for debugger to use for 13314 breakpoints. Causes wave to be halted 13315 with the PC at the trap instruction. 13316 The debugger is responsible to resume 13317 the wave, including the instruction 13318 that the breakpoint overwrote. 13319 ``llvm.trap`` ``s_trap 0x02`` ``SGPR0-1``: *none* Causes wave to be halted with the PC at 13320 ``queue_ptr`` the trap instruction. The associated 13321 queue is signalled to put it into the 13322 error state. When the queue is put in 13323 the error state, the waves executing 13324 dispatches on the queue will be 13325 terminated. 13326 ``llvm.debugtrap`` ``s_trap 0x03`` *none* *none* - If debugger not enabled then behaves 13327 as a no-operation. The trap handler 13328 is entered and immediately returns to 13329 continue execution of the wavefront. 13330 - If the debugger is enabled, causes 13331 the debug trap to be reported by the 13332 debugger and the wavefront is put in 13333 the halt state with the PC at the 13334 instruction. The debugger must 13335 increment the PC and resume the wave. 13336 reserved ``s_trap 0x04`` Reserved. 13337 reserved ``s_trap 0x05`` Reserved. 13338 reserved ``s_trap 0x06`` Reserved. 13339 reserved ``s_trap 0x07`` Reserved. 13340 reserved ``s_trap 0x08`` Reserved. 13341 reserved ``s_trap 0xfe`` Reserved. 13342 reserved ``s_trap 0xff`` Reserved. 13343 =================== =============== ================ ================= ======================================= 13344 13345.. _amdgpu-amdhsa-function-call-convention: 13346 13347Call Convention 13348~~~~~~~~~~~~~~~ 13349 13350.. note:: 13351 13352 This section is currently incomplete and has inaccuracies. It is WIP that will 13353 be updated as information is determined. 13354 13355See :ref:`amdgpu-dwarf-address-space-identifier` for information on swizzled 13356addresses. Unswizzled addresses are normal linear addresses. 13357 13358.. _amdgpu-amdhsa-function-call-convention-kernel-functions: 13359 13360Kernel Functions 13361++++++++++++++++ 13362 13363This section describes the call convention ABI for the outer kernel function. 13364 13365See :ref:`amdgpu-amdhsa-initial-kernel-execution-state` for the kernel call 13366convention. 13367 13368The following is not part of the AMDGPU kernel calling convention but describes 13369how the AMDGPU implements function calls: 13370 133711. Clang decides the kernarg layout to match the *HSA Programmer's Language 13372 Reference* [HSA]_. 13373 13374 - All structs are passed directly. 13375 - Lambda values are passed *TBA*. 13376 13377 .. TODO:: 13378 13379 - Does this really follow HSA rules? Or are structs >16 bytes passed 13380 by-value struct? 13381 - What is ABI for lambda values? 13382 133834. The kernel performs certain setup in its prolog, as described in 13384 :ref:`amdgpu-amdhsa-kernel-prolog`. 13385 13386.. _amdgpu-amdhsa-function-call-convention-non-kernel-functions: 13387 13388Non-Kernel Functions 13389++++++++++++++++++++ 13390 13391This section describes the call convention ABI for functions other than the 13392outer kernel function. 13393 13394If a kernel has function calls then scratch is always allocated and used for 13395the call stack which grows from low address to high address using the swizzled 13396scratch address space. 13397 13398On entry to a function: 13399 134001. SGPR0-3 contain a V# with the following properties (see 13401 :ref:`amdgpu-amdhsa-kernel-prolog-private-segment-buffer`): 13402 13403 * Base address pointing to the beginning of the wavefront scratch backing 13404 memory. 13405 * Swizzled with dword element size and stride of wavefront size elements. 13406 134072. The FLAT_SCRATCH register pair is setup. See 13408 :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`. 134093. GFX6-GFX8: M0 register set to the size of LDS in bytes. See 13410 :ref:`amdgpu-amdhsa-kernel-prolog-m0`. 134114. The EXEC register is set to the lanes active on entry to the function. 134125. MODE register: *TBD* 134136. VGPR0-31 and SGPR4-29 are used to pass function input arguments as described 13414 below. 134157. SGPR30-31 return address (RA). The code address that the function must 13416 return to when it completes. The value is undefined if the function is *no 13417 return*. 134188. SGPR32 is used for the stack pointer (SP). It is an unswizzled scratch 13419 offset relative to the beginning of the wavefront scratch backing memory. 13420 13421 The unswizzled SP can be used with buffer instructions as an unswizzled SGPR 13422 offset with the scratch V# in SGPR0-3 to access the stack in a swizzled 13423 manner. 13424 13425 The unswizzled SP value can be converted into the swizzled SP value by: 13426 13427 | swizzled SP = unswizzled SP / wavefront size 13428 13429 This may be used to obtain the private address space address of stack 13430 objects and to convert this address to a flat address by adding the flat 13431 scratch aperture base address. 13432 13433 The swizzled SP value is always 4 bytes aligned for the ``r600`` 13434 architecture and 16 byte aligned for the ``amdgcn`` architecture. 13435 13436 .. note:: 13437 13438 The ``amdgcn`` value is selected to avoid dynamic stack alignment for the 13439 OpenCL language which has the largest base type defined as 16 bytes. 13440 13441 On entry, the swizzled SP value is the address of the first function 13442 argument passed on the stack. Other stack passed arguments are positive 13443 offsets from the entry swizzled SP value. 13444 13445 The function may use positive offsets beyond the last stack passed argument 13446 for stack allocated local variables and register spill slots. If necessary, 13447 the function may align these to greater alignment than 16 bytes. After these 13448 the function may dynamically allocate space for such things as runtime sized 13449 ``alloca`` local allocations. 13450 13451 If the function calls another function, it will place any stack allocated 13452 arguments after the last local allocation and adjust SGPR32 to the address 13453 after the last local allocation. 13454 134559. All other registers are unspecified. 1345610. Any necessary ``s_waitcnt`` has been performed to ensure memory is available 13457 to the function. 13458 13459On exit from a function: 13460 134611. VGPR0-31 and SGPR4-29 are used to pass function result arguments as 13462 described below. Any registers used are considered clobbered registers. 134632. The following registers are preserved and have the same value as on entry: 13464 13465 * FLAT_SCRATCH 13466 * EXEC 13467 * GFX6-GFX8: M0 13468 * All SGPR registers except the clobbered registers of SGPR4-31. 13469 * VGPR40-47 13470 * VGPR56-63 13471 * VGPR72-79 13472 * VGPR88-95 13473 * VGPR104-111 13474 * VGPR120-127 13475 * VGPR136-143 13476 * VGPR152-159 13477 * VGPR168-175 13478 * VGPR184-191 13479 * VGPR200-207 13480 * VGPR216-223 13481 * VGPR232-239 13482 * VGPR248-255 13483 13484 .. note:: 13485 13486 Except the argument registers, the VGPRs clobbered and the preserved 13487 registers are intermixed at regular intervals in order to keep a 13488 similar ratio independent of the number of allocated VGPRs. 13489 13490 * GFX90A: All AGPR registers except the clobbered registers AGPR0-31. 13491 * Lanes of all VGPRs that are inactive at the call site. 13492 13493 For the AMDGPU backend, an inter-procedural register allocation (IPRA) 13494 optimization may mark some of clobbered SGPR and VGPR registers as 13495 preserved if it can be determined that the called function does not change 13496 their value. 13497 134982. The PC is set to the RA provided on entry. 134993. MODE register: *TBD*. 135004. All other registers are clobbered. 135015. Any necessary ``s_waitcnt`` has been performed to ensure memory accessed by 13502 function is available to the caller. 13503 13504.. TODO:: 13505 13506 - How are function results returned? The address of structured types is passed 13507 by reference, but what about other types? 13508 13509The function input arguments are made up of the formal arguments explicitly 13510declared by the source language function plus the implicit input arguments used 13511by the implementation. 13512 13513The source language input arguments are: 13514 135151. Any source language implicit ``this`` or ``self`` argument comes first as a 13516 pointer type. 135172. Followed by the function formal arguments in left to right source order. 13518 13519The source language result arguments are: 13520 135211. The function result argument. 13522 13523The source language input or result struct type arguments that are less than or 13524equal to 16 bytes, are decomposed recursively into their base type fields, and 13525each field is passed as if a separate argument. For input arguments, if the 13526called function requires the struct to be in memory, for example because its 13527address is taken, then the function body is responsible for allocating a stack 13528location and copying the field arguments into it. Clang terms this *direct 13529struct*. 13530 13531The source language input struct type arguments that are greater than 16 bytes, 13532are passed by reference. The caller is responsible for allocating a stack 13533location to make a copy of the struct value and pass the address as the input 13534argument. The called function is responsible to perform the dereference when 13535accessing the input argument. Clang terms this *by-value struct*. 13536 13537A source language result struct type argument that is greater than 16 bytes, is 13538returned by reference. The caller is responsible for allocating a stack location 13539to hold the result value and passes the address as the last input argument 13540(before the implicit input arguments). In this case there are no result 13541arguments. The called function is responsible to perform the dereference when 13542storing the result value. Clang terms this *structured return (sret)*. 13543 13544*TODO: correct the ``sret`` definition.* 13545 13546.. TODO:: 13547 13548 Is this definition correct? Or is ``sret`` only used if passing in registers, and 13549 pass as non-decomposed struct as stack argument? Or something else? Is the 13550 memory location in the caller stack frame, or a stack memory argument and so 13551 no address is passed as the caller can directly write to the argument stack 13552 location? But then the stack location is still live after return. If an 13553 argument stack location is it the first stack argument or the last one? 13554 13555Lambda argument types are treated as struct types with an implementation defined 13556set of fields. 13557 13558.. TODO:: 13559 13560 Need to specify the ABI for lambda types for AMDGPU. 13561 13562For AMDGPU backend all source language arguments (including the decomposed 13563struct type arguments) are passed in VGPRs unless marked ``inreg`` in which case 13564they are passed in SGPRs. 13565 13566The AMDGPU backend walks the function call graph from the leaves to determine 13567which implicit input arguments are used, propagating to each caller of the 13568function. The used implicit arguments are appended to the function arguments 13569after the source language arguments in the following order: 13570 13571.. TODO:: 13572 13573 Is recursion or external functions supported? 13574 135751. Work-Item ID (1 VGPR) 13576 13577 The X, Y and Z work-item ID are packed into a single VGRP with the following 13578 layout. Only fields actually used by the function are set. The other bits 13579 are undefined. 13580 13581 The values come from the initial kernel execution state. See 13582 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`. 13583 13584 .. table:: Work-item implicit argument layout 13585 :name: amdgpu-amdhsa-workitem-implicit-argument-layout-table 13586 13587 ======= ======= ============== 13588 Bits Size Field Name 13589 ======= ======= ============== 13590 9:0 10 bits X Work-Item ID 13591 19:10 10 bits Y Work-Item ID 13592 29:20 10 bits Z Work-Item ID 13593 31:30 2 bits Unused 13594 ======= ======= ============== 13595 135962. Dispatch Ptr (2 SGPRs) 13597 13598 The value comes from the initial kernel execution state. See 13599 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 13600 136013. Queue Ptr (2 SGPRs) 13602 13603 The value comes from the initial kernel execution state. See 13604 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 13605 136064. Kernarg Segment Ptr (2 SGPRs) 13607 13608 The value comes from the initial kernel execution state. See 13609 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 13610 136115. Dispatch id (2 SGPRs) 13612 13613 The value comes from the initial kernel execution state. See 13614 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 13615 136166. Work-Group ID X (1 SGPR) 13617 13618 The value comes from the initial kernel execution state. See 13619 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 13620 136217. Work-Group ID Y (1 SGPR) 13622 13623 The value comes from the initial kernel execution state. See 13624 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 13625 136268. Work-Group ID Z (1 SGPR) 13627 13628 The value comes from the initial kernel execution state. See 13629 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 13630 136319. Implicit Argument Ptr (2 SGPRs) 13632 13633 The value is computed by adding an offset to Kernarg Segment Ptr to get the 13634 global address space pointer to the first kernarg implicit argument. 13635 13636The input and result arguments are assigned in order in the following manner: 13637 13638.. note:: 13639 13640 There are likely some errors and omissions in the following description that 13641 need correction. 13642 13643 .. TODO:: 13644 13645 Check the Clang source code to decipher how function arguments and return 13646 results are handled. Also see the AMDGPU specific values used. 13647 13648* VGPR arguments are assigned to consecutive VGPRs starting at VGPR0 up to 13649 VGPR31. 13650 13651 If there are more arguments than will fit in these registers, the remaining 13652 arguments are allocated on the stack in order on naturally aligned 13653 addresses. 13654 13655 .. TODO:: 13656 13657 How are overly aligned structures allocated on the stack? 13658 13659* SGPR arguments are assigned to consecutive SGPRs starting at SGPR0 up to 13660 SGPR29. 13661 13662 If there are more arguments than will fit in these registers, the remaining 13663 arguments are allocated on the stack in order on naturally aligned 13664 addresses. 13665 13666Note that decomposed struct type arguments may have some fields passed in 13667registers and some in memory. 13668 13669.. TODO:: 13670 13671 So, a struct which can pass some fields as decomposed register arguments, will 13672 pass the rest as decomposed stack elements? But an argument that will not start 13673 in registers will not be decomposed and will be passed as a non-decomposed 13674 stack value? 13675 13676The following is not part of the AMDGPU function calling convention but 13677describes how the AMDGPU implements function calls: 13678 136791. SGPR33 is used as a frame pointer (FP) if necessary. Like the SP it is an 13680 unswizzled scratch address. It is only needed if runtime sized ``alloca`` 13681 are used, or for the reasons defined in ``SIFrameLowering``. 136822. Runtime stack alignment is supported. SGPR34 is used as a base pointer (BP) 13683 to access the incoming stack arguments in the function. The BP is needed 13684 only when the function requires the runtime stack alignment. 13685 136863. Allocating SGPR arguments on the stack are not supported. 13687 136884. No CFI is currently generated. See 13689 :ref:`amdgpu-dwarf-call-frame-information`. 13690 13691 .. note:: 13692 13693 CFI will be generated that defines the CFA as the unswizzled address 13694 relative to the wave scratch base in the unswizzled private address space 13695 of the lowest address stack allocated local variable. 13696 13697 ``DW_AT_frame_base`` will be defined as the swizzled address in the 13698 swizzled private address space by dividing the CFA by the wavefront size 13699 (since CFA is always at least dword aligned which matches the scratch 13700 swizzle element size). 13701 13702 If no dynamic stack alignment was performed, the stack allocated arguments 13703 are accessed as negative offsets relative to ``DW_AT_frame_base``, and the 13704 local variables and register spill slots are accessed as positive offsets 13705 relative to ``DW_AT_frame_base``. 13706 137075. Function argument passing is implemented by copying the input physical 13708 registers to virtual registers on entry. The register allocator can spill if 13709 necessary. These are copied back to physical registers at call sites. The 13710 net effect is that each function call can have these values in entirely 13711 distinct locations. The IPRA can help avoid shuffling argument registers. 137126. Call sites are implemented by setting up the arguments at positive offsets 13713 from SP. Then SP is incremented to account for the known frame size before 13714 the call and decremented after the call. 13715 13716 .. note:: 13717 13718 The CFI will reflect the changed calculation needed to compute the CFA 13719 from SP. 13720 137217. 4 byte spill slots are used in the stack frame. One slot is allocated for an 13722 emergency spill slot. Buffer instructions are used for stack accesses and 13723 not the ``flat_scratch`` instruction. 13724 13725 .. TODO:: 13726 13727 Explain when the emergency spill slot is used. 13728 13729.. TODO:: 13730 13731 Possible broken issues: 13732 13733 - Stack arguments must be aligned to required alignment. 13734 - Stack is aligned to max(16, max formal argument alignment) 13735 - Direct argument < 64 bits should check register budget. 13736 - Register budget calculation should respect ``inreg`` for SGPR. 13737 - SGPR overflow is not handled. 13738 - struct with 1 member unpeeling is not checking size of member. 13739 - ``sret`` is after ``this`` pointer. 13740 - Caller is not implementing stack realignment: need an extra pointer. 13741 - Should say AMDGPU passes FP rather than SP. 13742 - Should CFI define CFA as address of locals or arguments. Difference is 13743 apparent when have implemented dynamic alignment. 13744 - If ``SCRATCH`` instruction could allow negative offsets, then can make FP be 13745 highest address of stack frame and use negative offset for locals. Would 13746 allow SP to be the same as FP and could support signal-handler-like as now 13747 have a real SP for the top of the stack. 13748 - How is ``sret`` passed on the stack? In argument stack area? Can it overlay 13749 arguments? 13750 13751AMDPAL 13752------ 13753 13754This section provides code conventions used when the target triple OS is 13755``amdpal`` (see :ref:`amdgpu-target-triples`). 13756 13757.. _amdgpu-amdpal-code-object-metadata-section: 13758 13759Code Object Metadata 13760~~~~~~~~~~~~~~~~~~~~ 13761 13762.. note:: 13763 13764 The metadata is currently in development and is subject to major 13765 changes. Only the current version is supported. *When this document 13766 was generated the version was 2.6.* 13767 13768Code object metadata is specified by the ``NT_AMDGPU_METADATA`` note 13769record (see :ref:`amdgpu-note-records-v3-onwards`). 13770 13771The metadata is represented as Message Pack formatted binary data (see 13772[MsgPack]_). The top level is a Message Pack map that includes the keys 13773defined in table :ref:`amdgpu-amdpal-code-object-metadata-map-table` 13774and referenced tables. 13775 13776Additional information can be added to the maps. To avoid conflicts, any 13777key names should be prefixed by "*vendor-name*." where ``vendor-name`` 13778can be the name of the vendor and specific vendor tool that generates the 13779information. The prefix is abbreviated to simply "." when it appears 13780within a map that has been added by the same *vendor-name*. 13781 13782 .. table:: AMDPAL Code Object Metadata Map 13783 :name: amdgpu-amdpal-code-object-metadata-map-table 13784 13785 =================== ============== ========= ====================================================================== 13786 String Key Value Type Required? Description 13787 =================== ============== ========= ====================================================================== 13788 "amdpal.version" sequence of Required PAL code object metadata (major, minor) version. The current values 13789 2 integers are defined by *Util::Abi::PipelineMetadata(Major|Minor)Version*. 13790 "amdpal.pipelines" sequence of Required Per-pipeline metadata. See 13791 map :ref:`amdgpu-amdpal-code-object-pipeline-metadata-map-table` for the 13792 definition of the keys included in that map. 13793 =================== ============== ========= ====================================================================== 13794 13795.. 13796 13797 .. table:: AMDPAL Code Object Pipeline Metadata Map 13798 :name: amdgpu-amdpal-code-object-pipeline-metadata-map-table 13799 13800 ====================================== ============== ========= =================================================== 13801 String Key Value Type Required? Description 13802 ====================================== ============== ========= =================================================== 13803 ".name" string Source name of the pipeline. 13804 ".type" string Pipeline type, e.g. VsPs. Values include: 13805 13806 - "VsPs" 13807 - "Gs" 13808 - "Cs" 13809 - "Ngg" 13810 - "Tess" 13811 - "GsTess" 13812 - "NggTess" 13813 13814 ".internal_pipeline_hash" sequence of Required Internal compiler hash for this pipeline. Lower 13815 2 integers 64 bits is the "stable" portion of the hash, used 13816 for e.g. shader replacement lookup. Upper 64 bits 13817 is the "unique" portion of the hash, used for 13818 e.g. pipeline cache lookup. The value is 13819 implementation defined, and can not be relied on 13820 between different builds of the compiler. 13821 ".shaders" map Per-API shader metadata. See 13822 :ref:`amdgpu-amdpal-code-object-shader-map-table` 13823 for the definition of the keys included in that 13824 map. 13825 ".hardware_stages" map Per-hardware stage metadata. See 13826 :ref:`amdgpu-amdpal-code-object-hardware-stage-map-table` 13827 for the definition of the keys included in that 13828 map. 13829 ".shader_functions" map Per-shader function metadata. See 13830 :ref:`amdgpu-amdpal-code-object-shader-function-map-table` 13831 for the definition of the keys included in that 13832 map. 13833 ".registers" map Required Hardware register configuration. See 13834 :ref:`amdgpu-amdpal-code-object-register-map-table` 13835 for the definition of the keys included in that 13836 map. 13837 ".user_data_limit" integer Number of user data entries accessed by this 13838 pipeline. 13839 ".spill_threshold" integer The user data spill threshold. 0xFFFF for 13840 NoUserDataSpilling. 13841 ".uses_viewport_array_index" boolean Indicates whether or not the pipeline uses the 13842 viewport array index feature. Pipelines which use 13843 this feature can render into all 16 viewports, 13844 whereas pipelines which do not use it are 13845 restricted to viewport #0. 13846 ".es_gs_lds_size" integer Size in bytes of LDS space used internally for 13847 handling data-passing between the ES and GS 13848 shader stages. This can be zero if the data is 13849 passed using off-chip buffers. This value should 13850 be used to program all user-SGPRs which have been 13851 marked with "UserDataMapping::EsGsLdsSize" 13852 (typically only the GS and VS HW stages will ever 13853 have a user-SGPR so marked). 13854 ".nggSubgroupSize" integer Explicit maximum subgroup size for NGG shaders 13855 (maximum number of threads in a subgroup). 13856 ".num_interpolants" integer Graphics only. Number of PS interpolants. 13857 ".mesh_scratch_memory_size" integer Max mesh shader scratch memory used. 13858 ".api" string Name of the client graphics API. 13859 ".api_create_info" binary Graphics API shader create info binary blob. Can 13860 be defined by the driver using the compiler if 13861 they want to be able to correlate API-specific 13862 information used during creation at a later time. 13863 ====================================== ============== ========= =================================================== 13864 13865.. 13866 13867 .. table:: AMDPAL Code Object Shader Map 13868 :name: amdgpu-amdpal-code-object-shader-map-table 13869 13870 13871 +-------------+--------------+-------------------------------------------------------------------+ 13872 |String Key |Value Type |Description | 13873 +=============+==============+===================================================================+ 13874 |- ".compute" |map |See :ref:`amdgpu-amdpal-code-object-api-shader-metadata-map-table` | 13875 |- ".vertex" | |for the definition of the keys included in that map. | 13876 |- ".hull" | | | 13877 |- ".domain" | | | 13878 |- ".geometry"| | | 13879 |- ".pixel" | | | 13880 +-------------+--------------+-------------------------------------------------------------------+ 13881 13882.. 13883 13884 .. table:: AMDPAL Code Object API Shader Metadata Map 13885 :name: amdgpu-amdpal-code-object-api-shader-metadata-map-table 13886 13887 ==================== ============== ========= ===================================================================== 13888 String Key Value Type Required? Description 13889 ==================== ============== ========= ===================================================================== 13890 ".api_shader_hash" sequence of Required Input shader hash, typically passed in from the client. The value 13891 2 integers is implementation defined, and can not be relied on between 13892 different builds of the compiler. 13893 ".hardware_mapping" sequence of Required Flags indicating the HW stages this API shader maps to. Values 13894 string include: 13895 13896 - ".ls" 13897 - ".hs" 13898 - ".es" 13899 - ".gs" 13900 - ".vs" 13901 - ".ps" 13902 - ".cs" 13903 13904 ==================== ============== ========= ===================================================================== 13905 13906.. 13907 13908 .. table:: AMDPAL Code Object Hardware Stage Map 13909 :name: amdgpu-amdpal-code-object-hardware-stage-map-table 13910 13911 +-------------+--------------+-----------------------------------------------------------------------+ 13912 |String Key |Value Type |Description | 13913 +=============+==============+=======================================================================+ 13914 |- ".ls" |map |See :ref:`amdgpu-amdpal-code-object-hardware-stage-metadata-map-table` | 13915 |- ".hs" | |for the definition of the keys included in that map. | 13916 |- ".es" | | | 13917 |- ".gs" | | | 13918 |- ".vs" | | | 13919 |- ".ps" | | | 13920 |- ".cs" | | | 13921 +-------------+--------------+-----------------------------------------------------------------------+ 13922 13923.. 13924 13925 .. table:: AMDPAL Code Object Hardware Stage Metadata Map 13926 :name: amdgpu-amdpal-code-object-hardware-stage-metadata-map-table 13927 13928 ========================== ============== ========= =============================================================== 13929 String Key Value Type Required? Description 13930 ========================== ============== ========= =============================================================== 13931 ".entry_point" string The ELF symbol pointing to this pipeline's stage entry point. 13932 ".scratch_memory_size" integer Scratch memory size in bytes. 13933 ".lds_size" integer Local Data Share size in bytes. 13934 ".perf_data_buffer_size" integer Performance data buffer size in bytes. 13935 ".vgpr_count" integer Number of VGPRs used. 13936 ".agpr_count" integer Number of AGPRs used. 13937 ".sgpr_count" integer Number of SGPRs used. 13938 ".vgpr_limit" integer If non-zero, indicates the shader was compiled with a 13939 directive to instruct the compiler to limit the VGPR usage to 13940 be less than or equal to the specified value (only set if 13941 different from HW default). 13942 ".sgpr_limit" integer SGPR count upper limit (only set if different from HW 13943 default). 13944 ".threadgroup_dimensions" sequence of Thread-group X/Y/Z dimensions (Compute only). 13945 3 integers 13946 ".wavefront_size" integer Wavefront size (only set if different from HW default). 13947 ".uses_uavs" boolean The shader reads or writes UAVs. 13948 ".uses_rovs" boolean The shader reads or writes ROVs. 13949 ".writes_uavs" boolean The shader writes to one or more UAVs. 13950 ".writes_depth" boolean The shader writes out a depth value. 13951 ".uses_append_consume" boolean The shader uses append and/or consume operations, either 13952 memory or GDS. 13953 ".uses_prim_id" boolean The shader uses PrimID. 13954 ========================== ============== ========= =============================================================== 13955 13956.. 13957 13958 .. table:: AMDPAL Code Object Shader Function Map 13959 :name: amdgpu-amdpal-code-object-shader-function-map-table 13960 13961 =============== ============== ==================================================================== 13962 String Key Value Type Description 13963 =============== ============== ==================================================================== 13964 *symbol name* map *symbol name* is the ELF symbol name of the shader function code 13965 entry address. The value is the function's metadata. See 13966 :ref:`amdgpu-amdpal-code-object-shader-function-metadata-map-table`. 13967 =============== ============== ==================================================================== 13968 13969.. 13970 13971 .. table:: AMDPAL Code Object Shader Function Metadata Map 13972 :name: amdgpu-amdpal-code-object-shader-function-metadata-map-table 13973 13974 ============================= ============== ================================================================= 13975 String Key Value Type Description 13976 ============================= ============== ================================================================= 13977 ".api_shader_hash" sequence of Input shader hash, typically passed in from the client. The value 13978 2 integers is implementation defined, and can not be relied on between 13979 different builds of the compiler. 13980 ".scratch_memory_size" integer Size in bytes of scratch memory used by the shader. 13981 ".lds_size" integer Size in bytes of LDS memory. 13982 ".vgpr_count" integer Number of VGPRs used by the shader. 13983 ".sgpr_count" integer Number of SGPRs used by the shader. 13984 ".stack_frame_size_in_bytes" integer Amount of stack size used by the shader. 13985 ".shader_subtype" string Shader subtype/kind. Values include: 13986 13987 - "Unknown" 13988 13989 ============================= ============== ================================================================= 13990 13991.. 13992 13993 .. table:: AMDPAL Code Object Register Map 13994 :name: amdgpu-amdpal-code-object-register-map-table 13995 13996 ========================== ============== ==================================================================== 13997 32-bit Integer Key Value Type Description 13998 ========================== ============== ==================================================================== 13999 ``reg offset`` 32-bit integer ``reg offset`` is the dword offset into the GFXIP register space of 14000 a GRBM register (i.e., driver accessible GPU register number, not 14001 shader GPR register number). The driver is required to program each 14002 specified register to the corresponding specified value when 14003 executing this pipeline. Typically, the ``reg offsets`` are the 14004 ``uint16_t`` offsets to each register as defined by the hardware 14005 chip headers. The register is set to the provided value. However, a 14006 ``reg offset`` that specifies a user data register (e.g., 14007 COMPUTE_USER_DATA_0) needs special treatment. See 14008 :ref:`amdgpu-amdpal-code-object-user-data-section` section for more 14009 information. 14010 ========================== ============== ==================================================================== 14011 14012.. _amdgpu-amdpal-code-object-user-data-section: 14013 14014User Data 14015+++++++++ 14016 14017Each hardware stage has a set of 32-bit physical SPI *user data registers* 14018(either 16 or 32 based on graphics IP and the stage) which can be 14019written from a command buffer and then loaded into SGPRs when waves are 14020launched via a subsequent dispatch or draw operation. This is the way 14021most arguments are passed from the application/runtime to a hardware 14022shader. 14023 14024PAL abstracts this functionality by exposing a set of 128 *user data 14025entries* per pipeline a client can use to pass arguments from a command 14026buffer to one or more shaders in that pipeline. The ELF code object must 14027specify a mapping from virtualized *user data entries* to physical *user 14028data registers*, and PAL is responsible for implementing that mapping, 14029including spilling overflow *user data entries* to memory if needed. 14030 14031Since the *user data registers* are GRBM-accessible SPI registers, this 14032mapping is actually embedded in the ``.registers`` metadata entry. For 14033most registers, the value in that map is a literal 32-bit value that 14034should be written to the register by the driver. However, when the 14035register is a *user data register* (any USER_DATA register e.g., 14036SPI_SHADER_USER_DATA_PS_5), the value is instead an encoding that tells 14037the driver to write either a *user data entry* value or one of several 14038driver-internal values to the register. This encoding is described in 14039the following table: 14040 14041.. note:: 14042 14043 Currently, *user data registers* 0 and 1 (e.g., SPI_SHADER_USER_DATA_PS_0, 14044 and SPI_SHADER_USER_DATA_PS_1) are reserved. *User data register* 0 must 14045 always be programmed to the address of the GlobalTable, and *user data 14046 register* 1 must always be programmed to the address of the PerShaderTable. 14047 14048.. 14049 14050 .. table:: AMDPAL User Data Mapping 14051 :name: amdgpu-amdpal-code-object-metadata-user-data-mapping-table 14052 14053 ========== ================= =============================================================================== 14054 Value Name Description 14055 ========== ================= =============================================================================== 14056 0..127 *User Data Entry* 32-bit value of user_data_entry[N] as specified via *CmdSetUserData()* 14057 0x10000000 GlobalTable 32-bit pointer to GPU memory containing the global internal table (should 14058 always point to *user data register* 0). 14059 0x10000001 PerShaderTable 32-bit pointer to GPU memory containing the per-shader internal table. See 14060 :ref:`amdgpu-amdpal-code-object-metadata-user-data-per-shader-table-section` 14061 for more detail (should always point to *user data register* 1). 14062 0x10000002 SpillTable 32-bit pointer to GPU memory containing the user data spill table. See 14063 :ref:`amdgpu-amdpal-code-object-metadata-user-data-spill-table-section` for 14064 more detail. 14065 0x10000003 BaseVertex Vertex offset (32-bit unsigned integer). Not needed if the pipeline doesn't 14066 reference the draw index in the vertex shader. Only supported by the first 14067 stage in a graphics pipeline. 14068 0x10000004 BaseInstance Instance offset (32-bit unsigned integer). Only supported by the first stage in 14069 a graphics pipeline. 14070 0x10000005 DrawIndex Draw index (32-bit unsigned integer). Only supported by the first stage in a 14071 graphics pipeline. 14072 0x10000006 Workgroup Thread group count (32-bit unsigned integer). Low half of a 64-bit address of 14073 a buffer containing the grid dimensions for a Compute dispatch operation. The 14074 high half of the address is stored in the next sequential user-SGPR. Only 14075 supported by compute pipelines. 14076 0x1000000A EsGsLdsSize Indicates that PAL will program this user-SGPR to contain the amount of LDS 14077 space used for the ES/GS pseudo-ring-buffer for passing data between shader 14078 stages. 14079 0x1000000B ViewId View id (32-bit unsigned integer) identifies a view of graphic 14080 pipeline instancing. 14081 0x1000000C StreamOutTable 32-bit pointer to GPU memory containing the stream out target SRD table. This 14082 can only appear for one shader stage per pipeline. 14083 0x1000000D PerShaderPerfData 32-bit pointer to GPU memory containing the per-shader performance data buffer. 14084 0x1000000F VertexBufferTable 32-bit pointer to GPU memory containing the vertex buffer SRD table. This can 14085 only appear for one shader stage per pipeline. 14086 0x10000010 UavExportTable 32-bit pointer to GPU memory containing the UAV export SRD table. This can 14087 only appear for one shader stage per pipeline (PS). These replace color targets 14088 and are completely separate from any UAVs used by the shader. This is optional, 14089 and only used by the PS when UAV exports are used to replace color-target 14090 exports to optimize specific shaders. 14091 0x10000011 NggCullingData 64-bit pointer to GPU memory containing the hardware register data needed by 14092 some NGG pipelines to perform culling. This value contains the address of the 14093 first of two consecutive registers which provide the full GPU address. 14094 0x10000015 FetchShaderPtr 64-bit pointer to GPU memory containing the fetch shader subroutine. 14095 ========== ================= =============================================================================== 14096 14097.. _amdgpu-amdpal-code-object-metadata-user-data-per-shader-table-section: 14098 14099Per-Shader Table 14100################ 14101 14102Low 32 bits of the GPU address for an optional buffer in the ``.data`` 14103section of the ELF. The high 32 bits of the address match the high 32 bits 14104of the shader's program counter. 14105 14106The buffer can be anything the shader compiler needs it for, and 14107allows each shader to have its own region of the ``.data`` section. 14108Typically, this could be a table of buffer SRD's and the data pointed to 14109by the buffer SRD's, but it could be a flat-address region of memory as 14110well. Its layout and usage are defined by the shader compiler. 14111 14112Each shader's table in the ``.data`` section is referenced by the symbol 14113``_amdgpu_``\ *xs*\ ``_shdr_intrl_data`` where *xs* corresponds with the 14114hardware shader stage the data is for. E.g., 14115``_amdgpu_cs_shdr_intrl_data`` for the compute shader hardware stage. 14116 14117.. _amdgpu-amdpal-code-object-metadata-user-data-spill-table-section: 14118 14119Spill Table 14120########### 14121 14122It is possible for a hardware shader to need access to more *user data 14123entries* than there are slots available in user data registers for one 14124or more hardware shader stages. In that case, the PAL runtime expects 14125the necessary *user data entries* to be spilled to GPU memory and use 14126one user data register to point to the spilled user data memory. The 14127value of the *user data entry* must then represent the location where 14128a shader expects to read the low 32-bits of the table's GPU virtual 14129address. The *spill table* itself represents a set of 32-bit values 14130managed by the PAL runtime in GPU-accessible memory that can be made 14131indirectly accessible to a hardware shader. 14132 14133Unspecified OS 14134-------------- 14135 14136This section provides code conventions used when the target triple OS is 14137empty (see :ref:`amdgpu-target-triples`). 14138 14139Trap Handler ABI 14140~~~~~~~~~~~~~~~~ 14141 14142For code objects generated by AMDGPU backend for non-amdhsa OS, the runtime does 14143not install a trap handler. The ``llvm.trap`` and ``llvm.debugtrap`` 14144instructions are handled as follows: 14145 14146 .. table:: AMDGPU Trap Handler for Non-AMDHSA OS 14147 :name: amdgpu-trap-handler-for-non-amdhsa-os-table 14148 14149 =============== =============== =========================================== 14150 Usage Code Sequence Description 14151 =============== =============== =========================================== 14152 llvm.trap s_endpgm Causes wavefront to be terminated. 14153 llvm.debugtrap *none* Compiler warning given that there is no 14154 trap handler installed. 14155 =============== =============== =========================================== 14156 14157Source Languages 14158================ 14159 14160.. _amdgpu-opencl: 14161 14162OpenCL 14163------ 14164 14165When the language is OpenCL the following differences occur: 14166 141671. The OpenCL memory model is used (see :ref:`amdgpu-amdhsa-memory-model`). 141682. The AMDGPU backend appends additional arguments to the kernel's explicit 14169 arguments for the AMDHSA OS (see 14170 :ref:`opencl-kernel-implicit-arguments-appended-for-amdhsa-os-table`). 141713. Additional metadata is generated 14172 (see :ref:`amdgpu-amdhsa-code-object-metadata`). 14173 14174 .. table:: OpenCL kernel implicit arguments appended for AMDHSA OS 14175 :name: opencl-kernel-implicit-arguments-appended-for-amdhsa-os-table 14176 14177 ======== ==== ========= =========================================== 14178 Position Byte Byte Description 14179 Size Alignment 14180 ======== ==== ========= =========================================== 14181 1 8 8 OpenCL Global Offset X 14182 2 8 8 OpenCL Global Offset Y 14183 3 8 8 OpenCL Global Offset Z 14184 4 8 8 OpenCL address of printf buffer 14185 5 8 8 OpenCL address of virtual queue used by 14186 enqueue_kernel. 14187 6 8 8 OpenCL address of AqlWrap struct used by 14188 enqueue_kernel. 14189 7 8 8 Pointer argument used for Multi-gird 14190 synchronization. 14191 ======== ==== ========= =========================================== 14192 14193.. _amdgpu-hcc: 14194 14195HCC 14196--- 14197 14198When the language is HCC the following differences occur: 14199 142001. The HSA memory model is used (see :ref:`amdgpu-amdhsa-memory-model`). 14201 14202.. _amdgpu-assembler: 14203 14204Assembler 14205--------- 14206 14207AMDGPU backend has LLVM-MC based assembler which is currently in development. 14208It supports AMDGCN GFX6-GFX10. 14209 14210This section describes general syntax for instructions and operands. 14211 14212Instructions 14213~~~~~~~~~~~~ 14214 14215An instruction has the following :doc:`syntax<AMDGPUInstructionSyntax>`: 14216 14217 | ``<``\ *opcode*\ ``> <``\ *operand0*\ ``>, <``\ *operand1*\ ``>,... 14218 <``\ *modifier0*\ ``> <``\ *modifier1*\ ``>...`` 14219 14220:doc:`Operands<AMDGPUOperandSyntax>` are comma-separated while 14221:doc:`modifiers<AMDGPUModifierSyntax>` are space-separated. 14222 14223The order of operands and modifiers is fixed. 14224Most modifiers are optional and may be omitted. 14225 14226Links to detailed instruction syntax description may be found in the following 14227table. Note that features under development are not included 14228in this description. 14229 14230 ============= ============================================= ======================================= 14231 Architecture Core ISA ISA Variants and Extensions 14232 ============= ============================================= ======================================= 14233 GCN 2 :doc:`GFX7<AMDGPU/AMDGPUAsmGFX7>` \- 14234 GCN 3, GCN 4 :doc:`GFX8<AMDGPU/AMDGPUAsmGFX8>` \- 14235 GCN 5 :doc:`GFX9<AMDGPU/AMDGPUAsmGFX9>` :doc:`gfx900<AMDGPU/AMDGPUAsmGFX900>` 14236 14237 :doc:`gfx902<AMDGPU/AMDGPUAsmGFX900>` 14238 14239 :doc:`gfx904<AMDGPU/AMDGPUAsmGFX904>` 14240 14241 :doc:`gfx906<AMDGPU/AMDGPUAsmGFX906>` 14242 14243 :doc:`gfx909<AMDGPU/AMDGPUAsmGFX900>` 14244 14245 CDNA 1 :doc:`GFX9<AMDGPU/AMDGPUAsmGFX9>` :doc:`gfx908<AMDGPU/AMDGPUAsmGFX908>` 14246 CDNA 2 :doc:`GFX9<AMDGPU/AMDGPUAsmGFX9>` :doc:`gfx90a<AMDGPU/AMDGPUAsmGFX90a>` 14247 RDNA 1 :doc:`GFX10 RDNA1<AMDGPU/AMDGPUAsmGFX10>` :doc:`gfx1010<AMDGPU/AMDGPUAsmGFX10>` 14248 14249 :doc:`gfx1011<AMDGPU/AMDGPUAsmGFX1011>` 14250 14251 :doc:`gfx1012<AMDGPU/AMDGPUAsmGFX1011>` 14252 14253 :doc:`gfx1013<AMDGPU/AMDGPUAsmGFX1013>` 14254 14255 RDNA 2 :doc:`GFX10 RDNA2<AMDGPU/AMDGPUAsmGFX1030>` :doc:`gfx1030<AMDGPU/AMDGPUAsmGFX1030>` 14256 14257 :doc:`gfx1031<AMDGPU/AMDGPUAsmGFX1030>` 14258 14259 :doc:`gfx1032<AMDGPU/AMDGPUAsmGFX1030>` 14260 14261 :doc:`gfx1033<AMDGPU/AMDGPUAsmGFX1030>` 14262 14263 :doc:`gfx1034<AMDGPU/AMDGPUAsmGFX1030>` 14264 14265 :doc:`gfx1035<AMDGPU/AMDGPUAsmGFX1030>` 14266 14267 :doc:`gfx1036<AMDGPU/AMDGPUAsmGFX1030>` 14268 ============= ============================================= ======================================= 14269 14270For more information about instructions, their semantics and supported 14271combinations of operands, refer to one of instruction set architecture manuals 14272[AMD-GCN-GFX6]_, [AMD-GCN-GFX7]_, [AMD-GCN-GFX8]_, 14273[AMD-GCN-GFX900-GFX904-VEGA]_, [AMD-GCN-GFX906-VEGA7NM]_, 14274[AMD-GCN-GFX908-CDNA1]_, [AMD-GCN-GFX90A-CDNA2]_, [AMD-GCN-GFX10-RDNA1]_ and 14275[AMD-GCN-GFX10-RDNA2]_. 14276 14277Operands 14278~~~~~~~~ 14279 14280Detailed description of operands may be found :doc:`here<AMDGPUOperandSyntax>`. 14281 14282Modifiers 14283~~~~~~~~~ 14284 14285Detailed description of modifiers may be found 14286:doc:`here<AMDGPUModifierSyntax>`. 14287 14288Instruction Examples 14289~~~~~~~~~~~~~~~~~~~~ 14290 14291DS 14292++ 14293 14294.. code-block:: nasm 14295 14296 ds_add_u32 v2, v4 offset:16 14297 ds_write_src2_b64 v2 offset0:4 offset1:8 14298 ds_cmpst_f32 v2, v4, v6 14299 ds_min_rtn_f64 v[8:9], v2, v[4:5] 14300 14301For full list of supported instructions, refer to "LDS/GDS instructions" in ISA 14302Manual. 14303 14304FLAT 14305++++ 14306 14307.. code-block:: nasm 14308 14309 flat_load_dword v1, v[3:4] 14310 flat_store_dwordx3 v[3:4], v[5:7] 14311 flat_atomic_swap v1, v[3:4], v5 glc 14312 flat_atomic_cmpswap v1, v[3:4], v[5:6] glc slc 14313 flat_atomic_fmax_x2 v[1:2], v[3:4], v[5:6] glc 14314 14315For full list of supported instructions, refer to "FLAT instructions" in ISA 14316Manual. 14317 14318MUBUF 14319+++++ 14320 14321.. code-block:: nasm 14322 14323 buffer_load_dword v1, off, s[4:7], s1 14324 buffer_store_dwordx4 v[1:4], v2, ttmp[4:7], s1 offen offset:4 glc tfe 14325 buffer_store_format_xy v[1:2], off, s[4:7], s1 14326 buffer_wbinvl1 14327 buffer_atomic_inc v1, v2, s[8:11], s4 idxen offset:4 slc 14328 14329For full list of supported instructions, refer to "MUBUF Instructions" in ISA 14330Manual. 14331 14332SMRD/SMEM 14333+++++++++ 14334 14335.. code-block:: nasm 14336 14337 s_load_dword s1, s[2:3], 0xfc 14338 s_load_dwordx8 s[8:15], s[2:3], s4 14339 s_load_dwordx16 s[88:103], s[2:3], s4 14340 s_dcache_inv_vol 14341 s_memtime s[4:5] 14342 14343For full list of supported instructions, refer to "Scalar Memory Operations" in 14344ISA Manual. 14345 14346SOP1 14347++++ 14348 14349.. code-block:: nasm 14350 14351 s_mov_b32 s1, s2 14352 s_mov_b64 s[0:1], 0x80000000 14353 s_cmov_b32 s1, 200 14354 s_wqm_b64 s[2:3], s[4:5] 14355 s_bcnt0_i32_b64 s1, s[2:3] 14356 s_swappc_b64 s[2:3], s[4:5] 14357 s_cbranch_join s[4:5] 14358 14359For full list of supported instructions, refer to "SOP1 Instructions" in ISA 14360Manual. 14361 14362SOP2 14363++++ 14364 14365.. code-block:: nasm 14366 14367 s_add_u32 s1, s2, s3 14368 s_and_b64 s[2:3], s[4:5], s[6:7] 14369 s_cselect_b32 s1, s2, s3 14370 s_andn2_b32 s2, s4, s6 14371 s_lshr_b64 s[2:3], s[4:5], s6 14372 s_ashr_i32 s2, s4, s6 14373 s_bfm_b64 s[2:3], s4, s6 14374 s_bfe_i64 s[2:3], s[4:5], s6 14375 s_cbranch_g_fork s[4:5], s[6:7] 14376 14377For full list of supported instructions, refer to "SOP2 Instructions" in ISA 14378Manual. 14379 14380SOPC 14381++++ 14382 14383.. code-block:: nasm 14384 14385 s_cmp_eq_i32 s1, s2 14386 s_bitcmp1_b32 s1, s2 14387 s_bitcmp0_b64 s[2:3], s4 14388 s_setvskip s3, s5 14389 14390For full list of supported instructions, refer to "SOPC Instructions" in ISA 14391Manual. 14392 14393SOPP 14394++++ 14395 14396.. code-block:: nasm 14397 14398 s_barrier 14399 s_nop 2 14400 s_endpgm 14401 s_waitcnt 0 ; Wait for all counters to be 0 14402 s_waitcnt vmcnt(0) & expcnt(0) & lgkmcnt(0) ; Equivalent to above 14403 s_waitcnt vmcnt(1) ; Wait for vmcnt counter to be 1. 14404 s_sethalt 9 14405 s_sleep 10 14406 s_sendmsg 0x1 14407 s_sendmsg sendmsg(MSG_INTERRUPT) 14408 s_trap 1 14409 14410For full list of supported instructions, refer to "SOPP Instructions" in ISA 14411Manual. 14412 14413Unless otherwise mentioned, little verification is performed on the operands 14414of SOPP Instructions, so it is up to the programmer to be familiar with the 14415range or acceptable values. 14416 14417VALU 14418++++ 14419 14420For vector ALU instruction opcodes (VOP1, VOP2, VOP3, VOPC, VOP_DPP, VOP_SDWA), 14421the assembler will automatically use optimal encoding based on its operands. To 14422force specific encoding, one can add a suffix to the opcode of the instruction: 14423 14424* _e32 for 32-bit VOP1/VOP2/VOPC 14425* _e64 for 64-bit VOP3 14426* _dpp for VOP_DPP 14427* _sdwa for VOP_SDWA 14428 14429VOP1/VOP2/VOP3/VOPC examples: 14430 14431.. code-block:: nasm 14432 14433 v_mov_b32 v1, v2 14434 v_mov_b32_e32 v1, v2 14435 v_nop 14436 v_cvt_f64_i32_e32 v[1:2], v2 14437 v_floor_f32_e32 v1, v2 14438 v_bfrev_b32_e32 v1, v2 14439 v_add_f32_e32 v1, v2, v3 14440 v_mul_i32_i24_e64 v1, v2, 3 14441 v_mul_i32_i24_e32 v1, -3, v3 14442 v_mul_i32_i24_e32 v1, -100, v3 14443 v_addc_u32 v1, s[0:1], v2, v3, s[2:3] 14444 v_max_f16_e32 v1, v2, v3 14445 14446VOP_DPP examples: 14447 14448.. code-block:: nasm 14449 14450 v_mov_b32 v0, v0 quad_perm:[0,2,1,1] 14451 v_sin_f32 v0, v0 row_shl:1 row_mask:0xa bank_mask:0x1 bound_ctrl:0 14452 v_mov_b32 v0, v0 wave_shl:1 14453 v_mov_b32 v0, v0 row_mirror 14454 v_mov_b32 v0, v0 row_bcast:31 14455 v_mov_b32 v0, v0 quad_perm:[1,3,0,1] row_mask:0xa bank_mask:0x1 bound_ctrl:0 14456 v_add_f32 v0, v0, |v0| row_shl:1 row_mask:0xa bank_mask:0x1 bound_ctrl:0 14457 v_max_f16 v1, v2, v3 row_shl:1 row_mask:0xa bank_mask:0x1 bound_ctrl:0 14458 14459VOP_SDWA examples: 14460 14461.. code-block:: nasm 14462 14463 v_mov_b32 v1, v2 dst_sel:BYTE_0 dst_unused:UNUSED_PRESERVE src0_sel:DWORD 14464 v_min_u32 v200, v200, v1 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:BYTE_1 src1_sel:DWORD 14465 v_sin_f32 v0, v0 dst_unused:UNUSED_PAD src0_sel:WORD_1 14466 v_fract_f32 v0, |v0| dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 14467 v_cmpx_le_u32 vcc, v1, v2 src0_sel:BYTE_2 src1_sel:WORD_0 14468 14469For full list of supported instructions, refer to "Vector ALU instructions". 14470 14471.. _amdgpu-amdhsa-assembler-predefined-symbols-v2: 14472 14473Code Object V2 Predefined Symbols 14474~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 14475 14476.. warning:: 14477 Code object V2 is not the default code object version emitted by 14478 this version of LLVM. 14479 14480The AMDGPU assembler defines and updates some symbols automatically. These 14481symbols do not affect code generation. 14482 14483.option.machine_version_major 14484+++++++++++++++++++++++++++++ 14485 14486Set to the GFX major generation number of the target being assembled for. For 14487example, when assembling for a "GFX9" target this will be set to the integer 14488value "9". The possible GFX major generation numbers are presented in 14489:ref:`amdgpu-processors`. 14490 14491.option.machine_version_minor 14492+++++++++++++++++++++++++++++ 14493 14494Set to the GFX minor generation number of the target being assembled for. For 14495example, when assembling for a "GFX810" target this will be set to the integer 14496value "1". The possible GFX minor generation numbers are presented in 14497:ref:`amdgpu-processors`. 14498 14499.option.machine_version_stepping 14500++++++++++++++++++++++++++++++++ 14501 14502Set to the GFX stepping generation number of the target being assembled for. 14503For example, when assembling for a "GFX704" target this will be set to the 14504integer value "4". The possible GFX stepping generation numbers are presented 14505in :ref:`amdgpu-processors`. 14506 14507.kernel.vgpr_count 14508++++++++++++++++++ 14509 14510Set to zero each time a 14511:ref:`amdgpu-amdhsa-assembler-directive-amdgpu_hsa_kernel` directive is 14512encountered. At each instruction, if the current value of this symbol is less 14513than or equal to the maximum VGPR number explicitly referenced within that 14514instruction then the symbol value is updated to equal that VGPR number plus 14515one. 14516 14517.kernel.sgpr_count 14518++++++++++++++++++ 14519 14520Set to zero each time a 14521:ref:`amdgpu-amdhsa-assembler-directive-amdgpu_hsa_kernel` directive is 14522encountered. At each instruction, if the current value of this symbol is less 14523than or equal to the maximum VGPR number explicitly referenced within that 14524instruction then the symbol value is updated to equal that SGPR number plus 14525one. 14526 14527.. _amdgpu-amdhsa-assembler-directives-v2: 14528 14529Code Object V2 Directives 14530~~~~~~~~~~~~~~~~~~~~~~~~~ 14531 14532.. warning:: 14533 Code object V2 is not the default code object version emitted by 14534 this version of LLVM. 14535 14536AMDGPU ABI defines auxiliary data in output code object. In assembly source, 14537one can specify them with assembler directives. 14538 14539.hsa_code_object_version major, minor 14540+++++++++++++++++++++++++++++++++++++ 14541 14542*major* and *minor* are integers that specify the version of the HSA code 14543object that will be generated by the assembler. 14544 14545.hsa_code_object_isa [major, minor, stepping, vendor, arch] 14546+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 14547 14548 14549*major*, *minor*, and *stepping* are all integers that describe the instruction 14550set architecture (ISA) version of the assembly program. 14551 14552*vendor* and *arch* are quoted strings. *vendor* should always be equal to 14553"AMD" and *arch* should always be equal to "AMDGPU". 14554 14555By default, the assembler will derive the ISA version, *vendor*, and *arch* 14556from the value of the -mcpu option that is passed to the assembler. 14557 14558.. _amdgpu-amdhsa-assembler-directive-amdgpu_hsa_kernel: 14559 14560.amdgpu_hsa_kernel (name) 14561+++++++++++++++++++++++++ 14562 14563This directives specifies that the symbol with given name is a kernel entry 14564point (label) and the object should contain corresponding symbol of type 14565STT_AMDGPU_HSA_KERNEL. 14566 14567.amd_kernel_code_t 14568++++++++++++++++++ 14569 14570This directive marks the beginning of a list of key / value pairs that are used 14571to specify the amd_kernel_code_t object that will be emitted by the assembler. 14572The list must be terminated by the *.end_amd_kernel_code_t* directive. For any 14573amd_kernel_code_t values that are unspecified a default value will be used. The 14574default value for all keys is 0, with the following exceptions: 14575 14576- *amd_code_version_major* defaults to 1. 14577- *amd_kernel_code_version_minor* defaults to 2. 14578- *amd_machine_kind* defaults to 1. 14579- *amd_machine_version_major*, *machine_version_minor*, and 14580 *amd_machine_version_stepping* are derived from the value of the -mcpu option 14581 that is passed to the assembler. 14582- *kernel_code_entry_byte_offset* defaults to 256. 14583- *wavefront_size* defaults 6 for all targets before GFX10. For GFX10 onwards 14584 defaults to 6 if target feature ``wavefrontsize64`` is enabled, otherwise 5. 14585 Note that wavefront size is specified as a power of two, so a value of **n** 14586 means a size of 2^ **n**. 14587- *call_convention* defaults to -1. 14588- *kernarg_segment_alignment*, *group_segment_alignment*, and 14589 *private_segment_alignment* default to 4. Note that alignments are specified 14590 as a power of 2, so a value of **n** means an alignment of 2^ **n**. 14591- *enable_tg_split* defaults to 1 if target feature ``tgsplit`` is enabled for 14592 GFX90A onwards. 14593- *enable_wgp_mode* defaults to 1 if target feature ``cumode`` is disabled for 14594 GFX10 onwards. 14595- *enable_mem_ordered* defaults to 1 for GFX10 onwards. 14596 14597The *.amd_kernel_code_t* directive must be placed immediately after the 14598function label and before any instructions. 14599 14600For a full list of amd_kernel_code_t keys, refer to AMDGPU ABI document, 14601comments in lib/Target/AMDGPU/AmdKernelCodeT.h and test/CodeGen/AMDGPU/hsa.s. 14602 14603.. _amdgpu-amdhsa-assembler-example-v2: 14604 14605Code Object V2 Example Source Code 14606~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 14607 14608.. warning:: 14609 Code Object V2 is not the default code object version emitted by 14610 this version of LLVM. 14611 14612Here is an example of a minimal assembly source file, defining one HSA kernel: 14613 14614.. code:: 14615 :number-lines: 14616 14617 .hsa_code_object_version 1,0 14618 .hsa_code_object_isa 14619 14620 .hsatext 14621 .globl hello_world 14622 .p2align 8 14623 .amdgpu_hsa_kernel hello_world 14624 14625 hello_world: 14626 14627 .amd_kernel_code_t 14628 enable_sgpr_kernarg_segment_ptr = 1 14629 is_ptr64 = 1 14630 compute_pgm_rsrc1_vgprs = 0 14631 compute_pgm_rsrc1_sgprs = 0 14632 compute_pgm_rsrc2_user_sgpr = 2 14633 compute_pgm_rsrc1_wgp_mode = 0 14634 compute_pgm_rsrc1_mem_ordered = 0 14635 compute_pgm_rsrc1_fwd_progress = 1 14636 .end_amd_kernel_code_t 14637 14638 s_load_dwordx2 s[0:1], s[0:1] 0x0 14639 v_mov_b32 v0, 3.14159 14640 s_waitcnt lgkmcnt(0) 14641 v_mov_b32 v1, s0 14642 v_mov_b32 v2, s1 14643 flat_store_dword v[1:2], v0 14644 s_endpgm 14645 .Lfunc_end0: 14646 .size hello_world, .Lfunc_end0-hello_world 14647 14648.. _amdgpu-amdhsa-assembler-predefined-symbols-v3-onwards: 14649 14650Code Object V3 and Above Predefined Symbols 14651~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 14652 14653The AMDGPU assembler defines and updates some symbols automatically. These 14654symbols do not affect code generation. 14655 14656.amdgcn.gfx_generation_number 14657+++++++++++++++++++++++++++++ 14658 14659Set to the GFX major generation number of the target being assembled for. For 14660example, when assembling for a "GFX9" target this will be set to the integer 14661value "9". The possible GFX major generation numbers are presented in 14662:ref:`amdgpu-processors`. 14663 14664.amdgcn.gfx_generation_minor 14665++++++++++++++++++++++++++++ 14666 14667Set to the GFX minor generation number of the target being assembled for. For 14668example, when assembling for a "GFX810" target this will be set to the integer 14669value "1". The possible GFX minor generation numbers are presented in 14670:ref:`amdgpu-processors`. 14671 14672.amdgcn.gfx_generation_stepping 14673+++++++++++++++++++++++++++++++ 14674 14675Set to the GFX stepping generation number of the target being assembled for. 14676For example, when assembling for a "GFX704" target this will be set to the 14677integer value "4". The possible GFX stepping generation numbers are presented 14678in :ref:`amdgpu-processors`. 14679 14680.. _amdgpu-amdhsa-assembler-symbol-next_free_vgpr: 14681 14682.amdgcn.next_free_vgpr 14683++++++++++++++++++++++ 14684 14685Set to zero before assembly begins. At each instruction, if the current value 14686of this symbol is less than or equal to the maximum VGPR number explicitly 14687referenced within that instruction then the symbol value is updated to equal 14688that VGPR number plus one. 14689 14690May be used to set the `.amdhsa_next_free_vgpr` directive in 14691:ref:`amdhsa-kernel-directives-table`. 14692 14693May be set at any time, e.g. manually set to zero at the start of each kernel. 14694 14695.. _amdgpu-amdhsa-assembler-symbol-next_free_sgpr: 14696 14697.amdgcn.next_free_sgpr 14698++++++++++++++++++++++ 14699 14700Set to zero before assembly begins. At each instruction, if the current value 14701of this symbol is less than or equal the maximum SGPR number explicitly 14702referenced within that instruction then the symbol value is updated to equal 14703that SGPR number plus one. 14704 14705May be used to set the `.amdhsa_next_free_spgr` directive in 14706:ref:`amdhsa-kernel-directives-table`. 14707 14708May be set at any time, e.g. manually set to zero at the start of each kernel. 14709 14710.. _amdgpu-amdhsa-assembler-directives-v3-onwards: 14711 14712Code Object V3 and Above Directives 14713~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 14714 14715Directives which begin with ``.amdgcn`` are valid for all ``amdgcn`` 14716architecture processors, and are not OS-specific. Directives which begin with 14717``.amdhsa`` are specific to ``amdgcn`` architecture processors when the 14718``amdhsa`` OS is specified. See :ref:`amdgpu-target-triples` and 14719:ref:`amdgpu-processors`. 14720 14721.. _amdgpu-assembler-directive-amdgcn-target: 14722 14723.amdgcn_target <target-triple> "-" <target-id> 14724++++++++++++++++++++++++++++++++++++++++++++++ 14725 14726Optional directive which declares the ``<target-triple>-<target-id>`` supported 14727by the containing assembler source file. Used by the assembler to validate 14728command-line options such as ``-triple``, ``-mcpu``, and 14729``--offload-arch=<target-id>``. A non-canonical target ID is allowed. See 14730:ref:`amdgpu-target-triples` and :ref:`amdgpu-target-id`. 14731 14732.. note:: 14733 14734 The target ID syntax used for code object V2 to V3 for this directive differs 14735 from that used elsewhere. See :ref:`amdgpu-target-id-v2-v3`. 14736 14737.amdhsa_kernel <name> 14738+++++++++++++++++++++ 14739 14740Creates a correctly aligned AMDHSA kernel descriptor and a symbol, 14741``<name>.kd``, in the current location of the current section. Only valid when 14742the OS is ``amdhsa``. ``<name>`` must be a symbol that labels the first 14743instruction to execute, and does not need to be previously defined. 14744 14745Marks the beginning of a list of directives used to generate the bytes of a 14746kernel descriptor, as described in :ref:`amdgpu-amdhsa-kernel-descriptor`. 14747Directives which may appear in this list are described in 14748:ref:`amdhsa-kernel-directives-table`. Directives may appear in any order, must 14749be valid for the target being assembled for, and cannot be repeated. Directives 14750support the range of values specified by the field they reference in 14751:ref:`amdgpu-amdhsa-kernel-descriptor`. If a directive is not specified, it is 14752assumed to have its default value, unless it is marked as "Required", in which 14753case it is an error to omit the directive. This list of directives is 14754terminated by an ``.end_amdhsa_kernel`` directive. 14755 14756 .. table:: AMDHSA Kernel Assembler Directives 14757 :name: amdhsa-kernel-directives-table 14758 14759 ======================================================== =================== ============ =================== 14760 Directive Default Supported On Description 14761 ======================================================== =================== ============ =================== 14762 ``.amdhsa_group_segment_fixed_size`` 0 GFX6-GFX10 Controls GROUP_SEGMENT_FIXED_SIZE in 14763 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 14764 ``.amdhsa_private_segment_fixed_size`` 0 GFX6-GFX10 Controls PRIVATE_SEGMENT_FIXED_SIZE in 14765 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 14766 ``.amdhsa_kernarg_size`` 0 GFX6-GFX10 Controls KERNARG_SIZE in 14767 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 14768 ``.amdhsa_user_sgpr_count`` 0 GFX6-GFX10 Controls USER_SGPR_COUNT in COMPUTE_PGM_RSRC2 14769 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table` 14770 ``.amdhsa_user_sgpr_private_segment_buffer`` 0 GFX6-GFX10 Controls ENABLE_SGPR_PRIVATE_SEGMENT_BUFFER in 14771 (except :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 14772 GFX940) 14773 ``.amdhsa_user_sgpr_dispatch_ptr`` 0 GFX6-GFX10 Controls ENABLE_SGPR_DISPATCH_PTR in 14774 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 14775 ``.amdhsa_user_sgpr_queue_ptr`` 0 GFX6-GFX10 Controls ENABLE_SGPR_QUEUE_PTR in 14776 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 14777 ``.amdhsa_user_sgpr_kernarg_segment_ptr`` 0 GFX6-GFX10 Controls ENABLE_SGPR_KERNARG_SEGMENT_PTR in 14778 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 14779 ``.amdhsa_user_sgpr_dispatch_id`` 0 GFX6-GFX10 Controls ENABLE_SGPR_DISPATCH_ID in 14780 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 14781 ``.amdhsa_user_sgpr_flat_scratch_init`` 0 GFX6-GFX10 Controls ENABLE_SGPR_FLAT_SCRATCH_INIT in 14782 (except :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 14783 GFX940) 14784 ``.amdhsa_user_sgpr_private_segment_size`` 0 GFX6-GFX10 Controls ENABLE_SGPR_PRIVATE_SEGMENT_SIZE in 14785 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 14786 ``.amdhsa_wavefront_size32`` Target GFX10 Controls ENABLE_WAVEFRONT_SIZE32 in 14787 Feature :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 14788 Specific 14789 (wavefrontsize64) 14790 ``.amdhsa_system_sgpr_private_segment_wavefront_offset`` 0 GFX6-GFX10 Controls ENABLE_PRIVATE_SEGMENT in 14791 (except :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 14792 GFX940) 14793 ``.amdhsa_enable_private_segment`` 0 GFX940 Controls ENABLE_PRIVATE_SEGMENT in 14794 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 14795 ``.amdhsa_system_sgpr_workgroup_id_x`` 1 GFX6-GFX10 Controls ENABLE_SGPR_WORKGROUP_ID_X in 14796 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 14797 ``.amdhsa_system_sgpr_workgroup_id_y`` 0 GFX6-GFX10 Controls ENABLE_SGPR_WORKGROUP_ID_Y in 14798 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 14799 ``.amdhsa_system_sgpr_workgroup_id_z`` 0 GFX6-GFX10 Controls ENABLE_SGPR_WORKGROUP_ID_Z in 14800 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 14801 ``.amdhsa_system_sgpr_workgroup_info`` 0 GFX6-GFX10 Controls ENABLE_SGPR_WORKGROUP_INFO in 14802 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 14803 ``.amdhsa_system_vgpr_workitem_id`` 0 GFX6-GFX10 Controls ENABLE_VGPR_WORKITEM_ID in 14804 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 14805 Possible values are defined in 14806 :ref:`amdgpu-amdhsa-system-vgpr-work-item-id-enumeration-values-table`. 14807 ``.amdhsa_next_free_vgpr`` Required GFX6-GFX10 Maximum VGPR number explicitly referenced, plus one. 14808 Used to calculate GRANULATED_WORKITEM_VGPR_COUNT in 14809 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 14810 ``.amdhsa_next_free_sgpr`` Required GFX6-GFX10 Maximum SGPR number explicitly referenced, plus one. 14811 Used to calculate GRANULATED_WAVEFRONT_SGPR_COUNT in 14812 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 14813 ``.amdhsa_accum_offset`` Required GFX90A, Offset of a first AccVGPR in the unified register file. 14814 GFX940 Used to calculate ACCUM_OFFSET in 14815 :ref:`amdgpu-amdhsa-compute_pgm_rsrc3-gfx90a-table`. 14816 ``.amdhsa_reserve_vcc`` 1 GFX6-GFX10 Whether the kernel may use the special VCC SGPR. 14817 Used to calculate GRANULATED_WAVEFRONT_SGPR_COUNT in 14818 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 14819 ``.amdhsa_reserve_flat_scratch`` 1 GFX7-GFX10 Whether the kernel may use flat instructions to access 14820 (except scratch memory. Used to calculate 14821 GFX940) GRANULATED_WAVEFRONT_SGPR_COUNT in 14822 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 14823 ``.amdhsa_reserve_xnack_mask`` Target GFX8-GFX10 Whether the kernel may trigger XNACK replay. 14824 Feature Used to calculate GRANULATED_WAVEFRONT_SGPR_COUNT in 14825 Specific :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 14826 (xnack) 14827 ``.amdhsa_float_round_mode_32`` 0 GFX6-GFX10 Controls FLOAT_ROUND_MODE_32 in 14828 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 14829 Possible values are defined in 14830 :ref:`amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table`. 14831 ``.amdhsa_float_round_mode_16_64`` 0 GFX6-GFX10 Controls FLOAT_ROUND_MODE_16_64 in 14832 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 14833 Possible values are defined in 14834 :ref:`amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table`. 14835 ``.amdhsa_float_denorm_mode_32`` 0 GFX6-GFX10 Controls FLOAT_DENORM_MODE_32 in 14836 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 14837 Possible values are defined in 14838 :ref:`amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table`. 14839 ``.amdhsa_float_denorm_mode_16_64`` 3 GFX6-GFX10 Controls FLOAT_DENORM_MODE_16_64 in 14840 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 14841 Possible values are defined in 14842 :ref:`amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table`. 14843 ``.amdhsa_dx10_clamp`` 1 GFX6-GFX10 Controls ENABLE_DX10_CLAMP in 14844 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 14845 ``.amdhsa_ieee_mode`` 1 GFX6-GFX10 Controls ENABLE_IEEE_MODE in 14846 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 14847 ``.amdhsa_fp16_overflow`` 0 GFX9-GFX10 Controls FP16_OVFL in 14848 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 14849 ``.amdhsa_tg_split`` Target GFX90A, Controls TG_SPLIT in 14850 Feature GFX940 :ref:`amdgpu-amdhsa-compute_pgm_rsrc3-gfx90a-table`. 14851 Specific 14852 (tgsplit) 14853 ``.amdhsa_workgroup_processor_mode`` Target GFX10 Controls ENABLE_WGP_MODE in 14854 Feature :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 14855 Specific 14856 (cumode) 14857 ``.amdhsa_memory_ordered`` 1 GFX10 Controls MEM_ORDERED in 14858 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 14859 ``.amdhsa_forward_progress`` 0 GFX10 Controls FWD_PROGRESS in 14860 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 14861 ``.amdhsa_shared_vgpr_count`` 0 GFX10 Controls SHARED_VGPR_COUNT in 14862 :ref:`amdgpu-amdhsa-compute_pgm_rsrc3-gfx10-table`. 14863 ``.amdhsa_exception_fp_ieee_invalid_op`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_INVALID_OPERATION in 14864 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 14865 ``.amdhsa_exception_fp_denorm_src`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_FP_DENORMAL_SOURCE in 14866 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 14867 ``.amdhsa_exception_fp_ieee_div_zero`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_DIVISION_BY_ZERO in 14868 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 14869 ``.amdhsa_exception_fp_ieee_overflow`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_OVERFLOW in 14870 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 14871 ``.amdhsa_exception_fp_ieee_underflow`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_UNDERFLOW in 14872 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 14873 ``.amdhsa_exception_fp_ieee_inexact`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_INEXACT in 14874 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 14875 ``.amdhsa_exception_int_div_zero`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_INT_DIVIDE_BY_ZERO in 14876 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 14877 ======================================================== =================== ============ =================== 14878 14879.amdgpu_metadata 14880++++++++++++++++ 14881 14882Optional directive which declares the contents of the ``NT_AMDGPU_METADATA`` 14883note record (see :ref:`amdgpu-elf-note-records-table-v3-onwards`). 14884 14885The contents must be in the [YAML]_ markup format, with the same structure and 14886semantics described in :ref:`amdgpu-amdhsa-code-object-metadata-v3`, 14887:ref:`amdgpu-amdhsa-code-object-metadata-v4` or 14888:ref:`amdgpu-amdhsa-code-object-metadata-v5`. 14889 14890This directive is terminated by an ``.end_amdgpu_metadata`` directive. 14891 14892.. _amdgpu-amdhsa-assembler-example-v3-onwards: 14893 14894Code Object V3 and Above Example Source Code 14895~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 14896 14897Here is an example of a minimal assembly source file, defining one HSA kernel: 14898 14899.. code:: 14900 :number-lines: 14901 14902 .amdgcn_target "amdgcn-amd-amdhsa--gfx900+xnack" // optional 14903 14904 .text 14905 .globl hello_world 14906 .p2align 8 14907 .type hello_world,@function 14908 hello_world: 14909 s_load_dwordx2 s[0:1], s[0:1] 0x0 14910 v_mov_b32 v0, 3.14159 14911 s_waitcnt lgkmcnt(0) 14912 v_mov_b32 v1, s0 14913 v_mov_b32 v2, s1 14914 flat_store_dword v[1:2], v0 14915 s_endpgm 14916 .Lfunc_end0: 14917 .size hello_world, .Lfunc_end0-hello_world 14918 14919 .rodata 14920 .p2align 6 14921 .amdhsa_kernel hello_world 14922 .amdhsa_user_sgpr_kernarg_segment_ptr 1 14923 .amdhsa_next_free_vgpr .amdgcn.next_free_vgpr 14924 .amdhsa_next_free_sgpr .amdgcn.next_free_sgpr 14925 .end_amdhsa_kernel 14926 14927 .amdgpu_metadata 14928 --- 14929 amdhsa.version: 14930 - 1 14931 - 0 14932 amdhsa.kernels: 14933 - .name: hello_world 14934 .symbol: hello_world.kd 14935 .kernarg_segment_size: 48 14936 .group_segment_fixed_size: 0 14937 .private_segment_fixed_size: 0 14938 .kernarg_segment_align: 4 14939 .wavefront_size: 64 14940 .sgpr_count: 2 14941 .vgpr_count: 3 14942 .max_flat_workgroup_size: 256 14943 .args: 14944 - .size: 8 14945 .offset: 0 14946 .value_kind: global_buffer 14947 .address_space: global 14948 .actual_access: write_only 14949 //... 14950 .end_amdgpu_metadata 14951 14952This kernel is equivalent to the following HIP program: 14953 14954.. code:: 14955 :number-lines: 14956 14957 __global__ void hello_world(float *p) { 14958 *p = 3.14159f; 14959 } 14960 14961If an assembly source file contains multiple kernels and/or functions, the 14962:ref:`amdgpu-amdhsa-assembler-symbol-next_free_vgpr` and 14963:ref:`amdgpu-amdhsa-assembler-symbol-next_free_sgpr` symbols may be reset using 14964the ``.set <symbol>, <expression>`` directive. For example, in the case of two 14965kernels, where ``function1`` is only called from ``kernel1`` it is sufficient 14966to group the function with the kernel that calls it and reset the symbols 14967between the two connected components: 14968 14969.. code:: 14970 :number-lines: 14971 14972 .amdgcn_target "amdgcn-amd-amdhsa--gfx900+xnack" // optional 14973 14974 // gpr tracking symbols are implicitly set to zero 14975 14976 .text 14977 .globl kern0 14978 .p2align 8 14979 .type kern0,@function 14980 kern0: 14981 // ... 14982 s_endpgm 14983 .Lkern0_end: 14984 .size kern0, .Lkern0_end-kern0 14985 14986 .rodata 14987 .p2align 6 14988 .amdhsa_kernel kern0 14989 // ... 14990 .amdhsa_next_free_vgpr .amdgcn.next_free_vgpr 14991 .amdhsa_next_free_sgpr .amdgcn.next_free_sgpr 14992 .end_amdhsa_kernel 14993 14994 // reset symbols to begin tracking usage in func1 and kern1 14995 .set .amdgcn.next_free_vgpr, 0 14996 .set .amdgcn.next_free_sgpr, 0 14997 14998 .text 14999 .hidden func1 15000 .global func1 15001 .p2align 2 15002 .type func1,@function 15003 func1: 15004 // ... 15005 s_setpc_b64 s[30:31] 15006 .Lfunc1_end: 15007 .size func1, .Lfunc1_end-func1 15008 15009 .globl kern1 15010 .p2align 8 15011 .type kern1,@function 15012 kern1: 15013 // ... 15014 s_getpc_b64 s[4:5] 15015 s_add_u32 s4, s4, func1@rel32@lo+4 15016 s_addc_u32 s5, s5, func1@rel32@lo+4 15017 s_swappc_b64 s[30:31], s[4:5] 15018 // ... 15019 s_endpgm 15020 .Lkern1_end: 15021 .size kern1, .Lkern1_end-kern1 15022 15023 .rodata 15024 .p2align 6 15025 .amdhsa_kernel kern1 15026 // ... 15027 .amdhsa_next_free_vgpr .amdgcn.next_free_vgpr 15028 .amdhsa_next_free_sgpr .amdgcn.next_free_sgpr 15029 .end_amdhsa_kernel 15030 15031These symbols cannot identify connected components in order to automatically 15032track the usage for each kernel. However, in some cases careful organization of 15033the kernels and functions in the source file means there is minimal additional 15034effort required to accurately calculate GPR usage. 15035 15036Additional Documentation 15037======================== 15038 15039.. [AMD-GCN-GFX6] `AMD Southern Islands Series ISA <http://developer.amd.com/wordpress/media/2012/12/AMD_Southern_Islands_Instruction_Set_Architecture.pdf>`__ 15040.. [AMD-GCN-GFX7] `AMD Sea Islands Series ISA <http://developer.amd.com/wordpress/media/2013/07/AMD_Sea_Islands_Instruction_Set_Architecture.pdf>`_ 15041.. [AMD-GCN-GFX8] `AMD GCN3 Instruction Set Architecture <http://amd-dev.wpengine.netdna-cdn.com/wordpress/media/2013/12/AMD_GCN3_Instruction_Set_Architecture_rev1.1.pdf>`__ 15042.. [AMD-GCN-GFX900-GFX904-VEGA] `AMD Vega Instruction Set Architecture <http://developer.amd.com/wordpress/media/2013/12/Vega_Shader_ISA_28July2017.pdf>`__ 15043.. [AMD-GCN-GFX906-VEGA7NM] `AMD Vega 7nm Instruction Set Architecture <https://gpuopen.com/wp-content/uploads/2019/11/Vega_7nm_Shader_ISA_26November2019.pdf>`__ 15044.. [AMD-GCN-GFX908-CDNA1] `AMD Instinct MI100 Instruction Set Architecture <https://developer.amd.com/wp-content/resources/CDNA1_Shader_ISA_14December2020.pdf>`__ 15045.. [AMD-GCN-GFX90A-CDNA2] `AMD Instinct MI200 Instruction Set Architecture <https://developer.amd.com/wp-content/resources/CDNA2_Shader_ISA_4February2022.pdf>`__ 15046.. [AMD-GCN-GFX10-RDNA1] `AMD RDNA 1.0 Instruction Set Architecture <https://gpuopen.com/wp-content/uploads/2019/08/RDNA_Shader_ISA_5August2019.pdf>`__ 15047.. [AMD-GCN-GFX10-RDNA2] `AMD RDNA 2 Instruction Set Architecture <https://developer.amd.com/wp-content/resources/RDNA2_Shader_ISA_November2020.pdf>`__ 15048.. [AMD-RADEON-HD-2000-3000] `AMD R6xx shader ISA <http://developer.amd.com/wordpress/media/2012/10/R600_Instruction_Set_Architecture.pdf>`__ 15049.. [AMD-RADEON-HD-4000] `AMD R7xx shader ISA <http://developer.amd.com/wordpress/media/2012/10/R700-Family_Instruction_Set_Architecture.pdf>`__ 15050.. [AMD-RADEON-HD-5000] `AMD Evergreen shader ISA <http://developer.amd.com/wordpress/media/2012/10/AMD_Evergreen-Family_Instruction_Set_Architecture.pdf>`__ 15051.. [AMD-RADEON-HD-6000] `AMD Cayman/Trinity shader ISA <http://developer.amd.com/wordpress/media/2012/10/AMD_HD_6900_Series_Instruction_Set_Architecture.pdf>`__ 15052.. [AMD-ROCm] `AMD ROCm™ Platform <https://rocmdocs.amd.com/>`__ 15053.. [AMD-ROCm-github] `AMD ROCm™ github <http://github.com/RadeonOpenCompute>`__ 15054.. [AMD-ROCm-Release-Notes] `AMD ROCm Release Notes <https://github.com/RadeonOpenCompute/ROCm>`__ 15055.. [CLANG-ATTR] `Attributes in Clang <https://clang.llvm.org/docs/AttributeReference.html>`__ 15056.. [DWARF] `DWARF Debugging Information Format <http://dwarfstd.org/>`__ 15057.. [ELF] `Executable and Linkable Format (ELF) <http://www.sco.com/developers/gabi/>`__ 15058.. [HRF] `Heterogeneous-race-free Memory Models <http://benedictgaster.org/wp-content/uploads/2014/01/asplos269-FINAL.pdf>`__ 15059.. [HSA] `Heterogeneous System Architecture (HSA) Foundation <http://www.hsafoundation.com/>`__ 15060.. [MsgPack] `Message Pack <http://www.msgpack.org/>`__ 15061.. [OpenCL] `The OpenCL Specification Version 2.0 <http://www.khronos.org/registry/cl/specs/opencl-2.0.pdf>`__ 15062.. [SEMVER] `Semantic Versioning <https://semver.org/>`__ 15063.. [YAML] `YAML Ain't Markup Language (YAML™) Version 1.2 <http://www.yaml.org/spec/1.2/spec.html>`__ 15064