1============================= 2User Guide for AMDGPU Backend 3============================= 4 5.. contents:: 6 :local: 7 8.. toctree:: 9 :hidden: 10 11 AMDGPU/AMDGPUAsmGFX7 12 AMDGPU/AMDGPUAsmGFX8 13 AMDGPU/AMDGPUAsmGFX9 14 AMDGPU/AMDGPUAsmGFX900 15 AMDGPU/AMDGPUAsmGFX904 16 AMDGPU/AMDGPUAsmGFX906 17 AMDGPU/AMDGPUAsmGFX908 18 AMDGPU/AMDGPUAsmGFX90a 19 AMDGPU/AMDGPUAsmGFX10 20 AMDGPU/AMDGPUAsmGFX1011 21 AMDGPUModifierSyntax 22 AMDGPUOperandSyntax 23 AMDGPUInstructionSyntax 24 AMDGPUInstructionNotation 25 AMDGPUDwarfExtensionsForHeterogeneousDebugging 26 AMDGPUDwarfExtensionAllowLocationDescriptionOnTheDwarfExpressionStack/AMDGPUDwarfExtensionAllowLocationDescriptionOnTheDwarfExpressionStack 27 28Introduction 29============ 30 31The AMDGPU backend provides ISA code generation for AMD GPUs, starting with the 32R600 family up until the current GCN families. It lives in the 33``llvm/lib/Target/AMDGPU`` directory. 34 35LLVM 36==== 37 38.. _amdgpu-target-triples: 39 40Target Triples 41-------------- 42 43Use the Clang option ``-target <Architecture>-<Vendor>-<OS>-<Environment>`` 44to specify the target triple: 45 46 .. table:: AMDGPU Architectures 47 :name: amdgpu-architecture-table 48 49 ============ ============================================================== 50 Architecture Description 51 ============ ============================================================== 52 ``r600`` AMD GPUs HD2XXX-HD6XXX for graphics and compute shaders. 53 ``amdgcn`` AMD GPUs GCN GFX6 onwards for graphics and compute shaders. 54 ============ ============================================================== 55 56 .. table:: AMDGPU Vendors 57 :name: amdgpu-vendor-table 58 59 ============ ============================================================== 60 Vendor Description 61 ============ ============================================================== 62 ``amd`` Can be used for all AMD GPU usage. 63 ``mesa3d`` Can be used if the OS is ``mesa3d``. 64 ============ ============================================================== 65 66 .. table:: AMDGPU Operating Systems 67 :name: amdgpu-os 68 69 ============== ============================================================ 70 OS Description 71 ============== ============================================================ 72 *<empty>* Defaults to the *unknown* OS. 73 ``amdhsa`` Compute kernels executed on HSA [HSA]_ compatible runtimes 74 such as: 75 76 - AMD's ROCm™ runtime [AMD-ROCm]_ using the *rocm-amdhsa* 77 loader on Linux. See *AMD ROCm Platform Release Notes* 78 [AMD-ROCm-Release-Notes]_ for supported hardware and 79 software. 80 - AMD's PAL runtime using the *pal-amdhsa* loader on 81 Windows. 82 83 ``amdpal`` Graphic shaders and compute kernels executed on AMD's PAL 84 runtime using the *pal-amdpal* loader on Windows and Linux 85 Pro. 86 ``mesa3d`` Graphic shaders and compute kernels executed on AMD's Mesa 87 3D runtime using the *mesa-mesa3d* loader on Linux. 88 ============== ============================================================ 89 90 .. table:: AMDGPU Environments 91 :name: amdgpu-environment-table 92 93 ============ ============================================================== 94 Environment Description 95 ============ ============================================================== 96 *<empty>* Default. 97 ============ ============================================================== 98 99.. _amdgpu-processors: 100 101Processors 102---------- 103 104Use the Clang options ``-mcpu=<target-id>`` or ``--offload-arch=<target-id>`` to 105specify the AMDGPU processor together with optional target features. See 106:ref:`amdgpu-target-id` and :ref:`amdgpu-target-features` for AMD GPU target 107specific information. 108 109Every processor supports every OS ABI (see :ref:`amdgpu-os`) with the following exceptions: 110 111* ``amdhsa`` is not supported in ``r600`` architecture (see :ref:`amdgpu-architecture-table`). 112 113 114 .. table:: AMDGPU Processors 115 :name: amdgpu-processor-table 116 117 =========== =============== ============ ===== ================= =============== =============== ====================== 118 Processor Alternative Target dGPU/ Target Target OS Support Example 119 Processor Triple APU Features Properties *(see* Products 120 Architecture Supported `amdgpu-os`_ 121 *and 122 corresponding 123 runtime release 124 notes for 125 current 126 information and 127 level of 128 support)* 129 =========== =============== ============ ===== ================= =============== =============== ====================== 130 **Radeon HD 2000/3000 Series (R600)** [AMD-RADEON-HD-2000-3000]_ 131 ----------------------------------------------------------------------------------------------------------------------- 132 ``r600`` ``r600`` dGPU - Does not 133 support 134 generic 135 address 136 space 137 ``r630`` ``r600`` dGPU - Does not 138 support 139 generic 140 address 141 space 142 ``rs880`` ``r600`` dGPU - Does not 143 support 144 generic 145 address 146 space 147 ``rv670`` ``r600`` dGPU - Does not 148 support 149 generic 150 address 151 space 152 **Radeon HD 4000 Series (R700)** [AMD-RADEON-HD-4000]_ 153 ----------------------------------------------------------------------------------------------------------------------- 154 ``rv710`` ``r600`` dGPU - Does not 155 support 156 generic 157 address 158 space 159 ``rv730`` ``r600`` dGPU - Does not 160 support 161 generic 162 address 163 space 164 ``rv770`` ``r600`` dGPU - Does not 165 support 166 generic 167 address 168 space 169 **Radeon HD 5000 Series (Evergreen)** [AMD-RADEON-HD-5000]_ 170 ----------------------------------------------------------------------------------------------------------------------- 171 ``cedar`` ``r600`` dGPU - Does not 172 support 173 generic 174 address 175 space 176 ``cypress`` ``r600`` dGPU - Does not 177 support 178 generic 179 address 180 space 181 ``juniper`` ``r600`` dGPU - Does not 182 support 183 generic 184 address 185 space 186 ``redwood`` ``r600`` dGPU - Does not 187 support 188 generic 189 address 190 space 191 ``sumo`` ``r600`` dGPU - Does not 192 support 193 generic 194 address 195 space 196 **Radeon HD 6000 Series (Northern Islands)** [AMD-RADEON-HD-6000]_ 197 ----------------------------------------------------------------------------------------------------------------------- 198 ``barts`` ``r600`` dGPU - Does not 199 support 200 generic 201 address 202 space 203 ``caicos`` ``r600`` dGPU - Does not 204 support 205 generic 206 address 207 space 208 ``cayman`` ``r600`` dGPU - Does not 209 support 210 generic 211 address 212 space 213 ``turks`` ``r600`` dGPU - Does not 214 support 215 generic 216 address 217 space 218 **GCN GFX6 (Southern Islands (SI))** [AMD-GCN-GFX6]_ 219 ----------------------------------------------------------------------------------------------------------------------- 220 ``gfx600`` - ``tahiti`` ``amdgcn`` dGPU - Does not - *pal-amdpal* 221 support 222 generic 223 address 224 space 225 ``gfx601`` - ``pitcairn`` ``amdgcn`` dGPU - Does not - *pal-amdpal* 226 - ``verde`` support 227 generic 228 address 229 space 230 ``gfx602`` - ``hainan`` ``amdgcn`` dGPU - Does not - *pal-amdpal* 231 - ``oland`` support 232 generic 233 address 234 space 235 **GCN GFX7 (Sea Islands (CI))** [AMD-GCN-GFX7]_ 236 ----------------------------------------------------------------------------------------------------------------------- 237 ``gfx700`` - ``kaveri`` ``amdgcn`` APU - Offset - *rocm-amdhsa* - A6-7000 238 flat - *pal-amdhsa* - A6 Pro-7050B 239 scratch - *pal-amdpal* - A8-7100 240 - A8 Pro-7150B 241 - A10-7300 242 - A10 Pro-7350B 243 - FX-7500 244 - A8-7200P 245 - A10-7400P 246 - FX-7600P 247 ``gfx701`` - ``hawaii`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - FirePro W8100 248 flat - *pal-amdhsa* - FirePro W9100 249 scratch - *pal-amdpal* - FirePro S9150 250 - FirePro S9170 251 ``gfx702`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - Radeon R9 290 252 flat - *pal-amdhsa* - Radeon R9 290x 253 scratch - *pal-amdpal* - Radeon R390 254 - Radeon R390x 255 ``gfx703`` - ``kabini`` ``amdgcn`` APU - Offset - *pal-amdhsa* - E1-2100 256 - ``mullins`` flat - *pal-amdpal* - E1-2200 257 scratch - E1-2500 258 - E2-3000 259 - E2-3800 260 - A4-5000 261 - A4-5100 262 - A6-5200 263 - A4 Pro-3340B 264 ``gfx704`` - ``bonaire`` ``amdgcn`` dGPU - Offset - *pal-amdhsa* - Radeon HD 7790 265 flat - *pal-amdpal* - Radeon HD 8770 266 scratch - R7 260 267 - R7 260X 268 ``gfx705`` ``amdgcn`` APU - Offset - *pal-amdhsa* *TBA* 269 flat - *pal-amdpal* 270 scratch .. TODO:: 271 272 Add product 273 names. 274 275 **GCN GFX8 (Volcanic Islands (VI))** [AMD-GCN-GFX8]_ 276 ----------------------------------------------------------------------------------------------------------------------- 277 ``gfx801`` - ``carrizo`` ``amdgcn`` APU - xnack - Offset - *rocm-amdhsa* - A6-8500P 278 flat - *pal-amdhsa* - Pro A6-8500B 279 scratch - *pal-amdpal* - A8-8600P 280 - Pro A8-8600B 281 - FX-8800P 282 - Pro A12-8800B 283 - A10-8700P 284 - Pro A10-8700B 285 - A10-8780P 286 - A10-9600P 287 - A10-9630P 288 - A12-9700P 289 - A12-9730P 290 - FX-9800P 291 - FX-9830P 292 - E2-9010 293 - A6-9210 294 - A9-9410 295 ``gfx802`` - ``iceland`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - Radeon R9 285 296 - ``tonga`` flat - *pal-amdhsa* - Radeon R9 380 297 scratch - *pal-amdpal* - Radeon R9 385 298 ``gfx803`` - ``fiji`` ``amdgcn`` dGPU - *rocm-amdhsa* - Radeon R9 Nano 299 - *pal-amdhsa* - Radeon R9 Fury 300 - *pal-amdpal* - Radeon R9 FuryX 301 - Radeon Pro Duo 302 - FirePro S9300x2 303 - Radeon Instinct MI8 304 \ - ``polaris10`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - Radeon RX 470 305 flat - *pal-amdhsa* - Radeon RX 480 306 scratch - *pal-amdpal* - Radeon Instinct MI6 307 \ - ``polaris11`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - Radeon RX 460 308 flat - *pal-amdhsa* 309 scratch - *pal-amdpal* 310 ``gfx805`` - ``tongapro`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - FirePro S7150 311 flat - *pal-amdhsa* - FirePro S7100 312 scratch - *pal-amdpal* - FirePro W7100 313 - Mobile FirePro 314 M7170 315 ``gfx810`` - ``stoney`` ``amdgcn`` APU - xnack - Offset - *rocm-amdhsa* *TBA* 316 flat - *pal-amdhsa* 317 scratch - *pal-amdpal* .. TODO:: 318 319 Add product 320 names. 321 322 **GCN GFX9 (Vega)** [AMD-GCN-GFX900-GFX904-VEGA]_ [AMD-GCN-GFX906-VEGA7NM]_ [AMD-GCN-GFX908-CDNA1]_ 323 ----------------------------------------------------------------------------------------------------------------------- 324 ``gfx900`` ``amdgcn`` dGPU - xnack - Absolute - *rocm-amdhsa* - Radeon Vega 325 flat - *pal-amdhsa* Frontier Edition 326 scratch - *pal-amdpal* - Radeon RX Vega 56 327 - Radeon RX Vega 64 328 - Radeon RX Vega 64 329 Liquid 330 - Radeon Instinct MI25 331 ``gfx902`` ``amdgcn`` APU - xnack - Absolute - *rocm-amdhsa* - Ryzen 3 2200G 332 flat - *pal-amdhsa* - Ryzen 5 2400G 333 scratch - *pal-amdpal* 334 ``gfx904`` ``amdgcn`` dGPU - xnack - *rocm-amdhsa* *TBA* 335 - *pal-amdhsa* 336 - *pal-amdpal* .. TODO:: 337 338 Add product 339 names. 340 341 ``gfx906`` ``amdgcn`` dGPU - sramecc - Absolute - *rocm-amdhsa* - Radeon Instinct MI50 342 - xnack flat - *pal-amdhsa* - Radeon Instinct MI60 343 scratch - *pal-amdpal* - Radeon VII 344 - Radeon Pro VII 345 ``gfx908`` ``amdgcn`` dGPU - sramecc - *rocm-amdhsa* - AMD Instinct MI100 Accelerator 346 - xnack - Absolute 347 flat 348 scratch 349 ``gfx909`` ``amdgcn`` APU - xnack - Absolute - *pal-amdpal* *TBA* 350 flat 351 scratch .. TODO:: 352 353 Add product 354 names. 355 356 ``gfx90a`` ``amdgcn`` dGPU - sramecc - Absolute - *rocm-amdhsa* *TBA* 357 - tgsplit flat 358 - xnack scratch .. TODO:: 359 - Packed 360 work-item Add product 361 IDs names. 362 363 ``gfx90c`` ``amdgcn`` APU - xnack - Absolute - *pal-amdpal* - Ryzen 7 4700G 364 flat - Ryzen 7 4700GE 365 scratch - Ryzen 5 4600G 366 - Ryzen 5 4600GE 367 - Ryzen 3 4300G 368 - Ryzen 3 4300GE 369 - Ryzen Pro 4000G 370 - Ryzen 7 Pro 4700G 371 - Ryzen 7 Pro 4750GE 372 - Ryzen 5 Pro 4650G 373 - Ryzen 5 Pro 4650GE 374 - Ryzen 3 Pro 4350G 375 - Ryzen 3 Pro 4350GE 376 377 ``gfx940`` ``amdgcn`` dGPU - sramecc - Architected *TBA* 378 - tgsplit flat 379 - xnack scratch .. TODO:: 380 - Packed 381 work-item Add product 382 IDs names. 383 384 **GCN GFX10.1 (RDNA 1)** [AMD-GCN-GFX10-RDNA1]_ 385 ----------------------------------------------------------------------------------------------------------------------- 386 ``gfx1010`` ``amdgcn`` dGPU - cumode - Absolute - *rocm-amdhsa* - Radeon RX 5700 387 - wavefrontsize64 flat - *pal-amdhsa* - Radeon RX 5700 XT 388 - xnack scratch - *pal-amdpal* - Radeon Pro 5600 XT 389 - Radeon Pro 5600M 390 ``gfx1011`` ``amdgcn`` dGPU - cumode - *rocm-amdhsa* - Radeon Pro V520 391 - wavefrontsize64 - Absolute - *pal-amdhsa* 392 - xnack flat - *pal-amdpal* 393 scratch 394 ``gfx1012`` ``amdgcn`` dGPU - cumode - Absolute - *rocm-amdhsa* - Radeon RX 5500 395 - wavefrontsize64 flat - *pal-amdhsa* - Radeon RX 5500 XT 396 - xnack scratch - *pal-amdpal* 397 ``gfx1013`` ``amdgcn`` APU - cumode - Absolute - *rocm-amdhsa* *TBA* 398 - wavefrontsize64 flat - *pal-amdhsa* 399 - xnack scratch - *pal-amdpal* .. TODO:: 400 401 Add product 402 names. 403 404 **GCN GFX10.3 (RDNA 2)** [AMD-GCN-GFX10-RDNA2]_ 405 ----------------------------------------------------------------------------------------------------------------------- 406 ``gfx1030`` ``amdgcn`` dGPU - cumode - Absolute - *rocm-amdhsa* - Radeon RX 6800 407 - wavefrontsize64 flat - *pal-amdhsa* - Radeon RX 6800 XT 408 scratch - *pal-amdpal* - Radeon RX 6900 XT 409 ``gfx1031`` ``amdgcn`` dGPU - cumode - Absolute - *rocm-amdhsa* - Radeon RX 6700 XT 410 - wavefrontsize64 flat - *pal-amdhsa* 411 scratch - *pal-amdpal* 412 ``gfx1032`` ``amdgcn`` dGPU - cumode - Absolute - *rocm-amdhsa* *TBA* 413 - wavefrontsize64 flat - *pal-amdhsa* 414 scratch - *pal-amdpal* .. TODO:: 415 416 Add product 417 names. 418 419 ``gfx1033`` ``amdgcn`` APU - cumode - Absolute - *pal-amdpal* *TBA* 420 - wavefrontsize64 flat 421 scratch .. TODO:: 422 423 Add product 424 names. 425 ``gfx1034`` ``amdgcn`` dGPU - cumode - Absolute - *pal-amdpal* *TBA* 426 - wavefrontsize64 flat 427 scratch .. TODO:: 428 429 Add product 430 names. 431 432 ``gfx1035`` ``amdgcn`` APU - cumode - Absolute - *pal-amdpal* *TBA* 433 - wavefrontsize64 flat 434 scratch .. TODO:: 435 Add product 436 names. 437 438 ``gfx1036`` ``amdgcn`` APU - cumode - Absolute - *pal-amdpal* *TBA* 439 - wavefrontsize64 flat 440 scratch .. TODO:: 441 442 Add product 443 names. 444 445 =========== =============== ============ ===== ================= =============== =============== ====================== 446 447.. _amdgpu-target-features: 448 449Target Features 450--------------- 451 452Target features control how code is generated to support certain 453processor specific features. Not all target features are supported by 454all processors. The runtime must ensure that the features supported by 455the device used to execute the code match the features enabled when 456generating the code. A mismatch of features may result in incorrect 457execution, or a reduction in performance. 458 459The target features supported by each processor is listed in 460:ref:`amdgpu-processor-table`. 461 462Target features are controlled by exactly one of the following Clang 463options: 464 465``-mcpu=<target-id>`` or ``--offload-arch=<target-id>`` 466 467 The ``-mcpu`` and ``--offload-arch`` can specify the target feature as 468 optional components of the target ID. If omitted, the target feature has the 469 ``any`` value. See :ref:`amdgpu-target-id`. 470 471``-m[no-]<target-feature>`` 472 473 Target features not specified by the target ID are specified using a 474 separate option. These target features can have an ``on`` or ``off`` 475 value. ``on`` is specified by omitting the ``no-`` prefix, and 476 ``off`` is specified by including the ``no-`` prefix. The default 477 if not specified is ``off``. 478 479For example: 480 481``-mcpu=gfx908:xnack+`` 482 Enable the ``xnack`` feature. 483``-mcpu=gfx908:xnack-`` 484 Disable the ``xnack`` feature. 485``-mcumode`` 486 Enable the ``cumode`` feature. 487``-mno-cumode`` 488 Disable the ``cumode`` feature. 489 490 .. table:: AMDGPU Target Features 491 :name: amdgpu-target-features-table 492 493 =============== ============================ ================================================== 494 Target Feature Clang Option to Control Description 495 Name 496 =============== ============================ ================================================== 497 cumode - ``-m[no-]cumode`` Control the wavefront execution mode used 498 when generating code for kernels. When disabled 499 native WGP wavefront execution mode is used, 500 when enabled CU wavefront execution mode is used 501 (see :ref:`amdgpu-amdhsa-memory-model`). 502 503 sramecc - ``-mcpu`` If specified, generate code that can only be 504 - ``--offload-arch`` loaded and executed in a process that has a 505 matching setting for SRAMECC. 506 507 If not specified for code object V2 to V3, generate 508 code that can be loaded and executed in a process 509 with SRAMECC enabled. 510 511 If not specified for code object V4 or above, generate 512 code that can be loaded and executed in a process 513 with either setting of SRAMECC. 514 515 tgsplit ``-m[no-]tgsplit`` Enable/disable generating code that assumes 516 work-groups are launched in threadgroup split mode. 517 When enabled the waves of a work-group may be 518 launched in different CUs. 519 520 wavefrontsize64 - ``-m[no-]wavefrontsize64`` Control the wavefront size used when 521 generating code for kernels. When disabled 522 native wavefront size 32 is used, when enabled 523 wavefront size 64 is used. 524 525 xnack - ``-mcpu`` If specified, generate code that can only be 526 - ``--offload-arch`` loaded and executed in a process that has a 527 matching setting for XNACK replay. 528 529 If not specified for code object V2 to V3, generate 530 code that can be loaded and executed in a process 531 with XNACK replay enabled. 532 533 If not specified for code object V4 or above, generate 534 code that can be loaded and executed in a process 535 with either setting of XNACK replay. 536 537 XNACK replay can be used for demand paging and 538 page migration. If enabled in the device, then if 539 a page fault occurs the code may execute 540 incorrectly unless generated with XNACK replay 541 enabled, or generated for code object V4 or above without 542 specifying XNACK replay. Executing code that was 543 generated with XNACK replay enabled, or generated 544 for code object V4 or above without specifying XNACK replay, 545 on a device that does not have XNACK replay 546 enabled will execute correctly but may be less 547 performant than code generated for XNACK replay 548 disabled. 549 =============== ============================ ================================================== 550 551.. _amdgpu-target-id: 552 553Target ID 554--------- 555 556AMDGPU supports target IDs. See `Clang Offload Bundler 557<https://clang.llvm.org/docs/ClangOffloadBundler.html>`_ for a general 558description. The AMDGPU target specific information is: 559 560**processor** 561 Is an AMDGPU processor or alternative processor name specified in 562 :ref:`amdgpu-processor-table`. The non-canonical form target ID allows both 563 the primary processor and alternative processor names. The canonical form 564 target ID only allow the primary processor name. 565 566**target-feature** 567 Is a target feature name specified in :ref:`amdgpu-target-features-table` that 568 is supported by the processor. The target features supported by each processor 569 is specified in :ref:`amdgpu-processor-table`. Those that can be specified in 570 a target ID are marked as being controlled by ``-mcpu`` and 571 ``--offload-arch``. Each target feature must appear at most once in a target 572 ID. The non-canonical form target ID allows the target features to be 573 specified in any order. The canonical form target ID requires the target 574 features to be specified in alphabetic order. 575 576.. _amdgpu-target-id-v2-v3: 577 578Code Object V2 to V3 Target ID 579~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 580 581The target ID syntax for code object V2 to V3 is the same as defined in `Clang 582Offload Bundler <https://clang.llvm.org/docs/ClangOffloadBundler.html>`_ except 583when used in the :ref:`amdgpu-assembler-directive-amdgcn-target` assembler 584directive and the bundle entry ID. In those cases it has the following BNF 585syntax: 586 587.. code:: 588 589 <target-id> ::== <processor> ( "+" <target-feature> )* 590 591Where a target feature is omitted if *Off* and present if *On* or *Any*. 592 593.. note:: 594 595 The code object V2 to V3 cannot represent *Any* and treats it the same as 596 *On*. 597 598.. _amdgpu-embedding-bundled-objects: 599 600Embedding Bundled Code Objects 601------------------------------ 602 603AMDGPU supports the HIP and OpenMP languages that perform code object embedding 604as described in `Clang Offload Bundler 605<https://clang.llvm.org/docs/ClangOffloadBundler.html>`_. 606 607.. note:: 608 609 The target ID syntax used for code object V2 to V3 for a bundle entry ID 610 differs from that used elsewhere. See :ref:`amdgpu-target-id-v2-v3`. 611 612.. _amdgpu-address-spaces: 613 614Address Spaces 615-------------- 616 617The AMDGPU architecture supports a number of memory address spaces. The address 618space names use the OpenCL standard names, with some additions. 619 620The AMDGPU address spaces correspond to target architecture specific LLVM 621address space numbers used in LLVM IR. 622 623The AMDGPU address spaces are described in 624:ref:`amdgpu-address-spaces-table`. Only 64-bit process address spaces are 625supported for the ``amdgcn`` target. 626 627 .. table:: AMDGPU Address Spaces 628 :name: amdgpu-address-spaces-table 629 630 ================================= =============== =========== ================ ======= ============================ 631 .. 64-Bit Process Address Space 632 --------------------------------- --------------- ----------- ---------------- ------------------------------------ 633 Address Space Name LLVM IR Address HSA Segment Hardware Address NULL Value 634 Space Number Name Name Size 635 ================================= =============== =========== ================ ======= ============================ 636 Generic 0 flat flat 64 0x0000000000000000 637 Global 1 global global 64 0x0000000000000000 638 Region 2 N/A GDS 32 *not implemented for AMDHSA* 639 Local 3 group LDS 32 0xFFFFFFFF 640 Constant 4 constant *same as global* 64 0x0000000000000000 641 Private 5 private scratch 32 0xFFFFFFFF 642 Constant 32-bit 6 *TODO* 0x00000000 643 Buffer Fat Pointer (experimental) 7 *TODO* 644 ================================= =============== =========== ================ ======= ============================ 645 646**Generic** 647 The generic address space is supported unless the *Target Properties* column 648 of :ref:`amdgpu-processor-table` specifies *Does not support generic address 649 space*. 650 651 The generic address space uses the hardware flat address support for two fixed 652 ranges of virtual addresses (the private and local apertures), that are 653 outside the range of addressable global memory, to map from a flat address to 654 a private or local address. This uses FLAT instructions that can take a flat 655 address and access global, private (scratch), and group (LDS) memory depending 656 on if the address is within one of the aperture ranges. 657 658 Flat access to scratch requires hardware aperture setup and setup in the 659 kernel prologue (see :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`). Flat 660 access to LDS requires hardware aperture setup and M0 (GFX7-GFX8) register 661 setup (see :ref:`amdgpu-amdhsa-kernel-prolog-m0`). 662 663 To convert between a private or group address space address (termed a segment 664 address) and a flat address the base address of the corresponding aperture 665 can be used. For GFX7-GFX8 these are available in the 666 :ref:`amdgpu-amdhsa-hsa-aql-queue` the address of which can be obtained with 667 Queue Ptr SGPR (see :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). For 668 GFX9-GFX10 the aperture base addresses are directly available as inline 669 constant registers ``SRC_SHARED_BASE/LIMIT`` and ``SRC_PRIVATE_BASE/LIMIT``. 670 In 64-bit address mode the aperture sizes are 2^32 bytes and the base is 671 aligned to 2^32 which makes it easier to convert from flat to segment or 672 segment to flat. 673 674 A global address space address has the same value when used as a flat address 675 so no conversion is needed. 676 677**Global and Constant** 678 The global and constant address spaces both use global virtual addresses, 679 which are the same virtual address space used by the CPU. However, some 680 virtual addresses may only be accessible to the CPU, some only accessible 681 by the GPU, and some by both. 682 683 Using the constant address space indicates that the data will not change 684 during the execution of the kernel. This allows scalar read instructions to 685 be used. As the constant address space could only be modified on the host 686 side, a generic pointer loaded from the constant address space is safe to be 687 assumed as a global pointer since only the device global memory is visible 688 and managed on the host side. The vector and scalar L1 caches are invalidated 689 of volatile data before each kernel dispatch execution to allow constant 690 memory to change values between kernel dispatches. 691 692**Region** 693 The region address space uses the hardware Global Data Store (GDS). All 694 wavefronts executing on the same device will access the same memory for any 695 given region address. However, the same region address accessed by wavefronts 696 executing on different devices will access different memory. It is higher 697 performance than global memory. It is allocated by the runtime. The data 698 store (DS) instructions can be used to access it. 699 700**Local** 701 The local address space uses the hardware Local Data Store (LDS) which is 702 automatically allocated when the hardware creates the wavefronts of a 703 work-group, and freed when all the wavefronts of a work-group have 704 terminated. All wavefronts belonging to the same work-group will access the 705 same memory for any given local address. However, the same local address 706 accessed by wavefronts belonging to different work-groups will access 707 different memory. It is higher performance than global memory. The data store 708 (DS) instructions can be used to access it. 709 710**Private** 711 The private address space uses the hardware scratch memory support which 712 automatically allocates memory when it creates a wavefront and frees it when 713 a wavefronts terminates. The memory accessed by a lane of a wavefront for any 714 given private address will be different to the memory accessed by another lane 715 of the same or different wavefront for the same private address. 716 717 If a kernel dispatch uses scratch, then the hardware allocates memory from a 718 pool of backing memory allocated by the runtime for each wavefront. The lanes 719 of the wavefront access this using dword (4 byte) interleaving. The mapping 720 used from private address to backing memory address is: 721 722 ``wavefront-scratch-base + 723 ((private-address / 4) * wavefront-size * 4) + 724 (wavefront-lane-id * 4) + (private-address % 4)`` 725 726 If each lane of a wavefront accesses the same private address, the 727 interleaving results in adjacent dwords being accessed and hence requires 728 fewer cache lines to be fetched. 729 730 There are different ways that the wavefront scratch base address is 731 determined by a wavefront (see 732 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 733 734 Scratch memory can be accessed in an interleaved manner using buffer 735 instructions with the scratch buffer descriptor and per wavefront scratch 736 offset, by the scratch instructions, or by flat instructions. Multi-dword 737 access is not supported except by flat and scratch instructions in 738 GFX9-GFX10. 739 740**Constant 32-bit** 741 *TODO* 742 743**Buffer Fat Pointer** 744 The buffer fat pointer is an experimental address space that is currently 745 unsupported in the backend. It exposes a non-integral pointer that is in 746 the future intended to support the modelling of 128-bit buffer descriptors 747 plus a 32-bit offset into the buffer (in total encapsulating a 160-bit 748 *pointer*), allowing normal LLVM load/store/atomic operations to be used to 749 model the buffer descriptors used heavily in graphics workloads targeting 750 the backend. 751 752.. _amdgpu-memory-scopes: 753 754Memory Scopes 755------------- 756 757This section provides LLVM memory synchronization scopes supported by the AMDGPU 758backend memory model when the target triple OS is ``amdhsa`` (see 759:ref:`amdgpu-amdhsa-memory-model` and :ref:`amdgpu-target-triples`). 760 761The memory model supported is based on the HSA memory model [HSA]_ which is 762based in turn on HRF-indirect with scope inclusion [HRF]_. The happens-before 763relation is transitive over the synchronizes-with relation independent of scope 764and synchronizes-with allows the memory scope instances to be inclusive (see 765table :ref:`amdgpu-amdhsa-llvm-sync-scopes-table`). 766 767This is different to the OpenCL [OpenCL]_ memory model which does not have scope 768inclusion and requires the memory scopes to exactly match. However, this 769is conservatively correct for OpenCL. 770 771 .. table:: AMDHSA LLVM Sync Scopes 772 :name: amdgpu-amdhsa-llvm-sync-scopes-table 773 774 ======================= =================================================== 775 LLVM Sync Scope Description 776 ======================= =================================================== 777 *none* The default: ``system``. 778 779 Synchronizes with, and participates in modification 780 and seq_cst total orderings with, other operations 781 (except image operations) for all address spaces 782 (except private, or generic that accesses private) 783 provided the other operation's sync scope is: 784 785 - ``system``. 786 - ``agent`` and executed by a thread on the same 787 agent. 788 - ``workgroup`` and executed by a thread in the 789 same work-group. 790 - ``wavefront`` and executed by a thread in the 791 same wavefront. 792 793 ``agent`` Synchronizes with, and participates in modification 794 and seq_cst total orderings with, other operations 795 (except image operations) for all address spaces 796 (except private, or generic that accesses private) 797 provided the other operation's sync scope is: 798 799 - ``system`` or ``agent`` and executed by a thread 800 on the same agent. 801 - ``workgroup`` and executed by a thread in the 802 same work-group. 803 - ``wavefront`` and executed by a thread in the 804 same wavefront. 805 806 ``workgroup`` Synchronizes with, and participates in modification 807 and seq_cst total orderings with, other operations 808 (except image operations) for all address spaces 809 (except private, or generic that accesses private) 810 provided the other operation's sync scope is: 811 812 - ``system``, ``agent`` or ``workgroup`` and 813 executed by a thread in the same work-group. 814 - ``wavefront`` and executed by a thread in the 815 same wavefront. 816 817 ``wavefront`` Synchronizes with, and participates in modification 818 and seq_cst total orderings with, other operations 819 (except image operations) for all address spaces 820 (except private, or generic that accesses private) 821 provided the other operation's sync scope is: 822 823 - ``system``, ``agent``, ``workgroup`` or 824 ``wavefront`` and executed by a thread in the 825 same wavefront. 826 827 ``singlethread`` Only synchronizes with and participates in 828 modification and seq_cst total orderings with, 829 other operations (except image operations) running 830 in the same thread for all address spaces (for 831 example, in signal handlers). 832 833 ``one-as`` Same as ``system`` but only synchronizes with other 834 operations within the same address space. 835 836 ``agent-one-as`` Same as ``agent`` but only synchronizes with other 837 operations within the same address space. 838 839 ``workgroup-one-as`` Same as ``workgroup`` but only synchronizes with 840 other operations within the same address space. 841 842 ``wavefront-one-as`` Same as ``wavefront`` but only synchronizes with 843 other operations within the same address space. 844 845 ``singlethread-one-as`` Same as ``singlethread`` but only synchronizes with 846 other operations within the same address space. 847 ======================= =================================================== 848 849LLVM IR Intrinsics 850------------------ 851 852The AMDGPU backend implements the following LLVM IR intrinsics. 853 854*This section is WIP.* 855 856.. TODO:: 857 858 List AMDGPU intrinsics. 859 860LLVM IR Attributes 861------------------ 862 863The AMDGPU backend supports the following LLVM IR attributes. 864 865 .. table:: AMDGPU LLVM IR Attributes 866 :name: amdgpu-llvm-ir-attributes-table 867 868 ======================================= ========================================================== 869 LLVM Attribute Description 870 ======================================= ========================================================== 871 "amdgpu-flat-work-group-size"="min,max" Specify the minimum and maximum flat work group sizes that 872 will be specified when the kernel is dispatched. Generated 873 by the ``amdgpu_flat_work_group_size`` CLANG attribute [CLANG-ATTR]_. 874 The implied default value is 1,1024. 875 876 "amdgpu-implicitarg-num-bytes"="n" Number of kernel argument bytes to add to the kernel 877 argument block size for the implicit arguments. This 878 varies by OS and language (for OpenCL see 879 :ref:`opencl-kernel-implicit-arguments-appended-for-amdhsa-os-table`). 880 "amdgpu-num-sgpr"="n" Specifies the number of SGPRs to use. Generated by 881 the ``amdgpu_num_sgpr`` CLANG attribute [CLANG-ATTR]_. 882 "amdgpu-num-vgpr"="n" Specifies the number of VGPRs to use. Generated by the 883 ``amdgpu_num_vgpr`` CLANG attribute [CLANG-ATTR]_. 884 "amdgpu-waves-per-eu"="m,n" Specify the minimum and maximum number of waves per 885 execution unit. Generated by the ``amdgpu_waves_per_eu`` 886 CLANG attribute [CLANG-ATTR]_. This is an optimization hint, 887 and the backend may not be able to satisfy the request. If 888 the specified range is incompatible with the function's 889 "amdgpu-flat-work-group-size" value, the implied occupancy 890 bounds by the workgroup size takes precedence. 891 892 "amdgpu-ieee" true/false. Specify whether the function expects the IEEE field of the 893 mode register to be set on entry. Overrides the default for 894 the calling convention. 895 "amdgpu-dx10-clamp" true/false. Specify whether the function expects the DX10_CLAMP field of 896 the mode register to be set on entry. Overrides the default 897 for the calling convention. 898 899 "amdgpu-no-workitem-id-x" Indicates the function does not depend on the value of the 900 llvm.amdgcn.workitem.id.x intrinsic. If a function is marked with this 901 attribute, or reached through a call site marked with this attribute, 902 the value returned by the intrinsic is undefined. The backend can 903 generally infer this during code generation, so typically there is no 904 benefit to frontends marking functions with this. 905 906 "amdgpu-no-workitem-id-y" The same as amdgpu-no-workitem-id-x, except for the 907 llvm.amdgcn.workitem.id.y intrinsic. 908 909 "amdgpu-no-workitem-id-z" The same as amdgpu-no-workitem-id-x, except for the 910 llvm.amdgcn.workitem.id.z intrinsic. 911 912 "amdgpu-no-workgroup-id-x" The same as amdgpu-no-workitem-id-x, except for the 913 llvm.amdgcn.workgroup.id.x intrinsic. 914 915 "amdgpu-no-workgroup-id-y" The same as amdgpu-no-workitem-id-x, except for the 916 llvm.amdgcn.workgroup.id.y intrinsic. 917 918 "amdgpu-no-workgroup-id-z" The same as amdgpu-no-workitem-id-x, except for the 919 llvm.amdgcn.workgroup.id.z intrinsic. 920 921 "amdgpu-no-dispatch-ptr" The same as amdgpu-no-workitem-id-x, except for the 922 llvm.amdgcn.dispatch.ptr intrinsic. 923 924 "amdgpu-no-implicitarg-ptr" The same as amdgpu-no-workitem-id-x, except for the 925 llvm.amdgcn.implicitarg.ptr intrinsic. 926 927 "amdgpu-no-dispatch-id" The same as amdgpu-no-workitem-id-x, except for the 928 llvm.amdgcn.dispatch.id intrinsic. 929 930 "amdgpu-no-queue-ptr" Similar to amdgpu-no-workitem-id-x, except for the 931 llvm.amdgcn.queue.ptr intrinsic. Note that unlike the other ABI hint 932 attributes, the queue pointer may be required in situations where the 933 intrinsic call does not directly appear in the program. Some subtargets 934 require the queue pointer for to handle some addrspacecasts, as well 935 as the llvm.amdgcn.is.shared, llvm.amdgcn.is.private, llvm.trap, and 936 llvm.debug intrinsics. 937 938 "amdgpu-no-hostcall-ptr" Similar to amdgpu-no-implicitarg-ptr, except specific to the implicit 939 kernel argument that holds the pointer to the hostcall buffer. If this 940 attribute is absent, then the amdgpu-no-implicitarg-ptr is also removed. 941 942 "amdgpu-no-heap-ptr" Similar to amdgpu-no-implicitarg-ptr, except specific to the implicit 943 kernel argument that holds the pointer to an initialized memory buffer 944 that conforms to the requirements of the malloc/free device library V1 945 version implementation. If this attribute is absent, then the 946 amdgpu-no-implicitarg-ptr is also removed. 947 948 ======================================= ========================================================== 949 950.. _amdgpu-elf-code-object: 951 952ELF Code Object 953=============== 954 955The AMDGPU backend generates a standard ELF [ELF]_ relocatable code object that 956can be linked by ``lld`` to produce a standard ELF shared code object which can 957be loaded and executed on an AMDGPU target. 958 959.. _amdgpu-elf-header: 960 961Header 962------ 963 964The AMDGPU backend uses the following ELF header: 965 966 .. table:: AMDGPU ELF Header 967 :name: amdgpu-elf-header-table 968 969 ========================== =============================== 970 Field Value 971 ========================== =============================== 972 ``e_ident[EI_CLASS]`` ``ELFCLASS64`` 973 ``e_ident[EI_DATA]`` ``ELFDATA2LSB`` 974 ``e_ident[EI_OSABI]`` - ``ELFOSABI_NONE`` 975 - ``ELFOSABI_AMDGPU_HSA`` 976 - ``ELFOSABI_AMDGPU_PAL`` 977 - ``ELFOSABI_AMDGPU_MESA3D`` 978 ``e_ident[EI_ABIVERSION]`` - ``ELFABIVERSION_AMDGPU_HSA_V2`` 979 - ``ELFABIVERSION_AMDGPU_HSA_V3`` 980 - ``ELFABIVERSION_AMDGPU_HSA_V4`` 981 - ``ELFABIVERSION_AMDGPU_HSA_V5`` 982 - ``ELFABIVERSION_AMDGPU_PAL`` 983 - ``ELFABIVERSION_AMDGPU_MESA3D`` 984 ``e_type`` - ``ET_REL`` 985 - ``ET_DYN`` 986 ``e_machine`` ``EM_AMDGPU`` 987 ``e_entry`` 0 988 ``e_flags`` See :ref:`amdgpu-elf-header-e_flags-v2-table`, 989 :ref:`amdgpu-elf-header-e_flags-table-v3`, 990 and :ref:`amdgpu-elf-header-e_flags-table-v4-onwards` 991 ========================== =============================== 992 993.. 994 995 .. table:: AMDGPU ELF Header Enumeration Values 996 :name: amdgpu-elf-header-enumeration-values-table 997 998 =============================== ===== 999 Name Value 1000 =============================== ===== 1001 ``EM_AMDGPU`` 224 1002 ``ELFOSABI_NONE`` 0 1003 ``ELFOSABI_AMDGPU_HSA`` 64 1004 ``ELFOSABI_AMDGPU_PAL`` 65 1005 ``ELFOSABI_AMDGPU_MESA3D`` 66 1006 ``ELFABIVERSION_AMDGPU_HSA_V2`` 0 1007 ``ELFABIVERSION_AMDGPU_HSA_V3`` 1 1008 ``ELFABIVERSION_AMDGPU_HSA_V4`` 2 1009 ``ELFABIVERSION_AMDGPU_HSA_V5`` 3 1010 ``ELFABIVERSION_AMDGPU_PAL`` 0 1011 ``ELFABIVERSION_AMDGPU_MESA3D`` 0 1012 =============================== ===== 1013 1014``e_ident[EI_CLASS]`` 1015 The ELF class is: 1016 1017 * ``ELFCLASS32`` for ``r600`` architecture. 1018 1019 * ``ELFCLASS64`` for ``amdgcn`` architecture which only supports 64-bit 1020 process address space applications. 1021 1022``e_ident[EI_DATA]`` 1023 All AMDGPU targets use ``ELFDATA2LSB`` for little-endian byte ordering. 1024 1025``e_ident[EI_OSABI]`` 1026 One of the following AMDGPU target architecture specific OS ABIs 1027 (see :ref:`amdgpu-os`): 1028 1029 * ``ELFOSABI_NONE`` for *unknown* OS. 1030 1031 * ``ELFOSABI_AMDGPU_HSA`` for ``amdhsa`` OS. 1032 1033 * ``ELFOSABI_AMDGPU_PAL`` for ``amdpal`` OS. 1034 1035 * ``ELFOSABI_AMDGPU_MESA3D`` for ``mesa3D`` OS. 1036 1037``e_ident[EI_ABIVERSION]`` 1038 The ABI version of the AMDGPU target architecture specific OS ABI to which the code 1039 object conforms: 1040 1041 * ``ELFABIVERSION_AMDGPU_HSA_V2`` is used to specify the version of AMD HSA 1042 runtime ABI for code object V2. Specify using the Clang option 1043 ``-mcode-object-version=2``. 1044 1045 * ``ELFABIVERSION_AMDGPU_HSA_V3`` is used to specify the version of AMD HSA 1046 runtime ABI for code object V3. Specify using the Clang option 1047 ``-mcode-object-version=3``. 1048 1049 * ``ELFABIVERSION_AMDGPU_HSA_V4`` is used to specify the version of AMD HSA 1050 runtime ABI for code object V4. Specify using the Clang option 1051 ``-mcode-object-version=4``. This is the default code object 1052 version if not specified. 1053 1054 * ``ELFABIVERSION_AMDGPU_HSA_V5`` is used to specify the version of AMD HSA 1055 runtime ABI for code object V5. Specify using the Clang option 1056 ``-mcode-object-version=5``. 1057 1058 * ``ELFABIVERSION_AMDGPU_PAL`` is used to specify the version of AMD PAL 1059 runtime ABI. 1060 1061 * ``ELFABIVERSION_AMDGPU_MESA3D`` is used to specify the version of AMD MESA 1062 3D runtime ABI. 1063 1064``e_type`` 1065 Can be one of the following values: 1066 1067 1068 ``ET_REL`` 1069 The type produced by the AMDGPU backend compiler as it is relocatable code 1070 object. 1071 1072 ``ET_DYN`` 1073 The type produced by the linker as it is a shared code object. 1074 1075 The AMD HSA runtime loader requires a ``ET_DYN`` code object. 1076 1077``e_machine`` 1078 The value ``EM_AMDGPU`` is used for the machine for all processors supported 1079 by the ``r600`` and ``amdgcn`` architectures (see 1080 :ref:`amdgpu-processor-table`). The specific processor is specified in the 1081 ``NT_AMD_HSA_ISA_VERSION`` note record for code object V2 (see 1082 :ref:`amdgpu-note-records-v2`) and in the ``EF_AMDGPU_MACH`` bit field of the 1083 ``e_flags`` for code object V3 and above (see 1084 :ref:`amdgpu-elf-header-e_flags-table-v3` and 1085 :ref:`amdgpu-elf-header-e_flags-table-v4-onwards`). 1086 1087``e_entry`` 1088 The entry point is 0 as the entry points for individual kernels must be 1089 selected in order to invoke them through AQL packets. 1090 1091``e_flags`` 1092 The AMDGPU backend uses the following ELF header flags: 1093 1094 .. table:: AMDGPU ELF Header ``e_flags`` for Code Object V2 1095 :name: amdgpu-elf-header-e_flags-v2-table 1096 1097 ===================================== ===== ============================= 1098 Name Value Description 1099 ===================================== ===== ============================= 1100 ``EF_AMDGPU_FEATURE_XNACK_V2`` 0x01 Indicates if the ``xnack`` 1101 target feature is 1102 enabled for all code 1103 contained in the code object. 1104 If the processor 1105 does not support the 1106 ``xnack`` target 1107 feature then must 1108 be 0. 1109 See 1110 :ref:`amdgpu-target-features`. 1111 ``EF_AMDGPU_FEATURE_TRAP_HANDLER_V2`` 0x02 Indicates if the trap 1112 handler is enabled for all 1113 code contained in the code 1114 object. If the processor 1115 does not support a trap 1116 handler then must be 0. 1117 See 1118 :ref:`amdgpu-target-features`. 1119 ===================================== ===== ============================= 1120 1121 .. table:: AMDGPU ELF Header ``e_flags`` for Code Object V3 1122 :name: amdgpu-elf-header-e_flags-table-v3 1123 1124 ================================= ===== ============================= 1125 Name Value Description 1126 ================================= ===== ============================= 1127 ``EF_AMDGPU_MACH`` 0x0ff AMDGPU processor selection 1128 mask for 1129 ``EF_AMDGPU_MACH_xxx`` values 1130 defined in 1131 :ref:`amdgpu-ef-amdgpu-mach-table`. 1132 ``EF_AMDGPU_FEATURE_XNACK_V3`` 0x100 Indicates if the ``xnack`` 1133 target feature is 1134 enabled for all code 1135 contained in the code object. 1136 If the processor 1137 does not support the 1138 ``xnack`` target 1139 feature then must 1140 be 0. 1141 See 1142 :ref:`amdgpu-target-features`. 1143 ``EF_AMDGPU_FEATURE_SRAMECC_V3`` 0x200 Indicates if the ``sramecc`` 1144 target feature is 1145 enabled for all code 1146 contained in the code object. 1147 If the processor 1148 does not support the 1149 ``sramecc`` target 1150 feature then must 1151 be 0. 1152 See 1153 :ref:`amdgpu-target-features`. 1154 ================================= ===== ============================= 1155 1156 .. table:: AMDGPU ELF Header ``e_flags`` for Code Object V4 and After 1157 :name: amdgpu-elf-header-e_flags-table-v4-onwards 1158 1159 ============================================ ===== =================================== 1160 Name Value Description 1161 ============================================ ===== =================================== 1162 ``EF_AMDGPU_MACH`` 0x0ff AMDGPU processor selection 1163 mask for 1164 ``EF_AMDGPU_MACH_xxx`` values 1165 defined in 1166 :ref:`amdgpu-ef-amdgpu-mach-table`. 1167 ``EF_AMDGPU_FEATURE_XNACK_V4`` 0x300 XNACK selection mask for 1168 ``EF_AMDGPU_FEATURE_XNACK_*_V4`` 1169 values. 1170 ``EF_AMDGPU_FEATURE_XNACK_UNSUPPORTED_V4`` 0x000 XNACK unsuppored. 1171 ``EF_AMDGPU_FEATURE_XNACK_ANY_V4`` 0x100 XNACK can have any value. 1172 ``EF_AMDGPU_FEATURE_XNACK_OFF_V4`` 0x200 XNACK disabled. 1173 ``EF_AMDGPU_FEATURE_XNACK_ON_V4`` 0x300 XNACK enabled. 1174 ``EF_AMDGPU_FEATURE_SRAMECC_V4`` 0xc00 SRAMECC selection mask for 1175 ``EF_AMDGPU_FEATURE_SRAMECC_*_V4`` 1176 values. 1177 ``EF_AMDGPU_FEATURE_SRAMECC_UNSUPPORTED_V4`` 0x000 SRAMECC unsuppored. 1178 ``EF_AMDGPU_FEATURE_SRAMECC_ANY_V4`` 0x400 SRAMECC can have any value. 1179 ``EF_AMDGPU_FEATURE_SRAMECC_OFF_V4`` 0x800 SRAMECC disabled, 1180 ``EF_AMDGPU_FEATURE_SRAMECC_ON_V4`` 0xc00 SRAMECC enabled. 1181 ============================================ ===== =================================== 1182 1183 .. table:: AMDGPU ``EF_AMDGPU_MACH`` Values 1184 :name: amdgpu-ef-amdgpu-mach-table 1185 1186 ==================================== ========== ============================= 1187 Name Value Description (see 1188 :ref:`amdgpu-processor-table`) 1189 ==================================== ========== ============================= 1190 ``EF_AMDGPU_MACH_NONE`` 0x000 *not specified* 1191 ``EF_AMDGPU_MACH_R600_R600`` 0x001 ``r600`` 1192 ``EF_AMDGPU_MACH_R600_R630`` 0x002 ``r630`` 1193 ``EF_AMDGPU_MACH_R600_RS880`` 0x003 ``rs880`` 1194 ``EF_AMDGPU_MACH_R600_RV670`` 0x004 ``rv670`` 1195 ``EF_AMDGPU_MACH_R600_RV710`` 0x005 ``rv710`` 1196 ``EF_AMDGPU_MACH_R600_RV730`` 0x006 ``rv730`` 1197 ``EF_AMDGPU_MACH_R600_RV770`` 0x007 ``rv770`` 1198 ``EF_AMDGPU_MACH_R600_CEDAR`` 0x008 ``cedar`` 1199 ``EF_AMDGPU_MACH_R600_CYPRESS`` 0x009 ``cypress`` 1200 ``EF_AMDGPU_MACH_R600_JUNIPER`` 0x00a ``juniper`` 1201 ``EF_AMDGPU_MACH_R600_REDWOOD`` 0x00b ``redwood`` 1202 ``EF_AMDGPU_MACH_R600_SUMO`` 0x00c ``sumo`` 1203 ``EF_AMDGPU_MACH_R600_BARTS`` 0x00d ``barts`` 1204 ``EF_AMDGPU_MACH_R600_CAICOS`` 0x00e ``caicos`` 1205 ``EF_AMDGPU_MACH_R600_CAYMAN`` 0x00f ``cayman`` 1206 ``EF_AMDGPU_MACH_R600_TURKS`` 0x010 ``turks`` 1207 *reserved* 0x011 - Reserved for ``r600`` 1208 0x01f architecture processors. 1209 ``EF_AMDGPU_MACH_AMDGCN_GFX600`` 0x020 ``gfx600`` 1210 ``EF_AMDGPU_MACH_AMDGCN_GFX601`` 0x021 ``gfx601`` 1211 ``EF_AMDGPU_MACH_AMDGCN_GFX700`` 0x022 ``gfx700`` 1212 ``EF_AMDGPU_MACH_AMDGCN_GFX701`` 0x023 ``gfx701`` 1213 ``EF_AMDGPU_MACH_AMDGCN_GFX702`` 0x024 ``gfx702`` 1214 ``EF_AMDGPU_MACH_AMDGCN_GFX703`` 0x025 ``gfx703`` 1215 ``EF_AMDGPU_MACH_AMDGCN_GFX704`` 0x026 ``gfx704`` 1216 *reserved* 0x027 Reserved. 1217 ``EF_AMDGPU_MACH_AMDGCN_GFX801`` 0x028 ``gfx801`` 1218 ``EF_AMDGPU_MACH_AMDGCN_GFX802`` 0x029 ``gfx802`` 1219 ``EF_AMDGPU_MACH_AMDGCN_GFX803`` 0x02a ``gfx803`` 1220 ``EF_AMDGPU_MACH_AMDGCN_GFX810`` 0x02b ``gfx810`` 1221 ``EF_AMDGPU_MACH_AMDGCN_GFX900`` 0x02c ``gfx900`` 1222 ``EF_AMDGPU_MACH_AMDGCN_GFX902`` 0x02d ``gfx902`` 1223 ``EF_AMDGPU_MACH_AMDGCN_GFX904`` 0x02e ``gfx904`` 1224 ``EF_AMDGPU_MACH_AMDGCN_GFX906`` 0x02f ``gfx906`` 1225 ``EF_AMDGPU_MACH_AMDGCN_GFX908`` 0x030 ``gfx908`` 1226 ``EF_AMDGPU_MACH_AMDGCN_GFX909`` 0x031 ``gfx909`` 1227 ``EF_AMDGPU_MACH_AMDGCN_GFX90C`` 0x032 ``gfx90c`` 1228 ``EF_AMDGPU_MACH_AMDGCN_GFX1010`` 0x033 ``gfx1010`` 1229 ``EF_AMDGPU_MACH_AMDGCN_GFX1011`` 0x034 ``gfx1011`` 1230 ``EF_AMDGPU_MACH_AMDGCN_GFX1012`` 0x035 ``gfx1012`` 1231 ``EF_AMDGPU_MACH_AMDGCN_GFX1030`` 0x036 ``gfx1030`` 1232 ``EF_AMDGPU_MACH_AMDGCN_GFX1031`` 0x037 ``gfx1031`` 1233 ``EF_AMDGPU_MACH_AMDGCN_GFX1032`` 0x038 ``gfx1032`` 1234 ``EF_AMDGPU_MACH_AMDGCN_GFX1033`` 0x039 ``gfx1033`` 1235 ``EF_AMDGPU_MACH_AMDGCN_GFX602`` 0x03a ``gfx602`` 1236 ``EF_AMDGPU_MACH_AMDGCN_GFX705`` 0x03b ``gfx705`` 1237 ``EF_AMDGPU_MACH_AMDGCN_GFX805`` 0x03c ``gfx805`` 1238 ``EF_AMDGPU_MACH_AMDGCN_GFX1035`` 0x03d ``gfx1035`` 1239 ``EF_AMDGPU_MACH_AMDGCN_GFX1034`` 0x03e ``gfx1034`` 1240 ``EF_AMDGPU_MACH_AMDGCN_GFX90A`` 0x03f ``gfx90a`` 1241 ``EF_AMDGPU_MACH_AMDGCN_GFX940`` 0x040 ``gfx940`` 1242 *reserved* 0x041 Reserved. 1243 ``EF_AMDGPU_MACH_AMDGCN_GFX1013`` 0x042 ``gfx1013`` 1244 *reserved* 0x043 Reserved. 1245 *reserved* 0x044 Reserved. 1246 ``EF_AMDGPU_MACH_AMDGCN_GFX1036`` 0x045 ``gfx1036`` 1247 ==================================== ========== ============================= 1248 1249Sections 1250-------- 1251 1252An AMDGPU target ELF code object has the standard ELF sections which include: 1253 1254 .. table:: AMDGPU ELF Sections 1255 :name: amdgpu-elf-sections-table 1256 1257 ================== ================ ================================= 1258 Name Type Attributes 1259 ================== ================ ================================= 1260 ``.bss`` ``SHT_NOBITS`` ``SHF_ALLOC`` + ``SHF_WRITE`` 1261 ``.data`` ``SHT_PROGBITS`` ``SHF_ALLOC`` + ``SHF_WRITE`` 1262 ``.debug_``\ *\** ``SHT_PROGBITS`` *none* 1263 ``.dynamic`` ``SHT_DYNAMIC`` ``SHF_ALLOC`` 1264 ``.dynstr`` ``SHT_PROGBITS`` ``SHF_ALLOC`` 1265 ``.dynsym`` ``SHT_PROGBITS`` ``SHF_ALLOC`` 1266 ``.got`` ``SHT_PROGBITS`` ``SHF_ALLOC`` + ``SHF_WRITE`` 1267 ``.hash`` ``SHT_HASH`` ``SHF_ALLOC`` 1268 ``.note`` ``SHT_NOTE`` *none* 1269 ``.rela``\ *name* ``SHT_RELA`` *none* 1270 ``.rela.dyn`` ``SHT_RELA`` *none* 1271 ``.rodata`` ``SHT_PROGBITS`` ``SHF_ALLOC`` 1272 ``.shstrtab`` ``SHT_STRTAB`` *none* 1273 ``.strtab`` ``SHT_STRTAB`` *none* 1274 ``.symtab`` ``SHT_SYMTAB`` *none* 1275 ``.text`` ``SHT_PROGBITS`` ``SHF_ALLOC`` + ``SHF_EXECINSTR`` 1276 ================== ================ ================================= 1277 1278These sections have their standard meanings (see [ELF]_) and are only generated 1279if needed. 1280 1281``.debug``\ *\** 1282 The standard DWARF sections. See :ref:`amdgpu-dwarf-debug-information` for 1283 information on the DWARF produced by the AMDGPU backend. 1284 1285``.dynamic``, ``.dynstr``, ``.dynsym``, ``.hash`` 1286 The standard sections used by a dynamic loader. 1287 1288``.note`` 1289 See :ref:`amdgpu-note-records` for the note records supported by the AMDGPU 1290 backend. 1291 1292``.rela``\ *name*, ``.rela.dyn`` 1293 For relocatable code objects, *name* is the name of the section that the 1294 relocation records apply. For example, ``.rela.text`` is the section name for 1295 relocation records associated with the ``.text`` section. 1296 1297 For linked shared code objects, ``.rela.dyn`` contains all the relocation 1298 records from each of the relocatable code object's ``.rela``\ *name* sections. 1299 1300 See :ref:`amdgpu-relocation-records` for the relocation records supported by 1301 the AMDGPU backend. 1302 1303``.text`` 1304 The executable machine code for the kernels and functions they call. Generated 1305 as position independent code. See :ref:`amdgpu-code-conventions` for 1306 information on conventions used in the isa generation. 1307 1308.. _amdgpu-note-records: 1309 1310Note Records 1311------------ 1312 1313The AMDGPU backend code object contains ELF note records in the ``.note`` 1314section. The set of generated notes and their semantics depend on the code 1315object version; see :ref:`amdgpu-note-records-v2` and 1316:ref:`amdgpu-note-records-v3-onwards`. 1317 1318As required by ``ELFCLASS32`` and ``ELFCLASS64``, minimal zero-byte padding 1319must be generated after the ``name`` field to ensure the ``desc`` field is 4 1320byte aligned. In addition, minimal zero-byte padding must be generated to 1321ensure the ``desc`` field size is a multiple of 4 bytes. The ``sh_addralign`` 1322field of the ``.note`` section must be at least 4 to indicate at least 8 byte 1323alignment. 1324 1325.. _amdgpu-note-records-v2: 1326 1327Code Object V2 Note Records 1328~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1329 1330.. warning:: 1331 Code object V2 is not the default code object version emitted by 1332 this version of LLVM. 1333 1334The AMDGPU backend code object uses the following ELF note record in the 1335``.note`` section when compiling for code object V2. 1336 1337The note record vendor field is "AMD". 1338 1339Additional note records may be present, but any which are not documented here 1340are deprecated and should not be used. 1341 1342 .. table:: AMDGPU Code Object V2 ELF Note Records 1343 :name: amdgpu-elf-note-records-v2-table 1344 1345 ===== ===================================== ====================================== 1346 Name Type Description 1347 ===== ===================================== ====================================== 1348 "AMD" ``NT_AMD_HSA_CODE_OBJECT_VERSION`` Code object version. 1349 "AMD" ``NT_AMD_HSA_HSAIL`` HSAIL properties generated by the HSAIL 1350 Finalizer and not the LLVM compiler. 1351 "AMD" ``NT_AMD_HSA_ISA_VERSION`` Target ISA version. 1352 "AMD" ``NT_AMD_HSA_METADATA`` Metadata null terminated string in 1353 YAML [YAML]_ textual format. 1354 "AMD" ``NT_AMD_HSA_ISA_NAME`` Target ISA name. 1355 ===== ===================================== ====================================== 1356 1357.. 1358 1359 .. table:: AMDGPU Code Object V2 ELF Note Record Enumeration Values 1360 :name: amdgpu-elf-note-record-enumeration-values-v2-table 1361 1362 ===================================== ===== 1363 Name Value 1364 ===================================== ===== 1365 ``NT_AMD_HSA_CODE_OBJECT_VERSION`` 1 1366 ``NT_AMD_HSA_HSAIL`` 2 1367 ``NT_AMD_HSA_ISA_VERSION`` 3 1368 *reserved* 4-9 1369 ``NT_AMD_HSA_METADATA`` 10 1370 ``NT_AMD_HSA_ISA_NAME`` 11 1371 ===================================== ===== 1372 1373``NT_AMD_HSA_CODE_OBJECT_VERSION`` 1374 Specifies the code object version number. The description field has the 1375 following layout: 1376 1377 .. code:: c 1378 1379 struct amdgpu_hsa_note_code_object_version_s { 1380 uint32_t major_version; 1381 uint32_t minor_version; 1382 }; 1383 1384 The ``major_version`` has a value less than or equal to 2. 1385 1386``NT_AMD_HSA_HSAIL`` 1387 Specifies the HSAIL properties used by the HSAIL Finalizer. The description 1388 field has the following layout: 1389 1390 .. code:: c 1391 1392 struct amdgpu_hsa_note_hsail_s { 1393 uint32_t hsail_major_version; 1394 uint32_t hsail_minor_version; 1395 uint8_t profile; 1396 uint8_t machine_model; 1397 uint8_t default_float_round; 1398 }; 1399 1400``NT_AMD_HSA_ISA_VERSION`` 1401 Specifies the target ISA version. The description field has the following layout: 1402 1403 .. code:: c 1404 1405 struct amdgpu_hsa_note_isa_s { 1406 uint16_t vendor_name_size; 1407 uint16_t architecture_name_size; 1408 uint32_t major; 1409 uint32_t minor; 1410 uint32_t stepping; 1411 char vendor_and_architecture_name[1]; 1412 }; 1413 1414 ``vendor_name_size`` and ``architecture_name_size`` are the length of the 1415 vendor and architecture names respectively, including the NUL character. 1416 1417 ``vendor_and_architecture_name`` contains the NUL terminates string for the 1418 vendor, immediately followed by the NUL terminated string for the 1419 architecture. 1420 1421 This note record is used by the HSA runtime loader. 1422 1423 Code object V2 only supports a limited number of processors and has fixed 1424 settings for target features. See 1425 :ref:`amdgpu-elf-note-record-supported_processors-v2-table` for a list of 1426 processors and the corresponding target ID. In the table the note record ISA 1427 name is a concatenation of the vendor name, architecture name, major, minor, 1428 and stepping separated by a ":". 1429 1430 The target ID column shows the processor name and fixed target features used 1431 by the LLVM compiler. The LLVM compiler does not generate a 1432 ``NT_AMD_HSA_HSAIL`` note record. 1433 1434 A code object generated by the Finalizer also uses code object V2 and always 1435 generates a ``NT_AMD_HSA_HSAIL`` note record. The processor name and 1436 ``sramecc`` target feature is as shown in 1437 :ref:`amdgpu-elf-note-record-supported_processors-v2-table` but the ``xnack`` 1438 target feature is specified by the ``EF_AMDGPU_FEATURE_XNACK_V2`` ``e_flags`` 1439 bit. 1440 1441``NT_AMD_HSA_ISA_NAME`` 1442 Specifies the target ISA name as a non-NUL terminated string. 1443 1444 This note record is not used by the HSA runtime loader. 1445 1446 See the ``NT_AMD_HSA_ISA_VERSION`` note record description of the code object 1447 V2's limited support of processors and fixed settings for target features. 1448 1449 See :ref:`amdgpu-elf-note-record-supported_processors-v2-table` for a mapping 1450 from the string to the corresponding target ID. If the ``xnack`` target 1451 feature is supported and enabled, the string produced by the LLVM compiler 1452 will may have a ``+xnack`` appended. The Finlizer did not do the appending and 1453 instead used the ``EF_AMDGPU_FEATURE_XNACK_V2`` ``e_flags`` bit. 1454 1455``NT_AMD_HSA_METADATA`` 1456 Specifies extensible metadata associated with the code objects executed on HSA 1457 [HSA]_ compatible runtimes (see :ref:`amdgpu-os`). It is required when the 1458 target triple OS is ``amdhsa`` (see :ref:`amdgpu-target-triples`). See 1459 :ref:`amdgpu-amdhsa-code-object-metadata-v2` for the syntax of the code object 1460 metadata string. 1461 1462 .. table:: AMDGPU Code Object V2 Supported Processors and Fixed Target Feature Settings 1463 :name: amdgpu-elf-note-record-supported_processors-v2-table 1464 1465 ===================== ========================== 1466 Note Record ISA Name Target ID 1467 ===================== ========================== 1468 ``AMD:AMDGPU:6:0:0`` ``gfx600`` 1469 ``AMD:AMDGPU:6:0:1`` ``gfx601`` 1470 ``AMD:AMDGPU:6:0:2`` ``gfx602`` 1471 ``AMD:AMDGPU:7:0:0`` ``gfx700`` 1472 ``AMD:AMDGPU:7:0:1`` ``gfx701`` 1473 ``AMD:AMDGPU:7:0:2`` ``gfx702`` 1474 ``AMD:AMDGPU:7:0:3`` ``gfx703`` 1475 ``AMD:AMDGPU:7:0:4`` ``gfx704`` 1476 ``AMD:AMDGPU:7:0:5`` ``gfx705`` 1477 ``AMD:AMDGPU:8:0:0`` ``gfx802`` 1478 ``AMD:AMDGPU:8:0:1`` ``gfx801:xnack+`` 1479 ``AMD:AMDGPU:8:0:2`` ``gfx802`` 1480 ``AMD:AMDGPU:8:0:3`` ``gfx803`` 1481 ``AMD:AMDGPU:8:0:4`` ``gfx803`` 1482 ``AMD:AMDGPU:8:0:5`` ``gfx805`` 1483 ``AMD:AMDGPU:8:1:0`` ``gfx810:xnack+`` 1484 ``AMD:AMDGPU:9:0:0`` ``gfx900:xnack-`` 1485 ``AMD:AMDGPU:9:0:1`` ``gfx900:xnack+`` 1486 ``AMD:AMDGPU:9:0:2`` ``gfx902:xnack-`` 1487 ``AMD:AMDGPU:9:0:3`` ``gfx902:xnack+`` 1488 ``AMD:AMDGPU:9:0:4`` ``gfx904:xnack-`` 1489 ``AMD:AMDGPU:9:0:5`` ``gfx904:xnack+`` 1490 ``AMD:AMDGPU:9:0:6`` ``gfx906:sramecc-:xnack-`` 1491 ``AMD:AMDGPU:9:0:7`` ``gfx906:sramecc-:xnack+`` 1492 ``AMD:AMDGPU:9:0:12`` ``gfx90c:xnack-`` 1493 ===================== ========================== 1494 1495.. _amdgpu-note-records-v3-onwards: 1496 1497Code Object V3 and Above Note Records 1498~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1499 1500The AMDGPU backend code object uses the following ELF note record in the 1501``.note`` section when compiling for code object V3 and above. 1502 1503The note record vendor field is "AMDGPU". 1504 1505Additional note records may be present, but any which are not documented here 1506are deprecated and should not be used. 1507 1508 .. table:: AMDGPU Code Object V3 and Above ELF Note Records 1509 :name: amdgpu-elf-note-records-table-v3-onwards 1510 1511 ======== ============================== ====================================== 1512 Name Type Description 1513 ======== ============================== ====================================== 1514 "AMDGPU" ``NT_AMDGPU_METADATA`` Metadata in Message Pack [MsgPack]_ 1515 binary format. 1516 ======== ============================== ====================================== 1517 1518.. 1519 1520 .. table:: AMDGPU Code Object V3 and Above ELF Note Record Enumeration Values 1521 :name: amdgpu-elf-note-record-enumeration-values-table-v3-onwards 1522 1523 ============================== ===== 1524 Name Value 1525 ============================== ===== 1526 *reserved* 0-31 1527 ``NT_AMDGPU_METADATA`` 32 1528 ============================== ===== 1529 1530``NT_AMDGPU_METADATA`` 1531 Specifies extensible metadata associated with an AMDGPU code object. It is 1532 encoded as a map in the Message Pack [MsgPack]_ binary data format. See 1533 :ref:`amdgpu-amdhsa-code-object-metadata-v3`, 1534 :ref:`amdgpu-amdhsa-code-object-metadata-v4` and 1535 :ref:`amdgpu-amdhsa-code-object-metadata-v5` for the map keys defined for the 1536 ``amdhsa`` OS. 1537 1538.. _amdgpu-symbols: 1539 1540Symbols 1541------- 1542 1543Symbols include the following: 1544 1545 .. table:: AMDGPU ELF Symbols 1546 :name: amdgpu-elf-symbols-table 1547 1548 ===================== ================== ================ ================== 1549 Name Type Section Description 1550 ===================== ================== ================ ================== 1551 *link-name* ``STT_OBJECT`` - ``.data`` Global variable 1552 - ``.rodata`` 1553 - ``.bss`` 1554 *link-name*\ ``.kd`` ``STT_OBJECT`` - ``.rodata`` Kernel descriptor 1555 *link-name* ``STT_FUNC`` - ``.text`` Kernel entry point 1556 *link-name* ``STT_OBJECT`` - SHN_AMDGPU_LDS Global variable in LDS 1557 ===================== ================== ================ ================== 1558 1559Global variable 1560 Global variables both used and defined by the compilation unit. 1561 1562 If the symbol is defined in the compilation unit then it is allocated in the 1563 appropriate section according to if it has initialized data or is readonly. 1564 1565 If the symbol is external then its section is ``STN_UNDEF`` and the loader 1566 will resolve relocations using the definition provided by another code object 1567 or explicitly defined by the runtime. 1568 1569 If the symbol resides in local/group memory (LDS) then its section is the 1570 special processor specific section name ``SHN_AMDGPU_LDS``, and the 1571 ``st_value`` field describes alignment requirements as it does for common 1572 symbols. 1573 1574 .. TODO:: 1575 1576 Add description of linked shared object symbols. Seems undefined symbols 1577 are marked as STT_NOTYPE. 1578 1579Kernel descriptor 1580 Every HSA kernel has an associated kernel descriptor. It is the address of the 1581 kernel descriptor that is used in the AQL dispatch packet used to invoke the 1582 kernel, not the kernel entry point. The layout of the HSA kernel descriptor is 1583 defined in :ref:`amdgpu-amdhsa-kernel-descriptor`. 1584 1585Kernel entry point 1586 Every HSA kernel also has a symbol for its machine code entry point. 1587 1588.. _amdgpu-relocation-records: 1589 1590Relocation Records 1591------------------ 1592 1593AMDGPU backend generates ``Elf64_Rela`` relocation records. Supported 1594relocatable fields are: 1595 1596``word32`` 1597 This specifies a 32-bit field occupying 4 bytes with arbitrary byte 1598 alignment. These values use the same byte order as other word values in the 1599 AMDGPU architecture. 1600 1601``word64`` 1602 This specifies a 64-bit field occupying 8 bytes with arbitrary byte 1603 alignment. These values use the same byte order as other word values in the 1604 AMDGPU architecture. 1605 1606Following notations are used for specifying relocation calculations: 1607 1608**A** 1609 Represents the addend used to compute the value of the relocatable field. 1610 1611**G** 1612 Represents the offset into the global offset table at which the relocation 1613 entry's symbol will reside during execution. 1614 1615**GOT** 1616 Represents the address of the global offset table. 1617 1618**P** 1619 Represents the place (section offset for ``et_rel`` or address for ``et_dyn``) 1620 of the storage unit being relocated (computed using ``r_offset``). 1621 1622**S** 1623 Represents the value of the symbol whose index resides in the relocation 1624 entry. Relocations not using this must specify a symbol index of 1625 ``STN_UNDEF``. 1626 1627**B** 1628 Represents the base address of a loaded executable or shared object which is 1629 the difference between the ELF address and the actual load address. 1630 Relocations using this are only valid in executable or shared objects. 1631 1632The following relocation types are supported: 1633 1634 .. table:: AMDGPU ELF Relocation Records 1635 :name: amdgpu-elf-relocation-records-table 1636 1637 ========================== ======= ===== ========== ============================== 1638 Relocation Type Kind Value Field Calculation 1639 ========================== ======= ===== ========== ============================== 1640 ``R_AMDGPU_NONE`` 0 *none* *none* 1641 ``R_AMDGPU_ABS32_LO`` Static, 1 ``word32`` (S + A) & 0xFFFFFFFF 1642 Dynamic 1643 ``R_AMDGPU_ABS32_HI`` Static, 2 ``word32`` (S + A) >> 32 1644 Dynamic 1645 ``R_AMDGPU_ABS64`` Static, 3 ``word64`` S + A 1646 Dynamic 1647 ``R_AMDGPU_REL32`` Static 4 ``word32`` S + A - P 1648 ``R_AMDGPU_REL64`` Static 5 ``word64`` S + A - P 1649 ``R_AMDGPU_ABS32`` Static, 6 ``word32`` S + A 1650 Dynamic 1651 ``R_AMDGPU_GOTPCREL`` Static 7 ``word32`` G + GOT + A - P 1652 ``R_AMDGPU_GOTPCREL32_LO`` Static 8 ``word32`` (G + GOT + A - P) & 0xFFFFFFFF 1653 ``R_AMDGPU_GOTPCREL32_HI`` Static 9 ``word32`` (G + GOT + A - P) >> 32 1654 ``R_AMDGPU_REL32_LO`` Static 10 ``word32`` (S + A - P) & 0xFFFFFFFF 1655 ``R_AMDGPU_REL32_HI`` Static 11 ``word32`` (S + A - P) >> 32 1656 *reserved* 12 1657 ``R_AMDGPU_RELATIVE64`` Dynamic 13 ``word64`` B + A 1658 ``R_AMDGPU_REL16`` Static 14 ``word16`` ((S + A - P) - 4) / 4 1659 ========================== ======= ===== ========== ============================== 1660 1661``R_AMDGPU_ABS32_LO`` and ``R_AMDGPU_ABS32_HI`` are only supported by 1662the ``mesa3d`` OS, which does not support ``R_AMDGPU_ABS64``. 1663 1664There is no current OS loader support for 32-bit programs and so 1665``R_AMDGPU_ABS32`` is not used. 1666 1667.. _amdgpu-loaded-code-object-path-uniform-resource-identifier: 1668 1669Loaded Code Object Path Uniform Resource Identifier (URI) 1670--------------------------------------------------------- 1671 1672The AMD GPU code object loader represents the path of the ELF shared object from 1673which the code object was loaded as a textual Uniform Resource Identifier (URI). 1674Note that the code object is the in memory loaded relocated form of the ELF 1675shared object. Multiple code objects may be loaded at different memory 1676addresses in the same process from the same ELF shared object. 1677 1678The loaded code object path URI syntax is defined by the following BNF syntax: 1679 1680.. code:: 1681 1682 code_object_uri ::== file_uri | memory_uri 1683 file_uri ::== "file://" file_path [ range_specifier ] 1684 memory_uri ::== "memory://" process_id range_specifier 1685 range_specifier ::== [ "#" | "?" ] "offset=" number "&" "size=" number 1686 file_path ::== URI_ENCODED_OS_FILE_PATH 1687 process_id ::== DECIMAL_NUMBER 1688 number ::== HEX_NUMBER | DECIMAL_NUMBER | OCTAL_NUMBER 1689 1690**number** 1691 Is a C integral literal where hexadecimal values are prefixed by "0x" or "0X", 1692 and octal values by "0". 1693 1694**file_path** 1695 Is the file's path specified as a URI encoded UTF-8 string. In URI encoding, 1696 every character that is not in the regular expression ``[a-zA-Z0-9/_.~-]`` is 1697 encoded as two uppercase hexadecimal digits proceeded by "%". Directories in 1698 the path are separated by "/". 1699 1700**offset** 1701 Is a 0-based byte offset to the start of the code object. For a file URI, it 1702 is from the start of the file specified by the ``file_path``, and if omitted 1703 defaults to 0. For a memory URI, it is the memory address and is required. 1704 1705**size** 1706 Is the number of bytes in the code object. For a file URI, if omitted it 1707 defaults to the size of the file. It is required for a memory URI. 1708 1709**process_id** 1710 Is the identity of the process owning the memory. For Linux it is the C 1711 unsigned integral decimal literal for the process ID (PID). 1712 1713For example: 1714 1715.. code:: 1716 1717 file:///dir1/dir2/file1 1718 file:///dir3/dir4/file2#offset=0x2000&size=3000 1719 memory://1234#offset=0x20000&size=3000 1720 1721.. _amdgpu-dwarf-debug-information: 1722 1723DWARF Debug Information 1724======================= 1725 1726.. warning:: 1727 1728 This section describes **provisional support** for AMDGPU DWARF [DWARF]_ that 1729 is not currently fully implemented and is subject to change. 1730 1731AMDGPU generates DWARF [DWARF]_ debugging information ELF sections (see 1732:ref:`amdgpu-elf-code-object`) which contain information that maps the code 1733object executable code and data to the source language constructs. It can be 1734used by tools such as debuggers and profilers. It uses features defined in 1735:doc:`AMDGPUDwarfExtensionsForHeterogeneousDebugging` that are made available in 1736DWARF Version 4 and DWARF Version 5 as an LLVM vendor extension. 1737 1738This section defines the AMDGPU target architecture specific DWARF mappings. 1739 1740.. _amdgpu-dwarf-register-identifier: 1741 1742Register Identifier 1743------------------- 1744 1745This section defines the AMDGPU target architecture register numbers used in 1746DWARF operation expressions (see DWARF Version 5 section 2.5 and 1747:ref:`amdgpu-dwarf-operation-expressions`) and Call Frame Information 1748instructions (see DWARF Version 5 section 6.4 and 1749:ref:`amdgpu-dwarf-call-frame-information`). 1750 1751A single code object can contain code for kernels that have different wavefront 1752sizes. The vector registers and some scalar registers are based on the wavefront 1753size. AMDGPU defines distinct DWARF registers for each wavefront size. This 1754simplifies the consumer of the DWARF so that each register has a fixed size, 1755rather than being dynamic according to the wavefront size mode. Similarly, 1756distinct DWARF registers are defined for those registers that vary in size 1757according to the process address size. This allows a consumer to treat a 1758specific AMDGPU processor as a single architecture regardless of how it is 1759configured at run time. The compiler explicitly specifies the DWARF registers 1760that match the mode in which the code it is generating will be executed. 1761 1762DWARF registers are encoded as numbers, which are mapped to architecture 1763registers. The mapping for AMDGPU is defined in 1764:ref:`amdgpu-dwarf-register-mapping-table`. All AMDGPU targets use the same 1765mapping. 1766 1767.. table:: AMDGPU DWARF Register Mapping 1768 :name: amdgpu-dwarf-register-mapping-table 1769 1770 ============== ================= ======== ================================== 1771 DWARF Register AMDGPU Register Bit Size Description 1772 ============== ================= ======== ================================== 1773 0 PC_32 32 Program Counter (PC) when 1774 executing in a 32-bit process 1775 address space. Used in the CFI to 1776 describe the PC of the calling 1777 frame. 1778 1 EXEC_MASK_32 32 Execution Mask Register when 1779 executing in wavefront 32 mode. 1780 2-15 *Reserved* *Reserved for highly accessed 1781 registers using DWARF shortcut.* 1782 16 PC_64 64 Program Counter (PC) when 1783 executing in a 64-bit process 1784 address space. Used in the CFI to 1785 describe the PC of the calling 1786 frame. 1787 17 EXEC_MASK_64 64 Execution Mask Register when 1788 executing in wavefront 64 mode. 1789 18-31 *Reserved* *Reserved for highly accessed 1790 registers using DWARF shortcut.* 1791 32-95 SGPR0-SGPR63 32 Scalar General Purpose 1792 Registers. 1793 96-127 *Reserved* *Reserved for frequently accessed 1794 registers using DWARF 1-byte ULEB.* 1795 128 STATUS 32 Status Register. 1796 129-511 *Reserved* *Reserved for future Scalar 1797 Architectural Registers.* 1798 512 VCC_32 32 Vector Condition Code Register 1799 when executing in wavefront 32 1800 mode. 1801 513-767 *Reserved* *Reserved for future Vector 1802 Architectural Registers when 1803 executing in wavefront 32 mode.* 1804 768 VCC_64 64 Vector Condition Code Register 1805 when executing in wavefront 64 1806 mode. 1807 769-1023 *Reserved* *Reserved for future Vector 1808 Architectural Registers when 1809 executing in wavefront 64 mode.* 1810 1024-1087 *Reserved* *Reserved for padding.* 1811 1088-1129 SGPR64-SGPR105 32 Scalar General Purpose Registers. 1812 1130-1535 *Reserved* *Reserved for future Scalar 1813 General Purpose Registers.* 1814 1536-1791 VGPR0-VGPR255 32*32 Vector General Purpose Registers 1815 when executing in wavefront 32 1816 mode. 1817 1792-2047 *Reserved* *Reserved for future Vector 1818 General Purpose Registers when 1819 executing in wavefront 32 mode.* 1820 2048-2303 AGPR0-AGPR255 32*32 Vector Accumulation Registers 1821 when executing in wavefront 32 1822 mode. 1823 2304-2559 *Reserved* *Reserved for future Vector 1824 Accumulation Registers when 1825 executing in wavefront 32 mode.* 1826 2560-2815 VGPR0-VGPR255 64*32 Vector General Purpose Registers 1827 when executing in wavefront 64 1828 mode. 1829 2816-3071 *Reserved* *Reserved for future Vector 1830 General Purpose Registers when 1831 executing in wavefront 64 mode.* 1832 3072-3327 AGPR0-AGPR255 64*32 Vector Accumulation Registers 1833 when executing in wavefront 64 1834 mode. 1835 3328-3583 *Reserved* *Reserved for future Vector 1836 Accumulation Registers when 1837 executing in wavefront 64 mode.* 1838 ============== ================= ======== ================================== 1839 1840The vector registers are represented as the full size for the wavefront. They 1841are organized as consecutive dwords (32-bits), one per lane, with the dword at 1842the least significant bit position corresponding to lane 0 and so forth. DWARF 1843location expressions involving the ``DW_OP_LLVM_offset`` and 1844``DW_OP_LLVM_push_lane`` operations are used to select the part of the vector 1845register corresponding to the lane that is executing the current thread of 1846execution in languages that are implemented using a SIMD or SIMT execution 1847model. 1848 1849If the wavefront size is 32 lanes then the wavefront 32 mode register 1850definitions are used. If the wavefront size is 64 lanes then the wavefront 64 1851mode register definitions are used. Some AMDGPU targets support executing in 1852both wavefront 32 and wavefront 64 mode. The register definitions corresponding 1853to the wavefront mode of the generated code will be used. 1854 1855If code is generated to execute in a 32-bit process address space, then the 185632-bit process address space register definitions are used. If code is generated 1857to execute in a 64-bit process address space, then the 64-bit process address 1858space register definitions are used. The ``amdgcn`` target only supports the 185964-bit process address space. 1860 1861.. _amdgpu-dwarf-address-class-identifier: 1862 1863Address Class Identifier 1864------------------------ 1865 1866The DWARF address class represents the source language memory space. See DWARF 1867Version 5 section 2.12 which is updated by the *DWARF Extensions For 1868Heterogeneous Debugging* section :ref:`amdgpu-dwarf-segment_addresses`. 1869 1870The DWARF address class mapping used for AMDGPU is defined in 1871:ref:`amdgpu-dwarf-address-class-mapping-table`. 1872 1873.. table:: AMDGPU DWARF Address Class Mapping 1874 :name: amdgpu-dwarf-address-class-mapping-table 1875 1876 ========================= ====== ================= 1877 DWARF AMDGPU 1878 -------------------------------- ----------------- 1879 Address Class Name Value Address Space 1880 ========================= ====== ================= 1881 ``DW_ADDR_none`` 0x0000 Generic (Flat) 1882 ``DW_ADDR_LLVM_global`` 0x0001 Global 1883 ``DW_ADDR_LLVM_constant`` 0x0002 Global 1884 ``DW_ADDR_LLVM_group`` 0x0003 Local (group/LDS) 1885 ``DW_ADDR_LLVM_private`` 0x0004 Private (Scratch) 1886 ``DW_ADDR_AMDGPU_region`` 0x8000 Region (GDS) 1887 ========================= ====== ================= 1888 1889The DWARF address class values defined in the *DWARF Extensions For 1890Heterogeneous Debugging* section :ref:`amdgpu-dwarf-segment_addresses` are used. 1891 1892In addition, ``DW_ADDR_AMDGPU_region`` is encoded as a vendor extension. This is 1893available for use for the AMD extension for access to the hardware GDS memory 1894which is scratchpad memory allocated per device. 1895 1896For AMDGPU if no ``DW_AT_address_class`` attribute is present, then the default 1897address class of ``DW_ADDR_none`` is used. 1898 1899See :ref:`amdgpu-dwarf-address-space-identifier` for information on the AMDGPU 1900mapping of DWARF address classes to DWARF address spaces, including address size 1901and NULL value. 1902 1903.. _amdgpu-dwarf-address-space-identifier: 1904 1905Address Space Identifier 1906------------------------ 1907 1908DWARF address spaces correspond to target architecture specific linear 1909addressable memory areas. See DWARF Version 5 section 2.12 and *DWARF Extensions 1910For Heterogeneous Debugging* section :ref:`amdgpu-dwarf-segment_addresses`. 1911 1912The DWARF address space mapping used for AMDGPU is defined in 1913:ref:`amdgpu-dwarf-address-space-mapping-table`. 1914 1915.. table:: AMDGPU DWARF Address Space Mapping 1916 :name: amdgpu-dwarf-address-space-mapping-table 1917 1918 ======================================= ===== ======= ======== ================= ======================= 1919 DWARF AMDGPU Notes 1920 --------------------------------------- ----- ---------------- ----------------- ----------------------- 1921 Address Space Name Value Address Bit Size Address Space 1922 --------------------------------------- ----- ------- -------- ----------------- ----------------------- 1923 .. 64-bit 32-bit 1924 process process 1925 address address 1926 space space 1927 ======================================= ===== ======= ======== ================= ======================= 1928 ``DW_ASPACE_none`` 0x00 64 32 Global *default address space* 1929 ``DW_ASPACE_AMDGPU_generic`` 0x01 64 32 Generic (Flat) 1930 ``DW_ASPACE_AMDGPU_region`` 0x02 32 32 Region (GDS) 1931 ``DW_ASPACE_AMDGPU_local`` 0x03 32 32 Local (group/LDS) 1932 *Reserved* 0x04 1933 ``DW_ASPACE_AMDGPU_private_lane`` 0x05 32 32 Private (Scratch) *focused lane* 1934 ``DW_ASPACE_AMDGPU_private_wave`` 0x06 32 32 Private (Scratch) *unswizzled wavefront* 1935 ======================================= ===== ======= ======== ================= ======================= 1936 1937See :ref:`amdgpu-address-spaces` for information on the AMDGPU address spaces 1938including address size and NULL value. 1939 1940The ``DW_ASPACE_none`` address space is the default target architecture address 1941space used in DWARF operations that do not specify an address space. It 1942therefore has to map to the global address space so that the ``DW_OP_addr*`` and 1943related operations can refer to addresses in the program code. 1944 1945The ``DW_ASPACE_AMDGPU_generic`` address space allows location expressions to 1946specify the flat address space. If the address corresponds to an address in the 1947local address space, then it corresponds to the wavefront that is executing the 1948focused thread of execution. If the address corresponds to an address in the 1949private address space, then it corresponds to the lane that is executing the 1950focused thread of execution for languages that are implemented using a SIMD or 1951SIMT execution model. 1952 1953.. note:: 1954 1955 CUDA-like languages such as HIP that do not have address spaces in the 1956 language type system, but do allow variables to be allocated in different 1957 address spaces, need to explicitly specify the ``DW_ASPACE_AMDGPU_generic`` 1958 address space in the DWARF expression operations as the default address space 1959 is the global address space. 1960 1961The ``DW_ASPACE_AMDGPU_local`` address space allows location expressions to 1962specify the local address space corresponding to the wavefront that is executing 1963the focused thread of execution. 1964 1965The ``DW_ASPACE_AMDGPU_private_lane`` address space allows location expressions 1966to specify the private address space corresponding to the lane that is executing 1967the focused thread of execution for languages that are implemented using a SIMD 1968or SIMT execution model. 1969 1970The ``DW_ASPACE_AMDGPU_private_wave`` address space allows location expressions 1971to specify the unswizzled private address space corresponding to the wavefront 1972that is executing the focused thread of execution. The wavefront view of private 1973memory is the per wavefront unswizzled backing memory layout defined in 1974:ref:`amdgpu-address-spaces`, such that address 0 corresponds to the first 1975location for the backing memory of the wavefront (namely the address is not 1976offset by ``wavefront-scratch-base``). The following formula can be used to 1977convert from a ``DW_ASPACE_AMDGPU_private_lane`` address to a 1978``DW_ASPACE_AMDGPU_private_wave`` address: 1979 1980:: 1981 1982 private-address-wavefront = 1983 ((private-address-lane / 4) * wavefront-size * 4) + 1984 (wavefront-lane-id * 4) + (private-address-lane % 4) 1985 1986If the ``DW_ASPACE_AMDGPU_private_lane`` address is dword aligned, and the start 1987of the dwords for each lane starting with lane 0 is required, then this 1988simplifies to: 1989 1990:: 1991 1992 private-address-wavefront = 1993 private-address-lane * wavefront-size 1994 1995A compiler can use the ``DW_ASPACE_AMDGPU_private_wave`` address space to read a 1996complete spilled vector register back into a complete vector register in the 1997CFI. The frame pointer can be a private lane address which is dword aligned, 1998which can be shifted to multiply by the wavefront size, and then used to form a 1999private wavefront address that gives a location for a contiguous set of dwords, 2000one per lane, where the vector register dwords are spilled. The compiler knows 2001the wavefront size since it generates the code. Note that the type of the 2002address may have to be converted as the size of a 2003``DW_ASPACE_AMDGPU_private_lane`` address may be smaller than the size of a 2004``DW_ASPACE_AMDGPU_private_wave`` address. 2005 2006.. _amdgpu-dwarf-lane-identifier: 2007 2008Lane identifier 2009--------------- 2010 2011DWARF lane identifies specify a target architecture lane position for hardware 2012that executes in a SIMD or SIMT manner, and on which a source language maps its 2013threads of execution onto those lanes. The DWARF lane identifier is pushed by 2014the ``DW_OP_LLVM_push_lane`` DWARF expression operation. See DWARF Version 5 2015section 2.5 which is updated by *DWARF Extensions For Heterogeneous Debugging* 2016section :ref:`amdgpu-dwarf-operation-expressions`. 2017 2018For AMDGPU, the lane identifier corresponds to the hardware lane ID of a 2019wavefront. It is numbered from 0 to the wavefront size minus 1. 2020 2021Operation Expressions 2022--------------------- 2023 2024DWARF expressions are used to compute program values and the locations of 2025program objects. See DWARF Version 5 section 2.5 and 2026:ref:`amdgpu-dwarf-operation-expressions`. 2027 2028DWARF location descriptions describe how to access storage which includes memory 2029and registers. When accessing storage on AMDGPU, bytes are ordered with least 2030significant bytes first, and bits are ordered within bytes with least 2031significant bits first. 2032 2033For AMDGPU CFI expressions, ``DW_OP_LLVM_select_bit_piece`` is used to describe 2034unwinding vector registers that are spilled under the execution mask to memory: 2035the zero-single location description is the vector register, and the one-single 2036location description is the spilled memory location description. The 2037``DW_OP_LLVM_form_aspace_address`` is used to specify the address space of the 2038memory location description. 2039 2040In AMDGPU expressions, ``DW_OP_LLVM_select_bit_piece`` is used by the 2041``DW_AT_LLVM_lane_pc`` attribute expression where divergent control flow is 2042controlled by the execution mask. An undefined location description together 2043with ``DW_OP_LLVM_extend`` is used to indicate the lane was not active on entry 2044to the subprogram. See :ref:`amdgpu-dwarf-dw-at-llvm-lane-pc` for an example. 2045 2046Debugger Information Entry Attributes 2047------------------------------------- 2048 2049This section describes how certain debugger information entry attributes are 2050used by AMDGPU. See the sections in DWARF Version 5 section 3.3.5 and 3.1.1 2051which are updated by *DWARF Extensions For Heterogeneous Debugging* section 2052:ref:`amdgpu-dwarf-low-level-information` and 2053:ref:`amdgpu-dwarf-full-and-partial-compilation-unit-entries`. 2054 2055.. _amdgpu-dwarf-dw-at-llvm-lane-pc: 2056 2057``DW_AT_LLVM_lane_pc`` 2058~~~~~~~~~~~~~~~~~~~~~~ 2059 2060For AMDGPU, the ``DW_AT_LLVM_lane_pc`` attribute is used to specify the program 2061location of the separate lanes of a SIMT thread. 2062 2063If the lane is an active lane then this will be the same as the current program 2064location. 2065 2066If the lane is inactive, but was active on entry to the subprogram, then this is 2067the program location in the subprogram at which execution of the lane is 2068conceptual positioned. 2069 2070If the lane was not active on entry to the subprogram, then this will be the 2071undefined location. A client debugger can check if the lane is part of a valid 2072work-group by checking that the lane is in the range of the associated 2073work-group within the grid, accounting for partial work-groups. If it is not, 2074then the debugger can omit any information for the lane. Otherwise, the debugger 2075may repeatedly unwind the stack and inspect the ``DW_AT_LLVM_lane_pc`` of the 2076calling subprogram until it finds a non-undefined location. Conceptually the 2077lane only has the call frames that it has a non-undefined 2078``DW_AT_LLVM_lane_pc``. 2079 2080The following example illustrates how the AMDGPU backend can generate a DWARF 2081location list expression for the nested ``IF/THEN/ELSE`` structures of the 2082following subprogram pseudo code for a target with 64 lanes per wavefront. 2083 2084.. code:: 2085 :number-lines: 2086 2087 SUBPROGRAM X 2088 BEGIN 2089 a; 2090 IF (c1) THEN 2091 b; 2092 IF (c2) THEN 2093 c; 2094 ELSE 2095 d; 2096 ENDIF 2097 e; 2098 ELSE 2099 f; 2100 ENDIF 2101 g; 2102 END 2103 2104The AMDGPU backend may generate the following pseudo LLVM MIR to manipulate the 2105execution mask (``EXEC``) to linearize the control flow. The condition is 2106evaluated to make a mask of the lanes for which the condition evaluates to true. 2107First the ``THEN`` region is executed by setting the ``EXEC`` mask to the 2108logical ``AND`` of the current ``EXEC`` mask with the condition mask. Then the 2109``ELSE`` region is executed by negating the ``EXEC`` mask and logical ``AND`` of 2110the saved ``EXEC`` mask at the start of the region. After the ``IF/THEN/ELSE`` 2111region the ``EXEC`` mask is restored to the value it had at the beginning of the 2112region. This is shown below. Other approaches are possible, but the basic 2113concept is the same. 2114 2115.. code:: 2116 :number-lines: 2117 2118 $lex_start: 2119 a; 2120 %1 = EXEC 2121 %2 = c1 2122 $lex_1_start: 2123 EXEC = %1 & %2 2124 $if_1_then: 2125 b; 2126 %3 = EXEC 2127 %4 = c2 2128 $lex_1_1_start: 2129 EXEC = %3 & %4 2130 $lex_1_1_then: 2131 c; 2132 EXEC = ~EXEC & %3 2133 $lex_1_1_else: 2134 d; 2135 EXEC = %3 2136 $lex_1_1_end: 2137 e; 2138 EXEC = ~EXEC & %1 2139 $lex_1_else: 2140 f; 2141 EXEC = %1 2142 $lex_1_end: 2143 g; 2144 $lex_end: 2145 2146To create the DWARF location list expression that defines the location 2147description of a vector of lane program locations, the LLVM MIR ``DBG_VALUE`` 2148pseudo instruction can be used to annotate the linearized control flow. This can 2149be done by defining an artificial variable for the lane PC. The DWARF location 2150list expression created for it is used as the value of the 2151``DW_AT_LLVM_lane_pc`` attribute on the subprogram's debugger information entry. 2152 2153A DWARF procedure is defined for each well nested structured control flow region 2154which provides the conceptual lane program location for a lane if it is not 2155active (namely it is divergent). The DWARF operation expression for each region 2156conceptually inherits the value of the immediately enclosing region and modifies 2157it according to the semantics of the region. 2158 2159For an ``IF/THEN/ELSE`` region the divergent program location is at the start of 2160the region for the ``THEN`` region since it is executed first. For the ``ELSE`` 2161region the divergent program location is at the end of the ``IF/THEN/ELSE`` 2162region since the ``THEN`` region has completed. 2163 2164The lane PC artificial variable is assigned at each region transition. It uses 2165the immediately enclosing region's DWARF procedure to compute the program 2166location for each lane assuming they are divergent, and then modifies the result 2167by inserting the current program location for each lane that the ``EXEC`` mask 2168indicates is active. 2169 2170By having separate DWARF procedures for each region, they can be reused to 2171define the value for any nested region. This reduces the total size of the DWARF 2172operation expressions. 2173 2174The following provides an example using pseudo LLVM MIR. 2175 2176.. code:: 2177 :number-lines: 2178 2179 $lex_start: 2180 DEFINE_DWARF %__uint_64 = DW_TAG_base_type[ 2181 DW_AT_name = "__uint64"; 2182 DW_AT_byte_size = 8; 2183 DW_AT_encoding = DW_ATE_unsigned; 2184 ]; 2185 DEFINE_DWARF %__active_lane_pc = DW_TAG_dwarf_procedure[ 2186 DW_AT_name = "__active_lane_pc"; 2187 DW_AT_location = [ 2188 DW_OP_regx PC; 2189 DW_OP_LLVM_extend 64, 64; 2190 DW_OP_regval_type EXEC, %uint_64; 2191 DW_OP_LLVM_select_bit_piece 64, 64; 2192 ]; 2193 ]; 2194 DEFINE_DWARF %__divergent_lane_pc = DW_TAG_dwarf_procedure[ 2195 DW_AT_name = "__divergent_lane_pc"; 2196 DW_AT_location = [ 2197 DW_OP_LLVM_undefined; 2198 DW_OP_LLVM_extend 64, 64; 2199 ]; 2200 ]; 2201 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 2202 DW_OP_call_ref %__divergent_lane_pc; 2203 DW_OP_call_ref %__active_lane_pc; 2204 ]; 2205 a; 2206 %1 = EXEC; 2207 DBG_VALUE %1, $noreg, %__lex_1_save_exec; 2208 %2 = c1; 2209 $lex_1_start: 2210 EXEC = %1 & %2; 2211 $lex_1_then: 2212 DEFINE_DWARF %__divergent_lane_pc_1_then = DW_TAG_dwarf_procedure[ 2213 DW_AT_name = "__divergent_lane_pc_1_then"; 2214 DW_AT_location = DIExpression[ 2215 DW_OP_call_ref %__divergent_lane_pc; 2216 DW_OP_addrx &lex_1_start; 2217 DW_OP_stack_value; 2218 DW_OP_LLVM_extend 64, 64; 2219 DW_OP_call_ref %__lex_1_save_exec; 2220 DW_OP_deref_type 64, %__uint_64; 2221 DW_OP_LLVM_select_bit_piece 64, 64; 2222 ]; 2223 ]; 2224 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 2225 DW_OP_call_ref %__divergent_lane_pc_1_then; 2226 DW_OP_call_ref %__active_lane_pc; 2227 ]; 2228 b; 2229 %3 = EXEC; 2230 DBG_VALUE %3, %__lex_1_1_save_exec; 2231 %4 = c2; 2232 $lex_1_1_start: 2233 EXEC = %3 & %4; 2234 $lex_1_1_then: 2235 DEFINE_DWARF %__divergent_lane_pc_1_1_then = DW_TAG_dwarf_procedure[ 2236 DW_AT_name = "__divergent_lane_pc_1_1_then"; 2237 DW_AT_location = DIExpression[ 2238 DW_OP_call_ref %__divergent_lane_pc_1_then; 2239 DW_OP_addrx &lex_1_1_start; 2240 DW_OP_stack_value; 2241 DW_OP_LLVM_extend 64, 64; 2242 DW_OP_call_ref %__lex_1_1_save_exec; 2243 DW_OP_deref_type 64, %__uint_64; 2244 DW_OP_LLVM_select_bit_piece 64, 64; 2245 ]; 2246 ]; 2247 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 2248 DW_OP_call_ref %__divergent_lane_pc_1_1_then; 2249 DW_OP_call_ref %__active_lane_pc; 2250 ]; 2251 c; 2252 EXEC = ~EXEC & %3; 2253 $lex_1_1_else: 2254 DEFINE_DWARF %__divergent_lane_pc_1_1_else = DW_TAG_dwarf_procedure[ 2255 DW_AT_name = "__divergent_lane_pc_1_1_else"; 2256 DW_AT_location = DIExpression[ 2257 DW_OP_call_ref %__divergent_lane_pc_1_then; 2258 DW_OP_addrx &lex_1_1_end; 2259 DW_OP_stack_value; 2260 DW_OP_LLVM_extend 64, 64; 2261 DW_OP_call_ref %__lex_1_1_save_exec; 2262 DW_OP_deref_type 64, %__uint_64; 2263 DW_OP_LLVM_select_bit_piece 64, 64; 2264 ]; 2265 ]; 2266 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 2267 DW_OP_call_ref %__divergent_lane_pc_1_1_else; 2268 DW_OP_call_ref %__active_lane_pc; 2269 ]; 2270 d; 2271 EXEC = %3; 2272 $lex_1_1_end: 2273 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 2274 DW_OP_call_ref %__divergent_lane_pc; 2275 DW_OP_call_ref %__active_lane_pc; 2276 ]; 2277 e; 2278 EXEC = ~EXEC & %1; 2279 $lex_1_else: 2280 DEFINE_DWARF %__divergent_lane_pc_1_else = DW_TAG_dwarf_procedure[ 2281 DW_AT_name = "__divergent_lane_pc_1_else"; 2282 DW_AT_location = DIExpression[ 2283 DW_OP_call_ref %__divergent_lane_pc; 2284 DW_OP_addrx &lex_1_end; 2285 DW_OP_stack_value; 2286 DW_OP_LLVM_extend 64, 64; 2287 DW_OP_call_ref %__lex_1_save_exec; 2288 DW_OP_deref_type 64, %__uint_64; 2289 DW_OP_LLVM_select_bit_piece 64, 64; 2290 ]; 2291 ]; 2292 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 2293 DW_OP_call_ref %__divergent_lane_pc_1_else; 2294 DW_OP_call_ref %__active_lane_pc; 2295 ]; 2296 f; 2297 EXEC = %1; 2298 $lex_1_end: 2299 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc DIExpression[ 2300 DW_OP_call_ref %__divergent_lane_pc; 2301 DW_OP_call_ref %__active_lane_pc; 2302 ]; 2303 g; 2304 $lex_end: 2305 2306The DWARF procedure ``%__active_lane_pc`` is used to update the lane pc elements 2307that are active, with the current program location. 2308 2309Artificial variables %__lex_1_save_exec and %__lex_1_1_save_exec are created for 2310the execution masks saved on entry to a region. Using the ``DBG_VALUE`` pseudo 2311instruction, location list entries will be created that describe where the 2312artificial variables are allocated at any given program location. The compiler 2313may allocate them to registers or spill them to memory. 2314 2315The DWARF procedures for each region use the values of the saved execution mask 2316artificial variables to only update the lanes that are active on entry to the 2317region. All other lanes retain the value of the enclosing region where they were 2318last active. If they were not active on entry to the subprogram, then will have 2319the undefined location description. 2320 2321Other structured control flow regions can be handled similarly. For example, 2322loops would set the divergent program location for the region at the end of the 2323loop. Any lanes active will be in the loop, and any lanes not active must have 2324exited the loop. 2325 2326An ``IF/THEN/ELSEIF/ELSEIF/...`` region can be treated as a nest of 2327``IF/THEN/ELSE`` regions. 2328 2329The DWARF procedures can use the active lane artificial variable described in 2330:ref:`amdgpu-dwarf-amdgpu-dw-at-llvm-active-lane` rather than the actual 2331``EXEC`` mask in order to support whole or quad wavefront mode. 2332 2333.. _amdgpu-dwarf-amdgpu-dw-at-llvm-active-lane: 2334 2335``DW_AT_LLVM_active_lane`` 2336~~~~~~~~~~~~~~~~~~~~~~~~~~ 2337 2338The ``DW_AT_LLVM_active_lane`` attribute on a subprogram debugger information 2339entry is used to specify the lanes that are conceptually active for a SIMT 2340thread. 2341 2342The execution mask may be modified to implement whole or quad wavefront mode 2343operations. For example, all lanes may need to temporarily be made active to 2344execute a whole wavefront operation. Such regions would save the ``EXEC`` mask, 2345update it to enable the necessary lanes, perform the operations, and then 2346restore the ``EXEC`` mask from the saved value. While executing the whole 2347wavefront region, the conceptual execution mask is the saved value, not the 2348``EXEC`` value. 2349 2350This is handled by defining an artificial variable for the active lane mask. The 2351active lane mask artificial variable would be the actual ``EXEC`` mask for 2352normal regions, and the saved execution mask for regions where the mask is 2353temporarily updated. The location list expression created for this artificial 2354variable is used to define the value of the ``DW_AT_LLVM_active_lane`` 2355attribute. 2356 2357``DW_AT_LLVM_augmentation`` 2358~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2359 2360For AMDGPU, the ``DW_AT_LLVM_augmentation`` attribute of a compilation unit 2361debugger information entry has the following value for the augmentation string: 2362 2363:: 2364 2365 [amdgpu:v0.0] 2366 2367The "vX.Y" specifies the major X and minor Y version number of the AMDGPU 2368extensions used in the DWARF of the compilation unit. The version number 2369conforms to [SEMVER]_. 2370 2371Call Frame Information 2372---------------------- 2373 2374DWARF Call Frame Information (CFI) describes how a consumer can virtually 2375*unwind* call frames in a running process or core dump. See DWARF Version 5 2376section 6.4 and :ref:`amdgpu-dwarf-call-frame-information`. 2377 2378For AMDGPU, the Common Information Entry (CIE) fields have the following values: 2379 23801. ``augmentation`` string contains the following null-terminated UTF-8 string: 2381 2382 :: 2383 2384 [amd:v0.0] 2385 2386 The ``vX.Y`` specifies the major X and minor Y version number of the AMDGPU 2387 extensions used in this CIE or to the FDEs that use it. The version number 2388 conforms to [SEMVER]_. 2389 23902. ``address_size`` for the ``Global`` address space is defined in 2391 :ref:`amdgpu-dwarf-address-space-identifier`. 2392 23933. ``segment_selector_size`` is 0 as AMDGPU does not use a segment selector. 2394 23954. ``code_alignment_factor`` is 4 bytes. 2396 2397 .. TODO:: 2398 2399 Add to :ref:`amdgpu-processor-table` table. 2400 24015. ``data_alignment_factor`` is 4 bytes. 2402 2403 .. TODO:: 2404 2405 Add to :ref:`amdgpu-processor-table` table. 2406 24076. ``return_address_register`` is ``PC_32`` for 32-bit processes and ``PC_64`` 2408 for 64-bit processes defined in :ref:`amdgpu-dwarf-register-identifier`. 2409 24107. ``initial_instructions`` Since a subprogram X with fewer registers can be 2411 called from subprogram Y that has more allocated, X will not change any of 2412 the extra registers as it cannot access them. Therefore, the default rule 2413 for all columns is ``same value``. 2414 2415For AMDGPU the register number follows the numbering defined in 2416:ref:`amdgpu-dwarf-register-identifier`. 2417 2418For AMDGPU the instructions are variable size. A consumer can subtract 1 from 2419the return address to get the address of a byte within the call site 2420instructions. See DWARF Version 5 section 6.4.4. 2421 2422Accelerated Access 2423------------------ 2424 2425See DWARF Version 5 section 6.1. 2426 2427Lookup By Name Section Header 2428~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2429 2430See DWARF Version 5 section 6.1.1.4.1 and :ref:`amdgpu-dwarf-lookup-by-name`. 2431 2432For AMDGPU the lookup by name section header table: 2433 2434``augmentation_string_size`` (uword) 2435 2436 Set to the length of the ``augmentation_string`` value which is always a 2437 multiple of 4. 2438 2439``augmentation_string`` (sequence of UTF-8 characters) 2440 2441 Contains the following UTF-8 string null padded to a multiple of 4 bytes: 2442 2443 :: 2444 2445 [amdgpu:v0.0] 2446 2447 The "vX.Y" specifies the major X and minor Y version number of the AMDGPU 2448 extensions used in the DWARF of this index. The version number conforms to 2449 [SEMVER]_. 2450 2451 .. note:: 2452 2453 This is different to the DWARF Version 5 definition that requires the first 2454 4 characters to be the vendor ID. But this is consistent with the other 2455 augmentation strings and does allow multiple vendor contributions. However, 2456 backwards compatibility may be more desirable. 2457 2458Lookup By Address Section Header 2459~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2460 2461See DWARF Version 5 section 6.1.2. 2462 2463For AMDGPU the lookup by address section header table: 2464 2465``address_size`` (ubyte) 2466 2467 Match the address size for the ``Global`` address space defined in 2468 :ref:`amdgpu-dwarf-address-space-identifier`. 2469 2470``segment_selector_size`` (ubyte) 2471 2472 AMDGPU does not use a segment selector so this is 0. The entries in the 2473 ``.debug_aranges`` do not have a segment selector. 2474 2475Line Number Information 2476----------------------- 2477 2478See DWARF Version 5 section 6.2 and :ref:`amdgpu-dwarf-line-number-information`. 2479 2480AMDGPU does not use the ``isa`` state machine registers and always sets it to 0. 2481The instruction set must be obtained from the ELF file header ``e_flags`` field 2482in the ``EF_AMDGPU_MACH`` bit position (see :ref:`ELF Header 2483<amdgpu-elf-header>`). See DWARF Version 5 section 6.2.2. 2484 2485.. TODO:: 2486 2487 Should the ``isa`` state machine register be used to indicate if the code is 2488 in wavefront32 or wavefront64 mode? Or used to specify the architecture ISA? 2489 2490For AMDGPU the line number program header fields have the following values (see 2491DWARF Version 5 section 6.2.4): 2492 2493``address_size`` (ubyte) 2494 Matches the address size for the ``Global`` address space defined in 2495 :ref:`amdgpu-dwarf-address-space-identifier`. 2496 2497``segment_selector_size`` (ubyte) 2498 AMDGPU does not use a segment selector so this is 0. 2499 2500``minimum_instruction_length`` (ubyte) 2501 For GFX9-GFX10 this is 4. 2502 2503``maximum_operations_per_instruction`` (ubyte) 2504 For GFX9-GFX10 this is 1. 2505 2506Source text for online-compiled programs (for example, those compiled by the 2507OpenCL language runtime) may be embedded into the DWARF Version 5 line table. 2508See DWARF Version 5 section 6.2.4.1 which is updated by *DWARF Extensions For 2509Heterogeneous Debugging* section :ref:`DW_LNCT_LLVM_source 2510<amdgpu-dwarf-line-number-information-dw-lnct-llvm-source>`. 2511 2512The Clang option used to control source embedding in AMDGPU is defined in 2513:ref:`amdgpu-clang-debug-options-table`. 2514 2515 .. table:: AMDGPU Clang Debug Options 2516 :name: amdgpu-clang-debug-options-table 2517 2518 ==================== ================================================== 2519 Debug Flag Description 2520 ==================== ================================================== 2521 -g[no-]embed-source Enable/disable embedding source text in DWARF 2522 debug sections. Useful for environments where 2523 source cannot be written to disk, such as 2524 when performing online compilation. 2525 ==================== ================================================== 2526 2527For example: 2528 2529``-gembed-source`` 2530 Enable the embedded source. 2531 2532``-gno-embed-source`` 2533 Disable the embedded source. 2534 253532-Bit and 64-Bit DWARF Formats 2536------------------------------- 2537 2538See DWARF Version 5 section 7.4 and 2539:ref:`amdgpu-dwarf-32-bit-and-64-bit-dwarf-formats`. 2540 2541For AMDGPU: 2542 2543* For the ``amdgcn`` target architecture only the 64-bit process address space 2544 is supported. 2545 2546* The producer can generate either 32-bit or 64-bit DWARF format. LLVM generates 2547 the 32-bit DWARF format. 2548 2549Unit Headers 2550------------ 2551 2552For AMDGPU the following values apply for each of the unit headers described in 2553DWARF Version 5 sections 7.5.1.1, 7.5.1.2, and 7.5.1.3: 2554 2555``address_size`` (ubyte) 2556 Matches the address size for the ``Global`` address space defined in 2557 :ref:`amdgpu-dwarf-address-space-identifier`. 2558 2559.. _amdgpu-code-conventions: 2560 2561Code Conventions 2562================ 2563 2564This section provides code conventions used for each supported target triple OS 2565(see :ref:`amdgpu-target-triples`). 2566 2567AMDHSA 2568------ 2569 2570This section provides code conventions used when the target triple OS is 2571``amdhsa`` (see :ref:`amdgpu-target-triples`). 2572 2573.. _amdgpu-amdhsa-code-object-metadata: 2574 2575Code Object Metadata 2576~~~~~~~~~~~~~~~~~~~~ 2577 2578The code object metadata specifies extensible metadata associated with the code 2579objects executed on HSA [HSA]_ compatible runtimes (see :ref:`amdgpu-os`). The 2580encoding and semantics of this metadata depends on the code object version; see 2581:ref:`amdgpu-amdhsa-code-object-metadata-v2`, 2582:ref:`amdgpu-amdhsa-code-object-metadata-v3`, 2583:ref:`amdgpu-amdhsa-code-object-metadata-v4` and 2584:ref:`amdgpu-amdhsa-code-object-metadata-v5`. 2585 2586Code object metadata is specified in a note record (see 2587:ref:`amdgpu-note-records`) and is required when the target triple OS is 2588``amdhsa`` (see :ref:`amdgpu-target-triples`). It must contain the minimum 2589information necessary to support the HSA compatible runtime kernel queries. For 2590example, the segment sizes needed in a dispatch packet. In addition, a 2591high-level language runtime may require other information to be included. For 2592example, the AMD OpenCL runtime records kernel argument information. 2593 2594.. _amdgpu-amdhsa-code-object-metadata-v2: 2595 2596Code Object V2 Metadata 2597+++++++++++++++++++++++ 2598 2599.. warning:: 2600 Code object V2 is not the default code object version emitted by this version 2601 of LLVM. 2602 2603Code object V2 metadata is specified by the ``NT_AMD_HSA_METADATA`` note record 2604(see :ref:`amdgpu-note-records-v2`). 2605 2606The metadata is specified as a YAML formatted string (see [YAML]_ and 2607:doc:`YamlIO`). 2608 2609.. TODO:: 2610 2611 Is the string null terminated? It probably should not if YAML allows it to 2612 contain null characters, otherwise it should be. 2613 2614The metadata is represented as a single YAML document comprised of the mapping 2615defined in table :ref:`amdgpu-amdhsa-code-object-metadata-map-v2-table` and 2616referenced tables. 2617 2618For boolean values, the string values of ``false`` and ``true`` are used for 2619false and true respectively. 2620 2621Additional information can be added to the mappings. To avoid conflicts, any 2622non-AMD key names should be prefixed by "*vendor-name*.". 2623 2624 .. table:: AMDHSA Code Object V2 Metadata Map 2625 :name: amdgpu-amdhsa-code-object-metadata-map-v2-table 2626 2627 ========== ============== ========= ======================================= 2628 String Key Value Type Required? Description 2629 ========== ============== ========= ======================================= 2630 "Version" sequence of Required - The first integer is the major 2631 2 integers version. Currently 1. 2632 - The second integer is the minor 2633 version. Currently 0. 2634 "Printf" sequence of Each string is encoded information 2635 strings about a printf function call. The 2636 encoded information is organized as 2637 fields separated by colon (':'): 2638 2639 ``ID:N:S[0]:S[1]:...:S[N-1]:FormatString`` 2640 2641 where: 2642 2643 ``ID`` 2644 A 32-bit integer as a unique id for 2645 each printf function call 2646 2647 ``N`` 2648 A 32-bit integer equal to the number 2649 of arguments of printf function call 2650 minus 1 2651 2652 ``S[i]`` (where i = 0, 1, ... , N-1) 2653 32-bit integers for the size in bytes 2654 of the i-th FormatString argument of 2655 the printf function call 2656 2657 FormatString 2658 The format string passed to the 2659 printf function call. 2660 "Kernels" sequence of Required Sequence of the mappings for each 2661 mapping kernel in the code object. See 2662 :ref:`amdgpu-amdhsa-code-object-kernel-metadata-map-v2-table` 2663 for the definition of the mapping. 2664 ========== ============== ========= ======================================= 2665 2666.. 2667 2668 .. table:: AMDHSA Code Object V2 Kernel Metadata Map 2669 :name: amdgpu-amdhsa-code-object-kernel-metadata-map-v2-table 2670 2671 ================= ============== ========= ================================ 2672 String Key Value Type Required? Description 2673 ================= ============== ========= ================================ 2674 "Name" string Required Source name of the kernel. 2675 "SymbolName" string Required Name of the kernel 2676 descriptor ELF symbol. 2677 "Language" string Source language of the kernel. 2678 Values include: 2679 2680 - "OpenCL C" 2681 - "OpenCL C++" 2682 - "HCC" 2683 - "OpenMP" 2684 2685 "LanguageVersion" sequence of - The first integer is the major 2686 2 integers version. 2687 - The second integer is the 2688 minor version. 2689 "Attrs" mapping Mapping of kernel attributes. 2690 See 2691 :ref:`amdgpu-amdhsa-code-object-kernel-attribute-metadata-map-v2-table` 2692 for the mapping definition. 2693 "Args" sequence of Sequence of mappings of the 2694 mapping kernel arguments. See 2695 :ref:`amdgpu-amdhsa-code-object-kernel-argument-metadata-map-v2-table` 2696 for the definition of the mapping. 2697 "CodeProps" mapping Mapping of properties related to 2698 the kernel code. See 2699 :ref:`amdgpu-amdhsa-code-object-kernel-code-properties-metadata-map-v2-table` 2700 for the mapping definition. 2701 ================= ============== ========= ================================ 2702 2703.. 2704 2705 .. table:: AMDHSA Code Object V2 Kernel Attribute Metadata Map 2706 :name: amdgpu-amdhsa-code-object-kernel-attribute-metadata-map-v2-table 2707 2708 =================== ============== ========= ============================== 2709 String Key Value Type Required? Description 2710 =================== ============== ========= ============================== 2711 "ReqdWorkGroupSize" sequence of If not 0, 0, 0 then all values 2712 3 integers must be >=1 and the dispatch 2713 work-group size X, Y, Z must 2714 correspond to the specified 2715 values. Defaults to 0, 0, 0. 2716 2717 Corresponds to the OpenCL 2718 ``reqd_work_group_size`` 2719 attribute. 2720 "WorkGroupSizeHint" sequence of The dispatch work-group size 2721 3 integers X, Y, Z is likely to be the 2722 specified values. 2723 2724 Corresponds to the OpenCL 2725 ``work_group_size_hint`` 2726 attribute. 2727 "VecTypeHint" string The name of a scalar or vector 2728 type. 2729 2730 Corresponds to the OpenCL 2731 ``vec_type_hint`` attribute. 2732 2733 "RuntimeHandle" string The external symbol name 2734 associated with a kernel. 2735 OpenCL runtime allocates a 2736 global buffer for the symbol 2737 and saves the kernel's address 2738 to it, which is used for 2739 device side enqueueing. Only 2740 available for device side 2741 enqueued kernels. 2742 =================== ============== ========= ============================== 2743 2744.. 2745 2746 .. table:: AMDHSA Code Object V2 Kernel Argument Metadata Map 2747 :name: amdgpu-amdhsa-code-object-kernel-argument-metadata-map-v2-table 2748 2749 ================= ============== ========= ================================ 2750 String Key Value Type Required? Description 2751 ================= ============== ========= ================================ 2752 "Name" string Kernel argument name. 2753 "TypeName" string Kernel argument type name. 2754 "Size" integer Required Kernel argument size in bytes. 2755 "Align" integer Required Kernel argument alignment in 2756 bytes. Must be a power of two. 2757 "ValueKind" string Required Kernel argument kind that 2758 specifies how to set up the 2759 corresponding argument. 2760 Values include: 2761 2762 "ByValue" 2763 The argument is copied 2764 directly into the kernarg. 2765 2766 "GlobalBuffer" 2767 A global address space pointer 2768 to the buffer data is passed 2769 in the kernarg. 2770 2771 "DynamicSharedPointer" 2772 A group address space pointer 2773 to dynamically allocated LDS 2774 is passed in the kernarg. 2775 2776 "Sampler" 2777 A global address space 2778 pointer to a S# is passed in 2779 the kernarg. 2780 2781 "Image" 2782 A global address space 2783 pointer to a T# is passed in 2784 the kernarg. 2785 2786 "Pipe" 2787 A global address space pointer 2788 to an OpenCL pipe is passed in 2789 the kernarg. 2790 2791 "Queue" 2792 A global address space pointer 2793 to an OpenCL device enqueue 2794 queue is passed in the 2795 kernarg. 2796 2797 "HiddenGlobalOffsetX" 2798 The OpenCL grid dispatch 2799 global offset for the X 2800 dimension is passed in the 2801 kernarg. 2802 2803 "HiddenGlobalOffsetY" 2804 The OpenCL grid dispatch 2805 global offset for the Y 2806 dimension is passed in the 2807 kernarg. 2808 2809 "HiddenGlobalOffsetZ" 2810 The OpenCL grid dispatch 2811 global offset for the Z 2812 dimension is passed in the 2813 kernarg. 2814 2815 "HiddenNone" 2816 An argument that is not used 2817 by the kernel. Space needs to 2818 be left for it, but it does 2819 not need to be set up. 2820 2821 "HiddenPrintfBuffer" 2822 A global address space pointer 2823 to the runtime printf buffer 2824 is passed in kernarg. 2825 2826 "HiddenHostcallBuffer" 2827 A global address space pointer 2828 to the runtime hostcall buffer 2829 is passed in kernarg. 2830 2831 "HiddenDefaultQueue" 2832 A global address space pointer 2833 to the OpenCL device enqueue 2834 queue that should be used by 2835 the kernel by default is 2836 passed in the kernarg. 2837 2838 "HiddenCompletionAction" 2839 A global address space pointer 2840 to help link enqueued kernels into 2841 the ancestor tree for determining 2842 when the parent kernel has finished. 2843 2844 "HiddenMultiGridSyncArg" 2845 A global address space pointer for 2846 multi-grid synchronization is 2847 passed in the kernarg. 2848 2849 "ValueType" string Unused and deprecated. This should no longer 2850 be emitted, but is accepted for compatibility. 2851 2852 2853 "PointeeAlign" integer Alignment in bytes of pointee 2854 type for pointer type kernel 2855 argument. Must be a power 2856 of 2. Only present if 2857 "ValueKind" is 2858 "DynamicSharedPointer". 2859 "AddrSpaceQual" string Kernel argument address space 2860 qualifier. Only present if 2861 "ValueKind" is "GlobalBuffer" or 2862 "DynamicSharedPointer". Values 2863 are: 2864 2865 - "Private" 2866 - "Global" 2867 - "Constant" 2868 - "Local" 2869 - "Generic" 2870 - "Region" 2871 2872 .. TODO:: 2873 2874 Is GlobalBuffer only Global 2875 or Constant? Is 2876 DynamicSharedPointer always 2877 Local? Can HCC allow Generic? 2878 How can Private or Region 2879 ever happen? 2880 2881 "AccQual" string Kernel argument access 2882 qualifier. Only present if 2883 "ValueKind" is "Image" or 2884 "Pipe". Values 2885 are: 2886 2887 - "ReadOnly" 2888 - "WriteOnly" 2889 - "ReadWrite" 2890 2891 .. TODO:: 2892 2893 Does this apply to 2894 GlobalBuffer? 2895 2896 "ActualAccQual" string The actual memory accesses 2897 performed by the kernel on the 2898 kernel argument. Only present if 2899 "ValueKind" is "GlobalBuffer", 2900 "Image", or "Pipe". This may be 2901 more restrictive than indicated 2902 by "AccQual" to reflect what the 2903 kernel actual does. If not 2904 present then the runtime must 2905 assume what is implied by 2906 "AccQual" and "IsConst". Values 2907 are: 2908 2909 - "ReadOnly" 2910 - "WriteOnly" 2911 - "ReadWrite" 2912 2913 "IsConst" boolean Indicates if the kernel argument 2914 is const qualified. Only present 2915 if "ValueKind" is 2916 "GlobalBuffer". 2917 2918 "IsRestrict" boolean Indicates if the kernel argument 2919 is restrict qualified. Only 2920 present if "ValueKind" is 2921 "GlobalBuffer". 2922 2923 "IsVolatile" boolean Indicates if the kernel argument 2924 is volatile qualified. Only 2925 present if "ValueKind" is 2926 "GlobalBuffer". 2927 2928 "IsPipe" boolean Indicates if the kernel argument 2929 is pipe qualified. Only present 2930 if "ValueKind" is "Pipe". 2931 2932 .. TODO:: 2933 2934 Can GlobalBuffer be pipe 2935 qualified? 2936 2937 ================= ============== ========= ================================ 2938 2939.. 2940 2941 .. table:: AMDHSA Code Object V2 Kernel Code Properties Metadata Map 2942 :name: amdgpu-amdhsa-code-object-kernel-code-properties-metadata-map-v2-table 2943 2944 ============================ ============== ========= ===================== 2945 String Key Value Type Required? Description 2946 ============================ ============== ========= ===================== 2947 "KernargSegmentSize" integer Required The size in bytes of 2948 the kernarg segment 2949 that holds the values 2950 of the arguments to 2951 the kernel. 2952 "GroupSegmentFixedSize" integer Required The amount of group 2953 segment memory 2954 required by a 2955 work-group in 2956 bytes. This does not 2957 include any 2958 dynamically allocated 2959 group segment memory 2960 that may be added 2961 when the kernel is 2962 dispatched. 2963 "PrivateSegmentFixedSize" integer Required The amount of fixed 2964 private address space 2965 memory required for a 2966 work-item in 2967 bytes. If the kernel 2968 uses a dynamic call 2969 stack then additional 2970 space must be added 2971 to this value for the 2972 call stack. 2973 "KernargSegmentAlign" integer Required The maximum byte 2974 alignment of 2975 arguments in the 2976 kernarg segment. Must 2977 be a power of 2. 2978 "WavefrontSize" integer Required Wavefront size. Must 2979 be a power of 2. 2980 "NumSGPRs" integer Required Number of scalar 2981 registers used by a 2982 wavefront for 2983 GFX6-GFX10. This 2984 includes the special 2985 SGPRs for VCC, Flat 2986 Scratch (GFX7-GFX10) 2987 and XNACK (for 2988 GFX8-GFX10). It does 2989 not include the 16 2990 SGPR added if a trap 2991 handler is 2992 enabled. It is not 2993 rounded up to the 2994 allocation 2995 granularity. 2996 "NumVGPRs" integer Required Number of vector 2997 registers used by 2998 each work-item for 2999 GFX6-GFX10 3000 "MaxFlatWorkGroupSize" integer Required Maximum flat 3001 work-group size 3002 supported by the 3003 kernel in work-items. 3004 Must be >=1 and 3005 consistent with 3006 ReqdWorkGroupSize if 3007 not 0, 0, 0. 3008 "NumSpilledSGPRs" integer Number of stores from 3009 a scalar register to 3010 a register allocator 3011 created spill 3012 location. 3013 "NumSpilledVGPRs" integer Number of stores from 3014 a vector register to 3015 a register allocator 3016 created spill 3017 location. 3018 ============================ ============== ========= ===================== 3019 3020.. _amdgpu-amdhsa-code-object-metadata-v3: 3021 3022Code Object V3 Metadata 3023+++++++++++++++++++++++ 3024 3025.. warning:: 3026 Code object V3 is not the default code object version emitted by this version 3027 of LLVM. 3028 3029Code object V3 and above metadata is specified by the ``NT_AMDGPU_METADATA`` note 3030record (see :ref:`amdgpu-note-records-v3-onwards`). 3031 3032The metadata is represented as Message Pack formatted binary data (see 3033[MsgPack]_). The top level is a Message Pack map that includes the 3034keys defined in table 3035:ref:`amdgpu-amdhsa-code-object-metadata-map-table-v3` and referenced 3036tables. 3037 3038Additional information can be added to the maps. To avoid conflicts, 3039any key names should be prefixed by "*vendor-name*." where 3040``vendor-name`` can be the name of the vendor and specific vendor 3041tool that generates the information. The prefix is abbreviated to 3042simply "." when it appears within a map that has been added by the 3043same *vendor-name*. 3044 3045 .. table:: AMDHSA Code Object V3 Metadata Map 3046 :name: amdgpu-amdhsa-code-object-metadata-map-table-v3 3047 3048 ================= ============== ========= ======================================= 3049 String Key Value Type Required? Description 3050 ================= ============== ========= ======================================= 3051 "amdhsa.version" sequence of Required - The first integer is the major 3052 2 integers version. Currently 1. 3053 - The second integer is the minor 3054 version. Currently 0. 3055 "amdhsa.printf" sequence of Each string is encoded information 3056 strings about a printf function call. The 3057 encoded information is organized as 3058 fields separated by colon (':'): 3059 3060 ``ID:N:S[0]:S[1]:...:S[N-1]:FormatString`` 3061 3062 where: 3063 3064 ``ID`` 3065 A 32-bit integer as a unique id for 3066 each printf function call 3067 3068 ``N`` 3069 A 32-bit integer equal to the number 3070 of arguments of printf function call 3071 minus 1 3072 3073 ``S[i]`` (where i = 0, 1, ... , N-1) 3074 32-bit integers for the size in bytes 3075 of the i-th FormatString argument of 3076 the printf function call 3077 3078 FormatString 3079 The format string passed to the 3080 printf function call. 3081 "amdhsa.kernels" sequence of Required Sequence of the maps for each 3082 map kernel in the code object. See 3083 :ref:`amdgpu-amdhsa-code-object-kernel-metadata-map-table-v3` 3084 for the definition of the keys included 3085 in that map. 3086 ================= ============== ========= ======================================= 3087 3088.. 3089 3090 .. table:: AMDHSA Code Object V3 Kernel Metadata Map 3091 :name: amdgpu-amdhsa-code-object-kernel-metadata-map-table-v3 3092 3093 =================================== ============== ========= ================================ 3094 String Key Value Type Required? Description 3095 =================================== ============== ========= ================================ 3096 ".name" string Required Source name of the kernel. 3097 ".symbol" string Required Name of the kernel 3098 descriptor ELF symbol. 3099 ".language" string Source language of the kernel. 3100 Values include: 3101 3102 - "OpenCL C" 3103 - "OpenCL C++" 3104 - "HCC" 3105 - "HIP" 3106 - "OpenMP" 3107 - "Assembler" 3108 3109 ".language_version" sequence of - The first integer is the major 3110 2 integers version. 3111 - The second integer is the 3112 minor version. 3113 ".args" sequence of Sequence of maps of the 3114 map kernel arguments. See 3115 :ref:`amdgpu-amdhsa-code-object-kernel-argument-metadata-map-table-v3` 3116 for the definition of the keys 3117 included in that map. 3118 ".reqd_workgroup_size" sequence of If not 0, 0, 0 then all values 3119 3 integers must be >=1 and the dispatch 3120 work-group size X, Y, Z must 3121 correspond to the specified 3122 values. Defaults to 0, 0, 0. 3123 3124 Corresponds to the OpenCL 3125 ``reqd_work_group_size`` 3126 attribute. 3127 ".workgroup_size_hint" sequence of The dispatch work-group size 3128 3 integers X, Y, Z is likely to be the 3129 specified values. 3130 3131 Corresponds to the OpenCL 3132 ``work_group_size_hint`` 3133 attribute. 3134 ".vec_type_hint" string The name of a scalar or vector 3135 type. 3136 3137 Corresponds to the OpenCL 3138 ``vec_type_hint`` attribute. 3139 3140 ".device_enqueue_symbol" string The external symbol name 3141 associated with a kernel. 3142 OpenCL runtime allocates a 3143 global buffer for the symbol 3144 and saves the kernel's address 3145 to it, which is used for 3146 device side enqueueing. Only 3147 available for device side 3148 enqueued kernels. 3149 ".kernarg_segment_size" integer Required The size in bytes of 3150 the kernarg segment 3151 that holds the values 3152 of the arguments to 3153 the kernel. 3154 ".group_segment_fixed_size" integer Required The amount of group 3155 segment memory 3156 required by a 3157 work-group in 3158 bytes. This does not 3159 include any 3160 dynamically allocated 3161 group segment memory 3162 that may be added 3163 when the kernel is 3164 dispatched. 3165 ".private_segment_fixed_size" integer Required The amount of fixed 3166 private address space 3167 memory required for a 3168 work-item in 3169 bytes. If the kernel 3170 uses a dynamic call 3171 stack then additional 3172 space must be added 3173 to this value for the 3174 call stack. 3175 ".kernarg_segment_align" integer Required The maximum byte 3176 alignment of 3177 arguments in the 3178 kernarg segment. Must 3179 be a power of 2. 3180 ".wavefront_size" integer Required Wavefront size. Must 3181 be a power of 2. 3182 ".sgpr_count" integer Required Number of scalar 3183 registers required by a 3184 wavefront for 3185 GFX6-GFX9. A register 3186 is required if it is 3187 used explicitly, or 3188 if a higher numbered 3189 register is used 3190 explicitly. This 3191 includes the special 3192 SGPRs for VCC, Flat 3193 Scratch (GFX7-GFX9) 3194 and XNACK (for 3195 GFX8-GFX9). It does 3196 not include the 16 3197 SGPR added if a trap 3198 handler is 3199 enabled. It is not 3200 rounded up to the 3201 allocation 3202 granularity. 3203 ".vgpr_count" integer Required Number of vector 3204 registers required by 3205 each work-item for 3206 GFX6-GFX9. A register 3207 is required if it is 3208 used explicitly, or 3209 if a higher numbered 3210 register is used 3211 explicitly. 3212 ".agpr_count" integer Required Number of accumulator 3213 registers required by 3214 each work-item for 3215 GFX90A, GFX908. 3216 ".max_flat_workgroup_size" integer Required Maximum flat 3217 work-group size 3218 supported by the 3219 kernel in work-items. 3220 Must be >=1 and 3221 consistent with 3222 ReqdWorkGroupSize if 3223 not 0, 0, 0. 3224 ".sgpr_spill_count" integer Number of stores from 3225 a scalar register to 3226 a register allocator 3227 created spill 3228 location. 3229 ".vgpr_spill_count" integer Number of stores from 3230 a vector register to 3231 a register allocator 3232 created spill 3233 location. 3234 ".kind" string The kind of the kernel 3235 with the following 3236 values: 3237 3238 "normal" 3239 Regular kernels. 3240 3241 "init" 3242 These kernels must be 3243 invoked after loading 3244 the containing code 3245 object and must 3246 complete before any 3247 normal and fini 3248 kernels in the same 3249 code object are 3250 invoked. 3251 3252 "fini" 3253 These kernels must be 3254 invoked before 3255 unloading the 3256 containing code object 3257 and after all init and 3258 normal kernels in the 3259 same code object have 3260 been invoked and 3261 completed. 3262 3263 If omitted, "normal" is 3264 assumed. 3265 =================================== ============== ========= ================================ 3266 3267.. 3268 3269 .. table:: AMDHSA Code Object V3 Kernel Argument Metadata Map 3270 :name: amdgpu-amdhsa-code-object-kernel-argument-metadata-map-table-v3 3271 3272 ====================== ============== ========= ================================ 3273 String Key Value Type Required? Description 3274 ====================== ============== ========= ================================ 3275 ".name" string Kernel argument name. 3276 ".type_name" string Kernel argument type name. 3277 ".size" integer Required Kernel argument size in bytes. 3278 ".offset" integer Required Kernel argument offset in 3279 bytes. The offset must be a 3280 multiple of the alignment 3281 required by the argument. 3282 ".value_kind" string Required Kernel argument kind that 3283 specifies how to set up the 3284 corresponding argument. 3285 Values include: 3286 3287 "by_value" 3288 The argument is copied 3289 directly into the kernarg. 3290 3291 "global_buffer" 3292 A global address space pointer 3293 to the buffer data is passed 3294 in the kernarg. 3295 3296 "dynamic_shared_pointer" 3297 A group address space pointer 3298 to dynamically allocated LDS 3299 is passed in the kernarg. 3300 3301 "sampler" 3302 A global address space 3303 pointer to a S# is passed in 3304 the kernarg. 3305 3306 "image" 3307 A global address space 3308 pointer to a T# is passed in 3309 the kernarg. 3310 3311 "pipe" 3312 A global address space pointer 3313 to an OpenCL pipe is passed in 3314 the kernarg. 3315 3316 "queue" 3317 A global address space pointer 3318 to an OpenCL device enqueue 3319 queue is passed in the 3320 kernarg. 3321 3322 "hidden_global_offset_x" 3323 The OpenCL grid dispatch 3324 global offset for the X 3325 dimension is passed in the 3326 kernarg. 3327 3328 "hidden_global_offset_y" 3329 The OpenCL grid dispatch 3330 global offset for the Y 3331 dimension is passed in the 3332 kernarg. 3333 3334 "hidden_global_offset_z" 3335 The OpenCL grid dispatch 3336 global offset for the Z 3337 dimension is passed in the 3338 kernarg. 3339 3340 "hidden_none" 3341 An argument that is not used 3342 by the kernel. Space needs to 3343 be left for it, but it does 3344 not need to be set up. 3345 3346 "hidden_printf_buffer" 3347 A global address space pointer 3348 to the runtime printf buffer 3349 is passed in kernarg. 3350 3351 "hidden_hostcall_buffer" 3352 A global address space pointer 3353 to the runtime hostcall buffer 3354 is passed in kernarg. 3355 3356 "hidden_default_queue" 3357 A global address space pointer 3358 to the OpenCL device enqueue 3359 queue that should be used by 3360 the kernel by default is 3361 passed in the kernarg. 3362 3363 "hidden_completion_action" 3364 A global address space pointer 3365 to help link enqueued kernels into 3366 the ancestor tree for determining 3367 when the parent kernel has finished. 3368 3369 "hidden_multigrid_sync_arg" 3370 A global address space pointer for 3371 multi-grid synchronization is 3372 passed in the kernarg. 3373 3374 ".value_type" string Unused and deprecated. This should no longer 3375 be emitted, but is accepted for compatibility. 3376 3377 ".pointee_align" integer Alignment in bytes of pointee 3378 type for pointer type kernel 3379 argument. Must be a power 3380 of 2. Only present if 3381 ".value_kind" is 3382 "dynamic_shared_pointer". 3383 ".address_space" string Kernel argument address space 3384 qualifier. Only present if 3385 ".value_kind" is "global_buffer" or 3386 "dynamic_shared_pointer". Values 3387 are: 3388 3389 - "private" 3390 - "global" 3391 - "constant" 3392 - "local" 3393 - "generic" 3394 - "region" 3395 3396 .. TODO:: 3397 3398 Is "global_buffer" only "global" 3399 or "constant"? Is 3400 "dynamic_shared_pointer" always 3401 "local"? Can HCC allow "generic"? 3402 How can "private" or "region" 3403 ever happen? 3404 3405 ".access" string Kernel argument access 3406 qualifier. Only present if 3407 ".value_kind" is "image" or 3408 "pipe". Values 3409 are: 3410 3411 - "read_only" 3412 - "write_only" 3413 - "read_write" 3414 3415 .. TODO:: 3416 3417 Does this apply to 3418 "global_buffer"? 3419 3420 ".actual_access" string The actual memory accesses 3421 performed by the kernel on the 3422 kernel argument. Only present if 3423 ".value_kind" is "global_buffer", 3424 "image", or "pipe". This may be 3425 more restrictive than indicated 3426 by ".access" to reflect what the 3427 kernel actual does. If not 3428 present then the runtime must 3429 assume what is implied by 3430 ".access" and ".is_const" . Values 3431 are: 3432 3433 - "read_only" 3434 - "write_only" 3435 - "read_write" 3436 3437 ".is_const" boolean Indicates if the kernel argument 3438 is const qualified. Only present 3439 if ".value_kind" is 3440 "global_buffer". 3441 3442 ".is_restrict" boolean Indicates if the kernel argument 3443 is restrict qualified. Only 3444 present if ".value_kind" is 3445 "global_buffer". 3446 3447 ".is_volatile" boolean Indicates if the kernel argument 3448 is volatile qualified. Only 3449 present if ".value_kind" is 3450 "global_buffer". 3451 3452 ".is_pipe" boolean Indicates if the kernel argument 3453 is pipe qualified. Only present 3454 if ".value_kind" is "pipe". 3455 3456 .. TODO:: 3457 3458 Can "global_buffer" be pipe 3459 qualified? 3460 3461 ====================== ============== ========= ================================ 3462 3463.. _amdgpu-amdhsa-code-object-metadata-v4: 3464 3465Code Object V4 Metadata 3466+++++++++++++++++++++++ 3467 3468Code object V4 metadata is the same as 3469:ref:`amdgpu-amdhsa-code-object-metadata-v3` with the changes and additions 3470defined in table :ref:`amdgpu-amdhsa-code-object-metadata-map-table-v4`. 3471 3472 .. table:: AMDHSA Code Object V4 Metadata Map Changes 3473 :name: amdgpu-amdhsa-code-object-metadata-map-table-v4 3474 3475 ================= ============== ========= ======================================= 3476 String Key Value Type Required? Description 3477 ================= ============== ========= ======================================= 3478 "amdhsa.version" sequence of Required - The first integer is the major 3479 2 integers version. Currently 1. 3480 - The second integer is the minor 3481 version. Currently 1. 3482 "amdhsa.target" string Required The target name of the code using the syntax: 3483 3484 .. code:: 3485 3486 <target-triple> [ "-" <target-id> ] 3487 3488 A canonical target ID must be 3489 used. See :ref:`amdgpu-target-triples` 3490 and :ref:`amdgpu-target-id`. 3491 ================= ============== ========= ======================================= 3492 3493.. _amdgpu-amdhsa-code-object-metadata-v5: 3494 3495Code Object V5 Metadata 3496+++++++++++++++++++++++ 3497 3498.. warning:: 3499 Code object V5 is not the default code object version emitted by this version 3500 of LLVM. 3501 3502 3503Code object V5 metadata is the same as 3504:ref:`amdgpu-amdhsa-code-object-metadata-v4` with the changes defined in table 3505:ref:`amdgpu-amdhsa-code-object-metadata-map-table-v5` and table 3506:ref:`amdgpu-amdhsa-code-object-kernel-argument-metadata-map-table-v5`. 3507 3508 .. table:: AMDHSA Code Object V5 Metadata Map Changes 3509 :name: amdgpu-amdhsa-code-object-metadata-map-table-v5 3510 3511 ================= ============== ========= ======================================= 3512 String Key Value Type Required? Description 3513 ================= ============== ========= ======================================= 3514 "amdhsa.version" sequence of Required - The first integer is the major 3515 2 integers version. Currently 1. 3516 - The second integer is the minor 3517 version. Currently 2. 3518 ================= ============== ========= ======================================= 3519 3520.. 3521 3522 .. table:: AMDHSA Code Object V5 Kernel Argument Metadata Map Additions and Changes 3523 :name: amdgpu-amdhsa-code-object-kernel-argument-metadata-map-table-v5 3524 3525 ====================== ============== ========= ================================ 3526 String Key Value Type Required? Description 3527 ====================== ============== ========= ================================ 3528 ".value_kind" string Required Kernel argument kind that 3529 specifies how to set up the 3530 corresponding argument. 3531 Values include: 3532 the same as code object V3 metadata 3533 (see :ref:`amdgpu-amdhsa-code-object-kernel-argument-metadata-map-table-v3`) 3534 with the following additions: 3535 3536 "hidden_block_count_x" 3537 The grid dispatch work-group count for the X dimension 3538 is passed in the kernarg. Some languages, such as OpenCL, 3539 support a last work-group in each dimension being partial. 3540 This count only includes the non-partial work-group count. 3541 This is not the same as the value in the AQL dispatch packet, 3542 which has the grid size in work-items. 3543 3544 "hidden_block_count_y" 3545 The grid dispatch work-group count for the Y dimension 3546 is passed in the kernarg. Some languages, such as OpenCL, 3547 support a last work-group in each dimension being partial. 3548 This count only includes the non-partial work-group count. 3549 This is not the same as the value in the AQL dispatch packet, 3550 which has the grid size in work-items. If the grid dimensionality 3551 is 1, then must be 1. 3552 3553 "hidden_block_count_z" 3554 The grid dispatch work-group count for the Z dimension 3555 is passed in the kernarg. Some languages, such as OpenCL, 3556 support a last work-group in each dimension being partial. 3557 This count only includes the non-partial work-group count. 3558 This is not the same as the value in the AQL dispatch packet, 3559 which has the grid size in work-items. If the grid dimensionality 3560 is 1 or 2, then must be 1. 3561 3562 "hidden_group_size_x" 3563 The grid dispatch work-group size for the X dimension is 3564 passed in the kernarg. This size only applies to the 3565 non-partial work-groups. This is the same value as the AQL 3566 dispatch packet work-group size. 3567 3568 "hidden_group_size_y" 3569 The grid dispatch work-group size for the Y dimension is 3570 passed in the kernarg. This size only applies to the 3571 non-partial work-groups. This is the same value as the AQL 3572 dispatch packet work-group size. If the grid dimensionality 3573 is 1, then must be 1. 3574 3575 "hidden_group_size_z" 3576 The grid dispatch work-group size for the Z dimension is 3577 passed in the kernarg. This size only applies to the 3578 non-partial work-groups. This is the same value as the AQL 3579 dispatch packet work-group size. If the grid dimensionality 3580 is 1 or 2, then must be 1. 3581 3582 "hidden_remainder_x" 3583 The grid dispatch work group size of the the partial work group 3584 of the X dimension, if it exists. Must be zero if a partial 3585 work group does not exist in the X dimension. 3586 3587 "hidden_remainder_y" 3588 The grid dispatch work group size of the the partial work group 3589 of the Y dimension, if it exists. Must be zero if a partial 3590 work group does not exist in the Y dimension. 3591 3592 "hidden_remainder_z" 3593 The grid dispatch work group size of the the partial work group 3594 of the Z dimension, if it exists. Must be zero if a partial 3595 work group does not exist in the Z dimension. 3596 3597 "hidden_grid_dims" 3598 The grid dispatch dimensionality. This is the same value 3599 as the AQL dispatch packet dimensionality. Must be a value 3600 between 1 and 3. 3601 3602 "hidden_heap_v1" 3603 A global address space pointer to an initialized memory 3604 buffer that conforms to the requirements of the malloc/free 3605 device library V1 version implementation. 3606 3607 "hidden_private_base" 3608 The high 32 bits of the flat addressing private aperture base. 3609 Only used by GFX8 to allow conversion between private segment 3610 and flat addresses. See :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`. 3611 3612 "hidden_shared_base" 3613 The high 32 bits of the flat addressing shared aperture base. 3614 Only used by GFX8 to allow conversion between shared segment 3615 and flat addresses. See :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`. 3616 3617 "hidden_queue_ptr" 3618 A global memory address space pointer to the ROCm runtime 3619 ``struct amd_queue_t`` structure for the HSA queue of the 3620 associated dispatch AQL packet. It is only required for pre-GFX9 3621 devices for the trap handler ABI (see :ref:`amdgpu-amdhsa-trap-handler-abi`). 3622 3623 ====================== ============== ========= ================================ 3624 3625.. 3626 3627Kernel Dispatch 3628~~~~~~~~~~~~~~~ 3629 3630The HSA architected queuing language (AQL) defines a user space memory interface 3631that can be used to control the dispatch of kernels, in an agent independent 3632way. An agent can have zero or more AQL queues created for it using an HSA 3633compatible runtime (see :ref:`amdgpu-os`), in which AQL packets (all of which 3634are 64 bytes) can be placed. See the *HSA Platform System Architecture 3635Specification* [HSA]_ for the AQL queue mechanics and packet layouts. 3636 3637The packet processor of a kernel agent is responsible for detecting and 3638dispatching HSA kernels from the AQL queues associated with it. For AMD GPUs the 3639packet processor is implemented by the hardware command processor (CP), 3640asynchronous dispatch controller (ADC) and shader processor input controller 3641(SPI). 3642 3643An HSA compatible runtime can be used to allocate an AQL queue object. It uses 3644the kernel mode driver to initialize and register the AQL queue with CP. 3645 3646To dispatch a kernel the following actions are performed. This can occur in the 3647CPU host program, or from an HSA kernel executing on a GPU. 3648 36491. A pointer to an AQL queue for the kernel agent on which the kernel is to be 3650 executed is obtained. 36512. A pointer to the kernel descriptor (see 3652 :ref:`amdgpu-amdhsa-kernel-descriptor`) of the kernel to execute is obtained. 3653 It must be for a kernel that is contained in a code object that was loaded 3654 by an HSA compatible runtime on the kernel agent with which the AQL queue is 3655 associated. 36563. Space is allocated for the kernel arguments using the HSA compatible runtime 3657 allocator for a memory region with the kernarg property for the kernel agent 3658 that will execute the kernel. It must be at least 16-byte aligned. 36594. Kernel argument values are assigned to the kernel argument memory 3660 allocation. The layout is defined in the *HSA Programmer's Language 3661 Reference* [HSA]_. For AMDGPU the kernel execution directly accesses the 3662 kernel argument memory in the same way constant memory is accessed. (Note 3663 that the HSA specification allows an implementation to copy the kernel 3664 argument contents to another location that is accessed by the kernel.) 36655. An AQL kernel dispatch packet is created on the AQL queue. The HSA compatible 3666 runtime api uses 64-bit atomic operations to reserve space in the AQL queue 3667 for the packet. The packet must be set up, and the final write must use an 3668 atomic store release to set the packet kind to ensure the packet contents are 3669 visible to the kernel agent. AQL defines a doorbell signal mechanism to 3670 notify the kernel agent that the AQL queue has been updated. These rules, and 3671 the layout of the AQL queue and kernel dispatch packet is defined in the *HSA 3672 System Architecture Specification* [HSA]_. 36736. A kernel dispatch packet includes information about the actual dispatch, 3674 such as grid and work-group size, together with information from the code 3675 object about the kernel, such as segment sizes. The HSA compatible runtime 3676 queries on the kernel symbol can be used to obtain the code object values 3677 which are recorded in the :ref:`amdgpu-amdhsa-code-object-metadata`. 36787. CP executes micro-code and is responsible for detecting and setting up the 3679 GPU to execute the wavefronts of a kernel dispatch. 36808. CP ensures that when the a wavefront starts executing the kernel machine 3681 code, the scalar general purpose registers (SGPR) and vector general purpose 3682 registers (VGPR) are set up as required by the machine code. The required 3683 setup is defined in the :ref:`amdgpu-amdhsa-kernel-descriptor`. The initial 3684 register state is defined in 3685 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`. 36869. The prolog of the kernel machine code (see 3687 :ref:`amdgpu-amdhsa-kernel-prolog`) sets up the machine state as necessary 3688 before continuing executing the machine code that corresponds to the kernel. 368910. When the kernel dispatch has completed execution, CP signals the completion 3690 signal specified in the kernel dispatch packet if not 0. 3691 3692.. _amdgpu-amdhsa-memory-spaces: 3693 3694Memory Spaces 3695~~~~~~~~~~~~~ 3696 3697The memory space properties are: 3698 3699 .. table:: AMDHSA Memory Spaces 3700 :name: amdgpu-amdhsa-memory-spaces-table 3701 3702 ================= =========== ======== ======= ================== 3703 Memory Space Name HSA Segment Hardware Address NULL Value 3704 Name Name Size 3705 ================= =========== ======== ======= ================== 3706 Private private scratch 32 0x00000000 3707 Local group LDS 32 0xFFFFFFFF 3708 Global global global 64 0x0000000000000000 3709 Constant constant *same as 64 0x0000000000000000 3710 global* 3711 Generic flat flat 64 0x0000000000000000 3712 Region N/A GDS 32 *not implemented 3713 for AMDHSA* 3714 ================= =========== ======== ======= ================== 3715 3716The global and constant memory spaces both use global virtual addresses, which 3717are the same virtual address space used by the CPU. However, some virtual 3718addresses may only be accessible to the CPU, some only accessible by the GPU, 3719and some by both. 3720 3721Using the constant memory space indicates that the data will not change during 3722the execution of the kernel. This allows scalar read instructions to be 3723used. The vector and scalar L1 caches are invalidated of volatile data before 3724each kernel dispatch execution to allow constant memory to change values between 3725kernel dispatches. 3726 3727The local memory space uses the hardware Local Data Store (LDS) which is 3728automatically allocated when the hardware creates work-groups of wavefronts, and 3729freed when all the wavefronts of a work-group have terminated. The data store 3730(DS) instructions can be used to access it. 3731 3732The private memory space uses the hardware scratch memory support. If the kernel 3733uses scratch, then the hardware allocates memory that is accessed using 3734wavefront lane dword (4 byte) interleaving. The mapping used from private 3735address to physical address is: 3736 3737 ``wavefront-scratch-base + 3738 (private-address * wavefront-size * 4) + 3739 (wavefront-lane-id * 4)`` 3740 3741There are different ways that the wavefront scratch base address is determined 3742by a wavefront (see :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). This 3743memory can be accessed in an interleaved manner using buffer instruction with 3744the scratch buffer descriptor and per wavefront scratch offset, by the scratch 3745instructions, or by flat instructions. If each lane of a wavefront accesses the 3746same private address, the interleaving results in adjacent dwords being accessed 3747and hence requires fewer cache lines to be fetched. Multi-dword access is not 3748supported except by flat and scratch instructions in GFX9-GFX10. 3749 3750The generic address space uses the hardware flat address support available in 3751GFX7-GFX10. This uses two fixed ranges of virtual addresses (the private and 3752local apertures), that are outside the range of addressible global memory, to 3753map from a flat address to a private or local address. 3754 3755FLAT instructions can take a flat address and access global, private (scratch) 3756and group (LDS) memory depending on if the address is within one of the 3757aperture ranges. Flat access to scratch requires hardware aperture setup and 3758setup in the kernel prologue (see 3759:ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`). Flat access to LDS requires 3760hardware aperture setup and M0 (GFX7-GFX8) register setup (see 3761:ref:`amdgpu-amdhsa-kernel-prolog-m0`). 3762 3763To convert between a segment address and a flat address the base address of the 3764apertures address can be used. For GFX7-GFX8 these are available in the 3765:ref:`amdgpu-amdhsa-hsa-aql-queue` the address of which can be obtained with 3766Queue Ptr SGPR (see :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). For 3767GFX9-GFX10 the aperture base addresses are directly available as inline constant 3768registers ``SRC_SHARED_BASE/LIMIT`` and ``SRC_PRIVATE_BASE/LIMIT``. In 64 bit 3769address mode the aperture sizes are 2^32 bytes and the base is aligned to 2^32 3770which makes it easier to convert from flat to segment or segment to flat. 3771 3772Image and Samplers 3773~~~~~~~~~~~~~~~~~~ 3774 3775Image and sample handles created by an HSA compatible runtime (see 3776:ref:`amdgpu-os`) are 64-bit addresses of a hardware 32-byte V# and 48 byte S# 3777object respectively. In order to support the HSA ``query_sampler`` operations 3778two extra dwords are used to store the HSA BRIG enumeration values for the 3779queries that are not trivially deducible from the S# representation. 3780 3781HSA Signals 3782~~~~~~~~~~~ 3783 3784HSA signal handles created by an HSA compatible runtime (see :ref:`amdgpu-os`) 3785are 64-bit addresses of a structure allocated in memory accessible from both the 3786CPU and GPU. The structure is defined by the runtime and subject to change 3787between releases. For example, see [AMD-ROCm-github]_. 3788 3789.. _amdgpu-amdhsa-hsa-aql-queue: 3790 3791HSA AQL Queue 3792~~~~~~~~~~~~~ 3793 3794The HSA AQL queue structure is defined by an HSA compatible runtime (see 3795:ref:`amdgpu-os`) and subject to change between releases. For example, see 3796[AMD-ROCm-github]_. For some processors it contains fields needed to implement 3797certain language features such as the flat address aperture bases. It also 3798contains fields used by CP such as managing the allocation of scratch memory. 3799 3800.. _amdgpu-amdhsa-kernel-descriptor: 3801 3802Kernel Descriptor 3803~~~~~~~~~~~~~~~~~ 3804 3805A kernel descriptor consists of the information needed by CP to initiate the 3806execution of a kernel, including the entry point address of the machine code 3807that implements the kernel. 3808 3809Code Object V3 Kernel Descriptor 3810++++++++++++++++++++++++++++++++ 3811 3812CP microcode requires the Kernel descriptor to be allocated on 64-byte 3813alignment. 3814 3815The fields used by CP for code objects before V3 also match those specified in 3816:ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 3817 3818 .. table:: Code Object V3 Kernel Descriptor 3819 :name: amdgpu-amdhsa-kernel-descriptor-v3-table 3820 3821 ======= ======= =============================== ============================ 3822 Bits Size Field Name Description 3823 ======= ======= =============================== ============================ 3824 31:0 4 bytes GROUP_SEGMENT_FIXED_SIZE The amount of fixed local 3825 address space memory 3826 required for a work-group 3827 in bytes. This does not 3828 include any dynamically 3829 allocated local address 3830 space memory that may be 3831 added when the kernel is 3832 dispatched. 3833 63:32 4 bytes PRIVATE_SEGMENT_FIXED_SIZE The amount of fixed 3834 private address space 3835 memory required for a 3836 work-item in bytes. 3837 Additional space may need to 3838 be added to this value if 3839 the call stack has 3840 non-inlined function calls. 3841 95:64 4 bytes KERNARG_SIZE The size of the kernarg 3842 memory pointed to by the 3843 AQL dispatch packet. The 3844 kernarg memory is used to 3845 pass arguments to the 3846 kernel. 3847 3848 * If the kernarg pointer in 3849 the dispatch packet is NULL 3850 then there are no kernel 3851 arguments. 3852 * If the kernarg pointer in 3853 the dispatch packet is 3854 not NULL and this value 3855 is 0 then the kernarg 3856 memory size is 3857 unspecified. 3858 * If the kernarg pointer in 3859 the dispatch packet is 3860 not NULL and this value 3861 is not 0 then the value 3862 specifies the kernarg 3863 memory size in bytes. It 3864 is recommended to provide 3865 a value as it may be used 3866 by CP to optimize making 3867 the kernarg memory 3868 visible to the kernel 3869 code. 3870 3871 127:96 4 bytes Reserved, must be 0. 3872 191:128 8 bytes KERNEL_CODE_ENTRY_BYTE_OFFSET Byte offset (possibly 3873 negative) from base 3874 address of kernel 3875 descriptor to kernel's 3876 entry point instruction 3877 which must be 256 byte 3878 aligned. 3879 351:272 20 Reserved, must be 0. 3880 bytes 3881 383:352 4 bytes COMPUTE_PGM_RSRC3 GFX6-GFX9 3882 Reserved, must be 0. 3883 GFX90A, GFX940 3884 Compute Shader (CS) 3885 program settings used by 3886 CP to set up 3887 ``COMPUTE_PGM_RSRC3`` 3888 configuration 3889 register. See 3890 :ref:`amdgpu-amdhsa-compute_pgm_rsrc3-gfx90a-table`. 3891 GFX10 3892 Compute Shader (CS) 3893 program settings used by 3894 CP to set up 3895 ``COMPUTE_PGM_RSRC3`` 3896 configuration 3897 register. See 3898 :ref:`amdgpu-amdhsa-compute_pgm_rsrc3-gfx10-table`. 3899 415:384 4 bytes COMPUTE_PGM_RSRC1 Compute Shader (CS) 3900 program settings used by 3901 CP to set up 3902 ``COMPUTE_PGM_RSRC1`` 3903 configuration 3904 register. See 3905 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 3906 447:416 4 bytes COMPUTE_PGM_RSRC2 Compute Shader (CS) 3907 program settings used by 3908 CP to set up 3909 ``COMPUTE_PGM_RSRC2`` 3910 configuration 3911 register. See 3912 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 3913 458:448 7 bits *See separate bits below.* Enable the setup of the 3914 SGPR user data registers 3915 (see 3916 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 3917 3918 The total number of SGPR 3919 user data registers 3920 requested must not exceed 3921 16 and match value in 3922 ``compute_pgm_rsrc2.user_sgpr.user_sgpr_count``. 3923 Any requests beyond 16 3924 will be ignored. 3925 >448 1 bit ENABLE_SGPR_PRIVATE_SEGMENT If the *Target Properties* 3926 _BUFFER column of 3927 :ref:`amdgpu-processor-table` 3928 specifies *Architected flat 3929 scratch* then not supported 3930 and must be 0, 3931 >449 1 bit ENABLE_SGPR_DISPATCH_PTR 3932 >450 1 bit ENABLE_SGPR_QUEUE_PTR 3933 >451 1 bit ENABLE_SGPR_KERNARG_SEGMENT_PTR 3934 >452 1 bit ENABLE_SGPR_DISPATCH_ID 3935 >453 1 bit ENABLE_SGPR_FLAT_SCRATCH_INIT If the *Target Properties* 3936 column of 3937 :ref:`amdgpu-processor-table` 3938 specifies *Architected flat 3939 scratch* then not supported 3940 and must be 0, 3941 >454 1 bit ENABLE_SGPR_PRIVATE_SEGMENT 3942 _SIZE 3943 457:455 3 bits Reserved, must be 0. 3944 458 1 bit ENABLE_WAVEFRONT_SIZE32 GFX6-GFX9 3945 Reserved, must be 0. 3946 GFX10 3947 - If 0 execute in 3948 wavefront size 64 mode. 3949 - If 1 execute in 3950 native wavefront size 3951 32 mode. 3952 463:459 1 bit Reserved, must be 0. 3953 464 1 bit RESERVED_464 Deprecated, must be 0. 3954 467:465 3 bits Reserved, must be 0. 3955 468 1 bit RESERVED_468 Deprecated, must be 0. 3956 469:471 3 bits Reserved, must be 0. 3957 511:472 5 bytes Reserved, must be 0. 3958 512 **Total size 64 bytes.** 3959 ======= ==================================================================== 3960 3961.. 3962 3963 .. table:: compute_pgm_rsrc1 for GFX6-GFX10 3964 :name: amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table 3965 3966 ======= ======= =============================== =========================================================================== 3967 Bits Size Field Name Description 3968 ======= ======= =============================== =========================================================================== 3969 5:0 6 bits GRANULATED_WORKITEM_VGPR_COUNT Number of vector register 3970 blocks used by each work-item; 3971 granularity is device 3972 specific: 3973 3974 GFX6-GFX9 3975 - vgprs_used 0..256 3976 - max(0, ceil(vgprs_used / 4) - 1) 3977 GFX90A, GFX940 3978 - vgprs_used 0..512 3979 - vgprs_used = align(arch_vgprs, 4) 3980 + acc_vgprs 3981 - max(0, ceil(vgprs_used / 8) - 1) 3982 GFX10 (wavefront size 64) 3983 - max_vgpr 1..256 3984 - max(0, ceil(vgprs_used / 4) - 1) 3985 GFX10 (wavefront size 32) 3986 - max_vgpr 1..256 3987 - max(0, ceil(vgprs_used / 8) - 1) 3988 3989 Where vgprs_used is defined 3990 as the highest VGPR number 3991 explicitly referenced plus 3992 one. 3993 3994 Used by CP to set up 3995 ``COMPUTE_PGM_RSRC1.VGPRS``. 3996 3997 The 3998 :ref:`amdgpu-assembler` 3999 calculates this 4000 automatically for the 4001 selected processor from 4002 values provided to the 4003 `.amdhsa_kernel` directive 4004 by the 4005 `.amdhsa_next_free_vgpr` 4006 nested directive (see 4007 :ref:`amdhsa-kernel-directives-table`). 4008 9:6 4 bits GRANULATED_WAVEFRONT_SGPR_COUNT Number of scalar register 4009 blocks used by a wavefront; 4010 granularity is device 4011 specific: 4012 4013 GFX6-GFX8 4014 - sgprs_used 0..112 4015 - max(0, ceil(sgprs_used / 8) - 1) 4016 GFX9 4017 - sgprs_used 0..112 4018 - 2 * max(0, ceil(sgprs_used / 16) - 1) 4019 GFX10 4020 Reserved, must be 0. 4021 (128 SGPRs always 4022 allocated.) 4023 4024 Where sgprs_used is 4025 defined as the highest 4026 SGPR number explicitly 4027 referenced plus one, plus 4028 a target specific number 4029 of additional special 4030 SGPRs for VCC, 4031 FLAT_SCRATCH (GFX7+) and 4032 XNACK_MASK (GFX8+), and 4033 any additional 4034 target specific 4035 limitations. It does not 4036 include the 16 SGPRs added 4037 if a trap handler is 4038 enabled. 4039 4040 The target specific 4041 limitations and special 4042 SGPR layout are defined in 4043 the hardware 4044 documentation, which can 4045 be found in the 4046 :ref:`amdgpu-processors` 4047 table. 4048 4049 Used by CP to set up 4050 ``COMPUTE_PGM_RSRC1.SGPRS``. 4051 4052 The 4053 :ref:`amdgpu-assembler` 4054 calculates this 4055 automatically for the 4056 selected processor from 4057 values provided to the 4058 `.amdhsa_kernel` directive 4059 by the 4060 `.amdhsa_next_free_sgpr` 4061 and `.amdhsa_reserve_*` 4062 nested directives (see 4063 :ref:`amdhsa-kernel-directives-table`). 4064 11:10 2 bits PRIORITY Must be 0. 4065 4066 Start executing wavefront 4067 at the specified priority. 4068 4069 CP is responsible for 4070 filling in 4071 ``COMPUTE_PGM_RSRC1.PRIORITY``. 4072 13:12 2 bits FLOAT_ROUND_MODE_32 Wavefront starts execution 4073 with specified rounding 4074 mode for single (32 4075 bit) floating point 4076 precision floating point 4077 operations. 4078 4079 Floating point rounding 4080 mode values are defined in 4081 :ref:`amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table`. 4082 4083 Used by CP to set up 4084 ``COMPUTE_PGM_RSRC1.FLOAT_MODE``. 4085 15:14 2 bits FLOAT_ROUND_MODE_16_64 Wavefront starts execution 4086 with specified rounding 4087 denorm mode for half/double (16 4088 and 64-bit) floating point 4089 precision floating point 4090 operations. 4091 4092 Floating point rounding 4093 mode values are defined in 4094 :ref:`amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table`. 4095 4096 Used by CP to set up 4097 ``COMPUTE_PGM_RSRC1.FLOAT_MODE``. 4098 17:16 2 bits FLOAT_DENORM_MODE_32 Wavefront starts execution 4099 with specified denorm mode 4100 for single (32 4101 bit) floating point 4102 precision floating point 4103 operations. 4104 4105 Floating point denorm mode 4106 values are defined in 4107 :ref:`amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table`. 4108 4109 Used by CP to set up 4110 ``COMPUTE_PGM_RSRC1.FLOAT_MODE``. 4111 19:18 2 bits FLOAT_DENORM_MODE_16_64 Wavefront starts execution 4112 with specified denorm mode 4113 for half/double (16 4114 and 64-bit) floating point 4115 precision floating point 4116 operations. 4117 4118 Floating point denorm mode 4119 values are defined in 4120 :ref:`amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table`. 4121 4122 Used by CP to set up 4123 ``COMPUTE_PGM_RSRC1.FLOAT_MODE``. 4124 20 1 bit PRIV Must be 0. 4125 4126 Start executing wavefront 4127 in privilege trap handler 4128 mode. 4129 4130 CP is responsible for 4131 filling in 4132 ``COMPUTE_PGM_RSRC1.PRIV``. 4133 21 1 bit ENABLE_DX10_CLAMP Wavefront starts execution 4134 with DX10 clamp mode 4135 enabled. Used by the vector 4136 ALU to force DX10 style 4137 treatment of NaN's (when 4138 set, clamp NaN to zero, 4139 otherwise pass NaN 4140 through). 4141 4142 Used by CP to set up 4143 ``COMPUTE_PGM_RSRC1.DX10_CLAMP``. 4144 22 1 bit DEBUG_MODE Must be 0. 4145 4146 Start executing wavefront 4147 in single step mode. 4148 4149 CP is responsible for 4150 filling in 4151 ``COMPUTE_PGM_RSRC1.DEBUG_MODE``. 4152 23 1 bit ENABLE_IEEE_MODE Wavefront starts execution 4153 with IEEE mode 4154 enabled. Floating point 4155 opcodes that support 4156 exception flag gathering 4157 will quiet and propagate 4158 signaling-NaN inputs per 4159 IEEE 754-2008. Min_dx10 and 4160 max_dx10 become IEEE 4161 754-2008 compliant due to 4162 signaling-NaN propagation 4163 and quieting. 4164 4165 Used by CP to set up 4166 ``COMPUTE_PGM_RSRC1.IEEE_MODE``. 4167 24 1 bit BULKY Must be 0. 4168 4169 Only one work-group allowed 4170 to execute on a compute 4171 unit. 4172 4173 CP is responsible for 4174 filling in 4175 ``COMPUTE_PGM_RSRC1.BULKY``. 4176 25 1 bit CDBG_USER Must be 0. 4177 4178 Flag that can be used to 4179 control debugging code. 4180 4181 CP is responsible for 4182 filling in 4183 ``COMPUTE_PGM_RSRC1.CDBG_USER``. 4184 26 1 bit FP16_OVFL GFX6-GFX8 4185 Reserved, must be 0. 4186 GFX9-GFX10 4187 Wavefront starts execution 4188 with specified fp16 overflow 4189 mode. 4190 4191 - If 0, fp16 overflow generates 4192 +/-INF values. 4193 - If 1, fp16 overflow that is the 4194 result of an +/-INF input value 4195 or divide by 0 produces a +/-INF, 4196 otherwise clamps computed 4197 overflow to +/-MAX_FP16 as 4198 appropriate. 4199 4200 Used by CP to set up 4201 ``COMPUTE_PGM_RSRC1.FP16_OVFL``. 4202 28:27 2 bits Reserved, must be 0. 4203 29 1 bit WGP_MODE GFX6-GFX9 4204 Reserved, must be 0. 4205 GFX10 4206 - If 0 execute work-groups in 4207 CU wavefront execution mode. 4208 - If 1 execute work-groups on 4209 in WGP wavefront execution mode. 4210 4211 See :ref:`amdgpu-amdhsa-memory-model`. 4212 4213 Used by CP to set up 4214 ``COMPUTE_PGM_RSRC1.WGP_MODE``. 4215 30 1 bit MEM_ORDERED GFX6-GFX9 4216 Reserved, must be 0. 4217 GFX10 4218 Controls the behavior of the 4219 s_waitcnt's vmcnt and vscnt 4220 counters. 4221 4222 - If 0 vmcnt reports completion 4223 of load and atomic with return 4224 out of order with sample 4225 instructions, and the vscnt 4226 reports the completion of 4227 store and atomic without 4228 return in order. 4229 - If 1 vmcnt reports completion 4230 of load, atomic with return 4231 and sample instructions in 4232 order, and the vscnt reports 4233 the completion of store and 4234 atomic without return in order. 4235 4236 Used by CP to set up 4237 ``COMPUTE_PGM_RSRC1.MEM_ORDERED``. 4238 31 1 bit FWD_PROGRESS GFX6-GFX9 4239 Reserved, must be 0. 4240 GFX10 4241 - If 0 execute SIMD wavefronts 4242 using oldest first policy. 4243 - If 1 execute SIMD wavefronts to 4244 ensure wavefronts will make some 4245 forward progress. 4246 4247 Used by CP to set up 4248 ``COMPUTE_PGM_RSRC1.FWD_PROGRESS``. 4249 32 **Total size 4 bytes** 4250 ======= =================================================================================================================== 4251 4252.. 4253 4254 .. table:: compute_pgm_rsrc2 for GFX6-GFX10 4255 :name: amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table 4256 4257 ======= ======= =============================== =========================================================================== 4258 Bits Size Field Name Description 4259 ======= ======= =============================== =========================================================================== 4260 0 1 bit ENABLE_PRIVATE_SEGMENT * Enable the setup of the 4261 private segment. 4262 * If the *Target Properties* 4263 column of 4264 :ref:`amdgpu-processor-table` 4265 does not specify 4266 *Architected flat 4267 scratch* then enable the 4268 setup of the SGPR 4269 wavefront scratch offset 4270 system register (see 4271 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4272 * If the *Target Properties* 4273 column of 4274 :ref:`amdgpu-processor-table` 4275 specifies *Architected 4276 flat scratch* then enable 4277 the setup of the 4278 FLAT_SCRATCH register 4279 pair (see 4280 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4281 4282 Used by CP to set up 4283 ``COMPUTE_PGM_RSRC2.SCRATCH_EN``. 4284 5:1 5 bits USER_SGPR_COUNT The total number of SGPR 4285 user data 4286 registers requested. This 4287 number must be greater than 4288 or equal to the number of user 4289 data registers enabled. 4290 4291 Used by CP to set up 4292 ``COMPUTE_PGM_RSRC2.USER_SGPR``. 4293 6 1 bit ENABLE_TRAP_HANDLER Must be 0. 4294 4295 This bit represents 4296 ``COMPUTE_PGM_RSRC2.TRAP_PRESENT``, 4297 which is set by the CP if 4298 the runtime has installed a 4299 trap handler. 4300 7 1 bit ENABLE_SGPR_WORKGROUP_ID_X Enable the setup of the 4301 system SGPR register for 4302 the work-group id in the X 4303 dimension (see 4304 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4305 4306 Used by CP to set up 4307 ``COMPUTE_PGM_RSRC2.TGID_X_EN``. 4308 8 1 bit ENABLE_SGPR_WORKGROUP_ID_Y Enable the setup of the 4309 system SGPR register for 4310 the work-group id in the Y 4311 dimension (see 4312 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4313 4314 Used by CP to set up 4315 ``COMPUTE_PGM_RSRC2.TGID_Y_EN``. 4316 9 1 bit ENABLE_SGPR_WORKGROUP_ID_Z Enable the setup of the 4317 system SGPR register for 4318 the work-group id in the Z 4319 dimension (see 4320 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4321 4322 Used by CP to set up 4323 ``COMPUTE_PGM_RSRC2.TGID_Z_EN``. 4324 10 1 bit ENABLE_SGPR_WORKGROUP_INFO Enable the setup of the 4325 system SGPR register for 4326 work-group information (see 4327 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4328 4329 Used by CP to set up 4330 ``COMPUTE_PGM_RSRC2.TGID_SIZE_EN``. 4331 12:11 2 bits ENABLE_VGPR_WORKITEM_ID Enable the setup of the 4332 VGPR system registers used 4333 for the work-item ID. 4334 :ref:`amdgpu-amdhsa-system-vgpr-work-item-id-enumeration-values-table` 4335 defines the values. 4336 4337 Used by CP to set up 4338 ``COMPUTE_PGM_RSRC2.TIDIG_CMP_CNT``. 4339 13 1 bit ENABLE_EXCEPTION_ADDRESS_WATCH Must be 0. 4340 4341 Wavefront starts execution 4342 with address watch 4343 exceptions enabled which 4344 are generated when L1 has 4345 witnessed a thread access 4346 an *address of 4347 interest*. 4348 4349 CP is responsible for 4350 filling in the address 4351 watch bit in 4352 ``COMPUTE_PGM_RSRC2.EXCP_EN_MSB`` 4353 according to what the 4354 runtime requests. 4355 14 1 bit ENABLE_EXCEPTION_MEMORY Must be 0. 4356 4357 Wavefront starts execution 4358 with memory violation 4359 exceptions exceptions 4360 enabled which are generated 4361 when a memory violation has 4362 occurred for this wavefront from 4363 L1 or LDS 4364 (write-to-read-only-memory, 4365 mis-aligned atomic, LDS 4366 address out of range, 4367 illegal address, etc.). 4368 4369 CP sets the memory 4370 violation bit in 4371 ``COMPUTE_PGM_RSRC2.EXCP_EN_MSB`` 4372 according to what the 4373 runtime requests. 4374 23:15 9 bits GRANULATED_LDS_SIZE Must be 0. 4375 4376 CP uses the rounded value 4377 from the dispatch packet, 4378 not this value, as the 4379 dispatch may contain 4380 dynamically allocated group 4381 segment memory. CP writes 4382 directly to 4383 ``COMPUTE_PGM_RSRC2.LDS_SIZE``. 4384 4385 Amount of group segment 4386 (LDS) to allocate for each 4387 work-group. Granularity is 4388 device specific: 4389 4390 GFX6 4391 roundup(lds-size / (64 * 4)) 4392 GFX7-GFX10 4393 roundup(lds-size / (128 * 4)) 4394 4395 24 1 bit ENABLE_EXCEPTION_IEEE_754_FP Wavefront starts execution 4396 _INVALID_OPERATION with specified exceptions 4397 enabled. 4398 4399 Used by CP to set up 4400 ``COMPUTE_PGM_RSRC2.EXCP_EN`` 4401 (set from bits 0..6). 4402 4403 IEEE 754 FP Invalid 4404 Operation 4405 25 1 bit ENABLE_EXCEPTION_FP_DENORMAL FP Denormal one or more 4406 _SOURCE input operands is a 4407 denormal number 4408 26 1 bit ENABLE_EXCEPTION_IEEE_754_FP IEEE 754 FP Division by 4409 _DIVISION_BY_ZERO Zero 4410 27 1 bit ENABLE_EXCEPTION_IEEE_754_FP IEEE 754 FP FP Overflow 4411 _OVERFLOW 4412 28 1 bit ENABLE_EXCEPTION_IEEE_754_FP IEEE 754 FP Underflow 4413 _UNDERFLOW 4414 29 1 bit ENABLE_EXCEPTION_IEEE_754_FP IEEE 754 FP Inexact 4415 _INEXACT 4416 30 1 bit ENABLE_EXCEPTION_INT_DIVIDE_BY Integer Division by Zero 4417 _ZERO (rcp_iflag_f32 instruction 4418 only) 4419 31 1 bit Reserved, must be 0. 4420 32 **Total size 4 bytes.** 4421 ======= =================================================================================================================== 4422 4423.. 4424 4425 .. table:: compute_pgm_rsrc3 for GFX90A, GFX940 4426 :name: amdgpu-amdhsa-compute_pgm_rsrc3-gfx90a-table 4427 4428 ======= ======= =============================== =========================================================================== 4429 Bits Size Field Name Description 4430 ======= ======= =============================== =========================================================================== 4431 5:0 6 bits ACCUM_OFFSET Offset of a first AccVGPR in the unified register file. Granularity 4. 4432 Value 0-63. 0 - accum-offset = 4, 1 - accum-offset = 8, ..., 4433 63 - accum-offset = 256. 4434 6:15 10 Reserved, must be 0. 4435 bits 4436 16 1 bit TG_SPLIT - If 0 the waves of a work-group are 4437 launched in the same CU. 4438 - If 1 the waves of a work-group can be 4439 launched in different CUs. The waves 4440 cannot use S_BARRIER or LDS. 4441 17:31 15 Reserved, must be 0. 4442 bits 4443 32 **Total size 4 bytes.** 4444 ======= =================================================================================================================== 4445 4446.. 4447 4448 .. table:: compute_pgm_rsrc3 for GFX10 4449 :name: amdgpu-amdhsa-compute_pgm_rsrc3-gfx10-table 4450 4451 ======= ======= =============================== =========================================================================== 4452 Bits Size Field Name Description 4453 ======= ======= =============================== =========================================================================== 4454 3:0 4 bits SHARED_VGPR_COUNT Number of shared VGPRs for wavefront size 64. Granularity 8. Value 0-120. 4455 compute_pgm_rsrc1.vgprs + shared_vgpr_cnt cannot exceed 64. 4456 31:4 28 Reserved, must be 0. 4457 bits 4458 32 **Total size 4 bytes.** 4459 ======= =================================================================================================================== 4460 4461.. 4462 4463 .. table:: Floating Point Rounding Mode Enumeration Values 4464 :name: amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table 4465 4466 ====================================== ===== ============================== 4467 Enumeration Name Value Description 4468 ====================================== ===== ============================== 4469 FLOAT_ROUND_MODE_NEAR_EVEN 0 Round Ties To Even 4470 FLOAT_ROUND_MODE_PLUS_INFINITY 1 Round Toward +infinity 4471 FLOAT_ROUND_MODE_MINUS_INFINITY 2 Round Toward -infinity 4472 FLOAT_ROUND_MODE_ZERO 3 Round Toward 0 4473 ====================================== ===== ============================== 4474 4475.. 4476 4477 .. table:: Floating Point Denorm Mode Enumeration Values 4478 :name: amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table 4479 4480 ====================================== ===== ============================== 4481 Enumeration Name Value Description 4482 ====================================== ===== ============================== 4483 FLOAT_DENORM_MODE_FLUSH_SRC_DST 0 Flush Source and Destination 4484 Denorms 4485 FLOAT_DENORM_MODE_FLUSH_DST 1 Flush Output Denorms 4486 FLOAT_DENORM_MODE_FLUSH_SRC 2 Flush Source Denorms 4487 FLOAT_DENORM_MODE_FLUSH_NONE 3 No Flush 4488 ====================================== ===== ============================== 4489 4490.. 4491 4492 .. table:: System VGPR Work-Item ID Enumeration Values 4493 :name: amdgpu-amdhsa-system-vgpr-work-item-id-enumeration-values-table 4494 4495 ======================================== ===== ============================ 4496 Enumeration Name Value Description 4497 ======================================== ===== ============================ 4498 SYSTEM_VGPR_WORKITEM_ID_X 0 Set work-item X dimension 4499 ID. 4500 SYSTEM_VGPR_WORKITEM_ID_X_Y 1 Set work-item X and Y 4501 dimensions ID. 4502 SYSTEM_VGPR_WORKITEM_ID_X_Y_Z 2 Set work-item X, Y and Z 4503 dimensions ID. 4504 SYSTEM_VGPR_WORKITEM_ID_UNDEFINED 3 Undefined. 4505 ======================================== ===== ============================ 4506 4507.. _amdgpu-amdhsa-initial-kernel-execution-state: 4508 4509Initial Kernel Execution State 4510~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 4511 4512This section defines the register state that will be set up by the packet 4513processor prior to the start of execution of every wavefront. This is limited by 4514the constraints of the hardware controllers of CP/ADC/SPI. 4515 4516The order of the SGPR registers is defined, but the compiler can specify which 4517ones are actually setup in the kernel descriptor using the ``enable_sgpr_*`` bit 4518fields (see :ref:`amdgpu-amdhsa-kernel-descriptor`). The register numbers used 4519for enabled registers are dense starting at SGPR0: the first enabled register is 4520SGPR0, the next enabled register is SGPR1 etc.; disabled registers do not have 4521an SGPR number. 4522 4523The initial SGPRs comprise up to 16 User SRGPs that are set by CP and apply to 4524all wavefronts of the grid. It is possible to specify more than 16 User SGPRs 4525using the ``enable_sgpr_*`` bit fields, in which case only the first 16 are 4526actually initialized. These are then immediately followed by the System SGPRs 4527that are set up by ADC/SPI and can have different values for each wavefront of 4528the grid dispatch. 4529 4530SGPR register initial state is defined in 4531:ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 4532 4533 .. table:: SGPR Register Set Up Order 4534 :name: amdgpu-amdhsa-sgpr-register-set-up-order-table 4535 4536 ========== ========================== ====== ============================== 4537 SGPR Order Name Number Description 4538 (kernel descriptor enable of 4539 field) SGPRs 4540 ========== ========================== ====== ============================== 4541 First Private Segment Buffer 4 See 4542 (enable_sgpr_private :ref:`amdgpu-amdhsa-kernel-prolog-private-segment-buffer`. 4543 _segment_buffer) 4544 then Dispatch Ptr 2 64-bit address of AQL dispatch 4545 (enable_sgpr_dispatch_ptr) packet for kernel dispatch 4546 actually executing. 4547 then Queue Ptr 2 64-bit address of amd_queue_t 4548 (enable_sgpr_queue_ptr) object for AQL queue on which 4549 the dispatch packet was 4550 queued. 4551 then Kernarg Segment Ptr 2 64-bit address of Kernarg 4552 (enable_sgpr_kernarg segment. This is directly 4553 _segment_ptr) copied from the 4554 kernarg_address in the kernel 4555 dispatch packet. 4556 4557 Having CP load it once avoids 4558 loading it at the beginning of 4559 every wavefront. 4560 then Dispatch Id 2 64-bit Dispatch ID of the 4561 (enable_sgpr_dispatch_id) dispatch packet being 4562 executed. 4563 then Flat Scratch Init 2 See 4564 (enable_sgpr_flat_scratch :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`. 4565 _init) 4566 then Private Segment Size 1 The 32-bit byte size of a 4567 (enable_sgpr_private single work-item's memory 4568 _segment_size) allocation. This is the 4569 value from the kernel 4570 dispatch packet Private 4571 Segment Byte Size rounded up 4572 by CP to a multiple of 4573 DWORD. 4574 4575 Having CP load it once avoids 4576 loading it at the beginning of 4577 every wavefront. 4578 4579 This is not used for 4580 GFX7-GFX8 since it is the same 4581 value as the second SGPR of 4582 Flat Scratch Init. However, it 4583 may be needed for GFX9-GFX10 which 4584 changes the meaning of the 4585 Flat Scratch Init value. 4586 then Work-Group Id X 1 32-bit work-group id in X 4587 (enable_sgpr_workgroup_id dimension of grid for 4588 _X) wavefront. 4589 then Work-Group Id Y 1 32-bit work-group id in Y 4590 (enable_sgpr_workgroup_id dimension of grid for 4591 _Y) wavefront. 4592 then Work-Group Id Z 1 32-bit work-group id in Z 4593 (enable_sgpr_workgroup_id dimension of grid for 4594 _Z) wavefront. 4595 then Work-Group Info 1 {first_wavefront, 14'b0000, 4596 (enable_sgpr_workgroup ordered_append_term[10:0], 4597 _info) threadgroup_size_in_wavefronts[5:0]} 4598 then Scratch Wavefront Offset 1 See 4599 (enable_sgpr_private :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`. 4600 _segment_wavefront_offset) and 4601 :ref:`amdgpu-amdhsa-kernel-prolog-private-segment-buffer`. 4602 ========== ========================== ====== ============================== 4603 4604The order of the VGPR registers is defined, but the compiler can specify which 4605ones are actually setup in the kernel descriptor using the ``enable_vgpr*`` bit 4606fields (see :ref:`amdgpu-amdhsa-kernel-descriptor`). The register numbers used 4607for enabled registers are dense starting at VGPR0: the first enabled register is 4608VGPR0, the next enabled register is VGPR1 etc.; disabled registers do not have a 4609VGPR number. 4610 4611There are different methods used for the VGPR initial state: 4612 4613* Unless the *Target Properties* column of :ref:`amdgpu-processor-table` 4614 specifies otherwise, a separate VGPR register is used per work-item ID. The 4615 VGPR register initial state for this method is defined in 4616 :ref:`amdgpu-amdhsa-vgpr-register-set-up-order-for-unpacked-work-item-id-method-table`. 4617* If *Target Properties* column of :ref:`amdgpu-processor-table` 4618 specifies *Packed work-item IDs*, the initial value of VGPR0 register is used 4619 for all work-item IDs. The register layout for this method is defined in 4620 :ref:`amdgpu-amdhsa-register-layout-for-packed-work-item-id-method-table`. 4621 4622 .. table:: VGPR Register Set Up Order for Unpacked Work-Item ID Method 4623 :name: amdgpu-amdhsa-vgpr-register-set-up-order-for-unpacked-work-item-id-method-table 4624 4625 ========== ========================== ====== ============================== 4626 VGPR Order Name Number Description 4627 (kernel descriptor enable of 4628 field) VGPRs 4629 ========== ========================== ====== ============================== 4630 First Work-Item Id X 1 32-bit work-item id in X 4631 (Always initialized) dimension of work-group for 4632 wavefront lane. 4633 then Work-Item Id Y 1 32-bit work-item id in Y 4634 (enable_vgpr_workitem_id dimension of work-group for 4635 > 0) wavefront lane. 4636 then Work-Item Id Z 1 32-bit work-item id in Z 4637 (enable_vgpr_workitem_id dimension of work-group for 4638 > 1) wavefront lane. 4639 ========== ========================== ====== ============================== 4640 4641.. 4642 4643 .. table:: Register Layout for Packed Work-Item ID Method 4644 :name: amdgpu-amdhsa-register-layout-for-packed-work-item-id-method-table 4645 4646 ======= ======= ================ ========================================= 4647 Bits Size Field Name Description 4648 ======= ======= ================ ========================================= 4649 0:9 10 bits Work-Item Id X Work-item id in X 4650 dimension of work-group for 4651 wavefront lane. 4652 4653 Always initialized. 4654 4655 10:19 10 bits Work-Item Id Y Work-item id in Y 4656 dimension of work-group for 4657 wavefront lane. 4658 4659 Initialized if enable_vgpr_workitem_id > 4660 0, otherwise set to 0. 4661 20:29 10 bits Work-Item Id Z Work-item id in Z 4662 dimension of work-group for 4663 wavefront lane. 4664 4665 Initialized if enable_vgpr_workitem_id > 4666 1, otherwise set to 0. 4667 30:31 2 bits Reserved, set to 0. 4668 ======= ======= ================ ========================================= 4669 4670The setting of registers is done by GPU CP/ADC/SPI hardware as follows: 4671 46721. SGPRs before the Work-Group Ids are set by CP using the 16 User Data 4673 registers. 46742. Work-group Id registers X, Y, Z are set by ADC which supports any 4675 combination including none. 46763. Scratch Wavefront Offset is set by SPI in a per wavefront basis which is why 4677 its value cannot be included with the flat scratch init value which is per 4678 queue (see :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`). 46794. The VGPRs are set by SPI which only supports specifying either (X), (X, Y) 4680 or (X, Y, Z). 46815. Flat Scratch register pair initialization is described in 4682 :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`. 4683 4684The global segment can be accessed either using buffer instructions (GFX6 which 4685has V# 64-bit address support), flat instructions (GFX7-GFX10), or global 4686instructions (GFX9-GFX10). 4687 4688If buffer operations are used, then the compiler can generate a V# with the 4689following properties: 4690 4691* base address of 0 4692* no swizzle 4693* ATC: 1 if IOMMU present (such as APU) 4694* ptr64: 1 4695* MTYPE set to support memory coherence that matches the runtime (such as CC for 4696 APU and NC for dGPU). 4697 4698.. _amdgpu-amdhsa-kernel-prolog: 4699 4700Kernel Prolog 4701~~~~~~~~~~~~~ 4702 4703The compiler performs initialization in the kernel prologue depending on the 4704target and information about things like stack usage in the kernel and called 4705functions. Some of this initialization requires the compiler to request certain 4706User and System SGPRs be present in the 4707:ref:`amdgpu-amdhsa-initial-kernel-execution-state` via the 4708:ref:`amdgpu-amdhsa-kernel-descriptor`. 4709 4710.. _amdgpu-amdhsa-kernel-prolog-cfi: 4711 4712CFI 4713+++ 4714 47151. The CFI return address is undefined. 4716 47172. The CFI CFA is defined using an expression which evaluates to a location 4718 description that comprises one memory location description for the 4719 ``DW_ASPACE_AMDGPU_private_lane`` address space address ``0``. 4720 4721.. _amdgpu-amdhsa-kernel-prolog-m0: 4722 4723M0 4724++ 4725 4726GFX6-GFX8 4727 The M0 register must be initialized with a value at least the total LDS size 4728 if the kernel may access LDS via DS or flat operations. Total LDS size is 4729 available in dispatch packet. For M0, it is also possible to use maximum 4730 possible value of LDS for given target (0x7FFF for GFX6 and 0xFFFF for 4731 GFX7-GFX8). 4732GFX9-GFX10 4733 The M0 register is not used for range checking LDS accesses and so does not 4734 need to be initialized in the prolog. 4735 4736.. _amdgpu-amdhsa-kernel-prolog-stack-pointer: 4737 4738Stack Pointer 4739+++++++++++++ 4740 4741If the kernel has function calls it must set up the ABI stack pointer described 4742in :ref:`amdgpu-amdhsa-function-call-convention-non-kernel-functions` by setting 4743SGPR32 to the unswizzled scratch offset of the address past the last local 4744allocation. 4745 4746.. _amdgpu-amdhsa-kernel-prolog-frame-pointer: 4747 4748Frame Pointer 4749+++++++++++++ 4750 4751If the kernel needs a frame pointer for the reasons defined in 4752``SIFrameLowering`` then SGPR33 is used and is always set to ``0`` in the 4753kernel prolog. If a frame pointer is not required then all uses of the frame 4754pointer are replaced with immediate ``0`` offsets. 4755 4756.. _amdgpu-amdhsa-kernel-prolog-flat-scratch: 4757 4758Flat Scratch 4759++++++++++++ 4760 4761There are different methods used for initializing flat scratch: 4762 4763* If the *Target Properties* column of :ref:`amdgpu-processor-table` 4764 specifies *Does not support generic address space*: 4765 4766 Flat scratch is not supported and there is no flat scratch register pair. 4767 4768* If the *Target Properties* column of :ref:`amdgpu-processor-table` 4769 specifies *Offset flat scratch*: 4770 4771 If the kernel or any function it calls may use flat operations to access 4772 scratch memory, the prolog code must set up the FLAT_SCRATCH register pair 4773 (FLAT_SCRATCH_LO/FLAT_SCRATCH_HI). Initialization uses Flat Scratch Init and 4774 Scratch Wavefront Offset SGPR registers (see 4775 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`): 4776 4777 1. The low word of Flat Scratch Init is the 32-bit byte offset from 4778 ``SH_HIDDEN_PRIVATE_BASE_VIMID`` to the base of scratch backing memory 4779 being managed by SPI for the queue executing the kernel dispatch. This is 4780 the same value used in the Scratch Segment Buffer V# base address. 4781 4782 CP obtains this from the runtime. (The Scratch Segment Buffer base address 4783 is ``SH_HIDDEN_PRIVATE_BASE_VIMID`` plus this offset.) 4784 4785 The prolog must add the value of Scratch Wavefront Offset to get the 4786 wavefront's byte scratch backing memory offset from 4787 ``SH_HIDDEN_PRIVATE_BASE_VIMID``. 4788 4789 The Scratch Wavefront Offset must also be used as an offset with Private 4790 segment address when using the Scratch Segment Buffer. 4791 4792 Since FLAT_SCRATCH_LO is in units of 256 bytes, the offset must be right 4793 shifted by 8 before moving into FLAT_SCRATCH_HI. 4794 4795 FLAT_SCRATCH_HI corresponds to SGPRn-4 on GFX7, and SGPRn-6 on GFX8 (where 4796 SGPRn is the highest numbered SGPR allocated to the wavefront). 4797 FLAT_SCRATCH_HI is multiplied by 256 (as it is in units of 256 bytes) and 4798 added to ``SH_HIDDEN_PRIVATE_BASE_VIMID`` to calculate the per wavefront 4799 FLAT SCRATCH BASE in flat memory instructions that access the scratch 4800 aperture. 4801 2. The second word of Flat Scratch Init is 32-bit byte size of a single 4802 work-items scratch memory usage. 4803 4804 CP obtains this from the runtime, and it is always a multiple of DWORD. CP 4805 checks that the value in the kernel dispatch packet Private Segment Byte 4806 Size is not larger and requests the runtime to increase the queue's scratch 4807 size if necessary. 4808 4809 CP directly loads from the kernel dispatch packet Private Segment Byte Size 4810 field and rounds up to a multiple of DWORD. Having CP load it once avoids 4811 loading it at the beginning of every wavefront. 4812 4813 The kernel prolog code must move it to FLAT_SCRATCH_LO which is SGPRn-3 on 4814 GFX7 and SGPRn-5 on GFX8. FLAT_SCRATCH_LO is used as the FLAT SCRATCH SIZE 4815 in flat memory instructions. 4816 4817* If the *Target Properties* column of :ref:`amdgpu-processor-table` 4818 specifies *Absolute flat scratch*: 4819 4820 If the kernel or any function it calls may use flat operations to access 4821 scratch memory, the prolog code must set up the FLAT_SCRATCH register pair 4822 (FLAT_SCRATCH_LO/FLAT_SCRATCH_HI which are in SGPRn-4/SGPRn-3). Initialization 4823 uses Flat Scratch Init and Scratch Wavefront Offset SGPR registers (see 4824 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`): 4825 4826 The Flat Scratch Init is the 64-bit address of the base of scratch backing 4827 memory being managed by SPI for the queue executing the kernel dispatch. 4828 4829 CP obtains this from the runtime. 4830 4831 The kernel prolog must add the value of the wave's Scratch Wavefront Offset 4832 and move the result as a 64-bit value to the FLAT_SCRATCH SGPR register pair 4833 which is SGPRn-6 and SGPRn-5. It is used as the FLAT SCRATCH BASE in flat 4834 memory instructions. 4835 4836 The Scratch Wavefront Offset must also be used as an offset with Private 4837 segment address when using the Scratch Segment Buffer (see 4838 :ref:`amdgpu-amdhsa-kernel-prolog-private-segment-buffer`). 4839 4840* If the *Target Properties* column of :ref:`amdgpu-processor-table` 4841 specifies *Architected flat scratch*: 4842 4843 If ENABLE_PRIVATE_SEGMENT is enabled in 4844 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table` then the FLAT_SCRATCH 4845 register pair will be initialized to the 64-bit address of the base of scratch 4846 backing memory being managed by SPI for the queue executing the kernel 4847 dispatch plus the value of the wave's Scratch Wavefront Offset for use as the 4848 flat scratch base in flat memory instructions. 4849 4850.. _amdgpu-amdhsa-kernel-prolog-private-segment-buffer: 4851 4852Private Segment Buffer 4853++++++++++++++++++++++ 4854 4855If the *Target Properties* column of :ref:`amdgpu-processor-table` specifies 4856*Architected flat scratch* then a Private Segment Buffer is not supported. 4857Instead the flat SCRATCH instructions are used. 4858 4859Otherwise, Private Segment Buffer SGPR register is used to initialize 4 SGPRs 4860that are used as a V# to access scratch. CP uses the value provided by the 4861runtime. It is used, together with Scratch Wavefront Offset as an offset, to 4862access the private memory space using a segment address. See 4863:ref:`amdgpu-amdhsa-initial-kernel-execution-state`. 4864 4865The scratch V# is a four-aligned SGPR and always selected for the kernel as 4866follows: 4867 4868 - If it is known during instruction selection that there is stack usage, 4869 SGPR0-3 is reserved for use as the scratch V#. Stack usage is assumed if 4870 optimizations are disabled (``-O0``), if stack objects already exist (for 4871 locals, etc.), or if there are any function calls. 4872 4873 - Otherwise, four high numbered SGPRs beginning at a four-aligned SGPR index 4874 are reserved for the tentative scratch V#. These will be used if it is 4875 determined that spilling is needed. 4876 4877 - If no use is made of the tentative scratch V#, then it is unreserved, 4878 and the register count is determined ignoring it. 4879 - If use is made of the tentative scratch V#, then its register numbers 4880 are shifted to the first four-aligned SGPR index after the highest one 4881 allocated by the register allocator, and all uses are updated. The 4882 register count includes them in the shifted location. 4883 - In either case, if the processor has the SGPR allocation bug, the 4884 tentative allocation is not shifted or unreserved in order to ensure 4885 the register count is higher to workaround the bug. 4886 4887 .. note:: 4888 4889 This approach of using a tentative scratch V# and shifting the register 4890 numbers if used avoids having to perform register allocation a second 4891 time if the tentative V# is eliminated. This is more efficient and 4892 avoids the problem that the second register allocation may perform 4893 spilling which will fail as there is no longer a scratch V#. 4894 4895When the kernel prolog code is being emitted it is known whether the scratch V# 4896described above is actually used. If it is, the prolog code must set it up by 4897copying the Private Segment Buffer to the scratch V# registers and then adding 4898the Private Segment Wavefront Offset to the queue base address in the V#. The 4899result is a V# with a base address pointing to the beginning of the wavefront 4900scratch backing memory. 4901 4902The Private Segment Buffer is always requested, but the Private Segment 4903Wavefront Offset is only requested if it is used (see 4904:ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4905 4906.. _amdgpu-amdhsa-memory-model: 4907 4908Memory Model 4909~~~~~~~~~~~~ 4910 4911This section describes the mapping of the LLVM memory model onto AMDGPU machine 4912code (see :ref:`memmodel`). 4913 4914The AMDGPU backend supports the memory synchronization scopes specified in 4915:ref:`amdgpu-memory-scopes`. 4916 4917The code sequences used to implement the memory model specify the order of 4918instructions that a single thread must execute. The ``s_waitcnt`` and cache 4919management instructions such as ``buffer_wbinvl1_vol`` are defined with respect 4920to other memory instructions executed by the same thread. This allows them to be 4921moved earlier or later which can allow them to be combined with other instances 4922of the same instruction, or hoisted/sunk out of loops to improve performance. 4923Only the instructions related to the memory model are given; additional 4924``s_waitcnt`` instructions are required to ensure registers are defined before 4925being used. These may be able to be combined with the memory model ``s_waitcnt`` 4926instructions as described above. 4927 4928The AMDGPU backend supports the following memory models: 4929 4930 HSA Memory Model [HSA]_ 4931 The HSA memory model uses a single happens-before relation for all address 4932 spaces (see :ref:`amdgpu-address-spaces`). 4933 OpenCL Memory Model [OpenCL]_ 4934 The OpenCL memory model which has separate happens-before relations for the 4935 global and local address spaces. Only a fence specifying both global and 4936 local address space, and seq_cst instructions join the relationships. Since 4937 the LLVM ``memfence`` instruction does not allow an address space to be 4938 specified the OpenCL fence has to conservatively assume both local and 4939 global address space was specified. However, optimizations can often be 4940 done to eliminate the additional ``s_waitcnt`` instructions when there are 4941 no intervening memory instructions which access the corresponding address 4942 space. The code sequences in the table indicate what can be omitted for the 4943 OpenCL memory. The target triple environment is used to determine if the 4944 source language is OpenCL (see :ref:`amdgpu-opencl`). 4945 4946``ds/flat_load/store/atomic`` instructions to local memory are termed LDS 4947operations. 4948 4949``buffer/global/flat_load/store/atomic`` instructions to global memory are 4950termed vector memory operations. 4951 4952Private address space uses ``buffer_load/store`` using the scratch V# 4953(GFX6-GFX8), or ``scratch_load/store`` (GFX9-GFX10). Since only a single thread 4954is accessing the memory, atomic memory orderings are not meaningful, and all 4955accesses are treated as non-atomic. 4956 4957Constant address space uses ``buffer/global_load`` instructions (or equivalent 4958scalar memory instructions). Since the constant address space contents do not 4959change during the execution of a kernel dispatch it is not legal to perform 4960stores, and atomic memory orderings are not meaningful, and all accesses are 4961treated as non-atomic. 4962 4963A memory synchronization scope wider than work-group is not meaningful for the 4964group (LDS) address space and is treated as work-group. 4965 4966The memory model does not support the region address space which is treated as 4967non-atomic. 4968 4969Acquire memory ordering is not meaningful on store atomic instructions and is 4970treated as non-atomic. 4971 4972Release memory ordering is not meaningful on load atomic instructions and is 4973treated a non-atomic. 4974 4975Acquire-release memory ordering is not meaningful on load or store atomic 4976instructions and is treated as acquire and release respectively. 4977 4978The memory order also adds the single thread optimization constraints defined in 4979table 4980:ref:`amdgpu-amdhsa-memory-model-single-thread-optimization-constraints-table`. 4981 4982 .. table:: AMDHSA Memory Model Single Thread Optimization Constraints 4983 :name: amdgpu-amdhsa-memory-model-single-thread-optimization-constraints-table 4984 4985 ============ ============================================================== 4986 LLVM Memory Optimization Constraints 4987 Ordering 4988 ============ ============================================================== 4989 unordered *none* 4990 monotonic *none* 4991 acquire - If a load atomic/atomicrmw then no following load/load 4992 atomic/store/store atomic/atomicrmw/fence instruction can be 4993 moved before the acquire. 4994 - If a fence then same as load atomic, plus no preceding 4995 associated fence-paired-atomic can be moved after the fence. 4996 release - If a store atomic/atomicrmw then no preceding load/load 4997 atomic/store/store atomic/atomicrmw/fence instruction can be 4998 moved after the release. 4999 - If a fence then same as store atomic, plus no following 5000 associated fence-paired-atomic can be moved before the 5001 fence. 5002 acq_rel Same constraints as both acquire and release. 5003 seq_cst - If a load atomic then same constraints as acquire, plus no 5004 preceding sequentially consistent load atomic/store 5005 atomic/atomicrmw/fence instruction can be moved after the 5006 seq_cst. 5007 - If a store atomic then the same constraints as release, plus 5008 no following sequentially consistent load atomic/store 5009 atomic/atomicrmw/fence instruction can be moved before the 5010 seq_cst. 5011 - If an atomicrmw/fence then same constraints as acq_rel. 5012 ============ ============================================================== 5013 5014The code sequences used to implement the memory model are defined in the 5015following sections: 5016 5017* :ref:`amdgpu-amdhsa-memory-model-gfx6-gfx9` 5018* :ref:`amdgpu-amdhsa-memory-model-gfx90a` 5019* :ref:`amdgpu-amdhsa-memory-model-gfx10` 5020 5021.. _amdgpu-amdhsa-memory-model-gfx6-gfx9: 5022 5023Memory Model GFX6-GFX9 5024++++++++++++++++++++++ 5025 5026For GFX6-GFX9: 5027 5028* Each agent has multiple shader arrays (SA). 5029* Each SA has multiple compute units (CU). 5030* Each CU has multiple SIMDs that execute wavefronts. 5031* The wavefronts for a single work-group are executed in the same CU but may be 5032 executed by different SIMDs. 5033* Each CU has a single LDS memory shared by the wavefronts of the work-groups 5034 executing on it. 5035* All LDS operations of a CU are performed as wavefront wide operations in a 5036 global order and involve no caching. Completion is reported to a wavefront in 5037 execution order. 5038* The LDS memory has multiple request queues shared by the SIMDs of a 5039 CU. Therefore, the LDS operations performed by different wavefronts of a 5040 work-group can be reordered relative to each other, which can result in 5041 reordering the visibility of vector memory operations with respect to LDS 5042 operations of other wavefronts in the same work-group. A ``s_waitcnt 5043 lgkmcnt(0)`` is required to ensure synchronization between LDS operations and 5044 vector memory operations between wavefronts of a work-group, but not between 5045 operations performed by the same wavefront. 5046* The vector memory operations are performed as wavefront wide operations and 5047 completion is reported to a wavefront in execution order. The exception is 5048 that for GFX7-GFX9 ``flat_load/store/atomic`` instructions can report out of 5049 vector memory order if they access LDS memory, and out of LDS operation order 5050 if they access global memory. 5051* The vector memory operations access a single vector L1 cache shared by all 5052 SIMDs a CU. Therefore, no special action is required for coherence between the 5053 lanes of a single wavefront, or for coherence between wavefronts in the same 5054 work-group. A ``buffer_wbinvl1_vol`` is required for coherence between 5055 wavefronts executing in different work-groups as they may be executing on 5056 different CUs. 5057* The scalar memory operations access a scalar L1 cache shared by all wavefronts 5058 on a group of CUs. The scalar and vector L1 caches are not coherent. However, 5059 scalar operations are used in a restricted way so do not impact the memory 5060 model. See :ref:`amdgpu-amdhsa-memory-spaces`. 5061* The vector and scalar memory operations use an L2 cache shared by all CUs on 5062 the same agent. 5063* The L2 cache has independent channels to service disjoint ranges of virtual 5064 addresses. 5065* Each CU has a separate request queue per channel. Therefore, the vector and 5066 scalar memory operations performed by wavefronts executing in different 5067 work-groups (which may be executing on different CUs) of an agent can be 5068 reordered relative to each other. A ``s_waitcnt vmcnt(0)`` is required to 5069 ensure synchronization between vector memory operations of different CUs. It 5070 ensures a previous vector memory operation has completed before executing a 5071 subsequent vector memory or LDS operation and so can be used to meet the 5072 requirements of acquire and release. 5073* The L2 cache can be kept coherent with other agents on some targets, or ranges 5074 of virtual addresses can be set up to bypass it to ensure system coherence. 5075 5076Scalar memory operations are only used to access memory that is proven to not 5077change during the execution of the kernel dispatch. This includes constant 5078address space and global address space for program scope ``const`` variables. 5079Therefore, the kernel machine code does not have to maintain the scalar cache to 5080ensure it is coherent with the vector caches. The scalar and vector caches are 5081invalidated between kernel dispatches by CP since constant address space data 5082may change between kernel dispatch executions. See 5083:ref:`amdgpu-amdhsa-memory-spaces`. 5084 5085The one exception is if scalar writes are used to spill SGPR registers. In this 5086case the AMDGPU backend ensures the memory location used to spill is never 5087accessed by vector memory operations at the same time. If scalar writes are used 5088then a ``s_dcache_wb`` is inserted before the ``s_endpgm`` and before a function 5089return since the locations may be used for vector memory instructions by a 5090future wavefront that uses the same scratch area, or a function call that 5091creates a frame at the same address, respectively. There is no need for a 5092``s_dcache_inv`` as all scalar writes are write-before-read in the same thread. 5093 5094For kernarg backing memory: 5095 5096* CP invalidates the L1 cache at the start of each kernel dispatch. 5097* On dGPU the kernarg backing memory is allocated in host memory accessed as 5098 MTYPE UC (uncached) to avoid needing to invalidate the L2 cache. This also 5099 causes it to be treated as non-volatile and so is not invalidated by 5100 ``*_vol``. 5101* On APU the kernarg backing memory it is accessed as MTYPE CC (cache coherent) 5102 and so the L2 cache will be coherent with the CPU and other agents. 5103 5104Scratch backing memory (which is used for the private address space) is accessed 5105with MTYPE NC_NV (non-coherent non-volatile). Since the private address space is 5106only accessed by a single thread, and is always write-before-read, there is 5107never a need to invalidate these entries from the L1 cache. Hence all cache 5108invalidates are done as ``*_vol`` to only invalidate the volatile cache lines. 5109 5110The code sequences used to implement the memory model for GFX6-GFX9 are defined 5111in table :ref:`amdgpu-amdhsa-memory-model-code-sequences-gfx6-gfx9-table`. 5112 5113 .. table:: AMDHSA Memory Model Code Sequences GFX6-GFX9 5114 :name: amdgpu-amdhsa-memory-model-code-sequences-gfx6-gfx9-table 5115 5116 ============ ============ ============== ========== ================================ 5117 LLVM Instr LLVM Memory LLVM Memory AMDGPU AMDGPU Machine Code 5118 Ordering Sync Scope Address GFX6-GFX9 5119 Space 5120 ============ ============ ============== ========== ================================ 5121 **Non-Atomic** 5122 ------------------------------------------------------------------------------------ 5123 load *none* *none* - global - !volatile & !nontemporal 5124 - generic 5125 - private 1. buffer/global/flat_load 5126 - constant 5127 - !volatile & nontemporal 5128 5129 1. buffer/global/flat_load 5130 glc=1 slc=1 5131 5132 - volatile 5133 5134 1. buffer/global/flat_load 5135 glc=1 5136 2. s_waitcnt vmcnt(0) 5137 5138 - Must happen before 5139 any following volatile 5140 global/generic 5141 load/store. 5142 - Ensures that 5143 volatile 5144 operations to 5145 different 5146 addresses will not 5147 be reordered by 5148 hardware. 5149 5150 load *none* *none* - local 1. ds_load 5151 store *none* *none* - global - !volatile & !nontemporal 5152 - generic 5153 - private 1. buffer/global/flat_store 5154 - constant 5155 - !volatile & nontemporal 5156 5157 1. buffer/global/flat_store 5158 glc=1 slc=1 5159 5160 - volatile 5161 5162 1. buffer/global/flat_store 5163 2. s_waitcnt vmcnt(0) 5164 5165 - Must happen before 5166 any following volatile 5167 global/generic 5168 load/store. 5169 - Ensures that 5170 volatile 5171 operations to 5172 different 5173 addresses will not 5174 be reordered by 5175 hardware. 5176 5177 store *none* *none* - local 1. ds_store 5178 **Unordered Atomic** 5179 ------------------------------------------------------------------------------------ 5180 load atomic unordered *any* *any* *Same as non-atomic*. 5181 store atomic unordered *any* *any* *Same as non-atomic*. 5182 atomicrmw unordered *any* *any* *Same as monotonic atomic*. 5183 **Monotonic Atomic** 5184 ------------------------------------------------------------------------------------ 5185 load atomic monotonic - singlethread - global 1. buffer/global/ds/flat_load 5186 - wavefront - local 5187 - workgroup - generic 5188 load atomic monotonic - agent - global 1. buffer/global/flat_load 5189 - system - generic glc=1 5190 store atomic monotonic - singlethread - global 1. buffer/global/flat_store 5191 - wavefront - generic 5192 - workgroup 5193 - agent 5194 - system 5195 store atomic monotonic - singlethread - local 1. ds_store 5196 - wavefront 5197 - workgroup 5198 atomicrmw monotonic - singlethread - global 1. buffer/global/flat_atomic 5199 - wavefront - generic 5200 - workgroup 5201 - agent 5202 - system 5203 atomicrmw monotonic - singlethread - local 1. ds_atomic 5204 - wavefront 5205 - workgroup 5206 **Acquire Atomic** 5207 ------------------------------------------------------------------------------------ 5208 load atomic acquire - singlethread - global 1. buffer/global/ds/flat_load 5209 - wavefront - local 5210 - generic 5211 load atomic acquire - workgroup - global 1. buffer/global_load 5212 load atomic acquire - workgroup - local 1. ds/flat_load 5213 - generic 2. s_waitcnt lgkmcnt(0) 5214 5215 - If OpenCL, omit. 5216 - Must happen before 5217 any following 5218 global/generic 5219 load/load 5220 atomic/store/store 5221 atomic/atomicrmw. 5222 - Ensures any 5223 following global 5224 data read is no 5225 older than a local load 5226 atomic value being 5227 acquired. 5228 5229 load atomic acquire - agent - global 1. buffer/global_load 5230 - system glc=1 5231 2. s_waitcnt vmcnt(0) 5232 5233 - Must happen before 5234 following 5235 buffer_wbinvl1_vol. 5236 - Ensures the load 5237 has completed 5238 before invalidating 5239 the cache. 5240 5241 3. buffer_wbinvl1_vol 5242 5243 - Must happen before 5244 any following 5245 global/generic 5246 load/load 5247 atomic/atomicrmw. 5248 - Ensures that 5249 following 5250 loads will not see 5251 stale global data. 5252 5253 load atomic acquire - agent - generic 1. flat_load glc=1 5254 - system 2. s_waitcnt vmcnt(0) & 5255 lgkmcnt(0) 5256 5257 - If OpenCL omit 5258 lgkmcnt(0). 5259 - Must happen before 5260 following 5261 buffer_wbinvl1_vol. 5262 - Ensures the flat_load 5263 has completed 5264 before invalidating 5265 the cache. 5266 5267 3. buffer_wbinvl1_vol 5268 5269 - Must happen before 5270 any following 5271 global/generic 5272 load/load 5273 atomic/atomicrmw. 5274 - Ensures that 5275 following loads 5276 will not see stale 5277 global data. 5278 5279 atomicrmw acquire - singlethread - global 1. buffer/global/ds/flat_atomic 5280 - wavefront - local 5281 - generic 5282 atomicrmw acquire - workgroup - global 1. buffer/global_atomic 5283 atomicrmw acquire - workgroup - local 1. ds/flat_atomic 5284 - generic 2. s_waitcnt lgkmcnt(0) 5285 5286 - If OpenCL, omit. 5287 - Must happen before 5288 any following 5289 global/generic 5290 load/load 5291 atomic/store/store 5292 atomic/atomicrmw. 5293 - Ensures any 5294 following global 5295 data read is no 5296 older than a local 5297 atomicrmw value 5298 being acquired. 5299 5300 atomicrmw acquire - agent - global 1. buffer/global_atomic 5301 - system 2. s_waitcnt vmcnt(0) 5302 5303 - Must happen before 5304 following 5305 buffer_wbinvl1_vol. 5306 - Ensures the 5307 atomicrmw has 5308 completed before 5309 invalidating the 5310 cache. 5311 5312 3. buffer_wbinvl1_vol 5313 5314 - Must happen before 5315 any following 5316 global/generic 5317 load/load 5318 atomic/atomicrmw. 5319 - Ensures that 5320 following loads 5321 will not see stale 5322 global data. 5323 5324 atomicrmw acquire - agent - generic 1. flat_atomic 5325 - system 2. s_waitcnt vmcnt(0) & 5326 lgkmcnt(0) 5327 5328 - If OpenCL, omit 5329 lgkmcnt(0). 5330 - Must happen before 5331 following 5332 buffer_wbinvl1_vol. 5333 - Ensures the 5334 atomicrmw has 5335 completed before 5336 invalidating the 5337 cache. 5338 5339 3. buffer_wbinvl1_vol 5340 5341 - Must happen before 5342 any following 5343 global/generic 5344 load/load 5345 atomic/atomicrmw. 5346 - Ensures that 5347 following loads 5348 will not see stale 5349 global data. 5350 5351 fence acquire - singlethread *none* *none* 5352 - wavefront 5353 fence acquire - workgroup *none* 1. s_waitcnt lgkmcnt(0) 5354 5355 - If OpenCL and 5356 address space is 5357 not generic, omit. 5358 - However, since LLVM 5359 currently has no 5360 address space on 5361 the fence need to 5362 conservatively 5363 always generate. If 5364 fence had an 5365 address space then 5366 set to address 5367 space of OpenCL 5368 fence flag, or to 5369 generic if both 5370 local and global 5371 flags are 5372 specified. 5373 - Must happen after 5374 any preceding 5375 local/generic load 5376 atomic/atomicrmw 5377 with an equal or 5378 wider sync scope 5379 and memory ordering 5380 stronger than 5381 unordered (this is 5382 termed the 5383 fence-paired-atomic). 5384 - Must happen before 5385 any following 5386 global/generic 5387 load/load 5388 atomic/store/store 5389 atomic/atomicrmw. 5390 - Ensures any 5391 following global 5392 data read is no 5393 older than the 5394 value read by the 5395 fence-paired-atomic. 5396 5397 fence acquire - agent *none* 1. s_waitcnt lgkmcnt(0) & 5398 - system vmcnt(0) 5399 5400 - If OpenCL and 5401 address space is 5402 not generic, omit 5403 lgkmcnt(0). 5404 - However, since LLVM 5405 currently has no 5406 address space on 5407 the fence need to 5408 conservatively 5409 always generate 5410 (see comment for 5411 previous fence). 5412 - Could be split into 5413 separate s_waitcnt 5414 vmcnt(0) and 5415 s_waitcnt 5416 lgkmcnt(0) to allow 5417 them to be 5418 independently moved 5419 according to the 5420 following rules. 5421 - s_waitcnt vmcnt(0) 5422 must happen after 5423 any preceding 5424 global/generic load 5425 atomic/atomicrmw 5426 with an equal or 5427 wider sync scope 5428 and memory ordering 5429 stronger than 5430 unordered (this is 5431 termed the 5432 fence-paired-atomic). 5433 - s_waitcnt lgkmcnt(0) 5434 must happen after 5435 any preceding 5436 local/generic load 5437 atomic/atomicrmw 5438 with an equal or 5439 wider sync scope 5440 and memory ordering 5441 stronger than 5442 unordered (this is 5443 termed the 5444 fence-paired-atomic). 5445 - Must happen before 5446 the following 5447 buffer_wbinvl1_vol. 5448 - Ensures that the 5449 fence-paired atomic 5450 has completed 5451 before invalidating 5452 the 5453 cache. Therefore 5454 any following 5455 locations read must 5456 be no older than 5457 the value read by 5458 the 5459 fence-paired-atomic. 5460 5461 2. buffer_wbinvl1_vol 5462 5463 - Must happen before any 5464 following global/generic 5465 load/load 5466 atomic/store/store 5467 atomic/atomicrmw. 5468 - Ensures that 5469 following loads 5470 will not see stale 5471 global data. 5472 5473 **Release Atomic** 5474 ------------------------------------------------------------------------------------ 5475 store atomic release - singlethread - global 1. buffer/global/ds/flat_store 5476 - wavefront - local 5477 - generic 5478 store atomic release - workgroup - global 1. s_waitcnt lgkmcnt(0) 5479 - generic 5480 - If OpenCL, omit. 5481 - Must happen after 5482 any preceding 5483 local/generic 5484 load/store/load 5485 atomic/store 5486 atomic/atomicrmw. 5487 - Must happen before 5488 the following 5489 store. 5490 - Ensures that all 5491 memory operations 5492 to local have 5493 completed before 5494 performing the 5495 store that is being 5496 released. 5497 5498 2. buffer/global/flat_store 5499 store atomic release - workgroup - local 1. ds_store 5500 store atomic release - agent - global 1. s_waitcnt lgkmcnt(0) & 5501 - system - generic vmcnt(0) 5502 5503 - If OpenCL and 5504 address space is 5505 not generic, omit 5506 lgkmcnt(0). 5507 - Could be split into 5508 separate s_waitcnt 5509 vmcnt(0) and 5510 s_waitcnt 5511 lgkmcnt(0) to allow 5512 them to be 5513 independently moved 5514 according to the 5515 following rules. 5516 - s_waitcnt vmcnt(0) 5517 must happen after 5518 any preceding 5519 global/generic 5520 load/store/load 5521 atomic/store 5522 atomic/atomicrmw. 5523 - s_waitcnt lgkmcnt(0) 5524 must happen after 5525 any preceding 5526 local/generic 5527 load/store/load 5528 atomic/store 5529 atomic/atomicrmw. 5530 - Must happen before 5531 the following 5532 store. 5533 - Ensures that all 5534 memory operations 5535 to memory have 5536 completed before 5537 performing the 5538 store that is being 5539 released. 5540 5541 2. buffer/global/flat_store 5542 atomicrmw release - singlethread - global 1. buffer/global/ds/flat_atomic 5543 - wavefront - local 5544 - generic 5545 atomicrmw release - workgroup - global 1. s_waitcnt lgkmcnt(0) 5546 - generic 5547 - If OpenCL, omit. 5548 - Must happen after 5549 any preceding 5550 local/generic 5551 load/store/load 5552 atomic/store 5553 atomic/atomicrmw. 5554 - Must happen before 5555 the following 5556 atomicrmw. 5557 - Ensures that all 5558 memory operations 5559 to local have 5560 completed before 5561 performing the 5562 atomicrmw that is 5563 being released. 5564 5565 2. buffer/global/flat_atomic 5566 atomicrmw release - workgroup - local 1. ds_atomic 5567 atomicrmw release - agent - global 1. s_waitcnt lgkmcnt(0) & 5568 - system - generic vmcnt(0) 5569 5570 - If OpenCL, omit 5571 lgkmcnt(0). 5572 - Could be split into 5573 separate s_waitcnt 5574 vmcnt(0) and 5575 s_waitcnt 5576 lgkmcnt(0) to allow 5577 them to be 5578 independently moved 5579 according to the 5580 following rules. 5581 - s_waitcnt vmcnt(0) 5582 must happen after 5583 any preceding 5584 global/generic 5585 load/store/load 5586 atomic/store 5587 atomic/atomicrmw. 5588 - s_waitcnt lgkmcnt(0) 5589 must happen after 5590 any preceding 5591 local/generic 5592 load/store/load 5593 atomic/store 5594 atomic/atomicrmw. 5595 - Must happen before 5596 the following 5597 atomicrmw. 5598 - Ensures that all 5599 memory operations 5600 to global and local 5601 have completed 5602 before performing 5603 the atomicrmw that 5604 is being released. 5605 5606 2. buffer/global/flat_atomic 5607 fence release - singlethread *none* *none* 5608 - wavefront 5609 fence release - workgroup *none* 1. s_waitcnt lgkmcnt(0) 5610 5611 - If OpenCL and 5612 address space is 5613 not generic, omit. 5614 - However, since LLVM 5615 currently has no 5616 address space on 5617 the fence need to 5618 conservatively 5619 always generate. If 5620 fence had an 5621 address space then 5622 set to address 5623 space of OpenCL 5624 fence flag, or to 5625 generic if both 5626 local and global 5627 flags are 5628 specified. 5629 - Must happen after 5630 any preceding 5631 local/generic 5632 load/load 5633 atomic/store/store 5634 atomic/atomicrmw. 5635 - Must happen before 5636 any following store 5637 atomic/atomicrmw 5638 with an equal or 5639 wider sync scope 5640 and memory ordering 5641 stronger than 5642 unordered (this is 5643 termed the 5644 fence-paired-atomic). 5645 - Ensures that all 5646 memory operations 5647 to local have 5648 completed before 5649 performing the 5650 following 5651 fence-paired-atomic. 5652 5653 fence release - agent *none* 1. s_waitcnt lgkmcnt(0) & 5654 - system vmcnt(0) 5655 5656 - If OpenCL and 5657 address space is 5658 not generic, omit 5659 lgkmcnt(0). 5660 - If OpenCL and 5661 address space is 5662 local, omit 5663 vmcnt(0). 5664 - However, since LLVM 5665 currently has no 5666 address space on 5667 the fence need to 5668 conservatively 5669 always generate. If 5670 fence had an 5671 address space then 5672 set to address 5673 space of OpenCL 5674 fence flag, or to 5675 generic if both 5676 local and global 5677 flags are 5678 specified. 5679 - Could be split into 5680 separate s_waitcnt 5681 vmcnt(0) and 5682 s_waitcnt 5683 lgkmcnt(0) to allow 5684 them to be 5685 independently moved 5686 according to the 5687 following rules. 5688 - s_waitcnt vmcnt(0) 5689 must happen after 5690 any preceding 5691 global/generic 5692 load/store/load 5693 atomic/store 5694 atomic/atomicrmw. 5695 - s_waitcnt lgkmcnt(0) 5696 must happen after 5697 any preceding 5698 local/generic 5699 load/store/load 5700 atomic/store 5701 atomic/atomicrmw. 5702 - Must happen before 5703 any following store 5704 atomic/atomicrmw 5705 with an equal or 5706 wider sync scope 5707 and memory ordering 5708 stronger than 5709 unordered (this is 5710 termed the 5711 fence-paired-atomic). 5712 - Ensures that all 5713 memory operations 5714 have 5715 completed before 5716 performing the 5717 following 5718 fence-paired-atomic. 5719 5720 **Acquire-Release Atomic** 5721 ------------------------------------------------------------------------------------ 5722 atomicrmw acq_rel - singlethread - global 1. buffer/global/ds/flat_atomic 5723 - wavefront - local 5724 - generic 5725 atomicrmw acq_rel - workgroup - global 1. s_waitcnt lgkmcnt(0) 5726 5727 - If OpenCL, omit. 5728 - Must happen after 5729 any preceding 5730 local/generic 5731 load/store/load 5732 atomic/store 5733 atomic/atomicrmw. 5734 - Must happen before 5735 the following 5736 atomicrmw. 5737 - Ensures that all 5738 memory operations 5739 to local have 5740 completed before 5741 performing the 5742 atomicrmw that is 5743 being released. 5744 5745 2. buffer/global_atomic 5746 5747 atomicrmw acq_rel - workgroup - local 1. ds_atomic 5748 2. s_waitcnt lgkmcnt(0) 5749 5750 - If OpenCL, omit. 5751 - Must happen before 5752 any following 5753 global/generic 5754 load/load 5755 atomic/store/store 5756 atomic/atomicrmw. 5757 - Ensures any 5758 following global 5759 data read is no 5760 older than the local load 5761 atomic value being 5762 acquired. 5763 5764 atomicrmw acq_rel - workgroup - generic 1. s_waitcnt lgkmcnt(0) 5765 5766 - If OpenCL, omit. 5767 - Must happen after 5768 any preceding 5769 local/generic 5770 load/store/load 5771 atomic/store 5772 atomic/atomicrmw. 5773 - Must happen before 5774 the following 5775 atomicrmw. 5776 - Ensures that all 5777 memory operations 5778 to local have 5779 completed before 5780 performing the 5781 atomicrmw that is 5782 being released. 5783 5784 2. flat_atomic 5785 3. s_waitcnt lgkmcnt(0) 5786 5787 - If OpenCL, omit. 5788 - Must happen before 5789 any following 5790 global/generic 5791 load/load 5792 atomic/store/store 5793 atomic/atomicrmw. 5794 - Ensures any 5795 following global 5796 data read is no 5797 older than a local load 5798 atomic value being 5799 acquired. 5800 5801 atomicrmw acq_rel - agent - global 1. s_waitcnt lgkmcnt(0) & 5802 - system vmcnt(0) 5803 5804 - If OpenCL, omit 5805 lgkmcnt(0). 5806 - Could be split into 5807 separate s_waitcnt 5808 vmcnt(0) and 5809 s_waitcnt 5810 lgkmcnt(0) to allow 5811 them to be 5812 independently moved 5813 according to the 5814 following rules. 5815 - s_waitcnt vmcnt(0) 5816 must happen after 5817 any preceding 5818 global/generic 5819 load/store/load 5820 atomic/store 5821 atomic/atomicrmw. 5822 - s_waitcnt lgkmcnt(0) 5823 must happen after 5824 any preceding 5825 local/generic 5826 load/store/load 5827 atomic/store 5828 atomic/atomicrmw. 5829 - Must happen before 5830 the following 5831 atomicrmw. 5832 - Ensures that all 5833 memory operations 5834 to global have 5835 completed before 5836 performing the 5837 atomicrmw that is 5838 being released. 5839 5840 2. buffer/global_atomic 5841 3. s_waitcnt vmcnt(0) 5842 5843 - Must happen before 5844 following 5845 buffer_wbinvl1_vol. 5846 - Ensures the 5847 atomicrmw has 5848 completed before 5849 invalidating the 5850 cache. 5851 5852 4. buffer_wbinvl1_vol 5853 5854 - Must happen before 5855 any following 5856 global/generic 5857 load/load 5858 atomic/atomicrmw. 5859 - Ensures that 5860 following loads 5861 will not see stale 5862 global data. 5863 5864 atomicrmw acq_rel - agent - generic 1. s_waitcnt lgkmcnt(0) & 5865 - system vmcnt(0) 5866 5867 - If OpenCL, omit 5868 lgkmcnt(0). 5869 - Could be split into 5870 separate s_waitcnt 5871 vmcnt(0) and 5872 s_waitcnt 5873 lgkmcnt(0) to allow 5874 them to be 5875 independently moved 5876 according to the 5877 following rules. 5878 - s_waitcnt vmcnt(0) 5879 must happen after 5880 any preceding 5881 global/generic 5882 load/store/load 5883 atomic/store 5884 atomic/atomicrmw. 5885 - s_waitcnt lgkmcnt(0) 5886 must happen after 5887 any preceding 5888 local/generic 5889 load/store/load 5890 atomic/store 5891 atomic/atomicrmw. 5892 - Must happen before 5893 the following 5894 atomicrmw. 5895 - Ensures that all 5896 memory operations 5897 to global have 5898 completed before 5899 performing the 5900 atomicrmw that is 5901 being released. 5902 5903 2. flat_atomic 5904 3. s_waitcnt vmcnt(0) & 5905 lgkmcnt(0) 5906 5907 - If OpenCL, omit 5908 lgkmcnt(0). 5909 - Must happen before 5910 following 5911 buffer_wbinvl1_vol. 5912 - Ensures the 5913 atomicrmw has 5914 completed before 5915 invalidating the 5916 cache. 5917 5918 4. buffer_wbinvl1_vol 5919 5920 - Must happen before 5921 any following 5922 global/generic 5923 load/load 5924 atomic/atomicrmw. 5925 - Ensures that 5926 following loads 5927 will not see stale 5928 global data. 5929 5930 fence acq_rel - singlethread *none* *none* 5931 - wavefront 5932 fence acq_rel - workgroup *none* 1. s_waitcnt lgkmcnt(0) 5933 5934 - If OpenCL and 5935 address space is 5936 not generic, omit. 5937 - However, 5938 since LLVM 5939 currently has no 5940 address space on 5941 the fence need to 5942 conservatively 5943 always generate 5944 (see comment for 5945 previous fence). 5946 - Must happen after 5947 any preceding 5948 local/generic 5949 load/load 5950 atomic/store/store 5951 atomic/atomicrmw. 5952 - Must happen before 5953 any following 5954 global/generic 5955 load/load 5956 atomic/store/store 5957 atomic/atomicrmw. 5958 - Ensures that all 5959 memory operations 5960 to local have 5961 completed before 5962 performing any 5963 following global 5964 memory operations. 5965 - Ensures that the 5966 preceding 5967 local/generic load 5968 atomic/atomicrmw 5969 with an equal or 5970 wider sync scope 5971 and memory ordering 5972 stronger than 5973 unordered (this is 5974 termed the 5975 acquire-fence-paired-atomic) 5976 has completed 5977 before following 5978 global memory 5979 operations. This 5980 satisfies the 5981 requirements of 5982 acquire. 5983 - Ensures that all 5984 previous memory 5985 operations have 5986 completed before a 5987 following 5988 local/generic store 5989 atomic/atomicrmw 5990 with an equal or 5991 wider sync scope 5992 and memory ordering 5993 stronger than 5994 unordered (this is 5995 termed the 5996 release-fence-paired-atomic). 5997 This satisfies the 5998 requirements of 5999 release. 6000 6001 fence acq_rel - agent *none* 1. s_waitcnt lgkmcnt(0) & 6002 - system vmcnt(0) 6003 6004 - If OpenCL and 6005 address space is 6006 not generic, omit 6007 lgkmcnt(0). 6008 - However, since LLVM 6009 currently has no 6010 address space on 6011 the fence need to 6012 conservatively 6013 always generate 6014 (see comment for 6015 previous fence). 6016 - Could be split into 6017 separate s_waitcnt 6018 vmcnt(0) and 6019 s_waitcnt 6020 lgkmcnt(0) to allow 6021 them to be 6022 independently moved 6023 according to the 6024 following rules. 6025 - s_waitcnt vmcnt(0) 6026 must happen after 6027 any preceding 6028 global/generic 6029 load/store/load 6030 atomic/store 6031 atomic/atomicrmw. 6032 - s_waitcnt lgkmcnt(0) 6033 must happen after 6034 any preceding 6035 local/generic 6036 load/store/load 6037 atomic/store 6038 atomic/atomicrmw. 6039 - Must happen before 6040 the following 6041 buffer_wbinvl1_vol. 6042 - Ensures that the 6043 preceding 6044 global/local/generic 6045 load 6046 atomic/atomicrmw 6047 with an equal or 6048 wider sync scope 6049 and memory ordering 6050 stronger than 6051 unordered (this is 6052 termed the 6053 acquire-fence-paired-atomic) 6054 has completed 6055 before invalidating 6056 the cache. This 6057 satisfies the 6058 requirements of 6059 acquire. 6060 - Ensures that all 6061 previous memory 6062 operations have 6063 completed before a 6064 following 6065 global/local/generic 6066 store 6067 atomic/atomicrmw 6068 with an equal or 6069 wider sync scope 6070 and memory ordering 6071 stronger than 6072 unordered (this is 6073 termed the 6074 release-fence-paired-atomic). 6075 This satisfies the 6076 requirements of 6077 release. 6078 6079 2. buffer_wbinvl1_vol 6080 6081 - Must happen before 6082 any following 6083 global/generic 6084 load/load 6085 atomic/store/store 6086 atomic/atomicrmw. 6087 - Ensures that 6088 following loads 6089 will not see stale 6090 global data. This 6091 satisfies the 6092 requirements of 6093 acquire. 6094 6095 **Sequential Consistent Atomic** 6096 ------------------------------------------------------------------------------------ 6097 load atomic seq_cst - singlethread - global *Same as corresponding 6098 - wavefront - local load atomic acquire, 6099 - generic except must generate 6100 all instructions even 6101 for OpenCL.* 6102 load atomic seq_cst - workgroup - global 1. s_waitcnt lgkmcnt(0) 6103 - generic 6104 6105 - Must 6106 happen after 6107 preceding 6108 local/generic load 6109 atomic/store 6110 atomic/atomicrmw 6111 with memory 6112 ordering of seq_cst 6113 and with equal or 6114 wider sync scope. 6115 (Note that seq_cst 6116 fences have their 6117 own s_waitcnt 6118 lgkmcnt(0) and so do 6119 not need to be 6120 considered.) 6121 - Ensures any 6122 preceding 6123 sequential 6124 consistent local 6125 memory instructions 6126 have completed 6127 before executing 6128 this sequentially 6129 consistent 6130 instruction. This 6131 prevents reordering 6132 a seq_cst store 6133 followed by a 6134 seq_cst load. (Note 6135 that seq_cst is 6136 stronger than 6137 acquire/release as 6138 the reordering of 6139 load acquire 6140 followed by a store 6141 release is 6142 prevented by the 6143 s_waitcnt of 6144 the release, but 6145 there is nothing 6146 preventing a store 6147 release followed by 6148 load acquire from 6149 completing out of 6150 order. The s_waitcnt 6151 could be placed after 6152 seq_store or before 6153 the seq_load. We 6154 choose the load to 6155 make the s_waitcnt be 6156 as late as possible 6157 so that the store 6158 may have already 6159 completed.) 6160 6161 2. *Following 6162 instructions same as 6163 corresponding load 6164 atomic acquire, 6165 except must generate 6166 all instructions even 6167 for OpenCL.* 6168 load atomic seq_cst - workgroup - local *Same as corresponding 6169 load atomic acquire, 6170 except must generate 6171 all instructions even 6172 for OpenCL.* 6173 6174 load atomic seq_cst - agent - global 1. s_waitcnt lgkmcnt(0) & 6175 - system - generic vmcnt(0) 6176 6177 - Could be split into 6178 separate s_waitcnt 6179 vmcnt(0) 6180 and s_waitcnt 6181 lgkmcnt(0) to allow 6182 them to be 6183 independently moved 6184 according to the 6185 following rules. 6186 - s_waitcnt lgkmcnt(0) 6187 must happen after 6188 preceding 6189 global/generic load 6190 atomic/store 6191 atomic/atomicrmw 6192 with memory 6193 ordering of seq_cst 6194 and with equal or 6195 wider sync scope. 6196 (Note that seq_cst 6197 fences have their 6198 own s_waitcnt 6199 lgkmcnt(0) and so do 6200 not need to be 6201 considered.) 6202 - s_waitcnt vmcnt(0) 6203 must happen after 6204 preceding 6205 global/generic load 6206 atomic/store 6207 atomic/atomicrmw 6208 with memory 6209 ordering of seq_cst 6210 and with equal or 6211 wider sync scope. 6212 (Note that seq_cst 6213 fences have their 6214 own s_waitcnt 6215 vmcnt(0) and so do 6216 not need to be 6217 considered.) 6218 - Ensures any 6219 preceding 6220 sequential 6221 consistent global 6222 memory instructions 6223 have completed 6224 before executing 6225 this sequentially 6226 consistent 6227 instruction. This 6228 prevents reordering 6229 a seq_cst store 6230 followed by a 6231 seq_cst load. (Note 6232 that seq_cst is 6233 stronger than 6234 acquire/release as 6235 the reordering of 6236 load acquire 6237 followed by a store 6238 release is 6239 prevented by the 6240 s_waitcnt of 6241 the release, but 6242 there is nothing 6243 preventing a store 6244 release followed by 6245 load acquire from 6246 completing out of 6247 order. The s_waitcnt 6248 could be placed after 6249 seq_store or before 6250 the seq_load. We 6251 choose the load to 6252 make the s_waitcnt be 6253 as late as possible 6254 so that the store 6255 may have already 6256 completed.) 6257 6258 2. *Following 6259 instructions same as 6260 corresponding load 6261 atomic acquire, 6262 except must generate 6263 all instructions even 6264 for OpenCL.* 6265 store atomic seq_cst - singlethread - global *Same as corresponding 6266 - wavefront - local store atomic release, 6267 - workgroup - generic except must generate 6268 - agent all instructions even 6269 - system for OpenCL.* 6270 atomicrmw seq_cst - singlethread - global *Same as corresponding 6271 - wavefront - local atomicrmw acq_rel, 6272 - workgroup - generic except must generate 6273 - agent all instructions even 6274 - system for OpenCL.* 6275 fence seq_cst - singlethread *none* *Same as corresponding 6276 - wavefront fence acq_rel, 6277 - workgroup except must generate 6278 - agent all instructions even 6279 - system for OpenCL.* 6280 ============ ============ ============== ========== ================================ 6281 6282.. _amdgpu-amdhsa-memory-model-gfx90a: 6283 6284Memory Model GFX90A 6285+++++++++++++++++++ 6286 6287For GFX90A: 6288 6289* Each agent has multiple shader arrays (SA). 6290* Each SA has multiple compute units (CU). 6291* Each CU has multiple SIMDs that execute wavefronts. 6292* The wavefronts for a single work-group are executed in the same CU but may be 6293 executed by different SIMDs. The exception is when in tgsplit execution mode 6294 when the wavefronts may be executed by different SIMDs in different CUs. 6295* Each CU has a single LDS memory shared by the wavefronts of the work-groups 6296 executing on it. The exception is when in tgsplit execution mode when no LDS 6297 is allocated as wavefronts of the same work-group can be in different CUs. 6298* All LDS operations of a CU are performed as wavefront wide operations in a 6299 global order and involve no caching. Completion is reported to a wavefront in 6300 execution order. 6301* The LDS memory has multiple request queues shared by the SIMDs of a 6302 CU. Therefore, the LDS operations performed by different wavefronts of a 6303 work-group can be reordered relative to each other, which can result in 6304 reordering the visibility of vector memory operations with respect to LDS 6305 operations of other wavefronts in the same work-group. A ``s_waitcnt 6306 lgkmcnt(0)`` is required to ensure synchronization between LDS operations and 6307 vector memory operations between wavefronts of a work-group, but not between 6308 operations performed by the same wavefront. 6309* The vector memory operations are performed as wavefront wide operations and 6310 completion is reported to a wavefront in execution order. The exception is 6311 that ``flat_load/store/atomic`` instructions can report out of vector memory 6312 order if they access LDS memory, and out of LDS operation order if they access 6313 global memory. 6314* The vector memory operations access a single vector L1 cache shared by all 6315 SIMDs a CU. Therefore: 6316 6317 * No special action is required for coherence between the lanes of a single 6318 wavefront. 6319 6320 * No special action is required for coherence between wavefronts in the same 6321 work-group since they execute on the same CU. The exception is when in 6322 tgsplit execution mode as wavefronts of the same work-group can be in 6323 different CUs and so a ``buffer_wbinvl1_vol`` is required as described in 6324 the following item. 6325 6326 * A ``buffer_wbinvl1_vol`` is required for coherence between wavefronts 6327 executing in different work-groups as they may be executing on different 6328 CUs. 6329 6330* The scalar memory operations access a scalar L1 cache shared by all wavefronts 6331 on a group of CUs. The scalar and vector L1 caches are not coherent. However, 6332 scalar operations are used in a restricted way so do not impact the memory 6333 model. See :ref:`amdgpu-amdhsa-memory-spaces`. 6334* The vector and scalar memory operations use an L2 cache shared by all CUs on 6335 the same agent. 6336 6337 * The L2 cache has independent channels to service disjoint ranges of virtual 6338 addresses. 6339 * Each CU has a separate request queue per channel. Therefore, the vector and 6340 scalar memory operations performed by wavefronts executing in different 6341 work-groups (which may be executing on different CUs), or the same 6342 work-group if executing in tgsplit mode, of an agent can be reordered 6343 relative to each other. A ``s_waitcnt vmcnt(0)`` is required to ensure 6344 synchronization between vector memory operations of different CUs. It 6345 ensures a previous vector memory operation has completed before executing a 6346 subsequent vector memory or LDS operation and so can be used to meet the 6347 requirements of acquire and release. 6348 * The L2 cache of one agent can be kept coherent with other agents by: 6349 using the MTYPE RW (read-write) or MTYPE CC (cache-coherent) with the PTE 6350 C-bit for memory local to the L2; and using the MTYPE NC (non-coherent) with 6351 the PTE C-bit set or MTYPE UC (uncached) for memory not local to the L2. 6352 6353 * Any local memory cache lines will be automatically invalidated by writes 6354 from CUs associated with other L2 caches, or writes from the CPU, due to 6355 the cache probe caused by coherent requests. Coherent requests are caused 6356 by GPU accesses to pages with the PTE C-bit set, by CPU accesses over 6357 XGMI, and by PCIe requests that are configured to be coherent requests. 6358 * XGMI accesses from the CPU to local memory may be cached on the CPU. 6359 Subsequent access from the GPU will automatically invalidate or writeback 6360 the CPU cache due to the L2 probe filter and and the PTE C-bit being set. 6361 * Since all work-groups on the same agent share the same L2, no L2 6362 invalidation or writeback is required for coherence. 6363 * To ensure coherence of local and remote memory writes of work-groups in 6364 different agents a ``buffer_wbl2`` is required. It will writeback dirty L2 6365 cache lines of MTYPE RW (used for local coarse grain memory) and MTYPE NC 6366 ()used for remote coarse grain memory). Note that MTYPE CC (used for local 6367 fine grain memory) causes write through to DRAM, and MTYPE UC (used for 6368 remote fine grain memory) bypasses the L2, so both will never result in 6369 dirty L2 cache lines. 6370 * To ensure coherence of local and remote memory reads of work-groups in 6371 different agents a ``buffer_invl2`` is required. It will invalidate L2 6372 cache lines with MTYPE NC (used for remote coarse grain memory). Note that 6373 MTYPE CC (used for local fine grain memory) and MTYPE RW (used for local 6374 coarse memory) cause local reads to be invalidated by remote writes with 6375 with the PTE C-bit so these cache lines are not invalidated. Note that 6376 MTYPE UC (used for remote fine grain memory) bypasses the L2, so will 6377 never result in L2 cache lines that need to be invalidated. 6378 6379 * PCIe access from the GPU to the CPU memory is kept coherent by using the 6380 MTYPE UC (uncached) which bypasses the L2. 6381 6382Scalar memory operations are only used to access memory that is proven to not 6383change during the execution of the kernel dispatch. This includes constant 6384address space and global address space for program scope ``const`` variables. 6385Therefore, the kernel machine code does not have to maintain the scalar cache to 6386ensure it is coherent with the vector caches. The scalar and vector caches are 6387invalidated between kernel dispatches by CP since constant address space data 6388may change between kernel dispatch executions. See 6389:ref:`amdgpu-amdhsa-memory-spaces`. 6390 6391The one exception is if scalar writes are used to spill SGPR registers. In this 6392case the AMDGPU backend ensures the memory location used to spill is never 6393accessed by vector memory operations at the same time. If scalar writes are used 6394then a ``s_dcache_wb`` is inserted before the ``s_endpgm`` and before a function 6395return since the locations may be used for vector memory instructions by a 6396future wavefront that uses the same scratch area, or a function call that 6397creates a frame at the same address, respectively. There is no need for a 6398``s_dcache_inv`` as all scalar writes are write-before-read in the same thread. 6399 6400For kernarg backing memory: 6401 6402* CP invalidates the L1 cache at the start of each kernel dispatch. 6403* On dGPU over XGMI or PCIe the kernarg backing memory is allocated in host 6404 memory accessed as MTYPE UC (uncached) to avoid needing to invalidate the L2 6405 cache. This also causes it to be treated as non-volatile and so is not 6406 invalidated by ``*_vol``. 6407* On APU the kernarg backing memory is accessed as MTYPE CC (cache coherent) and 6408 so the L2 cache will be coherent with the CPU and other agents. 6409 6410Scratch backing memory (which is used for the private address space) is accessed 6411with MTYPE NC_NV (non-coherent non-volatile). Since the private address space is 6412only accessed by a single thread, and is always write-before-read, there is 6413never a need to invalidate these entries from the L1 cache. Hence all cache 6414invalidates are done as ``*_vol`` to only invalidate the volatile cache lines. 6415 6416The code sequences used to implement the memory model for GFX90A are defined 6417in table :ref:`amdgpu-amdhsa-memory-model-code-sequences-gfx90a-table`. 6418 6419 .. table:: AMDHSA Memory Model Code Sequences GFX90A 6420 :name: amdgpu-amdhsa-memory-model-code-sequences-gfx90a-table 6421 6422 ============ ============ ============== ========== ================================ 6423 LLVM Instr LLVM Memory LLVM Memory AMDGPU AMDGPU Machine Code 6424 Ordering Sync Scope Address GFX90A 6425 Space 6426 ============ ============ ============== ========== ================================ 6427 **Non-Atomic** 6428 ------------------------------------------------------------------------------------ 6429 load *none* *none* - global - !volatile & !nontemporal 6430 - generic 6431 - private 1. buffer/global/flat_load 6432 - constant 6433 - !volatile & nontemporal 6434 6435 1. buffer/global/flat_load 6436 glc=1 slc=1 6437 6438 - volatile 6439 6440 1. buffer/global/flat_load 6441 glc=1 6442 2. s_waitcnt vmcnt(0) 6443 6444 - Must happen before 6445 any following volatile 6446 global/generic 6447 load/store. 6448 - Ensures that 6449 volatile 6450 operations to 6451 different 6452 addresses will not 6453 be reordered by 6454 hardware. 6455 6456 load *none* *none* - local 1. ds_load 6457 store *none* *none* - global - !volatile & !nontemporal 6458 - generic 6459 - private 1. buffer/global/flat_store 6460 - constant 6461 - !volatile & nontemporal 6462 6463 1. buffer/global/flat_store 6464 glc=1 slc=1 6465 6466 - volatile 6467 6468 1. buffer/global/flat_store 6469 2. s_waitcnt vmcnt(0) 6470 6471 - Must happen before 6472 any following volatile 6473 global/generic 6474 load/store. 6475 - Ensures that 6476 volatile 6477 operations to 6478 different 6479 addresses will not 6480 be reordered by 6481 hardware. 6482 6483 store *none* *none* - local 1. ds_store 6484 **Unordered Atomic** 6485 ------------------------------------------------------------------------------------ 6486 load atomic unordered *any* *any* *Same as non-atomic*. 6487 store atomic unordered *any* *any* *Same as non-atomic*. 6488 atomicrmw unordered *any* *any* *Same as monotonic atomic*. 6489 **Monotonic Atomic** 6490 ------------------------------------------------------------------------------------ 6491 load atomic monotonic - singlethread - global 1. buffer/global/flat_load 6492 - wavefront - generic 6493 load atomic monotonic - workgroup - global 1. buffer/global/flat_load 6494 - generic glc=1 6495 6496 - If not TgSplit execution 6497 mode, omit glc=1. 6498 6499 load atomic monotonic - singlethread - local *If TgSplit execution mode, 6500 - wavefront local address space cannot 6501 - workgroup be used.* 6502 6503 1. ds_load 6504 load atomic monotonic - agent - global 1. buffer/global/flat_load 6505 - generic glc=1 6506 load atomic monotonic - system - global 1. buffer/global/flat_load 6507 - generic glc=1 6508 store atomic monotonic - singlethread - global 1. buffer/global/flat_store 6509 - wavefront - generic 6510 - workgroup 6511 - agent 6512 store atomic monotonic - system - global 1. buffer/global/flat_store 6513 - generic 6514 store atomic monotonic - singlethread - local *If TgSplit execution mode, 6515 - wavefront local address space cannot 6516 - workgroup be used.* 6517 6518 1. ds_store 6519 atomicrmw monotonic - singlethread - global 1. buffer/global/flat_atomic 6520 - wavefront - generic 6521 - workgroup 6522 - agent 6523 atomicrmw monotonic - system - global 1. buffer/global/flat_atomic 6524 - generic 6525 atomicrmw monotonic - singlethread - local *If TgSplit execution mode, 6526 - wavefront local address space cannot 6527 - workgroup be used.* 6528 6529 1. ds_atomic 6530 **Acquire Atomic** 6531 ------------------------------------------------------------------------------------ 6532 load atomic acquire - singlethread - global 1. buffer/global/ds/flat_load 6533 - wavefront - local 6534 - generic 6535 load atomic acquire - workgroup - global 1. buffer/global_load glc=1 6536 6537 - If not TgSplit execution 6538 mode, omit glc=1. 6539 6540 2. s_waitcnt vmcnt(0) 6541 6542 - If not TgSplit execution 6543 mode, omit. 6544 - Must happen before the 6545 following buffer_wbinvl1_vol. 6546 6547 3. buffer_wbinvl1_vol 6548 6549 - If not TgSplit execution 6550 mode, omit. 6551 - Must happen before 6552 any following 6553 global/generic 6554 load/load 6555 atomic/store/store 6556 atomic/atomicrmw. 6557 - Ensures that 6558 following 6559 loads will not see 6560 stale data. 6561 6562 load atomic acquire - workgroup - local *If TgSplit execution mode, 6563 local address space cannot 6564 be used.* 6565 6566 1. ds_load 6567 2. s_waitcnt lgkmcnt(0) 6568 6569 - If OpenCL, omit. 6570 - Must happen before 6571 any following 6572 global/generic 6573 load/load 6574 atomic/store/store 6575 atomic/atomicrmw. 6576 - Ensures any 6577 following global 6578 data read is no 6579 older than the local load 6580 atomic value being 6581 acquired. 6582 6583 load atomic acquire - workgroup - generic 1. flat_load glc=1 6584 6585 - If not TgSplit execution 6586 mode, omit glc=1. 6587 6588 2. s_waitcnt lgkm/vmcnt(0) 6589 6590 - Use lgkmcnt(0) if not 6591 TgSplit execution mode 6592 and vmcnt(0) if TgSplit 6593 execution mode. 6594 - If OpenCL, omit lgkmcnt(0). 6595 - Must happen before 6596 the following 6597 buffer_wbinvl1_vol and any 6598 following global/generic 6599 load/load 6600 atomic/store/store 6601 atomic/atomicrmw. 6602 - Ensures any 6603 following global 6604 data read is no 6605 older than a local load 6606 atomic value being 6607 acquired. 6608 6609 3. buffer_wbinvl1_vol 6610 6611 - If not TgSplit execution 6612 mode, omit. 6613 - Ensures that 6614 following 6615 loads will not see 6616 stale data. 6617 6618 load atomic acquire - agent - global 1. buffer/global_load 6619 glc=1 6620 2. s_waitcnt vmcnt(0) 6621 6622 - Must happen before 6623 following 6624 buffer_wbinvl1_vol. 6625 - Ensures the load 6626 has completed 6627 before invalidating 6628 the cache. 6629 6630 3. buffer_wbinvl1_vol 6631 6632 - Must happen before 6633 any following 6634 global/generic 6635 load/load 6636 atomic/atomicrmw. 6637 - Ensures that 6638 following 6639 loads will not see 6640 stale global data. 6641 6642 load atomic acquire - system - global 1. buffer/global/flat_load 6643 glc=1 6644 2. s_waitcnt vmcnt(0) 6645 6646 - Must happen before 6647 following buffer_invl2 and 6648 buffer_wbinvl1_vol. 6649 - Ensures the load 6650 has completed 6651 before invalidating 6652 the cache. 6653 6654 3. buffer_invl2; 6655 buffer_wbinvl1_vol 6656 6657 - Must happen before 6658 any following 6659 global/generic 6660 load/load 6661 atomic/atomicrmw. 6662 - Ensures that 6663 following 6664 loads will not see 6665 stale L1 global data, 6666 nor see stale L2 MTYPE 6667 NC global data. 6668 MTYPE RW and CC memory will 6669 never be stale in L2 due to 6670 the memory probes. 6671 6672 load atomic acquire - agent - generic 1. flat_load glc=1 6673 2. s_waitcnt vmcnt(0) & 6674 lgkmcnt(0) 6675 6676 - If TgSplit execution mode, 6677 omit lgkmcnt(0). 6678 - If OpenCL omit 6679 lgkmcnt(0). 6680 - Must happen before 6681 following 6682 buffer_wbinvl1_vol. 6683 - Ensures the flat_load 6684 has completed 6685 before invalidating 6686 the cache. 6687 6688 3. buffer_wbinvl1_vol 6689 6690 - Must happen before 6691 any following 6692 global/generic 6693 load/load 6694 atomic/atomicrmw. 6695 - Ensures that 6696 following loads 6697 will not see stale 6698 global data. 6699 6700 load atomic acquire - system - generic 1. flat_load glc=1 6701 2. s_waitcnt vmcnt(0) & 6702 lgkmcnt(0) 6703 6704 - If TgSplit execution mode, 6705 omit lgkmcnt(0). 6706 - If OpenCL omit 6707 lgkmcnt(0). 6708 - Must happen before 6709 following 6710 buffer_invl2 and 6711 buffer_wbinvl1_vol. 6712 - Ensures the flat_load 6713 has completed 6714 before invalidating 6715 the caches. 6716 6717 3. buffer_invl2; 6718 buffer_wbinvl1_vol 6719 6720 - Must happen before 6721 any following 6722 global/generic 6723 load/load 6724 atomic/atomicrmw. 6725 - Ensures that 6726 following 6727 loads will not see 6728 stale L1 global data, 6729 nor see stale L2 MTYPE 6730 NC global data. 6731 MTYPE RW and CC memory will 6732 never be stale in L2 due to 6733 the memory probes. 6734 6735 atomicrmw acquire - singlethread - global 1. buffer/global/flat_atomic 6736 - wavefront - generic 6737 atomicrmw acquire - singlethread - local *If TgSplit execution mode, 6738 - wavefront local address space cannot 6739 be used.* 6740 6741 1. ds_atomic 6742 atomicrmw acquire - workgroup - global 1. buffer/global_atomic 6743 2. s_waitcnt vmcnt(0) 6744 6745 - If not TgSplit execution 6746 mode, omit. 6747 - Must happen before the 6748 following buffer_wbinvl1_vol. 6749 - Ensures the atomicrmw 6750 has completed 6751 before invalidating 6752 the cache. 6753 6754 3. buffer_wbinvl1_vol 6755 6756 - If not TgSplit execution 6757 mode, omit. 6758 - Must happen before 6759 any following 6760 global/generic 6761 load/load 6762 atomic/atomicrmw. 6763 - Ensures that 6764 following loads 6765 will not see stale 6766 global data. 6767 6768 atomicrmw acquire - workgroup - local *If TgSplit execution mode, 6769 local address space cannot 6770 be used.* 6771 6772 1. ds_atomic 6773 2. s_waitcnt lgkmcnt(0) 6774 6775 - If OpenCL, omit. 6776 - Must happen before 6777 any following 6778 global/generic 6779 load/load 6780 atomic/store/store 6781 atomic/atomicrmw. 6782 - Ensures any 6783 following global 6784 data read is no 6785 older than the local 6786 atomicrmw value 6787 being acquired. 6788 6789 atomicrmw acquire - workgroup - generic 1. flat_atomic 6790 2. s_waitcnt lgkm/vmcnt(0) 6791 6792 - Use lgkmcnt(0) if not 6793 TgSplit execution mode 6794 and vmcnt(0) if TgSplit 6795 execution mode. 6796 - If OpenCL, omit lgkmcnt(0). 6797 - Must happen before 6798 the following 6799 buffer_wbinvl1_vol and 6800 any following 6801 global/generic 6802 load/load 6803 atomic/store/store 6804 atomic/atomicrmw. 6805 - Ensures any 6806 following global 6807 data read is no 6808 older than a local 6809 atomicrmw value 6810 being acquired. 6811 6812 3. buffer_wbinvl1_vol 6813 6814 - If not TgSplit execution 6815 mode, omit. 6816 - Ensures that 6817 following 6818 loads will not see 6819 stale data. 6820 6821 atomicrmw acquire - agent - global 1. buffer/global_atomic 6822 2. s_waitcnt vmcnt(0) 6823 6824 - Must happen before 6825 following 6826 buffer_wbinvl1_vol. 6827 - Ensures the 6828 atomicrmw has 6829 completed before 6830 invalidating the 6831 cache. 6832 6833 3. buffer_wbinvl1_vol 6834 6835 - Must happen before 6836 any following 6837 global/generic 6838 load/load 6839 atomic/atomicrmw. 6840 - Ensures that 6841 following loads 6842 will not see stale 6843 global data. 6844 6845 atomicrmw acquire - system - global 1. buffer/global_atomic 6846 2. s_waitcnt vmcnt(0) 6847 6848 - Must happen before 6849 following buffer_invl2 and 6850 buffer_wbinvl1_vol. 6851 - Ensures the 6852 atomicrmw has 6853 completed before 6854 invalidating the 6855 caches. 6856 6857 3. buffer_invl2; 6858 buffer_wbinvl1_vol 6859 6860 - Must happen before 6861 any following 6862 global/generic 6863 load/load 6864 atomic/atomicrmw. 6865 - Ensures that 6866 following 6867 loads will not see 6868 stale L1 global data, 6869 nor see stale L2 MTYPE 6870 NC global data. 6871 MTYPE RW and CC memory will 6872 never be stale in L2 due to 6873 the memory probes. 6874 6875 atomicrmw acquire - agent - generic 1. flat_atomic 6876 2. s_waitcnt vmcnt(0) & 6877 lgkmcnt(0) 6878 6879 - If TgSplit execution mode, 6880 omit lgkmcnt(0). 6881 - If OpenCL, omit 6882 lgkmcnt(0). 6883 - Must happen before 6884 following 6885 buffer_wbinvl1_vol. 6886 - Ensures the 6887 atomicrmw has 6888 completed before 6889 invalidating the 6890 cache. 6891 6892 3. buffer_wbinvl1_vol 6893 6894 - Must happen before 6895 any following 6896 global/generic 6897 load/load 6898 atomic/atomicrmw. 6899 - Ensures that 6900 following loads 6901 will not see stale 6902 global data. 6903 6904 atomicrmw acquire - system - generic 1. flat_atomic 6905 2. s_waitcnt vmcnt(0) & 6906 lgkmcnt(0) 6907 6908 - If TgSplit execution mode, 6909 omit lgkmcnt(0). 6910 - If OpenCL, omit 6911 lgkmcnt(0). 6912 - Must happen before 6913 following 6914 buffer_invl2 and 6915 buffer_wbinvl1_vol. 6916 - Ensures the 6917 atomicrmw has 6918 completed before 6919 invalidating the 6920 caches. 6921 6922 3. buffer_invl2; 6923 buffer_wbinvl1_vol 6924 6925 - Must happen before 6926 any following 6927 global/generic 6928 load/load 6929 atomic/atomicrmw. 6930 - Ensures that 6931 following 6932 loads will not see 6933 stale L1 global data, 6934 nor see stale L2 MTYPE 6935 NC global data. 6936 MTYPE RW and CC memory will 6937 never be stale in L2 due to 6938 the memory probes. 6939 6940 fence acquire - singlethread *none* *none* 6941 - wavefront 6942 fence acquire - workgroup *none* 1. s_waitcnt lgkm/vmcnt(0) 6943 6944 - Use lgkmcnt(0) if not 6945 TgSplit execution mode 6946 and vmcnt(0) if TgSplit 6947 execution mode. 6948 - If OpenCL and 6949 address space is 6950 not generic, omit 6951 lgkmcnt(0). 6952 - If OpenCL and 6953 address space is 6954 local, omit 6955 vmcnt(0). 6956 - However, since LLVM 6957 currently has no 6958 address space on 6959 the fence need to 6960 conservatively 6961 always generate. If 6962 fence had an 6963 address space then 6964 set to address 6965 space of OpenCL 6966 fence flag, or to 6967 generic if both 6968 local and global 6969 flags are 6970 specified. 6971 - s_waitcnt vmcnt(0) 6972 must happen after 6973 any preceding 6974 global/generic load 6975 atomic/ 6976 atomicrmw 6977 with an equal or 6978 wider sync scope 6979 and memory ordering 6980 stronger than 6981 unordered (this is 6982 termed the 6983 fence-paired-atomic). 6984 - s_waitcnt lgkmcnt(0) 6985 must happen after 6986 any preceding 6987 local/generic load 6988 atomic/atomicrmw 6989 with an equal or 6990 wider sync scope 6991 and memory ordering 6992 stronger than 6993 unordered (this is 6994 termed the 6995 fence-paired-atomic). 6996 - Must happen before 6997 the following 6998 buffer_wbinvl1_vol and 6999 any following 7000 global/generic 7001 load/load 7002 atomic/store/store 7003 atomic/atomicrmw. 7004 - Ensures any 7005 following global 7006 data read is no 7007 older than the 7008 value read by the 7009 fence-paired-atomic. 7010 7011 2. buffer_wbinvl1_vol 7012 7013 - If not TgSplit execution 7014 mode, omit. 7015 - Ensures that 7016 following 7017 loads will not see 7018 stale data. 7019 7020 fence acquire - agent *none* 1. s_waitcnt lgkmcnt(0) & 7021 vmcnt(0) 7022 7023 - If TgSplit execution mode, 7024 omit lgkmcnt(0). 7025 - If OpenCL and 7026 address space is 7027 not generic, omit 7028 lgkmcnt(0). 7029 - However, since LLVM 7030 currently has no 7031 address space on 7032 the fence need to 7033 conservatively 7034 always generate 7035 (see comment for 7036 previous fence). 7037 - Could be split into 7038 separate s_waitcnt 7039 vmcnt(0) and 7040 s_waitcnt 7041 lgkmcnt(0) to allow 7042 them to be 7043 independently moved 7044 according to the 7045 following rules. 7046 - s_waitcnt vmcnt(0) 7047 must happen after 7048 any preceding 7049 global/generic load 7050 atomic/atomicrmw 7051 with an equal or 7052 wider sync scope 7053 and memory ordering 7054 stronger than 7055 unordered (this is 7056 termed the 7057 fence-paired-atomic). 7058 - s_waitcnt lgkmcnt(0) 7059 must happen after 7060 any preceding 7061 local/generic load 7062 atomic/atomicrmw 7063 with an equal or 7064 wider sync scope 7065 and memory ordering 7066 stronger than 7067 unordered (this is 7068 termed the 7069 fence-paired-atomic). 7070 - Must happen before 7071 the following 7072 buffer_wbinvl1_vol. 7073 - Ensures that the 7074 fence-paired atomic 7075 has completed 7076 before invalidating 7077 the 7078 cache. Therefore 7079 any following 7080 locations read must 7081 be no older than 7082 the value read by 7083 the 7084 fence-paired-atomic. 7085 7086 2. buffer_wbinvl1_vol 7087 7088 - Must happen before any 7089 following global/generic 7090 load/load 7091 atomic/store/store 7092 atomic/atomicrmw. 7093 - Ensures that 7094 following loads 7095 will not see stale 7096 global data. 7097 7098 fence acquire - system *none* 1. s_waitcnt lgkmcnt(0) & 7099 vmcnt(0) 7100 7101 - If TgSplit execution mode, 7102 omit lgkmcnt(0). 7103 - If OpenCL and 7104 address space is 7105 not generic, omit 7106 lgkmcnt(0). 7107 - However, since LLVM 7108 currently has no 7109 address space on 7110 the fence need to 7111 conservatively 7112 always generate 7113 (see comment for 7114 previous fence). 7115 - Could be split into 7116 separate s_waitcnt 7117 vmcnt(0) and 7118 s_waitcnt 7119 lgkmcnt(0) to allow 7120 them to be 7121 independently moved 7122 according to the 7123 following rules. 7124 - s_waitcnt vmcnt(0) 7125 must happen after 7126 any preceding 7127 global/generic load 7128 atomic/atomicrmw 7129 with an equal or 7130 wider sync scope 7131 and memory ordering 7132 stronger than 7133 unordered (this is 7134 termed the 7135 fence-paired-atomic). 7136 - s_waitcnt lgkmcnt(0) 7137 must happen after 7138 any preceding 7139 local/generic load 7140 atomic/atomicrmw 7141 with an equal or 7142 wider sync scope 7143 and memory ordering 7144 stronger than 7145 unordered (this is 7146 termed the 7147 fence-paired-atomic). 7148 - Must happen before 7149 the following buffer_invl2 and 7150 buffer_wbinvl1_vol. 7151 - Ensures that the 7152 fence-paired atomic 7153 has completed 7154 before invalidating 7155 the 7156 cache. Therefore 7157 any following 7158 locations read must 7159 be no older than 7160 the value read by 7161 the 7162 fence-paired-atomic. 7163 7164 2. buffer_invl2; 7165 buffer_wbinvl1_vol 7166 7167 - Must happen before any 7168 following global/generic 7169 load/load 7170 atomic/store/store 7171 atomic/atomicrmw. 7172 - Ensures that 7173 following 7174 loads will not see 7175 stale L1 global data, 7176 nor see stale L2 MTYPE 7177 NC global data. 7178 MTYPE RW and CC memory will 7179 never be stale in L2 due to 7180 the memory probes. 7181 **Release Atomic** 7182 ------------------------------------------------------------------------------------ 7183 store atomic release - singlethread - global 1. buffer/global/flat_store 7184 - wavefront - generic 7185 store atomic release - singlethread - local *If TgSplit execution mode, 7186 - wavefront local address space cannot 7187 be used.* 7188 7189 1. ds_store 7190 store atomic release - workgroup - global 1. s_waitcnt lgkm/vmcnt(0) 7191 - generic 7192 - Use lgkmcnt(0) if not 7193 TgSplit execution mode 7194 and vmcnt(0) if TgSplit 7195 execution mode. 7196 - If OpenCL, omit lgkmcnt(0). 7197 - s_waitcnt vmcnt(0) 7198 must happen after 7199 any preceding 7200 global/generic load/store/ 7201 load atomic/store atomic/ 7202 atomicrmw. 7203 - s_waitcnt lgkmcnt(0) 7204 must happen after 7205 any preceding 7206 local/generic 7207 load/store/load 7208 atomic/store 7209 atomic/atomicrmw. 7210 - Must happen before 7211 the following 7212 store. 7213 - Ensures that all 7214 memory operations 7215 have 7216 completed before 7217 performing the 7218 store that is being 7219 released. 7220 7221 2. buffer/global/flat_store 7222 store atomic release - workgroup - local *If TgSplit execution mode, 7223 local address space cannot 7224 be used.* 7225 7226 1. ds_store 7227 store atomic release - agent - global 1. s_waitcnt lgkmcnt(0) & 7228 - generic vmcnt(0) 7229 7230 - If TgSplit execution mode, 7231 omit lgkmcnt(0). 7232 - If OpenCL and 7233 address space is 7234 not generic, omit 7235 lgkmcnt(0). 7236 - Could be split into 7237 separate s_waitcnt 7238 vmcnt(0) and 7239 s_waitcnt 7240 lgkmcnt(0) to allow 7241 them to be 7242 independently moved 7243 according to the 7244 following rules. 7245 - s_waitcnt vmcnt(0) 7246 must happen after 7247 any preceding 7248 global/generic 7249 load/store/load 7250 atomic/store 7251 atomic/atomicrmw. 7252 - s_waitcnt lgkmcnt(0) 7253 must happen after 7254 any preceding 7255 local/generic 7256 load/store/load 7257 atomic/store 7258 atomic/atomicrmw. 7259 - Must happen before 7260 the following 7261 store. 7262 - Ensures that all 7263 memory operations 7264 to memory have 7265 completed before 7266 performing the 7267 store that is being 7268 released. 7269 7270 2. buffer/global/flat_store 7271 store atomic release - system - global 1. buffer_wbl2 7272 - generic 7273 - Must happen before 7274 following s_waitcnt. 7275 - Performs L2 writeback to 7276 ensure previous 7277 global/generic 7278 store/atomicrmw are 7279 visible at system scope. 7280 7281 2. s_waitcnt lgkmcnt(0) & 7282 vmcnt(0) 7283 7284 - If TgSplit execution mode, 7285 omit lgkmcnt(0). 7286 - If OpenCL and 7287 address space is 7288 not generic, omit 7289 lgkmcnt(0). 7290 - Could be split into 7291 separate s_waitcnt 7292 vmcnt(0) and 7293 s_waitcnt 7294 lgkmcnt(0) to allow 7295 them to be 7296 independently moved 7297 according to the 7298 following rules. 7299 - s_waitcnt vmcnt(0) 7300 must happen after any 7301 preceding 7302 global/generic 7303 load/store/load 7304 atomic/store 7305 atomic/atomicrmw. 7306 - s_waitcnt lgkmcnt(0) 7307 must happen after any 7308 preceding 7309 local/generic 7310 load/store/load 7311 atomic/store 7312 atomic/atomicrmw. 7313 - Must happen before 7314 the following 7315 store. 7316 - Ensures that all 7317 memory operations 7318 to memory and the L2 7319 writeback have 7320 completed before 7321 performing the 7322 store that is being 7323 released. 7324 7325 3. buffer/global/flat_store 7326 atomicrmw release - singlethread - global 1. buffer/global/flat_atomic 7327 - wavefront - generic 7328 atomicrmw release - singlethread - local *If TgSplit execution mode, 7329 - wavefront local address space cannot 7330 be used.* 7331 7332 1. ds_atomic 7333 atomicrmw release - workgroup - global 1. s_waitcnt lgkm/vmcnt(0) 7334 - generic 7335 - Use lgkmcnt(0) if not 7336 TgSplit execution mode 7337 and vmcnt(0) if TgSplit 7338 execution mode. 7339 - If OpenCL, omit 7340 lgkmcnt(0). 7341 - s_waitcnt vmcnt(0) 7342 must happen after 7343 any preceding 7344 global/generic load/store/ 7345 load atomic/store atomic/ 7346 atomicrmw. 7347 - s_waitcnt lgkmcnt(0) 7348 must happen after 7349 any preceding 7350 local/generic 7351 load/store/load 7352 atomic/store 7353 atomic/atomicrmw. 7354 - Must happen before 7355 the following 7356 atomicrmw. 7357 - Ensures that all 7358 memory operations 7359 have 7360 completed before 7361 performing the 7362 atomicrmw that is 7363 being released. 7364 7365 2. buffer/global/flat_atomic 7366 atomicrmw release - workgroup - local *If TgSplit execution mode, 7367 local address space cannot 7368 be used.* 7369 7370 1. ds_atomic 7371 atomicrmw release - agent - global 1. s_waitcnt lgkmcnt(0) & 7372 - generic vmcnt(0) 7373 7374 - If TgSplit execution mode, 7375 omit lgkmcnt(0). 7376 - If OpenCL, omit 7377 lgkmcnt(0). 7378 - Could be split into 7379 separate s_waitcnt 7380 vmcnt(0) and 7381 s_waitcnt 7382 lgkmcnt(0) to allow 7383 them to be 7384 independently moved 7385 according to the 7386 following rules. 7387 - s_waitcnt vmcnt(0) 7388 must happen after 7389 any preceding 7390 global/generic 7391 load/store/load 7392 atomic/store 7393 atomic/atomicrmw. 7394 - s_waitcnt lgkmcnt(0) 7395 must happen after 7396 any preceding 7397 local/generic 7398 load/store/load 7399 atomic/store 7400 atomic/atomicrmw. 7401 - Must happen before 7402 the following 7403 atomicrmw. 7404 - Ensures that all 7405 memory operations 7406 to global and local 7407 have completed 7408 before performing 7409 the atomicrmw that 7410 is being released. 7411 7412 2. buffer/global/flat_atomic 7413 atomicrmw release - system - global 1. buffer_wbl2 7414 - generic 7415 - Must happen before 7416 following s_waitcnt. 7417 - Performs L2 writeback to 7418 ensure previous 7419 global/generic 7420 store/atomicrmw are 7421 visible at system scope. 7422 7423 2. s_waitcnt lgkmcnt(0) & 7424 vmcnt(0) 7425 7426 - If TgSplit execution mode, 7427 omit lgkmcnt(0). 7428 - If OpenCL, omit 7429 lgkmcnt(0). 7430 - Could be split into 7431 separate s_waitcnt 7432 vmcnt(0) and 7433 s_waitcnt 7434 lgkmcnt(0) to allow 7435 them to be 7436 independently moved 7437 according to the 7438 following rules. 7439 - s_waitcnt vmcnt(0) 7440 must happen after 7441 any preceding 7442 global/generic 7443 load/store/load 7444 atomic/store 7445 atomic/atomicrmw. 7446 - s_waitcnt lgkmcnt(0) 7447 must happen after 7448 any preceding 7449 local/generic 7450 load/store/load 7451 atomic/store 7452 atomic/atomicrmw. 7453 - Must happen before 7454 the following 7455 atomicrmw. 7456 - Ensures that all 7457 memory operations 7458 to memory and the L2 7459 writeback have 7460 completed before 7461 performing the 7462 store that is being 7463 released. 7464 7465 3. buffer/global/flat_atomic 7466 fence release - singlethread *none* *none* 7467 - wavefront 7468 fence release - workgroup *none* 1. s_waitcnt lgkm/vmcnt(0) 7469 7470 - Use lgkmcnt(0) if not 7471 TgSplit execution mode 7472 and vmcnt(0) if TgSplit 7473 execution mode. 7474 - If OpenCL and 7475 address space is 7476 not generic, omit 7477 lgkmcnt(0). 7478 - If OpenCL and 7479 address space is 7480 local, omit 7481 vmcnt(0). 7482 - However, since LLVM 7483 currently has no 7484 address space on 7485 the fence need to 7486 conservatively 7487 always generate. If 7488 fence had an 7489 address space then 7490 set to address 7491 space of OpenCL 7492 fence flag, or to 7493 generic if both 7494 local and global 7495 flags are 7496 specified. 7497 - s_waitcnt vmcnt(0) 7498 must happen after 7499 any preceding 7500 global/generic 7501 load/store/ 7502 load atomic/store atomic/ 7503 atomicrmw. 7504 - s_waitcnt lgkmcnt(0) 7505 must happen after 7506 any preceding 7507 local/generic 7508 load/load 7509 atomic/store/store 7510 atomic/atomicrmw. 7511 - Must happen before 7512 any following store 7513 atomic/atomicrmw 7514 with an equal or 7515 wider sync scope 7516 and memory ordering 7517 stronger than 7518 unordered (this is 7519 termed the 7520 fence-paired-atomic). 7521 - Ensures that all 7522 memory operations 7523 have 7524 completed before 7525 performing the 7526 following 7527 fence-paired-atomic. 7528 7529 fence release - agent *none* 1. s_waitcnt lgkmcnt(0) & 7530 vmcnt(0) 7531 7532 - If TgSplit execution mode, 7533 omit lgkmcnt(0). 7534 - If OpenCL and 7535 address space is 7536 not generic, omit 7537 lgkmcnt(0). 7538 - If OpenCL and 7539 address space is 7540 local, omit 7541 vmcnt(0). 7542 - However, since LLVM 7543 currently has no 7544 address space on 7545 the fence need to 7546 conservatively 7547 always generate. If 7548 fence had an 7549 address space then 7550 set to address 7551 space of OpenCL 7552 fence flag, or to 7553 generic if both 7554 local and global 7555 flags are 7556 specified. 7557 - Could be split into 7558 separate s_waitcnt 7559 vmcnt(0) and 7560 s_waitcnt 7561 lgkmcnt(0) to allow 7562 them to be 7563 independently moved 7564 according to the 7565 following rules. 7566 - s_waitcnt vmcnt(0) 7567 must happen after 7568 any preceding 7569 global/generic 7570 load/store/load 7571 atomic/store 7572 atomic/atomicrmw. 7573 - s_waitcnt lgkmcnt(0) 7574 must happen after 7575 any preceding 7576 local/generic 7577 load/store/load 7578 atomic/store 7579 atomic/atomicrmw. 7580 - Must happen before 7581 any following store 7582 atomic/atomicrmw 7583 with an equal or 7584 wider sync scope 7585 and memory ordering 7586 stronger than 7587 unordered (this is 7588 termed the 7589 fence-paired-atomic). 7590 - Ensures that all 7591 memory operations 7592 have 7593 completed before 7594 performing the 7595 following 7596 fence-paired-atomic. 7597 7598 fence release - system *none* 1. buffer_wbl2 7599 7600 - If OpenCL and 7601 address space is 7602 local, omit. 7603 - Must happen before 7604 following s_waitcnt. 7605 - Performs L2 writeback to 7606 ensure previous 7607 global/generic 7608 store/atomicrmw are 7609 visible at system scope. 7610 7611 2. s_waitcnt lgkmcnt(0) & 7612 vmcnt(0) 7613 7614 - If TgSplit execution mode, 7615 omit lgkmcnt(0). 7616 - If OpenCL and 7617 address space is 7618 not generic, omit 7619 lgkmcnt(0). 7620 - If OpenCL and 7621 address space is 7622 local, omit 7623 vmcnt(0). 7624 - However, since LLVM 7625 currently has no 7626 address space on 7627 the fence need to 7628 conservatively 7629 always generate. If 7630 fence had an 7631 address space then 7632 set to address 7633 space of OpenCL 7634 fence flag, or to 7635 generic if both 7636 local and global 7637 flags are 7638 specified. 7639 - Could be split into 7640 separate s_waitcnt 7641 vmcnt(0) and 7642 s_waitcnt 7643 lgkmcnt(0) to allow 7644 them to be 7645 independently moved 7646 according to the 7647 following rules. 7648 - s_waitcnt vmcnt(0) 7649 must happen after 7650 any preceding 7651 global/generic 7652 load/store/load 7653 atomic/store 7654 atomic/atomicrmw. 7655 - s_waitcnt lgkmcnt(0) 7656 must happen after 7657 any preceding 7658 local/generic 7659 load/store/load 7660 atomic/store 7661 atomic/atomicrmw. 7662 - Must happen before 7663 any following store 7664 atomic/atomicrmw 7665 with an equal or 7666 wider sync scope 7667 and memory ordering 7668 stronger than 7669 unordered (this is 7670 termed the 7671 fence-paired-atomic). 7672 - Ensures that all 7673 memory operations 7674 have 7675 completed before 7676 performing the 7677 following 7678 fence-paired-atomic. 7679 7680 **Acquire-Release Atomic** 7681 ------------------------------------------------------------------------------------ 7682 atomicrmw acq_rel - singlethread - global 1. buffer/global/flat_atomic 7683 - wavefront - generic 7684 atomicrmw acq_rel - singlethread - local *If TgSplit execution mode, 7685 - wavefront local address space cannot 7686 be used.* 7687 7688 1. ds_atomic 7689 atomicrmw acq_rel - workgroup - global 1. s_waitcnt lgkm/vmcnt(0) 7690 7691 - Use lgkmcnt(0) if not 7692 TgSplit execution mode 7693 and vmcnt(0) if TgSplit 7694 execution mode. 7695 - If OpenCL, omit 7696 lgkmcnt(0). 7697 - Must happen after 7698 any preceding 7699 local/generic 7700 load/store/load 7701 atomic/store 7702 atomic/atomicrmw. 7703 - s_waitcnt vmcnt(0) 7704 must happen after 7705 any preceding 7706 global/generic load/store/ 7707 load atomic/store atomic/ 7708 atomicrmw. 7709 - s_waitcnt lgkmcnt(0) 7710 must happen after 7711 any preceding 7712 local/generic 7713 load/store/load 7714 atomic/store 7715 atomic/atomicrmw. 7716 - Must happen before 7717 the following 7718 atomicrmw. 7719 - Ensures that all 7720 memory operations 7721 have 7722 completed before 7723 performing the 7724 atomicrmw that is 7725 being released. 7726 7727 2. buffer/global_atomic 7728 3. s_waitcnt vmcnt(0) 7729 7730 - If not TgSplit execution 7731 mode, omit. 7732 - Must happen before 7733 the following 7734 buffer_wbinvl1_vol. 7735 - Ensures any 7736 following global 7737 data read is no 7738 older than the 7739 atomicrmw value 7740 being acquired. 7741 7742 4. buffer_wbinvl1_vol 7743 7744 - If not TgSplit execution 7745 mode, omit. 7746 - Ensures that 7747 following 7748 loads will not see 7749 stale data. 7750 7751 atomicrmw acq_rel - workgroup - local *If TgSplit execution mode, 7752 local address space cannot 7753 be used.* 7754 7755 1. ds_atomic 7756 2. s_waitcnt lgkmcnt(0) 7757 7758 - If OpenCL, omit. 7759 - Must happen before 7760 any following 7761 global/generic 7762 load/load 7763 atomic/store/store 7764 atomic/atomicrmw. 7765 - Ensures any 7766 following global 7767 data read is no 7768 older than the local load 7769 atomic value being 7770 acquired. 7771 7772 atomicrmw acq_rel - workgroup - generic 1. s_waitcnt lgkm/vmcnt(0) 7773 7774 - Use lgkmcnt(0) if not 7775 TgSplit execution mode 7776 and vmcnt(0) if TgSplit 7777 execution mode. 7778 - If OpenCL, omit 7779 lgkmcnt(0). 7780 - s_waitcnt vmcnt(0) 7781 must happen after 7782 any preceding 7783 global/generic load/store/ 7784 load atomic/store atomic/ 7785 atomicrmw. 7786 - s_waitcnt lgkmcnt(0) 7787 must happen after 7788 any preceding 7789 local/generic 7790 load/store/load 7791 atomic/store 7792 atomic/atomicrmw. 7793 - Must happen before 7794 the following 7795 atomicrmw. 7796 - Ensures that all 7797 memory operations 7798 have 7799 completed before 7800 performing the 7801 atomicrmw that is 7802 being released. 7803 7804 2. flat_atomic 7805 3. s_waitcnt lgkmcnt(0) & 7806 vmcnt(0) 7807 7808 - If not TgSplit execution 7809 mode, omit vmcnt(0). 7810 - If OpenCL, omit 7811 lgkmcnt(0). 7812 - Must happen before 7813 the following 7814 buffer_wbinvl1_vol and 7815 any following 7816 global/generic 7817 load/load 7818 atomic/store/store 7819 atomic/atomicrmw. 7820 - Ensures any 7821 following global 7822 data read is no 7823 older than a local load 7824 atomic value being 7825 acquired. 7826 7827 3. buffer_wbinvl1_vol 7828 7829 - If not TgSplit execution 7830 mode, omit. 7831 - Ensures that 7832 following 7833 loads will not see 7834 stale data. 7835 7836 atomicrmw acq_rel - agent - global 1. s_waitcnt lgkmcnt(0) & 7837 vmcnt(0) 7838 7839 - If TgSplit execution mode, 7840 omit lgkmcnt(0). 7841 - If OpenCL, omit 7842 lgkmcnt(0). 7843 - Could be split into 7844 separate s_waitcnt 7845 vmcnt(0) and 7846 s_waitcnt 7847 lgkmcnt(0) to allow 7848 them to be 7849 independently moved 7850 according to the 7851 following rules. 7852 - s_waitcnt vmcnt(0) 7853 must happen after 7854 any preceding 7855 global/generic 7856 load/store/load 7857 atomic/store 7858 atomic/atomicrmw. 7859 - s_waitcnt lgkmcnt(0) 7860 must happen after 7861 any preceding 7862 local/generic 7863 load/store/load 7864 atomic/store 7865 atomic/atomicrmw. 7866 - Must happen before 7867 the following 7868 atomicrmw. 7869 - Ensures that all 7870 memory operations 7871 to global have 7872 completed before 7873 performing the 7874 atomicrmw that is 7875 being released. 7876 7877 2. buffer/global_atomic 7878 3. s_waitcnt vmcnt(0) 7879 7880 - Must happen before 7881 following 7882 buffer_wbinvl1_vol. 7883 - Ensures the 7884 atomicrmw has 7885 completed before 7886 invalidating the 7887 cache. 7888 7889 4. buffer_wbinvl1_vol 7890 7891 - Must happen before 7892 any following 7893 global/generic 7894 load/load 7895 atomic/atomicrmw. 7896 - Ensures that 7897 following loads 7898 will not see stale 7899 global data. 7900 7901 atomicrmw acq_rel - system - global 1. buffer_wbl2 7902 7903 - Must happen before 7904 following s_waitcnt. 7905 - Performs L2 writeback to 7906 ensure previous 7907 global/generic 7908 store/atomicrmw are 7909 visible at system scope. 7910 7911 2. s_waitcnt lgkmcnt(0) & 7912 vmcnt(0) 7913 7914 - If TgSplit execution mode, 7915 omit lgkmcnt(0). 7916 - If OpenCL, omit 7917 lgkmcnt(0). 7918 - Could be split into 7919 separate s_waitcnt 7920 vmcnt(0) and 7921 s_waitcnt 7922 lgkmcnt(0) to allow 7923 them to be 7924 independently moved 7925 according to the 7926 following rules. 7927 - s_waitcnt vmcnt(0) 7928 must happen after 7929 any preceding 7930 global/generic 7931 load/store/load 7932 atomic/store 7933 atomic/atomicrmw. 7934 - s_waitcnt lgkmcnt(0) 7935 must happen after 7936 any preceding 7937 local/generic 7938 load/store/load 7939 atomic/store 7940 atomic/atomicrmw. 7941 - Must happen before 7942 the following 7943 atomicrmw. 7944 - Ensures that all 7945 memory operations 7946 to global and L2 writeback 7947 have completed before 7948 performing the 7949 atomicrmw that is 7950 being released. 7951 7952 3. buffer/global_atomic 7953 4. s_waitcnt vmcnt(0) 7954 7955 - Must happen before 7956 following buffer_invl2 and 7957 buffer_wbinvl1_vol. 7958 - Ensures the 7959 atomicrmw has 7960 completed before 7961 invalidating the 7962 caches. 7963 7964 5. buffer_invl2; 7965 buffer_wbinvl1_vol 7966 7967 - Must happen before 7968 any following 7969 global/generic 7970 load/load 7971 atomic/atomicrmw. 7972 - Ensures that 7973 following 7974 loads will not see 7975 stale L1 global data, 7976 nor see stale L2 MTYPE 7977 NC global data. 7978 MTYPE RW and CC memory will 7979 never be stale in L2 due to 7980 the memory probes. 7981 7982 atomicrmw acq_rel - agent - generic 1. s_waitcnt lgkmcnt(0) & 7983 vmcnt(0) 7984 7985 - If TgSplit execution mode, 7986 omit lgkmcnt(0). 7987 - If OpenCL, omit 7988 lgkmcnt(0). 7989 - Could be split into 7990 separate s_waitcnt 7991 vmcnt(0) and 7992 s_waitcnt 7993 lgkmcnt(0) to allow 7994 them to be 7995 independently moved 7996 according to the 7997 following rules. 7998 - s_waitcnt vmcnt(0) 7999 must happen after 8000 any preceding 8001 global/generic 8002 load/store/load 8003 atomic/store 8004 atomic/atomicrmw. 8005 - s_waitcnt lgkmcnt(0) 8006 must happen after 8007 any preceding 8008 local/generic 8009 load/store/load 8010 atomic/store 8011 atomic/atomicrmw. 8012 - Must happen before 8013 the following 8014 atomicrmw. 8015 - Ensures that all 8016 memory operations 8017 to global have 8018 completed before 8019 performing the 8020 atomicrmw that is 8021 being released. 8022 8023 2. flat_atomic 8024 3. s_waitcnt vmcnt(0) & 8025 lgkmcnt(0) 8026 8027 - If TgSplit execution mode, 8028 omit lgkmcnt(0). 8029 - If OpenCL, omit 8030 lgkmcnt(0). 8031 - Must happen before 8032 following 8033 buffer_wbinvl1_vol. 8034 - Ensures the 8035 atomicrmw has 8036 completed before 8037 invalidating the 8038 cache. 8039 8040 4. buffer_wbinvl1_vol 8041 8042 - Must happen before 8043 any following 8044 global/generic 8045 load/load 8046 atomic/atomicrmw. 8047 - Ensures that 8048 following loads 8049 will not see stale 8050 global data. 8051 8052 atomicrmw acq_rel - system - generic 1. buffer_wbl2 8053 8054 - Must happen before 8055 following s_waitcnt. 8056 - Performs L2 writeback to 8057 ensure previous 8058 global/generic 8059 store/atomicrmw are 8060 visible at system scope. 8061 8062 2. s_waitcnt lgkmcnt(0) & 8063 vmcnt(0) 8064 8065 - If TgSplit execution mode, 8066 omit lgkmcnt(0). 8067 - If OpenCL, omit 8068 lgkmcnt(0). 8069 - Could be split into 8070 separate s_waitcnt 8071 vmcnt(0) and 8072 s_waitcnt 8073 lgkmcnt(0) to allow 8074 them to be 8075 independently moved 8076 according to the 8077 following rules. 8078 - s_waitcnt vmcnt(0) 8079 must happen after 8080 any preceding 8081 global/generic 8082 load/store/load 8083 atomic/store 8084 atomic/atomicrmw. 8085 - s_waitcnt lgkmcnt(0) 8086 must happen after 8087 any preceding 8088 local/generic 8089 load/store/load 8090 atomic/store 8091 atomic/atomicrmw. 8092 - Must happen before 8093 the following 8094 atomicrmw. 8095 - Ensures that all 8096 memory operations 8097 to global and L2 writeback 8098 have completed before 8099 performing the 8100 atomicrmw that is 8101 being released. 8102 8103 3. flat_atomic 8104 4. s_waitcnt vmcnt(0) & 8105 lgkmcnt(0) 8106 8107 - If TgSplit execution mode, 8108 omit lgkmcnt(0). 8109 - If OpenCL, omit 8110 lgkmcnt(0). 8111 - Must happen before 8112 following buffer_invl2 and 8113 buffer_wbinvl1_vol. 8114 - Ensures the 8115 atomicrmw has 8116 completed before 8117 invalidating the 8118 caches. 8119 8120 5. buffer_invl2; 8121 buffer_wbinvl1_vol 8122 8123 - Must happen before 8124 any following 8125 global/generic 8126 load/load 8127 atomic/atomicrmw. 8128 - Ensures that 8129 following 8130 loads will not see 8131 stale L1 global data, 8132 nor see stale L2 MTYPE 8133 NC global data. 8134 MTYPE RW and CC memory will 8135 never be stale in L2 due to 8136 the memory probes. 8137 8138 fence acq_rel - singlethread *none* *none* 8139 - wavefront 8140 fence acq_rel - workgroup *none* 1. s_waitcnt lgkm/vmcnt(0) 8141 8142 - Use lgkmcnt(0) if not 8143 TgSplit execution mode 8144 and vmcnt(0) if TgSplit 8145 execution mode. 8146 - If OpenCL and 8147 address space is 8148 not generic, omit 8149 lgkmcnt(0). 8150 - If OpenCL and 8151 address space is 8152 local, omit 8153 vmcnt(0). 8154 - However, 8155 since LLVM 8156 currently has no 8157 address space on 8158 the fence need to 8159 conservatively 8160 always generate 8161 (see comment for 8162 previous fence). 8163 - s_waitcnt vmcnt(0) 8164 must happen after 8165 any preceding 8166 global/generic 8167 load/store/ 8168 load atomic/store atomic/ 8169 atomicrmw. 8170 - s_waitcnt lgkmcnt(0) 8171 must happen after 8172 any preceding 8173 local/generic 8174 load/load 8175 atomic/store/store 8176 atomic/atomicrmw. 8177 - Must happen before 8178 any following 8179 global/generic 8180 load/load 8181 atomic/store/store 8182 atomic/atomicrmw. 8183 - Ensures that all 8184 memory operations 8185 have 8186 completed before 8187 performing any 8188 following global 8189 memory operations. 8190 - Ensures that the 8191 preceding 8192 local/generic load 8193 atomic/atomicrmw 8194 with an equal or 8195 wider sync scope 8196 and memory ordering 8197 stronger than 8198 unordered (this is 8199 termed the 8200 acquire-fence-paired-atomic) 8201 has completed 8202 before following 8203 global memory 8204 operations. This 8205 satisfies the 8206 requirements of 8207 acquire. 8208 - Ensures that all 8209 previous memory 8210 operations have 8211 completed before a 8212 following 8213 local/generic store 8214 atomic/atomicrmw 8215 with an equal or 8216 wider sync scope 8217 and memory ordering 8218 stronger than 8219 unordered (this is 8220 termed the 8221 release-fence-paired-atomic). 8222 This satisfies the 8223 requirements of 8224 release. 8225 - Must happen before 8226 the following 8227 buffer_wbinvl1_vol. 8228 - Ensures that the 8229 acquire-fence-paired 8230 atomic has completed 8231 before invalidating 8232 the 8233 cache. Therefore 8234 any following 8235 locations read must 8236 be no older than 8237 the value read by 8238 the 8239 acquire-fence-paired-atomic. 8240 8241 2. buffer_wbinvl1_vol 8242 8243 - If not TgSplit execution 8244 mode, omit. 8245 - Ensures that 8246 following 8247 loads will not see 8248 stale data. 8249 8250 fence acq_rel - agent *none* 1. s_waitcnt lgkmcnt(0) & 8251 vmcnt(0) 8252 8253 - If TgSplit execution mode, 8254 omit lgkmcnt(0). 8255 - If OpenCL and 8256 address space is 8257 not generic, omit 8258 lgkmcnt(0). 8259 - However, since LLVM 8260 currently has no 8261 address space on 8262 the fence need to 8263 conservatively 8264 always generate 8265 (see comment for 8266 previous fence). 8267 - Could be split into 8268 separate s_waitcnt 8269 vmcnt(0) and 8270 s_waitcnt 8271 lgkmcnt(0) to allow 8272 them to be 8273 independently moved 8274 according to the 8275 following rules. 8276 - s_waitcnt vmcnt(0) 8277 must happen after 8278 any preceding 8279 global/generic 8280 load/store/load 8281 atomic/store 8282 atomic/atomicrmw. 8283 - s_waitcnt lgkmcnt(0) 8284 must happen after 8285 any preceding 8286 local/generic 8287 load/store/load 8288 atomic/store 8289 atomic/atomicrmw. 8290 - Must happen before 8291 the following 8292 buffer_wbinvl1_vol. 8293 - Ensures that the 8294 preceding 8295 global/local/generic 8296 load 8297 atomic/atomicrmw 8298 with an equal or 8299 wider sync scope 8300 and memory ordering 8301 stronger than 8302 unordered (this is 8303 termed the 8304 acquire-fence-paired-atomic) 8305 has completed 8306 before invalidating 8307 the cache. This 8308 satisfies the 8309 requirements of 8310 acquire. 8311 - Ensures that all 8312 previous memory 8313 operations have 8314 completed before a 8315 following 8316 global/local/generic 8317 store 8318 atomic/atomicrmw 8319 with an equal or 8320 wider sync scope 8321 and memory ordering 8322 stronger than 8323 unordered (this is 8324 termed the 8325 release-fence-paired-atomic). 8326 This satisfies the 8327 requirements of 8328 release. 8329 8330 2. buffer_wbinvl1_vol 8331 8332 - Must happen before 8333 any following 8334 global/generic 8335 load/load 8336 atomic/store/store 8337 atomic/atomicrmw. 8338 - Ensures that 8339 following loads 8340 will not see stale 8341 global data. This 8342 satisfies the 8343 requirements of 8344 acquire. 8345 8346 fence acq_rel - system *none* 1. buffer_wbl2 8347 8348 - If OpenCL and 8349 address space is 8350 local, omit. 8351 - Must happen before 8352 following s_waitcnt. 8353 - Performs L2 writeback to 8354 ensure previous 8355 global/generic 8356 store/atomicrmw are 8357 visible at system scope. 8358 8359 2. s_waitcnt lgkmcnt(0) & 8360 vmcnt(0) 8361 8362 - If TgSplit execution mode, 8363 omit lgkmcnt(0). 8364 - If OpenCL and 8365 address space is 8366 not generic, omit 8367 lgkmcnt(0). 8368 - However, since LLVM 8369 currently has no 8370 address space on 8371 the fence need to 8372 conservatively 8373 always generate 8374 (see comment for 8375 previous fence). 8376 - Could be split into 8377 separate s_waitcnt 8378 vmcnt(0) and 8379 s_waitcnt 8380 lgkmcnt(0) to allow 8381 them to be 8382 independently moved 8383 according to the 8384 following rules. 8385 - s_waitcnt vmcnt(0) 8386 must happen after 8387 any preceding 8388 global/generic 8389 load/store/load 8390 atomic/store 8391 atomic/atomicrmw. 8392 - s_waitcnt lgkmcnt(0) 8393 must happen after 8394 any preceding 8395 local/generic 8396 load/store/load 8397 atomic/store 8398 atomic/atomicrmw. 8399 - Must happen before 8400 the following buffer_invl2 and 8401 buffer_wbinvl1_vol. 8402 - Ensures that the 8403 preceding 8404 global/local/generic 8405 load 8406 atomic/atomicrmw 8407 with an equal or 8408 wider sync scope 8409 and memory ordering 8410 stronger than 8411 unordered (this is 8412 termed the 8413 acquire-fence-paired-atomic) 8414 has completed 8415 before invalidating 8416 the cache. This 8417 satisfies the 8418 requirements of 8419 acquire. 8420 - Ensures that all 8421 previous memory 8422 operations have 8423 completed before a 8424 following 8425 global/local/generic 8426 store 8427 atomic/atomicrmw 8428 with an equal or 8429 wider sync scope 8430 and memory ordering 8431 stronger than 8432 unordered (this is 8433 termed the 8434 release-fence-paired-atomic). 8435 This satisfies the 8436 requirements of 8437 release. 8438 8439 3. buffer_invl2; 8440 buffer_wbinvl1_vol 8441 8442 - Must happen before 8443 any following 8444 global/generic 8445 load/load 8446 atomic/store/store 8447 atomic/atomicrmw. 8448 - Ensures that 8449 following 8450 loads will not see 8451 stale L1 global data, 8452 nor see stale L2 MTYPE 8453 NC global data. 8454 MTYPE RW and CC memory will 8455 never be stale in L2 due to 8456 the memory probes. 8457 8458 **Sequential Consistent Atomic** 8459 ------------------------------------------------------------------------------------ 8460 load atomic seq_cst - singlethread - global *Same as corresponding 8461 - wavefront - local load atomic acquire, 8462 - generic except must generate 8463 all instructions even 8464 for OpenCL.* 8465 load atomic seq_cst - workgroup - global 1. s_waitcnt lgkm/vmcnt(0) 8466 - generic 8467 - Use lgkmcnt(0) if not 8468 TgSplit execution mode 8469 and vmcnt(0) if TgSplit 8470 execution mode. 8471 - s_waitcnt lgkmcnt(0) must 8472 happen after 8473 preceding 8474 local/generic load 8475 atomic/store 8476 atomic/atomicrmw 8477 with memory 8478 ordering of seq_cst 8479 and with equal or 8480 wider sync scope. 8481 (Note that seq_cst 8482 fences have their 8483 own s_waitcnt 8484 lgkmcnt(0) and so do 8485 not need to be 8486 considered.) 8487 - s_waitcnt vmcnt(0) 8488 must happen after 8489 preceding 8490 global/generic load 8491 atomic/store 8492 atomic/atomicrmw 8493 with memory 8494 ordering of seq_cst 8495 and with equal or 8496 wider sync scope. 8497 (Note that seq_cst 8498 fences have their 8499 own s_waitcnt 8500 vmcnt(0) and so do 8501 not need to be 8502 considered.) 8503 - Ensures any 8504 preceding 8505 sequential 8506 consistent global/local 8507 memory instructions 8508 have completed 8509 before executing 8510 this sequentially 8511 consistent 8512 instruction. This 8513 prevents reordering 8514 a seq_cst store 8515 followed by a 8516 seq_cst load. (Note 8517 that seq_cst is 8518 stronger than 8519 acquire/release as 8520 the reordering of 8521 load acquire 8522 followed by a store 8523 release is 8524 prevented by the 8525 s_waitcnt of 8526 the release, but 8527 there is nothing 8528 preventing a store 8529 release followed by 8530 load acquire from 8531 completing out of 8532 order. The s_waitcnt 8533 could be placed after 8534 seq_store or before 8535 the seq_load. We 8536 choose the load to 8537 make the s_waitcnt be 8538 as late as possible 8539 so that the store 8540 may have already 8541 completed.) 8542 8543 2. *Following 8544 instructions same as 8545 corresponding load 8546 atomic acquire, 8547 except must generate 8548 all instructions even 8549 for OpenCL.* 8550 load atomic seq_cst - workgroup - local *If TgSplit execution mode, 8551 local address space cannot 8552 be used.* 8553 8554 *Same as corresponding 8555 load atomic acquire, 8556 except must generate 8557 all instructions even 8558 for OpenCL.* 8559 8560 load atomic seq_cst - agent - global 1. s_waitcnt lgkmcnt(0) & 8561 - system - generic vmcnt(0) 8562 8563 - If TgSplit execution mode, 8564 omit lgkmcnt(0). 8565 - Could be split into 8566 separate s_waitcnt 8567 vmcnt(0) 8568 and s_waitcnt 8569 lgkmcnt(0) to allow 8570 them to be 8571 independently moved 8572 according to the 8573 following rules. 8574 - s_waitcnt lgkmcnt(0) 8575 must happen after 8576 preceding 8577 global/generic load 8578 atomic/store 8579 atomic/atomicrmw 8580 with memory 8581 ordering of seq_cst 8582 and with equal or 8583 wider sync scope. 8584 (Note that seq_cst 8585 fences have their 8586 own s_waitcnt 8587 lgkmcnt(0) and so do 8588 not need to be 8589 considered.) 8590 - s_waitcnt vmcnt(0) 8591 must happen after 8592 preceding 8593 global/generic load 8594 atomic/store 8595 atomic/atomicrmw 8596 with memory 8597 ordering of seq_cst 8598 and with equal or 8599 wider sync scope. 8600 (Note that seq_cst 8601 fences have their 8602 own s_waitcnt 8603 vmcnt(0) and so do 8604 not need to be 8605 considered.) 8606 - Ensures any 8607 preceding 8608 sequential 8609 consistent global 8610 memory instructions 8611 have completed 8612 before executing 8613 this sequentially 8614 consistent 8615 instruction. This 8616 prevents reordering 8617 a seq_cst store 8618 followed by a 8619 seq_cst load. (Note 8620 that seq_cst is 8621 stronger than 8622 acquire/release as 8623 the reordering of 8624 load acquire 8625 followed by a store 8626 release is 8627 prevented by the 8628 s_waitcnt of 8629 the release, but 8630 there is nothing 8631 preventing a store 8632 release followed by 8633 load acquire from 8634 completing out of 8635 order. The s_waitcnt 8636 could be placed after 8637 seq_store or before 8638 the seq_load. We 8639 choose the load to 8640 make the s_waitcnt be 8641 as late as possible 8642 so that the store 8643 may have already 8644 completed.) 8645 8646 2. *Following 8647 instructions same as 8648 corresponding load 8649 atomic acquire, 8650 except must generate 8651 all instructions even 8652 for OpenCL.* 8653 store atomic seq_cst - singlethread - global *Same as corresponding 8654 - wavefront - local store atomic release, 8655 - workgroup - generic except must generate 8656 - agent all instructions even 8657 - system for OpenCL.* 8658 atomicrmw seq_cst - singlethread - global *Same as corresponding 8659 - wavefront - local atomicrmw acq_rel, 8660 - workgroup - generic except must generate 8661 - agent all instructions even 8662 - system for OpenCL.* 8663 fence seq_cst - singlethread *none* *Same as corresponding 8664 - wavefront fence acq_rel, 8665 - workgroup except must generate 8666 - agent all instructions even 8667 - system for OpenCL.* 8668 ============ ============ ============== ========== ================================ 8669 8670.. _amdgpu-amdhsa-memory-model-gfx10: 8671 8672Memory Model GFX10 8673++++++++++++++++++ 8674 8675For GFX10: 8676 8677* Each agent has multiple shader arrays (SA). 8678* Each SA has multiple work-group processors (WGP). 8679* Each WGP has multiple compute units (CU). 8680* Each CU has multiple SIMDs that execute wavefronts. 8681* The wavefronts for a single work-group are executed in the same 8682 WGP. In CU wavefront execution mode the wavefronts may be executed by 8683 different SIMDs in the same CU. In WGP wavefront execution mode the 8684 wavefronts may be executed by different SIMDs in different CUs in the same 8685 WGP. 8686* Each WGP has a single LDS memory shared by the wavefronts of the work-groups 8687 executing on it. 8688* All LDS operations of a WGP are performed as wavefront wide operations in a 8689 global order and involve no caching. Completion is reported to a wavefront in 8690 execution order. 8691* The LDS memory has multiple request queues shared by the SIMDs of a 8692 WGP. Therefore, the LDS operations performed by different wavefronts of a 8693 work-group can be reordered relative to each other, which can result in 8694 reordering the visibility of vector memory operations with respect to LDS 8695 operations of other wavefronts in the same work-group. A ``s_waitcnt 8696 lgkmcnt(0)`` is required to ensure synchronization between LDS operations and 8697 vector memory operations between wavefronts of a work-group, but not between 8698 operations performed by the same wavefront. 8699* The vector memory operations are performed as wavefront wide operations. 8700 Completion of load/store/sample operations are reported to a wavefront in 8701 execution order of other load/store/sample operations performed by that 8702 wavefront. 8703* The vector memory operations access a vector L0 cache. There is a single L0 8704 cache per CU. Each SIMD of a CU accesses the same L0 cache. Therefore, no 8705 special action is required for coherence between the lanes of a single 8706 wavefront. However, a ``buffer_gl0_inv`` is required for coherence between 8707 wavefronts executing in the same work-group as they may be executing on SIMDs 8708 of different CUs that access different L0s. A ``buffer_gl0_inv`` is also 8709 required for coherence between wavefronts executing in different work-groups 8710 as they may be executing on different WGPs. 8711* The scalar memory operations access a scalar L0 cache shared by all wavefronts 8712 on a WGP. The scalar and vector L0 caches are not coherent. However, scalar 8713 operations are used in a restricted way so do not impact the memory model. See 8714 :ref:`amdgpu-amdhsa-memory-spaces`. 8715* The vector and scalar memory L0 caches use an L1 cache shared by all WGPs on 8716 the same SA. Therefore, no special action is required for coherence between 8717 the wavefronts of a single work-group. However, a ``buffer_gl1_inv`` is 8718 required for coherence between wavefronts executing in different work-groups 8719 as they may be executing on different SAs that access different L1s. 8720* The L1 caches have independent quadrants to service disjoint ranges of virtual 8721 addresses. 8722* Each L0 cache has a separate request queue per L1 quadrant. Therefore, the 8723 vector and scalar memory operations performed by different wavefronts, whether 8724 executing in the same or different work-groups (which may be executing on 8725 different CUs accessing different L0s), can be reordered relative to each 8726 other. A ``s_waitcnt vmcnt(0) & vscnt(0)`` is required to ensure 8727 synchronization between vector memory operations of different wavefronts. It 8728 ensures a previous vector memory operation has completed before executing a 8729 subsequent vector memory or LDS operation and so can be used to meet the 8730 requirements of acquire, release and sequential consistency. 8731* The L1 caches use an L2 cache shared by all SAs on the same agent. 8732* The L2 cache has independent channels to service disjoint ranges of virtual 8733 addresses. 8734* Each L1 quadrant of a single SA accesses a different L2 channel. Each L1 8735 quadrant has a separate request queue per L2 channel. Therefore, the vector 8736 and scalar memory operations performed by wavefronts executing in different 8737 work-groups (which may be executing on different SAs) of an agent can be 8738 reordered relative to each other. A ``s_waitcnt vmcnt(0) & vscnt(0)`` is 8739 required to ensure synchronization between vector memory operations of 8740 different SAs. It ensures a previous vector memory operation has completed 8741 before executing a subsequent vector memory and so can be used to meet the 8742 requirements of acquire, release and sequential consistency. 8743* The L2 cache can be kept coherent with other agents on some targets, or ranges 8744 of virtual addresses can be set up to bypass it to ensure system coherence. 8745* On GFX10.3 a memory attached last level (MALL) cache exists for GPU memory. 8746 The MALL cache is fully coherent with GPU memory and has no impact on system 8747 coherence. All agents (GPU and CPU) access GPU memory through the MALL cache. 8748 8749Scalar memory operations are only used to access memory that is proven to not 8750change during the execution of the kernel dispatch. This includes constant 8751address space and global address space for program scope ``const`` variables. 8752Therefore, the kernel machine code does not have to maintain the scalar cache to 8753ensure it is coherent with the vector caches. The scalar and vector caches are 8754invalidated between kernel dispatches by CP since constant address space data 8755may change between kernel dispatch executions. See 8756:ref:`amdgpu-amdhsa-memory-spaces`. 8757 8758The one exception is if scalar writes are used to spill SGPR registers. In this 8759case the AMDGPU backend ensures the memory location used to spill is never 8760accessed by vector memory operations at the same time. If scalar writes are used 8761then a ``s_dcache_wb`` is inserted before the ``s_endpgm`` and before a function 8762return since the locations may be used for vector memory instructions by a 8763future wavefront that uses the same scratch area, or a function call that 8764creates a frame at the same address, respectively. There is no need for a 8765``s_dcache_inv`` as all scalar writes are write-before-read in the same thread. 8766 8767For kernarg backing memory: 8768 8769* CP invalidates the L0 and L1 caches at the start of each kernel dispatch. 8770* On dGPU the kernarg backing memory is accessed as MTYPE UC (uncached) to avoid 8771 needing to invalidate the L2 cache. 8772* On APU the kernarg backing memory is accessed as MTYPE CC (cache coherent) and 8773 so the L2 cache will be coherent with the CPU and other agents. 8774 8775Scratch backing memory (which is used for the private address space) is accessed 8776with MTYPE NC (non-coherent). Since the private address space is only accessed 8777by a single thread, and is always write-before-read, there is never a need to 8778invalidate these entries from the L0 or L1 caches. 8779 8780Wavefronts are executed in native mode with in-order reporting of loads and 8781sample instructions. In this mode vmcnt reports completion of load, atomic with 8782return and sample instructions in order, and the vscnt reports the completion of 8783store and atomic without return in order. See ``MEM_ORDERED`` field in 8784:ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 8785 8786Wavefronts can be executed in WGP or CU wavefront execution mode: 8787 8788* In WGP wavefront execution mode the wavefronts of a work-group are executed 8789 on the SIMDs of both CUs of the WGP. Therefore, explicit management of the per 8790 CU L0 caches is required for work-group synchronization. Also accesses to L1 8791 at work-group scope need to be explicitly ordered as the accesses from 8792 different CUs are not ordered. 8793* In CU wavefront execution mode the wavefronts of a work-group are executed on 8794 the SIMDs of a single CU of the WGP. Therefore, all global memory access by 8795 the work-group access the same L0 which in turn ensures L1 accesses are 8796 ordered and so do not require explicit management of the caches for 8797 work-group synchronization. 8798 8799See ``WGP_MODE`` field in 8800:ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table` and 8801:ref:`amdgpu-target-features`. 8802 8803The code sequences used to implement the memory model for GFX10 are defined in 8804table :ref:`amdgpu-amdhsa-memory-model-code-sequences-gfx10-table`. 8805 8806 .. table:: AMDHSA Memory Model Code Sequences GFX10 8807 :name: amdgpu-amdhsa-memory-model-code-sequences-gfx10-table 8808 8809 ============ ============ ============== ========== ================================ 8810 LLVM Instr LLVM Memory LLVM Memory AMDGPU AMDGPU Machine Code 8811 Ordering Sync Scope Address GFX10 8812 Space 8813 ============ ============ ============== ========== ================================ 8814 **Non-Atomic** 8815 ------------------------------------------------------------------------------------ 8816 load *none* *none* - global - !volatile & !nontemporal 8817 - generic 8818 - private 1. buffer/global/flat_load 8819 - constant 8820 - !volatile & nontemporal 8821 8822 1. buffer/global/flat_load 8823 slc=1 8824 8825 - volatile 8826 8827 1. buffer/global/flat_load 8828 glc=1 dlc=1 8829 2. s_waitcnt vmcnt(0) 8830 8831 - Must happen before 8832 any following volatile 8833 global/generic 8834 load/store. 8835 - Ensures that 8836 volatile 8837 operations to 8838 different 8839 addresses will not 8840 be reordered by 8841 hardware. 8842 8843 load *none* *none* - local 1. ds_load 8844 store *none* *none* - global - !volatile & !nontemporal 8845 - generic 8846 - private 1. buffer/global/flat_store 8847 - constant 8848 - !volatile & nontemporal 8849 8850 1. buffer/global/flat_store 8851 glc=1 slc=1 8852 8853 - volatile 8854 8855 1. buffer/global/flat_store 8856 2. s_waitcnt vscnt(0) 8857 8858 - Must happen before 8859 any following volatile 8860 global/generic 8861 load/store. 8862 - Ensures that 8863 volatile 8864 operations to 8865 different 8866 addresses will not 8867 be reordered by 8868 hardware. 8869 8870 store *none* *none* - local 1. ds_store 8871 **Unordered Atomic** 8872 ------------------------------------------------------------------------------------ 8873 load atomic unordered *any* *any* *Same as non-atomic*. 8874 store atomic unordered *any* *any* *Same as non-atomic*. 8875 atomicrmw unordered *any* *any* *Same as monotonic atomic*. 8876 **Monotonic Atomic** 8877 ------------------------------------------------------------------------------------ 8878 load atomic monotonic - singlethread - global 1. buffer/global/flat_load 8879 - wavefront - generic 8880 load atomic monotonic - workgroup - global 1. buffer/global/flat_load 8881 - generic glc=1 8882 8883 - If CU wavefront execution 8884 mode, omit glc=1. 8885 8886 load atomic monotonic - singlethread - local 1. ds_load 8887 - wavefront 8888 - workgroup 8889 load atomic monotonic - agent - global 1. buffer/global/flat_load 8890 - system - generic glc=1 dlc=1 8891 store atomic monotonic - singlethread - global 1. buffer/global/flat_store 8892 - wavefront - generic 8893 - workgroup 8894 - agent 8895 - system 8896 store atomic monotonic - singlethread - local 1. ds_store 8897 - wavefront 8898 - workgroup 8899 atomicrmw monotonic - singlethread - global 1. buffer/global/flat_atomic 8900 - wavefront - generic 8901 - workgroup 8902 - agent 8903 - system 8904 atomicrmw monotonic - singlethread - local 1. ds_atomic 8905 - wavefront 8906 - workgroup 8907 **Acquire Atomic** 8908 ------------------------------------------------------------------------------------ 8909 load atomic acquire - singlethread - global 1. buffer/global/ds/flat_load 8910 - wavefront - local 8911 - generic 8912 load atomic acquire - workgroup - global 1. buffer/global_load glc=1 8913 8914 - If CU wavefront execution 8915 mode, omit glc=1. 8916 8917 2. s_waitcnt vmcnt(0) 8918 8919 - If CU wavefront execution 8920 mode, omit. 8921 - Must happen before 8922 the following buffer_gl0_inv 8923 and before any following 8924 global/generic 8925 load/load 8926 atomic/store/store 8927 atomic/atomicrmw. 8928 8929 3. buffer_gl0_inv 8930 8931 - If CU wavefront execution 8932 mode, omit. 8933 - Ensures that 8934 following 8935 loads will not see 8936 stale data. 8937 8938 load atomic acquire - workgroup - local 1. ds_load 8939 2. s_waitcnt lgkmcnt(0) 8940 8941 - If OpenCL, omit. 8942 - Must happen before 8943 the following buffer_gl0_inv 8944 and before any following 8945 global/generic load/load 8946 atomic/store/store 8947 atomic/atomicrmw. 8948 - Ensures any 8949 following global 8950 data read is no 8951 older than the local load 8952 atomic value being 8953 acquired. 8954 8955 3. buffer_gl0_inv 8956 8957 - If CU wavefront execution 8958 mode, omit. 8959 - If OpenCL, omit. 8960 - Ensures that 8961 following 8962 loads will not see 8963 stale data. 8964 8965 load atomic acquire - workgroup - generic 1. flat_load glc=1 8966 8967 - If CU wavefront execution 8968 mode, omit glc=1. 8969 8970 2. s_waitcnt lgkmcnt(0) & 8971 vmcnt(0) 8972 8973 - If CU wavefront execution 8974 mode, omit vmcnt(0). 8975 - If OpenCL, omit 8976 lgkmcnt(0). 8977 - Must happen before 8978 the following 8979 buffer_gl0_inv and any 8980 following global/generic 8981 load/load 8982 atomic/store/store 8983 atomic/atomicrmw. 8984 - Ensures any 8985 following global 8986 data read is no 8987 older than a local load 8988 atomic value being 8989 acquired. 8990 8991 3. buffer_gl0_inv 8992 8993 - If CU wavefront execution 8994 mode, omit. 8995 - Ensures that 8996 following 8997 loads will not see 8998 stale data. 8999 9000 load atomic acquire - agent - global 1. buffer/global_load 9001 - system glc=1 dlc=1 9002 2. s_waitcnt vmcnt(0) 9003 9004 - Must happen before 9005 following 9006 buffer_gl*_inv. 9007 - Ensures the load 9008 has completed 9009 before invalidating 9010 the caches. 9011 9012 3. buffer_gl0_inv; 9013 buffer_gl1_inv 9014 9015 - Must happen before 9016 any following 9017 global/generic 9018 load/load 9019 atomic/atomicrmw. 9020 - Ensures that 9021 following 9022 loads will not see 9023 stale global data. 9024 9025 load atomic acquire - agent - generic 1. flat_load glc=1 dlc=1 9026 - system 2. s_waitcnt vmcnt(0) & 9027 lgkmcnt(0) 9028 9029 - If OpenCL omit 9030 lgkmcnt(0). 9031 - Must happen before 9032 following 9033 buffer_gl*_invl. 9034 - Ensures the flat_load 9035 has completed 9036 before invalidating 9037 the caches. 9038 9039 3. buffer_gl0_inv; 9040 buffer_gl1_inv 9041 9042 - Must happen before 9043 any following 9044 global/generic 9045 load/load 9046 atomic/atomicrmw. 9047 - Ensures that 9048 following loads 9049 will not see stale 9050 global data. 9051 9052 atomicrmw acquire - singlethread - global 1. buffer/global/ds/flat_atomic 9053 - wavefront - local 9054 - generic 9055 atomicrmw acquire - workgroup - global 1. buffer/global_atomic 9056 2. s_waitcnt vm/vscnt(0) 9057 9058 - If CU wavefront execution 9059 mode, omit. 9060 - Use vmcnt(0) if atomic with 9061 return and vscnt(0) if 9062 atomic with no-return. 9063 - Must happen before 9064 the following buffer_gl0_inv 9065 and before any following 9066 global/generic 9067 load/load 9068 atomic/store/store 9069 atomic/atomicrmw. 9070 9071 3. buffer_gl0_inv 9072 9073 - If CU wavefront execution 9074 mode, omit. 9075 - Ensures that 9076 following 9077 loads will not see 9078 stale data. 9079 9080 atomicrmw acquire - workgroup - local 1. ds_atomic 9081 2. s_waitcnt lgkmcnt(0) 9082 9083 - If OpenCL, omit. 9084 - Must happen before 9085 the following 9086 buffer_gl0_inv. 9087 - Ensures any 9088 following global 9089 data read is no 9090 older than the local 9091 atomicrmw value 9092 being acquired. 9093 9094 3. buffer_gl0_inv 9095 9096 - If OpenCL omit. 9097 - Ensures that 9098 following 9099 loads will not see 9100 stale data. 9101 9102 atomicrmw acquire - workgroup - generic 1. flat_atomic 9103 2. s_waitcnt lgkmcnt(0) & 9104 vm/vscnt(0) 9105 9106 - If CU wavefront execution 9107 mode, omit vm/vscnt(0). 9108 - If OpenCL, omit lgkmcnt(0). 9109 - Use vmcnt(0) if atomic with 9110 return and vscnt(0) if 9111 atomic with no-return. 9112 - Must happen before 9113 the following 9114 buffer_gl0_inv. 9115 - Ensures any 9116 following global 9117 data read is no 9118 older than a local 9119 atomicrmw value 9120 being acquired. 9121 9122 3. buffer_gl0_inv 9123 9124 - If CU wavefront execution 9125 mode, omit. 9126 - Ensures that 9127 following 9128 loads will not see 9129 stale data. 9130 9131 atomicrmw acquire - agent - global 1. buffer/global_atomic 9132 - system 2. s_waitcnt vm/vscnt(0) 9133 9134 - Use vmcnt(0) if atomic with 9135 return and vscnt(0) if 9136 atomic with no-return. 9137 - Must happen before 9138 following 9139 buffer_gl*_inv. 9140 - Ensures the 9141 atomicrmw has 9142 completed before 9143 invalidating the 9144 caches. 9145 9146 3. buffer_gl0_inv; 9147 buffer_gl1_inv 9148 9149 - Must happen before 9150 any following 9151 global/generic 9152 load/load 9153 atomic/atomicrmw. 9154 - Ensures that 9155 following loads 9156 will not see stale 9157 global data. 9158 9159 atomicrmw acquire - agent - generic 1. flat_atomic 9160 - system 2. s_waitcnt vm/vscnt(0) & 9161 lgkmcnt(0) 9162 9163 - If OpenCL, omit 9164 lgkmcnt(0). 9165 - Use vmcnt(0) if atomic with 9166 return and vscnt(0) if 9167 atomic with no-return. 9168 - Must happen before 9169 following 9170 buffer_gl*_inv. 9171 - Ensures the 9172 atomicrmw has 9173 completed before 9174 invalidating the 9175 caches. 9176 9177 3. buffer_gl0_inv; 9178 buffer_gl1_inv 9179 9180 - Must happen before 9181 any following 9182 global/generic 9183 load/load 9184 atomic/atomicrmw. 9185 - Ensures that 9186 following loads 9187 will not see stale 9188 global data. 9189 9190 fence acquire - singlethread *none* *none* 9191 - wavefront 9192 fence acquire - workgroup *none* 1. s_waitcnt lgkmcnt(0) & 9193 vmcnt(0) & vscnt(0) 9194 9195 - If CU wavefront execution 9196 mode, omit vmcnt(0) and 9197 vscnt(0). 9198 - If OpenCL and 9199 address space is 9200 not generic, omit 9201 lgkmcnt(0). 9202 - If OpenCL and 9203 address space is 9204 local, omit 9205 vmcnt(0) and vscnt(0). 9206 - However, since LLVM 9207 currently has no 9208 address space on 9209 the fence need to 9210 conservatively 9211 always generate. If 9212 fence had an 9213 address space then 9214 set to address 9215 space of OpenCL 9216 fence flag, or to 9217 generic if both 9218 local and global 9219 flags are 9220 specified. 9221 - Could be split into 9222 separate s_waitcnt 9223 vmcnt(0), s_waitcnt 9224 vscnt(0) and s_waitcnt 9225 lgkmcnt(0) to allow 9226 them to be 9227 independently moved 9228 according to the 9229 following rules. 9230 - s_waitcnt vmcnt(0) 9231 must happen after 9232 any preceding 9233 global/generic load 9234 atomic/ 9235 atomicrmw-with-return-value 9236 with an equal or 9237 wider sync scope 9238 and memory ordering 9239 stronger than 9240 unordered (this is 9241 termed the 9242 fence-paired-atomic). 9243 - s_waitcnt vscnt(0) 9244 must happen after 9245 any preceding 9246 global/generic 9247 atomicrmw-no-return-value 9248 with an equal or 9249 wider sync scope 9250 and memory ordering 9251 stronger than 9252 unordered (this is 9253 termed the 9254 fence-paired-atomic). 9255 - s_waitcnt lgkmcnt(0) 9256 must happen after 9257 any preceding 9258 local/generic load 9259 atomic/atomicrmw 9260 with an equal or 9261 wider sync scope 9262 and memory ordering 9263 stronger than 9264 unordered (this is 9265 termed the 9266 fence-paired-atomic). 9267 - Must happen before 9268 the following 9269 buffer_gl0_inv. 9270 - Ensures that the 9271 fence-paired atomic 9272 has completed 9273 before invalidating 9274 the 9275 cache. Therefore 9276 any following 9277 locations read must 9278 be no older than 9279 the value read by 9280 the 9281 fence-paired-atomic. 9282 9283 3. buffer_gl0_inv 9284 9285 - If CU wavefront execution 9286 mode, omit. 9287 - Ensures that 9288 following 9289 loads will not see 9290 stale data. 9291 9292 fence acquire - agent *none* 1. s_waitcnt lgkmcnt(0) & 9293 - system vmcnt(0) & vscnt(0) 9294 9295 - If OpenCL and 9296 address space is 9297 not generic, omit 9298 lgkmcnt(0). 9299 - If OpenCL and 9300 address space is 9301 local, omit 9302 vmcnt(0) and vscnt(0). 9303 - However, since LLVM 9304 currently has no 9305 address space on 9306 the fence need to 9307 conservatively 9308 always generate 9309 (see comment for 9310 previous fence). 9311 - Could be split into 9312 separate s_waitcnt 9313 vmcnt(0), s_waitcnt 9314 vscnt(0) and s_waitcnt 9315 lgkmcnt(0) to allow 9316 them to be 9317 independently moved 9318 according to the 9319 following rules. 9320 - s_waitcnt vmcnt(0) 9321 must happen after 9322 any preceding 9323 global/generic load 9324 atomic/ 9325 atomicrmw-with-return-value 9326 with an equal or 9327 wider sync scope 9328 and memory ordering 9329 stronger than 9330 unordered (this is 9331 termed the 9332 fence-paired-atomic). 9333 - s_waitcnt vscnt(0) 9334 must happen after 9335 any preceding 9336 global/generic 9337 atomicrmw-no-return-value 9338 with an equal or 9339 wider sync scope 9340 and memory ordering 9341 stronger than 9342 unordered (this is 9343 termed the 9344 fence-paired-atomic). 9345 - s_waitcnt lgkmcnt(0) 9346 must happen after 9347 any preceding 9348 local/generic load 9349 atomic/atomicrmw 9350 with an equal or 9351 wider sync scope 9352 and memory ordering 9353 stronger than 9354 unordered (this is 9355 termed the 9356 fence-paired-atomic). 9357 - Must happen before 9358 the following 9359 buffer_gl*_inv. 9360 - Ensures that the 9361 fence-paired atomic 9362 has completed 9363 before invalidating 9364 the 9365 caches. Therefore 9366 any following 9367 locations read must 9368 be no older than 9369 the value read by 9370 the 9371 fence-paired-atomic. 9372 9373 2. buffer_gl0_inv; 9374 buffer_gl1_inv 9375 9376 - Must happen before any 9377 following global/generic 9378 load/load 9379 atomic/store/store 9380 atomic/atomicrmw. 9381 - Ensures that 9382 following loads 9383 will not see stale 9384 global data. 9385 9386 **Release Atomic** 9387 ------------------------------------------------------------------------------------ 9388 store atomic release - singlethread - global 1. buffer/global/ds/flat_store 9389 - wavefront - local 9390 - generic 9391 store atomic release - workgroup - global 1. s_waitcnt lgkmcnt(0) & 9392 - generic vmcnt(0) & vscnt(0) 9393 9394 - If CU wavefront execution 9395 mode, omit vmcnt(0) and 9396 vscnt(0). 9397 - If OpenCL, omit 9398 lgkmcnt(0). 9399 - Could be split into 9400 separate s_waitcnt 9401 vmcnt(0), s_waitcnt 9402 vscnt(0) and s_waitcnt 9403 lgkmcnt(0) to allow 9404 them to be 9405 independently moved 9406 according to the 9407 following rules. 9408 - s_waitcnt vmcnt(0) 9409 must happen after 9410 any preceding 9411 global/generic load/load 9412 atomic/ 9413 atomicrmw-with-return-value. 9414 - s_waitcnt vscnt(0) 9415 must happen after 9416 any preceding 9417 global/generic 9418 store/store 9419 atomic/ 9420 atomicrmw-no-return-value. 9421 - s_waitcnt lgkmcnt(0) 9422 must happen after 9423 any preceding 9424 local/generic 9425 load/store/load 9426 atomic/store 9427 atomic/atomicrmw. 9428 - Must happen before 9429 the following 9430 store. 9431 - Ensures that all 9432 memory operations 9433 have 9434 completed before 9435 performing the 9436 store that is being 9437 released. 9438 9439 2. buffer/global/flat_store 9440 store atomic release - workgroup - local 1. s_waitcnt vmcnt(0) & vscnt(0) 9441 9442 - If CU wavefront execution 9443 mode, omit. 9444 - If OpenCL, omit. 9445 - Could be split into 9446 separate s_waitcnt 9447 vmcnt(0) and s_waitcnt 9448 vscnt(0) to allow 9449 them to be 9450 independently moved 9451 according to the 9452 following rules. 9453 - s_waitcnt vmcnt(0) 9454 must happen after 9455 any preceding 9456 global/generic load/load 9457 atomic/ 9458 atomicrmw-with-return-value. 9459 - s_waitcnt vscnt(0) 9460 must happen after 9461 any preceding 9462 global/generic 9463 store/store atomic/ 9464 atomicrmw-no-return-value. 9465 - Must happen before 9466 the following 9467 store. 9468 - Ensures that all 9469 global memory 9470 operations have 9471 completed before 9472 performing the 9473 store that is being 9474 released. 9475 9476 2. ds_store 9477 store atomic release - agent - global 1. s_waitcnt lgkmcnt(0) & 9478 - system - generic vmcnt(0) & vscnt(0) 9479 9480 - If OpenCL and 9481 address space is 9482 not generic, omit 9483 lgkmcnt(0). 9484 - Could be split into 9485 separate s_waitcnt 9486 vmcnt(0), s_waitcnt vscnt(0) 9487 and s_waitcnt 9488 lgkmcnt(0) to allow 9489 them to be 9490 independently moved 9491 according to the 9492 following rules. 9493 - s_waitcnt vmcnt(0) 9494 must happen after 9495 any preceding 9496 global/generic 9497 load/load 9498 atomic/ 9499 atomicrmw-with-return-value. 9500 - s_waitcnt vscnt(0) 9501 must happen after 9502 any preceding 9503 global/generic 9504 store/store atomic/ 9505 atomicrmw-no-return-value. 9506 - s_waitcnt lgkmcnt(0) 9507 must happen after 9508 any preceding 9509 local/generic 9510 load/store/load 9511 atomic/store 9512 atomic/atomicrmw. 9513 - Must happen before 9514 the following 9515 store. 9516 - Ensures that all 9517 memory operations 9518 have 9519 completed before 9520 performing the 9521 store that is being 9522 released. 9523 9524 2. buffer/global/flat_store 9525 atomicrmw release - singlethread - global 1. buffer/global/ds/flat_atomic 9526 - wavefront - local 9527 - generic 9528 atomicrmw release - workgroup - global 1. s_waitcnt lgkmcnt(0) & 9529 - generic vmcnt(0) & vscnt(0) 9530 9531 - If CU wavefront execution 9532 mode, omit vmcnt(0) and 9533 vscnt(0). 9534 - If OpenCL, omit lgkmcnt(0). 9535 - Could be split into 9536 separate s_waitcnt 9537 vmcnt(0), s_waitcnt 9538 vscnt(0) and s_waitcnt 9539 lgkmcnt(0) to allow 9540 them to be 9541 independently moved 9542 according to the 9543 following rules. 9544 - s_waitcnt vmcnt(0) 9545 must happen after 9546 any preceding 9547 global/generic load/load 9548 atomic/ 9549 atomicrmw-with-return-value. 9550 - s_waitcnt vscnt(0) 9551 must happen after 9552 any preceding 9553 global/generic 9554 store/store 9555 atomic/ 9556 atomicrmw-no-return-value. 9557 - s_waitcnt lgkmcnt(0) 9558 must happen after 9559 any preceding 9560 local/generic 9561 load/store/load 9562 atomic/store 9563 atomic/atomicrmw. 9564 - Must happen before 9565 the following 9566 atomicrmw. 9567 - Ensures that all 9568 memory operations 9569 have 9570 completed before 9571 performing the 9572 atomicrmw that is 9573 being released. 9574 9575 2. buffer/global/flat_atomic 9576 atomicrmw release - workgroup - local 1. s_waitcnt vmcnt(0) & vscnt(0) 9577 9578 - If CU wavefront execution 9579 mode, omit. 9580 - If OpenCL, omit. 9581 - Could be split into 9582 separate s_waitcnt 9583 vmcnt(0) and s_waitcnt 9584 vscnt(0) to allow 9585 them to be 9586 independently moved 9587 according to the 9588 following rules. 9589 - s_waitcnt vmcnt(0) 9590 must happen after 9591 any preceding 9592 global/generic load/load 9593 atomic/ 9594 atomicrmw-with-return-value. 9595 - s_waitcnt vscnt(0) 9596 must happen after 9597 any preceding 9598 global/generic 9599 store/store atomic/ 9600 atomicrmw-no-return-value. 9601 - Must happen before 9602 the following 9603 store. 9604 - Ensures that all 9605 global memory 9606 operations have 9607 completed before 9608 performing the 9609 store that is being 9610 released. 9611 9612 2. ds_atomic 9613 atomicrmw release - agent - global 1. s_waitcnt lgkmcnt(0) & 9614 - system - generic vmcnt(0) & vscnt(0) 9615 9616 - If OpenCL, omit 9617 lgkmcnt(0). 9618 - Could be split into 9619 separate s_waitcnt 9620 vmcnt(0), s_waitcnt 9621 vscnt(0) and s_waitcnt 9622 lgkmcnt(0) to allow 9623 them to be 9624 independently moved 9625 according to the 9626 following rules. 9627 - s_waitcnt vmcnt(0) 9628 must happen after 9629 any preceding 9630 global/generic 9631 load/load atomic/ 9632 atomicrmw-with-return-value. 9633 - s_waitcnt vscnt(0) 9634 must happen after 9635 any preceding 9636 global/generic 9637 store/store atomic/ 9638 atomicrmw-no-return-value. 9639 - s_waitcnt lgkmcnt(0) 9640 must happen after 9641 any preceding 9642 local/generic 9643 load/store/load 9644 atomic/store 9645 atomic/atomicrmw. 9646 - Must happen before 9647 the following 9648 atomicrmw. 9649 - Ensures that all 9650 memory operations 9651 to global and local 9652 have completed 9653 before performing 9654 the atomicrmw that 9655 is being released. 9656 9657 2. buffer/global/flat_atomic 9658 fence release - singlethread *none* *none* 9659 - wavefront 9660 fence release - workgroup *none* 1. s_waitcnt lgkmcnt(0) & 9661 vmcnt(0) & vscnt(0) 9662 9663 - If CU wavefront execution 9664 mode, omit vmcnt(0) and 9665 vscnt(0). 9666 - If OpenCL and 9667 address space is 9668 not generic, omit 9669 lgkmcnt(0). 9670 - If OpenCL and 9671 address space is 9672 local, omit 9673 vmcnt(0) and vscnt(0). 9674 - However, since LLVM 9675 currently has no 9676 address space on 9677 the fence need to 9678 conservatively 9679 always generate. If 9680 fence had an 9681 address space then 9682 set to address 9683 space of OpenCL 9684 fence flag, or to 9685 generic if both 9686 local and global 9687 flags are 9688 specified. 9689 - Could be split into 9690 separate s_waitcnt 9691 vmcnt(0), s_waitcnt 9692 vscnt(0) and s_waitcnt 9693 lgkmcnt(0) to allow 9694 them to be 9695 independently moved 9696 according to the 9697 following rules. 9698 - s_waitcnt vmcnt(0) 9699 must happen after 9700 any preceding 9701 global/generic 9702 load/load 9703 atomic/ 9704 atomicrmw-with-return-value. 9705 - s_waitcnt vscnt(0) 9706 must happen after 9707 any preceding 9708 global/generic 9709 store/store atomic/ 9710 atomicrmw-no-return-value. 9711 - s_waitcnt lgkmcnt(0) 9712 must happen after 9713 any preceding 9714 local/generic 9715 load/store/load 9716 atomic/store atomic/ 9717 atomicrmw. 9718 - Must happen before 9719 any following store 9720 atomic/atomicrmw 9721 with an equal or 9722 wider sync scope 9723 and memory ordering 9724 stronger than 9725 unordered (this is 9726 termed the 9727 fence-paired-atomic). 9728 - Ensures that all 9729 memory operations 9730 have 9731 completed before 9732 performing the 9733 following 9734 fence-paired-atomic. 9735 9736 fence release - agent *none* 1. s_waitcnt lgkmcnt(0) & 9737 - system vmcnt(0) & vscnt(0) 9738 9739 - If OpenCL and 9740 address space is 9741 not generic, omit 9742 lgkmcnt(0). 9743 - If OpenCL and 9744 address space is 9745 local, omit 9746 vmcnt(0) and vscnt(0). 9747 - However, since LLVM 9748 currently has no 9749 address space on 9750 the fence need to 9751 conservatively 9752 always generate. If 9753 fence had an 9754 address space then 9755 set to address 9756 space of OpenCL 9757 fence flag, or to 9758 generic if both 9759 local and global 9760 flags are 9761 specified. 9762 - Could be split into 9763 separate s_waitcnt 9764 vmcnt(0), s_waitcnt 9765 vscnt(0) and s_waitcnt 9766 lgkmcnt(0) to allow 9767 them to be 9768 independently moved 9769 according to the 9770 following rules. 9771 - s_waitcnt vmcnt(0) 9772 must happen after 9773 any preceding 9774 global/generic 9775 load/load atomic/ 9776 atomicrmw-with-return-value. 9777 - s_waitcnt vscnt(0) 9778 must happen after 9779 any preceding 9780 global/generic 9781 store/store atomic/ 9782 atomicrmw-no-return-value. 9783 - s_waitcnt lgkmcnt(0) 9784 must happen after 9785 any preceding 9786 local/generic 9787 load/store/load 9788 atomic/store 9789 atomic/atomicrmw. 9790 - Must happen before 9791 any following store 9792 atomic/atomicrmw 9793 with an equal or 9794 wider sync scope 9795 and memory ordering 9796 stronger than 9797 unordered (this is 9798 termed the 9799 fence-paired-atomic). 9800 - Ensures that all 9801 memory operations 9802 have 9803 completed before 9804 performing the 9805 following 9806 fence-paired-atomic. 9807 9808 **Acquire-Release Atomic** 9809 ------------------------------------------------------------------------------------ 9810 atomicrmw acq_rel - singlethread - global 1. buffer/global/ds/flat_atomic 9811 - wavefront - local 9812 - generic 9813 atomicrmw acq_rel - workgroup - global 1. s_waitcnt lgkmcnt(0) & 9814 vmcnt(0) & vscnt(0) 9815 9816 - If CU wavefront execution 9817 mode, omit vmcnt(0) and 9818 vscnt(0). 9819 - If OpenCL, omit 9820 lgkmcnt(0). 9821 - Must happen after 9822 any preceding 9823 local/generic 9824 load/store/load 9825 atomic/store 9826 atomic/atomicrmw. 9827 - Could be split into 9828 separate s_waitcnt 9829 vmcnt(0), s_waitcnt 9830 vscnt(0), and s_waitcnt 9831 lgkmcnt(0) to allow 9832 them to be 9833 independently moved 9834 according to the 9835 following rules. 9836 - s_waitcnt vmcnt(0) 9837 must happen after 9838 any preceding 9839 global/generic load/load 9840 atomic/ 9841 atomicrmw-with-return-value. 9842 - s_waitcnt vscnt(0) 9843 must happen after 9844 any preceding 9845 global/generic 9846 store/store 9847 atomic/ 9848 atomicrmw-no-return-value. 9849 - s_waitcnt lgkmcnt(0) 9850 must happen after 9851 any preceding 9852 local/generic 9853 load/store/load 9854 atomic/store 9855 atomic/atomicrmw. 9856 - Must happen before 9857 the following 9858 atomicrmw. 9859 - Ensures that all 9860 memory operations 9861 have 9862 completed before 9863 performing the 9864 atomicrmw that is 9865 being released. 9866 9867 2. buffer/global_atomic 9868 3. s_waitcnt vm/vscnt(0) 9869 9870 - If CU wavefront execution 9871 mode, omit. 9872 - Use vmcnt(0) if atomic with 9873 return and vscnt(0) if 9874 atomic with no-return. 9875 - Must happen before 9876 the following 9877 buffer_gl0_inv. 9878 - Ensures any 9879 following global 9880 data read is no 9881 older than the 9882 atomicrmw value 9883 being acquired. 9884 9885 4. buffer_gl0_inv 9886 9887 - If CU wavefront execution 9888 mode, omit. 9889 - Ensures that 9890 following 9891 loads will not see 9892 stale data. 9893 9894 atomicrmw acq_rel - workgroup - local 1. s_waitcnt vmcnt(0) & vscnt(0) 9895 9896 - If CU wavefront execution 9897 mode, omit. 9898 - If OpenCL, omit. 9899 - Could be split into 9900 separate s_waitcnt 9901 vmcnt(0) and s_waitcnt 9902 vscnt(0) to allow 9903 them to be 9904 independently moved 9905 according to the 9906 following rules. 9907 - s_waitcnt vmcnt(0) 9908 must happen after 9909 any preceding 9910 global/generic load/load 9911 atomic/ 9912 atomicrmw-with-return-value. 9913 - s_waitcnt vscnt(0) 9914 must happen after 9915 any preceding 9916 global/generic 9917 store/store atomic/ 9918 atomicrmw-no-return-value. 9919 - Must happen before 9920 the following 9921 store. 9922 - Ensures that all 9923 global memory 9924 operations have 9925 completed before 9926 performing the 9927 store that is being 9928 released. 9929 9930 2. ds_atomic 9931 3. s_waitcnt lgkmcnt(0) 9932 9933 - If OpenCL, omit. 9934 - Must happen before 9935 the following 9936 buffer_gl0_inv. 9937 - Ensures any 9938 following global 9939 data read is no 9940 older than the local load 9941 atomic value being 9942 acquired. 9943 9944 4. buffer_gl0_inv 9945 9946 - If CU wavefront execution 9947 mode, omit. 9948 - If OpenCL omit. 9949 - Ensures that 9950 following 9951 loads will not see 9952 stale data. 9953 9954 atomicrmw acq_rel - workgroup - generic 1. s_waitcnt lgkmcnt(0) & 9955 vmcnt(0) & vscnt(0) 9956 9957 - If CU wavefront execution 9958 mode, omit vmcnt(0) and 9959 vscnt(0). 9960 - If OpenCL, omit lgkmcnt(0). 9961 - Could be split into 9962 separate s_waitcnt 9963 vmcnt(0), s_waitcnt 9964 vscnt(0) and s_waitcnt 9965 lgkmcnt(0) to allow 9966 them to be 9967 independently moved 9968 according to the 9969 following rules. 9970 - s_waitcnt vmcnt(0) 9971 must happen after 9972 any preceding 9973 global/generic load/load 9974 atomic/ 9975 atomicrmw-with-return-value. 9976 - s_waitcnt vscnt(0) 9977 must happen after 9978 any preceding 9979 global/generic 9980 store/store 9981 atomic/ 9982 atomicrmw-no-return-value. 9983 - s_waitcnt lgkmcnt(0) 9984 must happen after 9985 any preceding 9986 local/generic 9987 load/store/load 9988 atomic/store 9989 atomic/atomicrmw. 9990 - Must happen before 9991 the following 9992 atomicrmw. 9993 - Ensures that all 9994 memory operations 9995 have 9996 completed before 9997 performing the 9998 atomicrmw that is 9999 being released. 10000 10001 2. flat_atomic 10002 3. s_waitcnt lgkmcnt(0) & 10003 vmcnt(0) & vscnt(0) 10004 10005 - If CU wavefront execution 10006 mode, omit vmcnt(0) and 10007 vscnt(0). 10008 - If OpenCL, omit lgkmcnt(0). 10009 - Must happen before 10010 the following 10011 buffer_gl0_inv. 10012 - Ensures any 10013 following global 10014 data read is no 10015 older than the load 10016 atomic value being 10017 acquired. 10018 10019 3. buffer_gl0_inv 10020 10021 - If CU wavefront execution 10022 mode, omit. 10023 - Ensures that 10024 following 10025 loads will not see 10026 stale data. 10027 10028 atomicrmw acq_rel - agent - global 1. s_waitcnt lgkmcnt(0) & 10029 - system vmcnt(0) & vscnt(0) 10030 10031 - If OpenCL, omit 10032 lgkmcnt(0). 10033 - Could be split into 10034 separate s_waitcnt 10035 vmcnt(0), s_waitcnt 10036 vscnt(0) and s_waitcnt 10037 lgkmcnt(0) to allow 10038 them to be 10039 independently moved 10040 according to the 10041 following rules. 10042 - s_waitcnt vmcnt(0) 10043 must happen after 10044 any preceding 10045 global/generic 10046 load/load atomic/ 10047 atomicrmw-with-return-value. 10048 - s_waitcnt vscnt(0) 10049 must happen after 10050 any preceding 10051 global/generic 10052 store/store atomic/ 10053 atomicrmw-no-return-value. 10054 - s_waitcnt lgkmcnt(0) 10055 must happen after 10056 any preceding 10057 local/generic 10058 load/store/load 10059 atomic/store 10060 atomic/atomicrmw. 10061 - Must happen before 10062 the following 10063 atomicrmw. 10064 - Ensures that all 10065 memory operations 10066 to global have 10067 completed before 10068 performing the 10069 atomicrmw that is 10070 being released. 10071 10072 2. buffer/global_atomic 10073 3. s_waitcnt vm/vscnt(0) 10074 10075 - Use vmcnt(0) if atomic with 10076 return and vscnt(0) if 10077 atomic with no-return. 10078 - Must happen before 10079 following 10080 buffer_gl*_inv. 10081 - Ensures the 10082 atomicrmw has 10083 completed before 10084 invalidating the 10085 caches. 10086 10087 4. buffer_gl0_inv; 10088 buffer_gl1_inv 10089 10090 - Must happen before 10091 any following 10092 global/generic 10093 load/load 10094 atomic/atomicrmw. 10095 - Ensures that 10096 following loads 10097 will not see stale 10098 global data. 10099 10100 atomicrmw acq_rel - agent - generic 1. s_waitcnt lgkmcnt(0) & 10101 - system vmcnt(0) & vscnt(0) 10102 10103 - If OpenCL, omit 10104 lgkmcnt(0). 10105 - Could be split into 10106 separate s_waitcnt 10107 vmcnt(0), s_waitcnt 10108 vscnt(0), and s_waitcnt 10109 lgkmcnt(0) to allow 10110 them to be 10111 independently moved 10112 according to the 10113 following rules. 10114 - s_waitcnt vmcnt(0) 10115 must happen after 10116 any preceding 10117 global/generic 10118 load/load atomic 10119 atomicrmw-with-return-value. 10120 - s_waitcnt vscnt(0) 10121 must happen after 10122 any preceding 10123 global/generic 10124 store/store atomic/ 10125 atomicrmw-no-return-value. 10126 - s_waitcnt lgkmcnt(0) 10127 must happen after 10128 any preceding 10129 local/generic 10130 load/store/load 10131 atomic/store 10132 atomic/atomicrmw. 10133 - Must happen before 10134 the following 10135 atomicrmw. 10136 - Ensures that all 10137 memory operations 10138 have 10139 completed before 10140 performing the 10141 atomicrmw that is 10142 being released. 10143 10144 2. flat_atomic 10145 3. s_waitcnt vm/vscnt(0) & 10146 lgkmcnt(0) 10147 10148 - If OpenCL, omit 10149 lgkmcnt(0). 10150 - Use vmcnt(0) if atomic with 10151 return and vscnt(0) if 10152 atomic with no-return. 10153 - Must happen before 10154 following 10155 buffer_gl*_inv. 10156 - Ensures the 10157 atomicrmw has 10158 completed before 10159 invalidating the 10160 caches. 10161 10162 4. buffer_gl0_inv; 10163 buffer_gl1_inv 10164 10165 - Must happen before 10166 any following 10167 global/generic 10168 load/load 10169 atomic/atomicrmw. 10170 - Ensures that 10171 following loads 10172 will not see stale 10173 global data. 10174 10175 fence acq_rel - singlethread *none* *none* 10176 - wavefront 10177 fence acq_rel - workgroup *none* 1. s_waitcnt lgkmcnt(0) & 10178 vmcnt(0) & vscnt(0) 10179 10180 - If CU wavefront execution 10181 mode, omit vmcnt(0) and 10182 vscnt(0). 10183 - If OpenCL and 10184 address space is 10185 not generic, omit 10186 lgkmcnt(0). 10187 - If OpenCL and 10188 address space is 10189 local, omit 10190 vmcnt(0) and vscnt(0). 10191 - However, 10192 since LLVM 10193 currently has no 10194 address space on 10195 the fence need to 10196 conservatively 10197 always generate 10198 (see comment for 10199 previous fence). 10200 - Could be split into 10201 separate s_waitcnt 10202 vmcnt(0), s_waitcnt 10203 vscnt(0) and s_waitcnt 10204 lgkmcnt(0) to allow 10205 them to be 10206 independently moved 10207 according to the 10208 following rules. 10209 - s_waitcnt vmcnt(0) 10210 must happen after 10211 any preceding 10212 global/generic 10213 load/load 10214 atomic/ 10215 atomicrmw-with-return-value. 10216 - s_waitcnt vscnt(0) 10217 must happen after 10218 any preceding 10219 global/generic 10220 store/store atomic/ 10221 atomicrmw-no-return-value. 10222 - s_waitcnt lgkmcnt(0) 10223 must happen after 10224 any preceding 10225 local/generic 10226 load/store/load 10227 atomic/store atomic/ 10228 atomicrmw. 10229 - Must happen before 10230 any following 10231 global/generic 10232 load/load 10233 atomic/store/store 10234 atomic/atomicrmw. 10235 - Ensures that all 10236 memory operations 10237 have 10238 completed before 10239 performing any 10240 following global 10241 memory operations. 10242 - Ensures that the 10243 preceding 10244 local/generic load 10245 atomic/atomicrmw 10246 with an equal or 10247 wider sync scope 10248 and memory ordering 10249 stronger than 10250 unordered (this is 10251 termed the 10252 acquire-fence-paired-atomic) 10253 has completed 10254 before following 10255 global memory 10256 operations. This 10257 satisfies the 10258 requirements of 10259 acquire. 10260 - Ensures that all 10261 previous memory 10262 operations have 10263 completed before a 10264 following 10265 local/generic store 10266 atomic/atomicrmw 10267 with an equal or 10268 wider sync scope 10269 and memory ordering 10270 stronger than 10271 unordered (this is 10272 termed the 10273 release-fence-paired-atomic). 10274 This satisfies the 10275 requirements of 10276 release. 10277 - Must happen before 10278 the following 10279 buffer_gl0_inv. 10280 - Ensures that the 10281 acquire-fence-paired 10282 atomic has completed 10283 before invalidating 10284 the 10285 cache. Therefore 10286 any following 10287 locations read must 10288 be no older than 10289 the value read by 10290 the 10291 acquire-fence-paired-atomic. 10292 10293 3. buffer_gl0_inv 10294 10295 - If CU wavefront execution 10296 mode, omit. 10297 - Ensures that 10298 following 10299 loads will not see 10300 stale data. 10301 10302 fence acq_rel - agent *none* 1. s_waitcnt lgkmcnt(0) & 10303 - system vmcnt(0) & vscnt(0) 10304 10305 - If OpenCL and 10306 address space is 10307 not generic, omit 10308 lgkmcnt(0). 10309 - If OpenCL and 10310 address space is 10311 local, omit 10312 vmcnt(0) and vscnt(0). 10313 - However, since LLVM 10314 currently has no 10315 address space on 10316 the fence need to 10317 conservatively 10318 always generate 10319 (see comment for 10320 previous fence). 10321 - Could be split into 10322 separate s_waitcnt 10323 vmcnt(0), s_waitcnt 10324 vscnt(0) and s_waitcnt 10325 lgkmcnt(0) to allow 10326 them to be 10327 independently moved 10328 according to the 10329 following rules. 10330 - s_waitcnt vmcnt(0) 10331 must happen after 10332 any preceding 10333 global/generic 10334 load/load 10335 atomic/ 10336 atomicrmw-with-return-value. 10337 - s_waitcnt vscnt(0) 10338 must happen after 10339 any preceding 10340 global/generic 10341 store/store atomic/ 10342 atomicrmw-no-return-value. 10343 - s_waitcnt lgkmcnt(0) 10344 must happen after 10345 any preceding 10346 local/generic 10347 load/store/load 10348 atomic/store 10349 atomic/atomicrmw. 10350 - Must happen before 10351 the following 10352 buffer_gl*_inv. 10353 - Ensures that the 10354 preceding 10355 global/local/generic 10356 load 10357 atomic/atomicrmw 10358 with an equal or 10359 wider sync scope 10360 and memory ordering 10361 stronger than 10362 unordered (this is 10363 termed the 10364 acquire-fence-paired-atomic) 10365 has completed 10366 before invalidating 10367 the caches. This 10368 satisfies the 10369 requirements of 10370 acquire. 10371 - Ensures that all 10372 previous memory 10373 operations have 10374 completed before a 10375 following 10376 global/local/generic 10377 store 10378 atomic/atomicrmw 10379 with an equal or 10380 wider sync scope 10381 and memory ordering 10382 stronger than 10383 unordered (this is 10384 termed the 10385 release-fence-paired-atomic). 10386 This satisfies the 10387 requirements of 10388 release. 10389 10390 2. buffer_gl0_inv; 10391 buffer_gl1_inv 10392 10393 - Must happen before 10394 any following 10395 global/generic 10396 load/load 10397 atomic/store/store 10398 atomic/atomicrmw. 10399 - Ensures that 10400 following loads 10401 will not see stale 10402 global data. This 10403 satisfies the 10404 requirements of 10405 acquire. 10406 10407 **Sequential Consistent Atomic** 10408 ------------------------------------------------------------------------------------ 10409 load atomic seq_cst - singlethread - global *Same as corresponding 10410 - wavefront - local load atomic acquire, 10411 - generic except must generate 10412 all instructions even 10413 for OpenCL.* 10414 load atomic seq_cst - workgroup - global 1. s_waitcnt lgkmcnt(0) & 10415 - generic vmcnt(0) & vscnt(0) 10416 10417 - If CU wavefront execution 10418 mode, omit vmcnt(0) and 10419 vscnt(0). 10420 - Could be split into 10421 separate s_waitcnt 10422 vmcnt(0), s_waitcnt 10423 vscnt(0), and s_waitcnt 10424 lgkmcnt(0) to allow 10425 them to be 10426 independently moved 10427 according to the 10428 following rules. 10429 - s_waitcnt lgkmcnt(0) must 10430 happen after 10431 preceding 10432 local/generic load 10433 atomic/store 10434 atomic/atomicrmw 10435 with memory 10436 ordering of seq_cst 10437 and with equal or 10438 wider sync scope. 10439 (Note that seq_cst 10440 fences have their 10441 own s_waitcnt 10442 lgkmcnt(0) and so do 10443 not need to be 10444 considered.) 10445 - s_waitcnt vmcnt(0) 10446 must happen after 10447 preceding 10448 global/generic load 10449 atomic/ 10450 atomicrmw-with-return-value 10451 with memory 10452 ordering of seq_cst 10453 and with equal or 10454 wider sync scope. 10455 (Note that seq_cst 10456 fences have their 10457 own s_waitcnt 10458 vmcnt(0) and so do 10459 not need to be 10460 considered.) 10461 - s_waitcnt vscnt(0) 10462 Must happen after 10463 preceding 10464 global/generic store 10465 atomic/ 10466 atomicrmw-no-return-value 10467 with memory 10468 ordering of seq_cst 10469 and with equal or 10470 wider sync scope. 10471 (Note that seq_cst 10472 fences have their 10473 own s_waitcnt 10474 vscnt(0) and so do 10475 not need to be 10476 considered.) 10477 - Ensures any 10478 preceding 10479 sequential 10480 consistent global/local 10481 memory instructions 10482 have completed 10483 before executing 10484 this sequentially 10485 consistent 10486 instruction. This 10487 prevents reordering 10488 a seq_cst store 10489 followed by a 10490 seq_cst load. (Note 10491 that seq_cst is 10492 stronger than 10493 acquire/release as 10494 the reordering of 10495 load acquire 10496 followed by a store 10497 release is 10498 prevented by the 10499 s_waitcnt of 10500 the release, but 10501 there is nothing 10502 preventing a store 10503 release followed by 10504 load acquire from 10505 completing out of 10506 order. The s_waitcnt 10507 could be placed after 10508 seq_store or before 10509 the seq_load. We 10510 choose the load to 10511 make the s_waitcnt be 10512 as late as possible 10513 so that the store 10514 may have already 10515 completed.) 10516 10517 2. *Following 10518 instructions same as 10519 corresponding load 10520 atomic acquire, 10521 except must generate 10522 all instructions even 10523 for OpenCL.* 10524 load atomic seq_cst - workgroup - local 10525 10526 1. s_waitcnt vmcnt(0) & vscnt(0) 10527 10528 - If CU wavefront execution 10529 mode, omit. 10530 - Could be split into 10531 separate s_waitcnt 10532 vmcnt(0) and s_waitcnt 10533 vscnt(0) to allow 10534 them to be 10535 independently moved 10536 according to the 10537 following rules. 10538 - s_waitcnt vmcnt(0) 10539 Must happen after 10540 preceding 10541 global/generic load 10542 atomic/ 10543 atomicrmw-with-return-value 10544 with memory 10545 ordering of seq_cst 10546 and with equal or 10547 wider sync scope. 10548 (Note that seq_cst 10549 fences have their 10550 own s_waitcnt 10551 vmcnt(0) and so do 10552 not need to be 10553 considered.) 10554 - s_waitcnt vscnt(0) 10555 Must happen after 10556 preceding 10557 global/generic store 10558 atomic/ 10559 atomicrmw-no-return-value 10560 with memory 10561 ordering of seq_cst 10562 and with equal or 10563 wider sync scope. 10564 (Note that seq_cst 10565 fences have their 10566 own s_waitcnt 10567 vscnt(0) and so do 10568 not need to be 10569 considered.) 10570 - Ensures any 10571 preceding 10572 sequential 10573 consistent global 10574 memory instructions 10575 have completed 10576 before executing 10577 this sequentially 10578 consistent 10579 instruction. This 10580 prevents reordering 10581 a seq_cst store 10582 followed by a 10583 seq_cst load. (Note 10584 that seq_cst is 10585 stronger than 10586 acquire/release as 10587 the reordering of 10588 load acquire 10589 followed by a store 10590 release is 10591 prevented by the 10592 s_waitcnt of 10593 the release, but 10594 there is nothing 10595 preventing a store 10596 release followed by 10597 load acquire from 10598 completing out of 10599 order. The s_waitcnt 10600 could be placed after 10601 seq_store or before 10602 the seq_load. We 10603 choose the load to 10604 make the s_waitcnt be 10605 as late as possible 10606 so that the store 10607 may have already 10608 completed.) 10609 10610 2. *Following 10611 instructions same as 10612 corresponding load 10613 atomic acquire, 10614 except must generate 10615 all instructions even 10616 for OpenCL.* 10617 10618 load atomic seq_cst - agent - global 1. s_waitcnt lgkmcnt(0) & 10619 - system - generic vmcnt(0) & vscnt(0) 10620 10621 - Could be split into 10622 separate s_waitcnt 10623 vmcnt(0), s_waitcnt 10624 vscnt(0) and s_waitcnt 10625 lgkmcnt(0) to allow 10626 them to be 10627 independently moved 10628 according to the 10629 following rules. 10630 - s_waitcnt lgkmcnt(0) 10631 must happen after 10632 preceding 10633 local load 10634 atomic/store 10635 atomic/atomicrmw 10636 with memory 10637 ordering of seq_cst 10638 and with equal or 10639 wider sync scope. 10640 (Note that seq_cst 10641 fences have their 10642 own s_waitcnt 10643 lgkmcnt(0) and so do 10644 not need to be 10645 considered.) 10646 - s_waitcnt vmcnt(0) 10647 must happen after 10648 preceding 10649 global/generic load 10650 atomic/ 10651 atomicrmw-with-return-value 10652 with memory 10653 ordering of seq_cst 10654 and with equal or 10655 wider sync scope. 10656 (Note that seq_cst 10657 fences have their 10658 own s_waitcnt 10659 vmcnt(0) and so do 10660 not need to be 10661 considered.) 10662 - s_waitcnt vscnt(0) 10663 Must happen after 10664 preceding 10665 global/generic store 10666 atomic/ 10667 atomicrmw-no-return-value 10668 with memory 10669 ordering of seq_cst 10670 and with equal or 10671 wider sync scope. 10672 (Note that seq_cst 10673 fences have their 10674 own s_waitcnt 10675 vscnt(0) and so do 10676 not need to be 10677 considered.) 10678 - Ensures any 10679 preceding 10680 sequential 10681 consistent global 10682 memory instructions 10683 have completed 10684 before executing 10685 this sequentially 10686 consistent 10687 instruction. This 10688 prevents reordering 10689 a seq_cst store 10690 followed by a 10691 seq_cst load. (Note 10692 that seq_cst is 10693 stronger than 10694 acquire/release as 10695 the reordering of 10696 load acquire 10697 followed by a store 10698 release is 10699 prevented by the 10700 s_waitcnt of 10701 the release, but 10702 there is nothing 10703 preventing a store 10704 release followed by 10705 load acquire from 10706 completing out of 10707 order. The s_waitcnt 10708 could be placed after 10709 seq_store or before 10710 the seq_load. We 10711 choose the load to 10712 make the s_waitcnt be 10713 as late as possible 10714 so that the store 10715 may have already 10716 completed.) 10717 10718 2. *Following 10719 instructions same as 10720 corresponding load 10721 atomic acquire, 10722 except must generate 10723 all instructions even 10724 for OpenCL.* 10725 store atomic seq_cst - singlethread - global *Same as corresponding 10726 - wavefront - local store atomic release, 10727 - workgroup - generic except must generate 10728 - agent all instructions even 10729 - system for OpenCL.* 10730 atomicrmw seq_cst - singlethread - global *Same as corresponding 10731 - wavefront - local atomicrmw acq_rel, 10732 - workgroup - generic except must generate 10733 - agent all instructions even 10734 - system for OpenCL.* 10735 fence seq_cst - singlethread *none* *Same as corresponding 10736 - wavefront fence acq_rel, 10737 - workgroup except must generate 10738 - agent all instructions even 10739 - system for OpenCL.* 10740 ============ ============ ============== ========== ================================ 10741 10742.. _amdgpu-amdhsa-trap-handler-abi: 10743 10744Trap Handler ABI 10745~~~~~~~~~~~~~~~~ 10746 10747For code objects generated by the AMDGPU backend for HSA [HSA]_ compatible 10748runtimes (see :ref:`amdgpu-os`), the runtime installs a trap handler that 10749supports the ``s_trap`` instruction. For usage see: 10750 10751- :ref:`amdgpu-trap-handler-for-amdhsa-os-v2-table` 10752- :ref:`amdgpu-trap-handler-for-amdhsa-os-v3-table` 10753- :ref:`amdgpu-trap-handler-for-amdhsa-os-v4-onwards-table` 10754 10755 .. table:: AMDGPU Trap Handler for AMDHSA OS Code Object V2 10756 :name: amdgpu-trap-handler-for-amdhsa-os-v2-table 10757 10758 =================== =============== =============== ======================================= 10759 Usage Code Sequence Trap Handler Description 10760 Inputs 10761 =================== =============== =============== ======================================= 10762 reserved ``s_trap 0x00`` Reserved by hardware. 10763 ``debugtrap(arg)`` ``s_trap 0x01`` ``SGPR0-1``: Reserved for Finalizer HSA ``debugtrap`` 10764 ``queue_ptr`` intrinsic (not implemented). 10765 ``VGPR0``: 10766 ``arg`` 10767 ``llvm.trap`` ``s_trap 0x02`` ``SGPR0-1``: Causes wave to be halted with the PC at 10768 ``queue_ptr`` the trap instruction. The associated 10769 queue is signalled to put it into the 10770 error state. When the queue is put in 10771 the error state, the waves executing 10772 dispatches on the queue will be 10773 terminated. 10774 ``llvm.debugtrap`` ``s_trap 0x03`` *none* - If debugger not enabled then behaves 10775 as a no-operation. The trap handler 10776 is entered and immediately returns to 10777 continue execution of the wavefront. 10778 - If the debugger is enabled, causes 10779 the debug trap to be reported by the 10780 debugger and the wavefront is put in 10781 the halt state with the PC at the 10782 instruction. The debugger must 10783 increment the PC and resume the wave. 10784 reserved ``s_trap 0x04`` Reserved. 10785 reserved ``s_trap 0x05`` Reserved. 10786 reserved ``s_trap 0x06`` Reserved. 10787 reserved ``s_trap 0x07`` Reserved. 10788 reserved ``s_trap 0x08`` Reserved. 10789 reserved ``s_trap 0xfe`` Reserved. 10790 reserved ``s_trap 0xff`` Reserved. 10791 =================== =============== =============== ======================================= 10792 10793.. 10794 10795 .. table:: AMDGPU Trap Handler for AMDHSA OS Code Object V3 10796 :name: amdgpu-trap-handler-for-amdhsa-os-v3-table 10797 10798 =================== =============== =============== ======================================= 10799 Usage Code Sequence Trap Handler Description 10800 Inputs 10801 =================== =============== =============== ======================================= 10802 reserved ``s_trap 0x00`` Reserved by hardware. 10803 debugger breakpoint ``s_trap 0x01`` *none* Reserved for debugger to use for 10804 breakpoints. Causes wave to be halted 10805 with the PC at the trap instruction. 10806 The debugger is responsible to resume 10807 the wave, including the instruction 10808 that the breakpoint overwrote. 10809 ``llvm.trap`` ``s_trap 0x02`` ``SGPR0-1``: Causes wave to be halted with the PC at 10810 ``queue_ptr`` the trap instruction. The associated 10811 queue is signalled to put it into the 10812 error state. When the queue is put in 10813 the error state, the waves executing 10814 dispatches on the queue will be 10815 terminated. 10816 ``llvm.debugtrap`` ``s_trap 0x03`` *none* - If debugger not enabled then behaves 10817 as a no-operation. The trap handler 10818 is entered and immediately returns to 10819 continue execution of the wavefront. 10820 - If the debugger is enabled, causes 10821 the debug trap to be reported by the 10822 debugger and the wavefront is put in 10823 the halt state with the PC at the 10824 instruction. The debugger must 10825 increment the PC and resume the wave. 10826 reserved ``s_trap 0x04`` Reserved. 10827 reserved ``s_trap 0x05`` Reserved. 10828 reserved ``s_trap 0x06`` Reserved. 10829 reserved ``s_trap 0x07`` Reserved. 10830 reserved ``s_trap 0x08`` Reserved. 10831 reserved ``s_trap 0xfe`` Reserved. 10832 reserved ``s_trap 0xff`` Reserved. 10833 =================== =============== =============== ======================================= 10834 10835.. 10836 10837 .. table:: AMDGPU Trap Handler for AMDHSA OS Code Object V4 and Above 10838 :name: amdgpu-trap-handler-for-amdhsa-os-v4-onwards-table 10839 10840 =================== =============== ================ ================= ======================================= 10841 Usage Code Sequence GFX6-GFX8 Inputs GFX9-GFX10 Inputs Description 10842 =================== =============== ================ ================= ======================================= 10843 reserved ``s_trap 0x00`` Reserved by hardware. 10844 debugger breakpoint ``s_trap 0x01`` *none* *none* Reserved for debugger to use for 10845 breakpoints. Causes wave to be halted 10846 with the PC at the trap instruction. 10847 The debugger is responsible to resume 10848 the wave, including the instruction 10849 that the breakpoint overwrote. 10850 ``llvm.trap`` ``s_trap 0x02`` ``SGPR0-1``: *none* Causes wave to be halted with the PC at 10851 ``queue_ptr`` the trap instruction. The associated 10852 queue is signalled to put it into the 10853 error state. When the queue is put in 10854 the error state, the waves executing 10855 dispatches on the queue will be 10856 terminated. 10857 ``llvm.debugtrap`` ``s_trap 0x03`` *none* *none* - If debugger not enabled then behaves 10858 as a no-operation. The trap handler 10859 is entered and immediately returns to 10860 continue execution of the wavefront. 10861 - If the debugger is enabled, causes 10862 the debug trap to be reported by the 10863 debugger and the wavefront is put in 10864 the halt state with the PC at the 10865 instruction. The debugger must 10866 increment the PC and resume the wave. 10867 reserved ``s_trap 0x04`` Reserved. 10868 reserved ``s_trap 0x05`` Reserved. 10869 reserved ``s_trap 0x06`` Reserved. 10870 reserved ``s_trap 0x07`` Reserved. 10871 reserved ``s_trap 0x08`` Reserved. 10872 reserved ``s_trap 0xfe`` Reserved. 10873 reserved ``s_trap 0xff`` Reserved. 10874 =================== =============== ================ ================= ======================================= 10875 10876.. _amdgpu-amdhsa-function-call-convention: 10877 10878Call Convention 10879~~~~~~~~~~~~~~~ 10880 10881.. note:: 10882 10883 This section is currently incomplete and has inaccuracies. It is WIP that will 10884 be updated as information is determined. 10885 10886See :ref:`amdgpu-dwarf-address-space-identifier` for information on swizzled 10887addresses. Unswizzled addresses are normal linear addresses. 10888 10889.. _amdgpu-amdhsa-function-call-convention-kernel-functions: 10890 10891Kernel Functions 10892++++++++++++++++ 10893 10894This section describes the call convention ABI for the outer kernel function. 10895 10896See :ref:`amdgpu-amdhsa-initial-kernel-execution-state` for the kernel call 10897convention. 10898 10899The following is not part of the AMDGPU kernel calling convention but describes 10900how the AMDGPU implements function calls: 10901 109021. Clang decides the kernarg layout to match the *HSA Programmer's Language 10903 Reference* [HSA]_. 10904 10905 - All structs are passed directly. 10906 - Lambda values are passed *TBA*. 10907 10908 .. TODO:: 10909 10910 - Does this really follow HSA rules? Or are structs >16 bytes passed 10911 by-value struct? 10912 - What is ABI for lambda values? 10913 109144. The kernel performs certain setup in its prolog, as described in 10915 :ref:`amdgpu-amdhsa-kernel-prolog`. 10916 10917.. _amdgpu-amdhsa-function-call-convention-non-kernel-functions: 10918 10919Non-Kernel Functions 10920++++++++++++++++++++ 10921 10922This section describes the call convention ABI for functions other than the 10923outer kernel function. 10924 10925If a kernel has function calls then scratch is always allocated and used for 10926the call stack which grows from low address to high address using the swizzled 10927scratch address space. 10928 10929On entry to a function: 10930 109311. SGPR0-3 contain a V# with the following properties (see 10932 :ref:`amdgpu-amdhsa-kernel-prolog-private-segment-buffer`): 10933 10934 * Base address pointing to the beginning of the wavefront scratch backing 10935 memory. 10936 * Swizzled with dword element size and stride of wavefront size elements. 10937 109382. The FLAT_SCRATCH register pair is setup. See 10939 :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`. 109403. GFX6-GFX8: M0 register set to the size of LDS in bytes. See 10941 :ref:`amdgpu-amdhsa-kernel-prolog-m0`. 109424. The EXEC register is set to the lanes active on entry to the function. 109435. MODE register: *TBD* 109446. VGPR0-31 and SGPR4-29 are used to pass function input arguments as described 10945 below. 109467. SGPR30-31 return address (RA). The code address that the function must 10947 return to when it completes. The value is undefined if the function is *no 10948 return*. 109498. SGPR32 is used for the stack pointer (SP). It is an unswizzled scratch 10950 offset relative to the beginning of the wavefront scratch backing memory. 10951 10952 The unswizzled SP can be used with buffer instructions as an unswizzled SGPR 10953 offset with the scratch V# in SGPR0-3 to access the stack in a swizzled 10954 manner. 10955 10956 The unswizzled SP value can be converted into the swizzled SP value by: 10957 10958 | swizzled SP = unswizzled SP / wavefront size 10959 10960 This may be used to obtain the private address space address of stack 10961 objects and to convert this address to a flat address by adding the flat 10962 scratch aperture base address. 10963 10964 The swizzled SP value is always 4 bytes aligned for the ``r600`` 10965 architecture and 16 byte aligned for the ``amdgcn`` architecture. 10966 10967 .. note:: 10968 10969 The ``amdgcn`` value is selected to avoid dynamic stack alignment for the 10970 OpenCL language which has the largest base type defined as 16 bytes. 10971 10972 On entry, the swizzled SP value is the address of the first function 10973 argument passed on the stack. Other stack passed arguments are positive 10974 offsets from the entry swizzled SP value. 10975 10976 The function may use positive offsets beyond the last stack passed argument 10977 for stack allocated local variables and register spill slots. If necessary, 10978 the function may align these to greater alignment than 16 bytes. After these 10979 the function may dynamically allocate space for such things as runtime sized 10980 ``alloca`` local allocations. 10981 10982 If the function calls another function, it will place any stack allocated 10983 arguments after the last local allocation and adjust SGPR32 to the address 10984 after the last local allocation. 10985 109869. All other registers are unspecified. 1098710. Any necessary ``s_waitcnt`` has been performed to ensure memory is available 10988 to the function. 10989 10990On exit from a function: 10991 109921. VGPR0-31 and SGPR4-29 are used to pass function result arguments as 10993 described below. Any registers used are considered clobbered registers. 109942. The following registers are preserved and have the same value as on entry: 10995 10996 * FLAT_SCRATCH 10997 * EXEC 10998 * GFX6-GFX8: M0 10999 * All SGPR registers except the clobbered registers of SGPR4-31. 11000 * VGPR40-47 11001 * VGPR56-63 11002 * VGPR72-79 11003 * VGPR88-95 11004 * VGPR104-111 11005 * VGPR120-127 11006 * VGPR136-143 11007 * VGPR152-159 11008 * VGPR168-175 11009 * VGPR184-191 11010 * VGPR200-207 11011 * VGPR216-223 11012 * VGPR232-239 11013 * VGPR248-255 11014 11015 .. note:: 11016 11017 Except the argument registers, the VGPRs clobbered and the preserved 11018 registers are intermixed at regular intervals in order to keep a 11019 similar ratio independent of the number of allocated VGPRs. 11020 11021 * GFX90A: All AGPR registers except the clobbered registers AGPR0-31. 11022 * Lanes of all VGPRs that are inactive at the call site. 11023 11024 For the AMDGPU backend, an inter-procedural register allocation (IPRA) 11025 optimization may mark some of clobbered SGPR and VGPR registers as 11026 preserved if it can be determined that the called function does not change 11027 their value. 11028 110292. The PC is set to the RA provided on entry. 110303. MODE register: *TBD*. 110314. All other registers are clobbered. 110325. Any necessary ``s_waitcnt`` has been performed to ensure memory accessed by 11033 function is available to the caller. 11034 11035.. TODO:: 11036 11037 - How are function results returned? The address of structured types is passed 11038 by reference, but what about other types? 11039 11040The function input arguments are made up of the formal arguments explicitly 11041declared by the source language function plus the implicit input arguments used 11042by the implementation. 11043 11044The source language input arguments are: 11045 110461. Any source language implicit ``this`` or ``self`` argument comes first as a 11047 pointer type. 110482. Followed by the function formal arguments in left to right source order. 11049 11050The source language result arguments are: 11051 110521. The function result argument. 11053 11054The source language input or result struct type arguments that are less than or 11055equal to 16 bytes, are decomposed recursively into their base type fields, and 11056each field is passed as if a separate argument. For input arguments, if the 11057called function requires the struct to be in memory, for example because its 11058address is taken, then the function body is responsible for allocating a stack 11059location and copying the field arguments into it. Clang terms this *direct 11060struct*. 11061 11062The source language input struct type arguments that are greater than 16 bytes, 11063are passed by reference. The caller is responsible for allocating a stack 11064location to make a copy of the struct value and pass the address as the input 11065argument. The called function is responsible to perform the dereference when 11066accessing the input argument. Clang terms this *by-value struct*. 11067 11068A source language result struct type argument that is greater than 16 bytes, is 11069returned by reference. The caller is responsible for allocating a stack location 11070to hold the result value and passes the address as the last input argument 11071(before the implicit input arguments). In this case there are no result 11072arguments. The called function is responsible to perform the dereference when 11073storing the result value. Clang terms this *structured return (sret)*. 11074 11075*TODO: correct the ``sret`` definition.* 11076 11077.. TODO:: 11078 11079 Is this definition correct? Or is ``sret`` only used if passing in registers, and 11080 pass as non-decomposed struct as stack argument? Or something else? Is the 11081 memory location in the caller stack frame, or a stack memory argument and so 11082 no address is passed as the caller can directly write to the argument stack 11083 location? But then the stack location is still live after return. If an 11084 argument stack location is it the first stack argument or the last one? 11085 11086Lambda argument types are treated as struct types with an implementation defined 11087set of fields. 11088 11089.. TODO:: 11090 11091 Need to specify the ABI for lambda types for AMDGPU. 11092 11093For AMDGPU backend all source language arguments (including the decomposed 11094struct type arguments) are passed in VGPRs unless marked ``inreg`` in which case 11095they are passed in SGPRs. 11096 11097The AMDGPU backend walks the function call graph from the leaves to determine 11098which implicit input arguments are used, propagating to each caller of the 11099function. The used implicit arguments are appended to the function arguments 11100after the source language arguments in the following order: 11101 11102.. TODO:: 11103 11104 Is recursion or external functions supported? 11105 111061. Work-Item ID (1 VGPR) 11107 11108 The X, Y and Z work-item ID are packed into a single VGRP with the following 11109 layout. Only fields actually used by the function are set. The other bits 11110 are undefined. 11111 11112 The values come from the initial kernel execution state. See 11113 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`. 11114 11115 .. table:: Work-item implicit argument layout 11116 :name: amdgpu-amdhsa-workitem-implicit-argument-layout-table 11117 11118 ======= ======= ============== 11119 Bits Size Field Name 11120 ======= ======= ============== 11121 9:0 10 bits X Work-Item ID 11122 19:10 10 bits Y Work-Item ID 11123 29:20 10 bits Z Work-Item ID 11124 31:30 2 bits Unused 11125 ======= ======= ============== 11126 111272. Dispatch Ptr (2 SGPRs) 11128 11129 The value comes from the initial kernel execution state. See 11130 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 11131 111323. Queue Ptr (2 SGPRs) 11133 11134 The value comes from the initial kernel execution state. See 11135 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 11136 111374. Kernarg Segment Ptr (2 SGPRs) 11138 11139 The value comes from the initial kernel execution state. See 11140 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 11141 111425. Dispatch id (2 SGPRs) 11143 11144 The value comes from the initial kernel execution state. See 11145 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 11146 111476. Work-Group ID X (1 SGPR) 11148 11149 The value comes from the initial kernel execution state. See 11150 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 11151 111527. Work-Group ID Y (1 SGPR) 11153 11154 The value comes from the initial kernel execution state. See 11155 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 11156 111578. Work-Group ID Z (1 SGPR) 11158 11159 The value comes from the initial kernel execution state. See 11160 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 11161 111629. Implicit Argument Ptr (2 SGPRs) 11163 11164 The value is computed by adding an offset to Kernarg Segment Ptr to get the 11165 global address space pointer to the first kernarg implicit argument. 11166 11167The input and result arguments are assigned in order in the following manner: 11168 11169.. note:: 11170 11171 There are likely some errors and omissions in the following description that 11172 need correction. 11173 11174 .. TODO:: 11175 11176 Check the Clang source code to decipher how function arguments and return 11177 results are handled. Also see the AMDGPU specific values used. 11178 11179* VGPR arguments are assigned to consecutive VGPRs starting at VGPR0 up to 11180 VGPR31. 11181 11182 If there are more arguments than will fit in these registers, the remaining 11183 arguments are allocated on the stack in order on naturally aligned 11184 addresses. 11185 11186 .. TODO:: 11187 11188 How are overly aligned structures allocated on the stack? 11189 11190* SGPR arguments are assigned to consecutive SGPRs starting at SGPR0 up to 11191 SGPR29. 11192 11193 If there are more arguments than will fit in these registers, the remaining 11194 arguments are allocated on the stack in order on naturally aligned 11195 addresses. 11196 11197Note that decomposed struct type arguments may have some fields passed in 11198registers and some in memory. 11199 11200.. TODO:: 11201 11202 So, a struct which can pass some fields as decomposed register arguments, will 11203 pass the rest as decomposed stack elements? But an argument that will not start 11204 in registers will not be decomposed and will be passed as a non-decomposed 11205 stack value? 11206 11207The following is not part of the AMDGPU function calling convention but 11208describes how the AMDGPU implements function calls: 11209 112101. SGPR33 is used as a frame pointer (FP) if necessary. Like the SP it is an 11211 unswizzled scratch address. It is only needed if runtime sized ``alloca`` 11212 are used, or for the reasons defined in ``SIFrameLowering``. 112132. Runtime stack alignment is supported. SGPR34 is used as a base pointer (BP) 11214 to access the incoming stack arguments in the function. The BP is needed 11215 only when the function requires the runtime stack alignment. 11216 112173. Allocating SGPR arguments on the stack are not supported. 11218 112194. No CFI is currently generated. See 11220 :ref:`amdgpu-dwarf-call-frame-information`. 11221 11222 .. note:: 11223 11224 CFI will be generated that defines the CFA as the unswizzled address 11225 relative to the wave scratch base in the unswizzled private address space 11226 of the lowest address stack allocated local variable. 11227 11228 ``DW_AT_frame_base`` will be defined as the swizzled address in the 11229 swizzled private address space by dividing the CFA by the wavefront size 11230 (since CFA is always at least dword aligned which matches the scratch 11231 swizzle element size). 11232 11233 If no dynamic stack alignment was performed, the stack allocated arguments 11234 are accessed as negative offsets relative to ``DW_AT_frame_base``, and the 11235 local variables and register spill slots are accessed as positive offsets 11236 relative to ``DW_AT_frame_base``. 11237 112385. Function argument passing is implemented by copying the input physical 11239 registers to virtual registers on entry. The register allocator can spill if 11240 necessary. These are copied back to physical registers at call sites. The 11241 net effect is that each function call can have these values in entirely 11242 distinct locations. The IPRA can help avoid shuffling argument registers. 112436. Call sites are implemented by setting up the arguments at positive offsets 11244 from SP. Then SP is incremented to account for the known frame size before 11245 the call and decremented after the call. 11246 11247 .. note:: 11248 11249 The CFI will reflect the changed calculation needed to compute the CFA 11250 from SP. 11251 112527. 4 byte spill slots are used in the stack frame. One slot is allocated for an 11253 emergency spill slot. Buffer instructions are used for stack accesses and 11254 not the ``flat_scratch`` instruction. 11255 11256 .. TODO:: 11257 11258 Explain when the emergency spill slot is used. 11259 11260.. TODO:: 11261 11262 Possible broken issues: 11263 11264 - Stack arguments must be aligned to required alignment. 11265 - Stack is aligned to max(16, max formal argument alignment) 11266 - Direct argument < 64 bits should check register budget. 11267 - Register budget calculation should respect ``inreg`` for SGPR. 11268 - SGPR overflow is not handled. 11269 - struct with 1 member unpeeling is not checking size of member. 11270 - ``sret`` is after ``this`` pointer. 11271 - Caller is not implementing stack realignment: need an extra pointer. 11272 - Should say AMDGPU passes FP rather than SP. 11273 - Should CFI define CFA as address of locals or arguments. Difference is 11274 apparent when have implemented dynamic alignment. 11275 - If ``SCRATCH`` instruction could allow negative offsets, then can make FP be 11276 highest address of stack frame and use negative offset for locals. Would 11277 allow SP to be the same as FP and could support signal-handler-like as now 11278 have a real SP for the top of the stack. 11279 - How is ``sret`` passed on the stack? In argument stack area? Can it overlay 11280 arguments? 11281 11282AMDPAL 11283------ 11284 11285This section provides code conventions used when the target triple OS is 11286``amdpal`` (see :ref:`amdgpu-target-triples`). 11287 11288.. _amdgpu-amdpal-code-object-metadata-section: 11289 11290Code Object Metadata 11291~~~~~~~~~~~~~~~~~~~~ 11292 11293.. note:: 11294 11295 The metadata is currently in development and is subject to major 11296 changes. Only the current version is supported. *When this document 11297 was generated the version was 2.6.* 11298 11299Code object metadata is specified by the ``NT_AMDGPU_METADATA`` note 11300record (see :ref:`amdgpu-note-records-v3-onwards`). 11301 11302The metadata is represented as Message Pack formatted binary data (see 11303[MsgPack]_). The top level is a Message Pack map that includes the keys 11304defined in table :ref:`amdgpu-amdpal-code-object-metadata-map-table` 11305and referenced tables. 11306 11307Additional information can be added to the maps. To avoid conflicts, any 11308key names should be prefixed by "*vendor-name*." where ``vendor-name`` 11309can be the name of the vendor and specific vendor tool that generates the 11310information. The prefix is abbreviated to simply "." when it appears 11311within a map that has been added by the same *vendor-name*. 11312 11313 .. table:: AMDPAL Code Object Metadata Map 11314 :name: amdgpu-amdpal-code-object-metadata-map-table 11315 11316 =================== ============== ========= ====================================================================== 11317 String Key Value Type Required? Description 11318 =================== ============== ========= ====================================================================== 11319 "amdpal.version" sequence of Required PAL code object metadata (major, minor) version. The current values 11320 2 integers are defined by *Util::Abi::PipelineMetadata(Major|Minor)Version*. 11321 "amdpal.pipelines" sequence of Required Per-pipeline metadata. See 11322 map :ref:`amdgpu-amdpal-code-object-pipeline-metadata-map-table` for the 11323 definition of the keys included in that map. 11324 =================== ============== ========= ====================================================================== 11325 11326.. 11327 11328 .. table:: AMDPAL Code Object Pipeline Metadata Map 11329 :name: amdgpu-amdpal-code-object-pipeline-metadata-map-table 11330 11331 ====================================== ============== ========= =================================================== 11332 String Key Value Type Required? Description 11333 ====================================== ============== ========= =================================================== 11334 ".name" string Source name of the pipeline. 11335 ".type" string Pipeline type, e.g. VsPs. Values include: 11336 11337 - "VsPs" 11338 - "Gs" 11339 - "Cs" 11340 - "Ngg" 11341 - "Tess" 11342 - "GsTess" 11343 - "NggTess" 11344 11345 ".internal_pipeline_hash" sequence of Required Internal compiler hash for this pipeline. Lower 11346 2 integers 64 bits is the "stable" portion of the hash, used 11347 for e.g. shader replacement lookup. Upper 64 bits 11348 is the "unique" portion of the hash, used for 11349 e.g. pipeline cache lookup. The value is 11350 implementation defined, and can not be relied on 11351 between different builds of the compiler. 11352 ".shaders" map Per-API shader metadata. See 11353 :ref:`amdgpu-amdpal-code-object-shader-map-table` 11354 for the definition of the keys included in that 11355 map. 11356 ".hardware_stages" map Per-hardware stage metadata. See 11357 :ref:`amdgpu-amdpal-code-object-hardware-stage-map-table` 11358 for the definition of the keys included in that 11359 map. 11360 ".shader_functions" map Per-shader function metadata. See 11361 :ref:`amdgpu-amdpal-code-object-shader-function-map-table` 11362 for the definition of the keys included in that 11363 map. 11364 ".registers" map Required Hardware register configuration. See 11365 :ref:`amdgpu-amdpal-code-object-register-map-table` 11366 for the definition of the keys included in that 11367 map. 11368 ".user_data_limit" integer Number of user data entries accessed by this 11369 pipeline. 11370 ".spill_threshold" integer The user data spill threshold. 0xFFFF for 11371 NoUserDataSpilling. 11372 ".uses_viewport_array_index" boolean Indicates whether or not the pipeline uses the 11373 viewport array index feature. Pipelines which use 11374 this feature can render into all 16 viewports, 11375 whereas pipelines which do not use it are 11376 restricted to viewport #0. 11377 ".es_gs_lds_size" integer Size in bytes of LDS space used internally for 11378 handling data-passing between the ES and GS 11379 shader stages. This can be zero if the data is 11380 passed using off-chip buffers. This value should 11381 be used to program all user-SGPRs which have been 11382 marked with "UserDataMapping::EsGsLdsSize" 11383 (typically only the GS and VS HW stages will ever 11384 have a user-SGPR so marked). 11385 ".nggSubgroupSize" integer Explicit maximum subgroup size for NGG shaders 11386 (maximum number of threads in a subgroup). 11387 ".num_interpolants" integer Graphics only. Number of PS interpolants. 11388 ".mesh_scratch_memory_size" integer Max mesh shader scratch memory used. 11389 ".api" string Name of the client graphics API. 11390 ".api_create_info" binary Graphics API shader create info binary blob. Can 11391 be defined by the driver using the compiler if 11392 they want to be able to correlate API-specific 11393 information used during creation at a later time. 11394 ====================================== ============== ========= =================================================== 11395 11396.. 11397 11398 .. table:: AMDPAL Code Object Shader Map 11399 :name: amdgpu-amdpal-code-object-shader-map-table 11400 11401 11402 +-------------+--------------+-------------------------------------------------------------------+ 11403 |String Key |Value Type |Description | 11404 +=============+==============+===================================================================+ 11405 |- ".compute" |map |See :ref:`amdgpu-amdpal-code-object-api-shader-metadata-map-table` | 11406 |- ".vertex" | |for the definition of the keys included in that map. | 11407 |- ".hull" | | | 11408 |- ".domain" | | | 11409 |- ".geometry"| | | 11410 |- ".pixel" | | | 11411 +-------------+--------------+-------------------------------------------------------------------+ 11412 11413.. 11414 11415 .. table:: AMDPAL Code Object API Shader Metadata Map 11416 :name: amdgpu-amdpal-code-object-api-shader-metadata-map-table 11417 11418 ==================== ============== ========= ===================================================================== 11419 String Key Value Type Required? Description 11420 ==================== ============== ========= ===================================================================== 11421 ".api_shader_hash" sequence of Required Input shader hash, typically passed in from the client. The value 11422 2 integers is implementation defined, and can not be relied on between 11423 different builds of the compiler. 11424 ".hardware_mapping" sequence of Required Flags indicating the HW stages this API shader maps to. Values 11425 string include: 11426 11427 - ".ls" 11428 - ".hs" 11429 - ".es" 11430 - ".gs" 11431 - ".vs" 11432 - ".ps" 11433 - ".cs" 11434 11435 ==================== ============== ========= ===================================================================== 11436 11437.. 11438 11439 .. table:: AMDPAL Code Object Hardware Stage Map 11440 :name: amdgpu-amdpal-code-object-hardware-stage-map-table 11441 11442 +-------------+--------------+-----------------------------------------------------------------------+ 11443 |String Key |Value Type |Description | 11444 +=============+==============+=======================================================================+ 11445 |- ".ls" |map |See :ref:`amdgpu-amdpal-code-object-hardware-stage-metadata-map-table` | 11446 |- ".hs" | |for the definition of the keys included in that map. | 11447 |- ".es" | | | 11448 |- ".gs" | | | 11449 |- ".vs" | | | 11450 |- ".ps" | | | 11451 |- ".cs" | | | 11452 +-------------+--------------+-----------------------------------------------------------------------+ 11453 11454.. 11455 11456 .. table:: AMDPAL Code Object Hardware Stage Metadata Map 11457 :name: amdgpu-amdpal-code-object-hardware-stage-metadata-map-table 11458 11459 ========================== ============== ========= =============================================================== 11460 String Key Value Type Required? Description 11461 ========================== ============== ========= =============================================================== 11462 ".entry_point" string The ELF symbol pointing to this pipeline's stage entry point. 11463 ".scratch_memory_size" integer Scratch memory size in bytes. 11464 ".lds_size" integer Local Data Share size in bytes. 11465 ".perf_data_buffer_size" integer Performance data buffer size in bytes. 11466 ".vgpr_count" integer Number of VGPRs used. 11467 ".agpr_count" integer Number of AGPRs used. 11468 ".sgpr_count" integer Number of SGPRs used. 11469 ".vgpr_limit" integer If non-zero, indicates the shader was compiled with a 11470 directive to instruct the compiler to limit the VGPR usage to 11471 be less than or equal to the specified value (only set if 11472 different from HW default). 11473 ".sgpr_limit" integer SGPR count upper limit (only set if different from HW 11474 default). 11475 ".threadgroup_dimensions" sequence of Thread-group X/Y/Z dimensions (Compute only). 11476 3 integers 11477 ".wavefront_size" integer Wavefront size (only set if different from HW default). 11478 ".uses_uavs" boolean The shader reads or writes UAVs. 11479 ".uses_rovs" boolean The shader reads or writes ROVs. 11480 ".writes_uavs" boolean The shader writes to one or more UAVs. 11481 ".writes_depth" boolean The shader writes out a depth value. 11482 ".uses_append_consume" boolean The shader uses append and/or consume operations, either 11483 memory or GDS. 11484 ".uses_prim_id" boolean The shader uses PrimID. 11485 ========================== ============== ========= =============================================================== 11486 11487.. 11488 11489 .. table:: AMDPAL Code Object Shader Function Map 11490 :name: amdgpu-amdpal-code-object-shader-function-map-table 11491 11492 =============== ============== ==================================================================== 11493 String Key Value Type Description 11494 =============== ============== ==================================================================== 11495 *symbol name* map *symbol name* is the ELF symbol name of the shader function code 11496 entry address. The value is the function's metadata. See 11497 :ref:`amdgpu-amdpal-code-object-shader-function-metadata-map-table`. 11498 =============== ============== ==================================================================== 11499 11500.. 11501 11502 .. table:: AMDPAL Code Object Shader Function Metadata Map 11503 :name: amdgpu-amdpal-code-object-shader-function-metadata-map-table 11504 11505 ============================= ============== ================================================================= 11506 String Key Value Type Description 11507 ============================= ============== ================================================================= 11508 ".api_shader_hash" sequence of Input shader hash, typically passed in from the client. The value 11509 2 integers is implementation defined, and can not be relied on between 11510 different builds of the compiler. 11511 ".scratch_memory_size" integer Size in bytes of scratch memory used by the shader. 11512 ".lds_size" integer Size in bytes of LDS memory. 11513 ".vgpr_count" integer Number of VGPRs used by the shader. 11514 ".sgpr_count" integer Number of SGPRs used by the shader. 11515 ".stack_frame_size_in_bytes" integer Amount of stack size used by the shader. 11516 ".shader_subtype" string Shader subtype/kind. Values include: 11517 11518 - "Unknown" 11519 11520 ============================= ============== ================================================================= 11521 11522.. 11523 11524 .. table:: AMDPAL Code Object Register Map 11525 :name: amdgpu-amdpal-code-object-register-map-table 11526 11527 ========================== ============== ==================================================================== 11528 32-bit Integer Key Value Type Description 11529 ========================== ============== ==================================================================== 11530 ``reg offset`` 32-bit integer ``reg offset`` is the dword offset into the GFXIP register space of 11531 a GRBM register (i.e., driver accessible GPU register number, not 11532 shader GPR register number). The driver is required to program each 11533 specified register to the corresponding specified value when 11534 executing this pipeline. Typically, the ``reg offsets`` are the 11535 ``uint16_t`` offsets to each register as defined by the hardware 11536 chip headers. The register is set to the provided value. However, a 11537 ``reg offset`` that specifies a user data register (e.g., 11538 COMPUTE_USER_DATA_0) needs special treatment. See 11539 :ref:`amdgpu-amdpal-code-object-user-data-section` section for more 11540 information. 11541 ========================== ============== ==================================================================== 11542 11543.. _amdgpu-amdpal-code-object-user-data-section: 11544 11545User Data 11546+++++++++ 11547 11548Each hardware stage has a set of 32-bit physical SPI *user data registers* 11549(either 16 or 32 based on graphics IP and the stage) which can be 11550written from a command buffer and then loaded into SGPRs when waves are 11551launched via a subsequent dispatch or draw operation. This is the way 11552most arguments are passed from the application/runtime to a hardware 11553shader. 11554 11555PAL abstracts this functionality by exposing a set of 128 *user data 11556entries* per pipeline a client can use to pass arguments from a command 11557buffer to one or more shaders in that pipeline. The ELF code object must 11558specify a mapping from virtualized *user data entries* to physical *user 11559data registers*, and PAL is responsible for implementing that mapping, 11560including spilling overflow *user data entries* to memory if needed. 11561 11562Since the *user data registers* are GRBM-accessible SPI registers, this 11563mapping is actually embedded in the ``.registers`` metadata entry. For 11564most registers, the value in that map is a literal 32-bit value that 11565should be written to the register by the driver. However, when the 11566register is a *user data register* (any USER_DATA register e.g., 11567SPI_SHADER_USER_DATA_PS_5), the value is instead an encoding that tells 11568the driver to write either a *user data entry* value or one of several 11569driver-internal values to the register. This encoding is described in 11570the following table: 11571 11572.. note:: 11573 11574 Currently, *user data registers* 0 and 1 (e.g., SPI_SHADER_USER_DATA_PS_0, 11575 and SPI_SHADER_USER_DATA_PS_1) are reserved. *User data register* 0 must 11576 always be programmed to the address of the GlobalTable, and *user data 11577 register* 1 must always be programmed to the address of the PerShaderTable. 11578 11579.. 11580 11581 .. table:: AMDPAL User Data Mapping 11582 :name: amdgpu-amdpal-code-object-metadata-user-data-mapping-table 11583 11584 ========== ================= =============================================================================== 11585 Value Name Description 11586 ========== ================= =============================================================================== 11587 0..127 *User Data Entry* 32-bit value of user_data_entry[N] as specified via *CmdSetUserData()* 11588 0x10000000 GlobalTable 32-bit pointer to GPU memory containing the global internal table (should 11589 always point to *user data register* 0). 11590 0x10000001 PerShaderTable 32-bit pointer to GPU memory containing the per-shader internal table. See 11591 :ref:`amdgpu-amdpal-code-object-metadata-user-data-per-shader-table-section` 11592 for more detail (should always point to *user data register* 1). 11593 0x10000002 SpillTable 32-bit pointer to GPU memory containing the user data spill table. See 11594 :ref:`amdgpu-amdpal-code-object-metadata-user-data-spill-table-section` for 11595 more detail. 11596 0x10000003 BaseVertex Vertex offset (32-bit unsigned integer). Not needed if the pipeline doesn't 11597 reference the draw index in the vertex shader. Only supported by the first 11598 stage in a graphics pipeline. 11599 0x10000004 BaseInstance Instance offset (32-bit unsigned integer). Only supported by the first stage in 11600 a graphics pipeline. 11601 0x10000005 DrawIndex Draw index (32-bit unsigned integer). Only supported by the first stage in a 11602 graphics pipeline. 11603 0x10000006 Workgroup Thread group count (32-bit unsigned integer). Low half of a 64-bit address of 11604 a buffer containing the grid dimensions for a Compute dispatch operation. The 11605 high half of the address is stored in the next sequential user-SGPR. Only 11606 supported by compute pipelines. 11607 0x1000000A EsGsLdsSize Indicates that PAL will program this user-SGPR to contain the amount of LDS 11608 space used for the ES/GS pseudo-ring-buffer for passing data between shader 11609 stages. 11610 0x1000000B ViewId View id (32-bit unsigned integer) identifies a view of graphic 11611 pipeline instancing. 11612 0x1000000C StreamOutTable 32-bit pointer to GPU memory containing the stream out target SRD table. This 11613 can only appear for one shader stage per pipeline. 11614 0x1000000D PerShaderPerfData 32-bit pointer to GPU memory containing the per-shader performance data buffer. 11615 0x1000000F VertexBufferTable 32-bit pointer to GPU memory containing the vertex buffer SRD table. This can 11616 only appear for one shader stage per pipeline. 11617 0x10000010 UavExportTable 32-bit pointer to GPU memory containing the UAV export SRD table. This can 11618 only appear for one shader stage per pipeline (PS). These replace color targets 11619 and are completely separate from any UAVs used by the shader. This is optional, 11620 and only used by the PS when UAV exports are used to replace color-target 11621 exports to optimize specific shaders. 11622 0x10000011 NggCullingData 64-bit pointer to GPU memory containing the hardware register data needed by 11623 some NGG pipelines to perform culling. This value contains the address of the 11624 first of two consecutive registers which provide the full GPU address. 11625 0x10000015 FetchShaderPtr 64-bit pointer to GPU memory containing the fetch shader subroutine. 11626 ========== ================= =============================================================================== 11627 11628.. _amdgpu-amdpal-code-object-metadata-user-data-per-shader-table-section: 11629 11630Per-Shader Table 11631################ 11632 11633Low 32 bits of the GPU address for an optional buffer in the ``.data`` 11634section of the ELF. The high 32 bits of the address match the high 32 bits 11635of the shader's program counter. 11636 11637The buffer can be anything the shader compiler needs it for, and 11638allows each shader to have its own region of the ``.data`` section. 11639Typically, this could be a table of buffer SRD's and the data pointed to 11640by the buffer SRD's, but it could be a flat-address region of memory as 11641well. Its layout and usage are defined by the shader compiler. 11642 11643Each shader's table in the ``.data`` section is referenced by the symbol 11644``_amdgpu_``\ *xs*\ ``_shdr_intrl_data`` where *xs* corresponds with the 11645hardware shader stage the data is for. E.g., 11646``_amdgpu_cs_shdr_intrl_data`` for the compute shader hardware stage. 11647 11648.. _amdgpu-amdpal-code-object-metadata-user-data-spill-table-section: 11649 11650Spill Table 11651########### 11652 11653It is possible for a hardware shader to need access to more *user data 11654entries* than there are slots available in user data registers for one 11655or more hardware shader stages. In that case, the PAL runtime expects 11656the necessary *user data entries* to be spilled to GPU memory and use 11657one user data register to point to the spilled user data memory. The 11658value of the *user data entry* must then represent the location where 11659a shader expects to read the low 32-bits of the table's GPU virtual 11660address. The *spill table* itself represents a set of 32-bit values 11661managed by the PAL runtime in GPU-accessible memory that can be made 11662indirectly accessible to a hardware shader. 11663 11664Unspecified OS 11665-------------- 11666 11667This section provides code conventions used when the target triple OS is 11668empty (see :ref:`amdgpu-target-triples`). 11669 11670Trap Handler ABI 11671~~~~~~~~~~~~~~~~ 11672 11673For code objects generated by AMDGPU backend for non-amdhsa OS, the runtime does 11674not install a trap handler. The ``llvm.trap`` and ``llvm.debugtrap`` 11675instructions are handled as follows: 11676 11677 .. table:: AMDGPU Trap Handler for Non-AMDHSA OS 11678 :name: amdgpu-trap-handler-for-non-amdhsa-os-table 11679 11680 =============== =============== =========================================== 11681 Usage Code Sequence Description 11682 =============== =============== =========================================== 11683 llvm.trap s_endpgm Causes wavefront to be terminated. 11684 llvm.debugtrap *none* Compiler warning given that there is no 11685 trap handler installed. 11686 =============== =============== =========================================== 11687 11688Source Languages 11689================ 11690 11691.. _amdgpu-opencl: 11692 11693OpenCL 11694------ 11695 11696When the language is OpenCL the following differences occur: 11697 116981. The OpenCL memory model is used (see :ref:`amdgpu-amdhsa-memory-model`). 116992. The AMDGPU backend appends additional arguments to the kernel's explicit 11700 arguments for the AMDHSA OS (see 11701 :ref:`opencl-kernel-implicit-arguments-appended-for-amdhsa-os-table`). 117023. Additional metadata is generated 11703 (see :ref:`amdgpu-amdhsa-code-object-metadata`). 11704 11705 .. table:: OpenCL kernel implicit arguments appended for AMDHSA OS 11706 :name: opencl-kernel-implicit-arguments-appended-for-amdhsa-os-table 11707 11708 ======== ==== ========= =========================================== 11709 Position Byte Byte Description 11710 Size Alignment 11711 ======== ==== ========= =========================================== 11712 1 8 8 OpenCL Global Offset X 11713 2 8 8 OpenCL Global Offset Y 11714 3 8 8 OpenCL Global Offset Z 11715 4 8 8 OpenCL address of printf buffer 11716 5 8 8 OpenCL address of virtual queue used by 11717 enqueue_kernel. 11718 6 8 8 OpenCL address of AqlWrap struct used by 11719 enqueue_kernel. 11720 7 8 8 Pointer argument used for Multi-gird 11721 synchronization. 11722 ======== ==== ========= =========================================== 11723 11724.. _amdgpu-hcc: 11725 11726HCC 11727--- 11728 11729When the language is HCC the following differences occur: 11730 117311. The HSA memory model is used (see :ref:`amdgpu-amdhsa-memory-model`). 11732 11733.. _amdgpu-assembler: 11734 11735Assembler 11736--------- 11737 11738AMDGPU backend has LLVM-MC based assembler which is currently in development. 11739It supports AMDGCN GFX6-GFX10. 11740 11741This section describes general syntax for instructions and operands. 11742 11743Instructions 11744~~~~~~~~~~~~ 11745 11746An instruction has the following :doc:`syntax<AMDGPUInstructionSyntax>`: 11747 11748 | ``<``\ *opcode*\ ``> <``\ *operand0*\ ``>, <``\ *operand1*\ ``>,... 11749 <``\ *modifier0*\ ``> <``\ *modifier1*\ ``>...`` 11750 11751:doc:`Operands<AMDGPUOperandSyntax>` are comma-separated while 11752:doc:`modifiers<AMDGPUModifierSyntax>` are space-separated. 11753 11754The order of operands and modifiers is fixed. 11755Most modifiers are optional and may be omitted. 11756 11757Links to detailed instruction syntax description may be found in the following 11758table. Note that features under development are not included 11759in this description. 11760 11761 =================================== ======================================= 11762 Core ISA ISA Extensions 11763 =================================== ======================================= 11764 :doc:`GFX7<AMDGPU/AMDGPUAsmGFX7>` \- 11765 :doc:`GFX8<AMDGPU/AMDGPUAsmGFX8>` \- 11766 :doc:`GFX9<AMDGPU/AMDGPUAsmGFX9>` :doc:`gfx900<AMDGPU/AMDGPUAsmGFX900>` 11767 11768 :doc:`gfx902<AMDGPU/AMDGPUAsmGFX900>` 11769 11770 :doc:`gfx904<AMDGPU/AMDGPUAsmGFX904>` 11771 11772 :doc:`gfx906<AMDGPU/AMDGPUAsmGFX906>` 11773 11774 :doc:`gfx908<AMDGPU/AMDGPUAsmGFX908>` 11775 11776 :doc:`gfx909<AMDGPU/AMDGPUAsmGFX900>` 11777 11778 :doc:`gfx90a<AMDGPU/AMDGPUAsmGFX90a>` 11779 11780 :doc:`GFX10<AMDGPU/AMDGPUAsmGFX10>` :doc:`gfx1011<AMDGPU/AMDGPUAsmGFX1011>` 11781 11782 :doc:`gfx1012<AMDGPU/AMDGPUAsmGFX1011>` 11783 =================================== ======================================= 11784 11785For more information about instructions, their semantics and supported 11786combinations of operands, refer to one of instruction set architecture manuals 11787[AMD-GCN-GFX6]_, [AMD-GCN-GFX7]_, [AMD-GCN-GFX8]_, 11788[AMD-GCN-GFX900-GFX904-VEGA]_, [AMD-GCN-GFX906-VEGA7NM]_ 11789[AMD-GCN-GFX908-CDNA1]_, [AMD-GCN-GFX10-RDNA1]_ and [AMD-GCN-GFX10-RDNA2]_. 11790 11791Operands 11792~~~~~~~~ 11793 11794Detailed description of operands may be found :doc:`here<AMDGPUOperandSyntax>`. 11795 11796Modifiers 11797~~~~~~~~~ 11798 11799Detailed description of modifiers may be found 11800:doc:`here<AMDGPUModifierSyntax>`. 11801 11802Instruction Examples 11803~~~~~~~~~~~~~~~~~~~~ 11804 11805DS 11806++ 11807 11808.. code-block:: nasm 11809 11810 ds_add_u32 v2, v4 offset:16 11811 ds_write_src2_b64 v2 offset0:4 offset1:8 11812 ds_cmpst_f32 v2, v4, v6 11813 ds_min_rtn_f64 v[8:9], v2, v[4:5] 11814 11815For full list of supported instructions, refer to "LDS/GDS instructions" in ISA 11816Manual. 11817 11818FLAT 11819++++ 11820 11821.. code-block:: nasm 11822 11823 flat_load_dword v1, v[3:4] 11824 flat_store_dwordx3 v[3:4], v[5:7] 11825 flat_atomic_swap v1, v[3:4], v5 glc 11826 flat_atomic_cmpswap v1, v[3:4], v[5:6] glc slc 11827 flat_atomic_fmax_x2 v[1:2], v[3:4], v[5:6] glc 11828 11829For full list of supported instructions, refer to "FLAT instructions" in ISA 11830Manual. 11831 11832MUBUF 11833+++++ 11834 11835.. code-block:: nasm 11836 11837 buffer_load_dword v1, off, s[4:7], s1 11838 buffer_store_dwordx4 v[1:4], v2, ttmp[4:7], s1 offen offset:4 glc tfe 11839 buffer_store_format_xy v[1:2], off, s[4:7], s1 11840 buffer_wbinvl1 11841 buffer_atomic_inc v1, v2, s[8:11], s4 idxen offset:4 slc 11842 11843For full list of supported instructions, refer to "MUBUF Instructions" in ISA 11844Manual. 11845 11846SMRD/SMEM 11847+++++++++ 11848 11849.. code-block:: nasm 11850 11851 s_load_dword s1, s[2:3], 0xfc 11852 s_load_dwordx8 s[8:15], s[2:3], s4 11853 s_load_dwordx16 s[88:103], s[2:3], s4 11854 s_dcache_inv_vol 11855 s_memtime s[4:5] 11856 11857For full list of supported instructions, refer to "Scalar Memory Operations" in 11858ISA Manual. 11859 11860SOP1 11861++++ 11862 11863.. code-block:: nasm 11864 11865 s_mov_b32 s1, s2 11866 s_mov_b64 s[0:1], 0x80000000 11867 s_cmov_b32 s1, 200 11868 s_wqm_b64 s[2:3], s[4:5] 11869 s_bcnt0_i32_b64 s1, s[2:3] 11870 s_swappc_b64 s[2:3], s[4:5] 11871 s_cbranch_join s[4:5] 11872 11873For full list of supported instructions, refer to "SOP1 Instructions" in ISA 11874Manual. 11875 11876SOP2 11877++++ 11878 11879.. code-block:: nasm 11880 11881 s_add_u32 s1, s2, s3 11882 s_and_b64 s[2:3], s[4:5], s[6:7] 11883 s_cselect_b32 s1, s2, s3 11884 s_andn2_b32 s2, s4, s6 11885 s_lshr_b64 s[2:3], s[4:5], s6 11886 s_ashr_i32 s2, s4, s6 11887 s_bfm_b64 s[2:3], s4, s6 11888 s_bfe_i64 s[2:3], s[4:5], s6 11889 s_cbranch_g_fork s[4:5], s[6:7] 11890 11891For full list of supported instructions, refer to "SOP2 Instructions" in ISA 11892Manual. 11893 11894SOPC 11895++++ 11896 11897.. code-block:: nasm 11898 11899 s_cmp_eq_i32 s1, s2 11900 s_bitcmp1_b32 s1, s2 11901 s_bitcmp0_b64 s[2:3], s4 11902 s_setvskip s3, s5 11903 11904For full list of supported instructions, refer to "SOPC Instructions" in ISA 11905Manual. 11906 11907SOPP 11908++++ 11909 11910.. code-block:: nasm 11911 11912 s_barrier 11913 s_nop 2 11914 s_endpgm 11915 s_waitcnt 0 ; Wait for all counters to be 0 11916 s_waitcnt vmcnt(0) & expcnt(0) & lgkmcnt(0) ; Equivalent to above 11917 s_waitcnt vmcnt(1) ; Wait for vmcnt counter to be 1. 11918 s_sethalt 9 11919 s_sleep 10 11920 s_sendmsg 0x1 11921 s_sendmsg sendmsg(MSG_INTERRUPT) 11922 s_trap 1 11923 11924For full list of supported instructions, refer to "SOPP Instructions" in ISA 11925Manual. 11926 11927Unless otherwise mentioned, little verification is performed on the operands 11928of SOPP Instructions, so it is up to the programmer to be familiar with the 11929range or acceptable values. 11930 11931VALU 11932++++ 11933 11934For vector ALU instruction opcodes (VOP1, VOP2, VOP3, VOPC, VOP_DPP, VOP_SDWA), 11935the assembler will automatically use optimal encoding based on its operands. To 11936force specific encoding, one can add a suffix to the opcode of the instruction: 11937 11938* _e32 for 32-bit VOP1/VOP2/VOPC 11939* _e64 for 64-bit VOP3 11940* _dpp for VOP_DPP 11941* _sdwa for VOP_SDWA 11942 11943VOP1/VOP2/VOP3/VOPC examples: 11944 11945.. code-block:: nasm 11946 11947 v_mov_b32 v1, v2 11948 v_mov_b32_e32 v1, v2 11949 v_nop 11950 v_cvt_f64_i32_e32 v[1:2], v2 11951 v_floor_f32_e32 v1, v2 11952 v_bfrev_b32_e32 v1, v2 11953 v_add_f32_e32 v1, v2, v3 11954 v_mul_i32_i24_e64 v1, v2, 3 11955 v_mul_i32_i24_e32 v1, -3, v3 11956 v_mul_i32_i24_e32 v1, -100, v3 11957 v_addc_u32 v1, s[0:1], v2, v3, s[2:3] 11958 v_max_f16_e32 v1, v2, v3 11959 11960VOP_DPP examples: 11961 11962.. code-block:: nasm 11963 11964 v_mov_b32 v0, v0 quad_perm:[0,2,1,1] 11965 v_sin_f32 v0, v0 row_shl:1 row_mask:0xa bank_mask:0x1 bound_ctrl:0 11966 v_mov_b32 v0, v0 wave_shl:1 11967 v_mov_b32 v0, v0 row_mirror 11968 v_mov_b32 v0, v0 row_bcast:31 11969 v_mov_b32 v0, v0 quad_perm:[1,3,0,1] row_mask:0xa bank_mask:0x1 bound_ctrl:0 11970 v_add_f32 v0, v0, |v0| row_shl:1 row_mask:0xa bank_mask:0x1 bound_ctrl:0 11971 v_max_f16 v1, v2, v3 row_shl:1 row_mask:0xa bank_mask:0x1 bound_ctrl:0 11972 11973VOP_SDWA examples: 11974 11975.. code-block:: nasm 11976 11977 v_mov_b32 v1, v2 dst_sel:BYTE_0 dst_unused:UNUSED_PRESERVE src0_sel:DWORD 11978 v_min_u32 v200, v200, v1 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:BYTE_1 src1_sel:DWORD 11979 v_sin_f32 v0, v0 dst_unused:UNUSED_PAD src0_sel:WORD_1 11980 v_fract_f32 v0, |v0| dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 11981 v_cmpx_le_u32 vcc, v1, v2 src0_sel:BYTE_2 src1_sel:WORD_0 11982 11983For full list of supported instructions, refer to "Vector ALU instructions". 11984 11985.. _amdgpu-amdhsa-assembler-predefined-symbols-v2: 11986 11987Code Object V2 Predefined Symbols 11988~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 11989 11990.. warning:: 11991 Code object V2 is not the default code object version emitted by 11992 this version of LLVM. 11993 11994The AMDGPU assembler defines and updates some symbols automatically. These 11995symbols do not affect code generation. 11996 11997.option.machine_version_major 11998+++++++++++++++++++++++++++++ 11999 12000Set to the GFX major generation number of the target being assembled for. For 12001example, when assembling for a "GFX9" target this will be set to the integer 12002value "9". The possible GFX major generation numbers are presented in 12003:ref:`amdgpu-processors`. 12004 12005.option.machine_version_minor 12006+++++++++++++++++++++++++++++ 12007 12008Set to the GFX minor generation number of the target being assembled for. For 12009example, when assembling for a "GFX810" target this will be set to the integer 12010value "1". The possible GFX minor generation numbers are presented in 12011:ref:`amdgpu-processors`. 12012 12013.option.machine_version_stepping 12014++++++++++++++++++++++++++++++++ 12015 12016Set to the GFX stepping generation number of the target being assembled for. 12017For example, when assembling for a "GFX704" target this will be set to the 12018integer value "4". The possible GFX stepping generation numbers are presented 12019in :ref:`amdgpu-processors`. 12020 12021.kernel.vgpr_count 12022++++++++++++++++++ 12023 12024Set to zero each time a 12025:ref:`amdgpu-amdhsa-assembler-directive-amdgpu_hsa_kernel` directive is 12026encountered. At each instruction, if the current value of this symbol is less 12027than or equal to the maximum VGPR number explicitly referenced within that 12028instruction then the symbol value is updated to equal that VGPR number plus 12029one. 12030 12031.kernel.sgpr_count 12032++++++++++++++++++ 12033 12034Set to zero each time a 12035:ref:`amdgpu-amdhsa-assembler-directive-amdgpu_hsa_kernel` directive is 12036encountered. At each instruction, if the current value of this symbol is less 12037than or equal to the maximum VGPR number explicitly referenced within that 12038instruction then the symbol value is updated to equal that SGPR number plus 12039one. 12040 12041.. _amdgpu-amdhsa-assembler-directives-v2: 12042 12043Code Object V2 Directives 12044~~~~~~~~~~~~~~~~~~~~~~~~~ 12045 12046.. warning:: 12047 Code object V2 is not the default code object version emitted by 12048 this version of LLVM. 12049 12050AMDGPU ABI defines auxiliary data in output code object. In assembly source, 12051one can specify them with assembler directives. 12052 12053.hsa_code_object_version major, minor 12054+++++++++++++++++++++++++++++++++++++ 12055 12056*major* and *minor* are integers that specify the version of the HSA code 12057object that will be generated by the assembler. 12058 12059.hsa_code_object_isa [major, minor, stepping, vendor, arch] 12060+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 12061 12062 12063*major*, *minor*, and *stepping* are all integers that describe the instruction 12064set architecture (ISA) version of the assembly program. 12065 12066*vendor* and *arch* are quoted strings. *vendor* should always be equal to 12067"AMD" and *arch* should always be equal to "AMDGPU". 12068 12069By default, the assembler will derive the ISA version, *vendor*, and *arch* 12070from the value of the -mcpu option that is passed to the assembler. 12071 12072.. _amdgpu-amdhsa-assembler-directive-amdgpu_hsa_kernel: 12073 12074.amdgpu_hsa_kernel (name) 12075+++++++++++++++++++++++++ 12076 12077This directives specifies that the symbol with given name is a kernel entry 12078point (label) and the object should contain corresponding symbol of type 12079STT_AMDGPU_HSA_KERNEL. 12080 12081.amd_kernel_code_t 12082++++++++++++++++++ 12083 12084This directive marks the beginning of a list of key / value pairs that are used 12085to specify the amd_kernel_code_t object that will be emitted by the assembler. 12086The list must be terminated by the *.end_amd_kernel_code_t* directive. For any 12087amd_kernel_code_t values that are unspecified a default value will be used. The 12088default value for all keys is 0, with the following exceptions: 12089 12090- *amd_code_version_major* defaults to 1. 12091- *amd_kernel_code_version_minor* defaults to 2. 12092- *amd_machine_kind* defaults to 1. 12093- *amd_machine_version_major*, *machine_version_minor*, and 12094 *amd_machine_version_stepping* are derived from the value of the -mcpu option 12095 that is passed to the assembler. 12096- *kernel_code_entry_byte_offset* defaults to 256. 12097- *wavefront_size* defaults 6 for all targets before GFX10. For GFX10 onwards 12098 defaults to 6 if target feature ``wavefrontsize64`` is enabled, otherwise 5. 12099 Note that wavefront size is specified as a power of two, so a value of **n** 12100 means a size of 2^ **n**. 12101- *call_convention* defaults to -1. 12102- *kernarg_segment_alignment*, *group_segment_alignment*, and 12103 *private_segment_alignment* default to 4. Note that alignments are specified 12104 as a power of 2, so a value of **n** means an alignment of 2^ **n**. 12105- *enable_tg_split* defaults to 1 if target feature ``tgsplit`` is enabled for 12106 GFX90A onwards. 12107- *enable_wgp_mode* defaults to 1 if target feature ``cumode`` is disabled for 12108 GFX10 onwards. 12109- *enable_mem_ordered* defaults to 1 for GFX10 onwards. 12110 12111The *.amd_kernel_code_t* directive must be placed immediately after the 12112function label and before any instructions. 12113 12114For a full list of amd_kernel_code_t keys, refer to AMDGPU ABI document, 12115comments in lib/Target/AMDGPU/AmdKernelCodeT.h and test/CodeGen/AMDGPU/hsa.s. 12116 12117.. _amdgpu-amdhsa-assembler-example-v2: 12118 12119Code Object V2 Example Source Code 12120~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 12121 12122.. warning:: 12123 Code Object V2 is not the default code object version emitted by 12124 this version of LLVM. 12125 12126Here is an example of a minimal assembly source file, defining one HSA kernel: 12127 12128.. code:: 12129 :number-lines: 12130 12131 .hsa_code_object_version 1,0 12132 .hsa_code_object_isa 12133 12134 .hsatext 12135 .globl hello_world 12136 .p2align 8 12137 .amdgpu_hsa_kernel hello_world 12138 12139 hello_world: 12140 12141 .amd_kernel_code_t 12142 enable_sgpr_kernarg_segment_ptr = 1 12143 is_ptr64 = 1 12144 compute_pgm_rsrc1_vgprs = 0 12145 compute_pgm_rsrc1_sgprs = 0 12146 compute_pgm_rsrc2_user_sgpr = 2 12147 compute_pgm_rsrc1_wgp_mode = 0 12148 compute_pgm_rsrc1_mem_ordered = 0 12149 compute_pgm_rsrc1_fwd_progress = 1 12150 .end_amd_kernel_code_t 12151 12152 s_load_dwordx2 s[0:1], s[0:1] 0x0 12153 v_mov_b32 v0, 3.14159 12154 s_waitcnt lgkmcnt(0) 12155 v_mov_b32 v1, s0 12156 v_mov_b32 v2, s1 12157 flat_store_dword v[1:2], v0 12158 s_endpgm 12159 .Lfunc_end0: 12160 .size hello_world, .Lfunc_end0-hello_world 12161 12162.. _amdgpu-amdhsa-assembler-predefined-symbols-v3-onwards: 12163 12164Code Object V3 and Above Predefined Symbols 12165~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 12166 12167The AMDGPU assembler defines and updates some symbols automatically. These 12168symbols do not affect code generation. 12169 12170.amdgcn.gfx_generation_number 12171+++++++++++++++++++++++++++++ 12172 12173Set to the GFX major generation number of the target being assembled for. For 12174example, when assembling for a "GFX9" target this will be set to the integer 12175value "9". The possible GFX major generation numbers are presented in 12176:ref:`amdgpu-processors`. 12177 12178.amdgcn.gfx_generation_minor 12179++++++++++++++++++++++++++++ 12180 12181Set to the GFX minor generation number of the target being assembled for. For 12182example, when assembling for a "GFX810" target this will be set to the integer 12183value "1". The possible GFX minor generation numbers are presented in 12184:ref:`amdgpu-processors`. 12185 12186.amdgcn.gfx_generation_stepping 12187+++++++++++++++++++++++++++++++ 12188 12189Set to the GFX stepping generation number of the target being assembled for. 12190For example, when assembling for a "GFX704" target this will be set to the 12191integer value "4". The possible GFX stepping generation numbers are presented 12192in :ref:`amdgpu-processors`. 12193 12194.. _amdgpu-amdhsa-assembler-symbol-next_free_vgpr: 12195 12196.amdgcn.next_free_vgpr 12197++++++++++++++++++++++ 12198 12199Set to zero before assembly begins. At each instruction, if the current value 12200of this symbol is less than or equal to the maximum VGPR number explicitly 12201referenced within that instruction then the symbol value is updated to equal 12202that VGPR number plus one. 12203 12204May be used to set the `.amdhsa_next_free_vgpr` directive in 12205:ref:`amdhsa-kernel-directives-table`. 12206 12207May be set at any time, e.g. manually set to zero at the start of each kernel. 12208 12209.. _amdgpu-amdhsa-assembler-symbol-next_free_sgpr: 12210 12211.amdgcn.next_free_sgpr 12212++++++++++++++++++++++ 12213 12214Set to zero before assembly begins. At each instruction, if the current value 12215of this symbol is less than or equal the maximum SGPR number explicitly 12216referenced within that instruction then the symbol value is updated to equal 12217that SGPR number plus one. 12218 12219May be used to set the `.amdhsa_next_free_spgr` directive in 12220:ref:`amdhsa-kernel-directives-table`. 12221 12222May be set at any time, e.g. manually set to zero at the start of each kernel. 12223 12224.. _amdgpu-amdhsa-assembler-directives-v3-onwards: 12225 12226Code Object V3 and Above Directives 12227~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 12228 12229Directives which begin with ``.amdgcn`` are valid for all ``amdgcn`` 12230architecture processors, and are not OS-specific. Directives which begin with 12231``.amdhsa`` are specific to ``amdgcn`` architecture processors when the 12232``amdhsa`` OS is specified. See :ref:`amdgpu-target-triples` and 12233:ref:`amdgpu-processors`. 12234 12235.. _amdgpu-assembler-directive-amdgcn-target: 12236 12237.amdgcn_target <target-triple> "-" <target-id> 12238++++++++++++++++++++++++++++++++++++++++++++++ 12239 12240Optional directive which declares the ``<target-triple>-<target-id>`` supported 12241by the containing assembler source file. Used by the assembler to validate 12242command-line options such as ``-triple``, ``-mcpu``, and 12243``--offload-arch=<target-id>``. A non-canonical target ID is allowed. See 12244:ref:`amdgpu-target-triples` and :ref:`amdgpu-target-id`. 12245 12246.. note:: 12247 12248 The target ID syntax used for code object V2 to V3 for this directive differs 12249 from that used elsewhere. See :ref:`amdgpu-target-id-v2-v3`. 12250 12251.amdhsa_kernel <name> 12252+++++++++++++++++++++ 12253 12254Creates a correctly aligned AMDHSA kernel descriptor and a symbol, 12255``<name>.kd``, in the current location of the current section. Only valid when 12256the OS is ``amdhsa``. ``<name>`` must be a symbol that labels the first 12257instruction to execute, and does not need to be previously defined. 12258 12259Marks the beginning of a list of directives used to generate the bytes of a 12260kernel descriptor, as described in :ref:`amdgpu-amdhsa-kernel-descriptor`. 12261Directives which may appear in this list are described in 12262:ref:`amdhsa-kernel-directives-table`. Directives may appear in any order, must 12263be valid for the target being assembled for, and cannot be repeated. Directives 12264support the range of values specified by the field they reference in 12265:ref:`amdgpu-amdhsa-kernel-descriptor`. If a directive is not specified, it is 12266assumed to have its default value, unless it is marked as "Required", in which 12267case it is an error to omit the directive. This list of directives is 12268terminated by an ``.end_amdhsa_kernel`` directive. 12269 12270 .. table:: AMDHSA Kernel Assembler Directives 12271 :name: amdhsa-kernel-directives-table 12272 12273 ======================================================== =================== ============ =================== 12274 Directive Default Supported On Description 12275 ======================================================== =================== ============ =================== 12276 ``.amdhsa_group_segment_fixed_size`` 0 GFX6-GFX10 Controls GROUP_SEGMENT_FIXED_SIZE in 12277 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 12278 ``.amdhsa_private_segment_fixed_size`` 0 GFX6-GFX10 Controls PRIVATE_SEGMENT_FIXED_SIZE in 12279 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 12280 ``.amdhsa_kernarg_size`` 0 GFX6-GFX10 Controls KERNARG_SIZE in 12281 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 12282 ``.amdhsa_user_sgpr_count`` 0 GFX6-GFX10 Controls USER_SGPR_COUNT in COMPUTE_PGM_RSRC2 12283 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table` 12284 ``.amdhsa_user_sgpr_private_segment_buffer`` 0 GFX6-GFX10 Controls ENABLE_SGPR_PRIVATE_SEGMENT_BUFFER in 12285 (except :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 12286 GFX940) 12287 ``.amdhsa_user_sgpr_dispatch_ptr`` 0 GFX6-GFX10 Controls ENABLE_SGPR_DISPATCH_PTR in 12288 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 12289 ``.amdhsa_user_sgpr_queue_ptr`` 0 GFX6-GFX10 Controls ENABLE_SGPR_QUEUE_PTR in 12290 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 12291 ``.amdhsa_user_sgpr_kernarg_segment_ptr`` 0 GFX6-GFX10 Controls ENABLE_SGPR_KERNARG_SEGMENT_PTR in 12292 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 12293 ``.amdhsa_user_sgpr_dispatch_id`` 0 GFX6-GFX10 Controls ENABLE_SGPR_DISPATCH_ID in 12294 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 12295 ``.amdhsa_user_sgpr_flat_scratch_init`` 0 GFX6-GFX10 Controls ENABLE_SGPR_FLAT_SCRATCH_INIT in 12296 (except :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 12297 GFX940) 12298 ``.amdhsa_user_sgpr_private_segment_size`` 0 GFX6-GFX10 Controls ENABLE_SGPR_PRIVATE_SEGMENT_SIZE in 12299 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 12300 ``.amdhsa_wavefront_size32`` Target GFX10 Controls ENABLE_WAVEFRONT_SIZE32 in 12301 Feature :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 12302 Specific 12303 (wavefrontsize64) 12304 ``.amdhsa_system_sgpr_private_segment_wavefront_offset`` 0 GFX6-GFX10 Controls ENABLE_PRIVATE_SEGMENT in 12305 (except :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 12306 GFX940) 12307 ``.amdhsa_enable_private_segment`` 0 GFX940 Controls ENABLE_PRIVATE_SEGMENT in 12308 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 12309 ``.amdhsa_system_sgpr_workgroup_id_x`` 1 GFX6-GFX10 Controls ENABLE_SGPR_WORKGROUP_ID_X in 12310 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 12311 ``.amdhsa_system_sgpr_workgroup_id_y`` 0 GFX6-GFX10 Controls ENABLE_SGPR_WORKGROUP_ID_Y in 12312 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 12313 ``.amdhsa_system_sgpr_workgroup_id_z`` 0 GFX6-GFX10 Controls ENABLE_SGPR_WORKGROUP_ID_Z in 12314 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 12315 ``.amdhsa_system_sgpr_workgroup_info`` 0 GFX6-GFX10 Controls ENABLE_SGPR_WORKGROUP_INFO in 12316 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 12317 ``.amdhsa_system_vgpr_workitem_id`` 0 GFX6-GFX10 Controls ENABLE_VGPR_WORKITEM_ID in 12318 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 12319 Possible values are defined in 12320 :ref:`amdgpu-amdhsa-system-vgpr-work-item-id-enumeration-values-table`. 12321 ``.amdhsa_next_free_vgpr`` Required GFX6-GFX10 Maximum VGPR number explicitly referenced, plus one. 12322 Used to calculate GRANULATED_WORKITEM_VGPR_COUNT in 12323 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 12324 ``.amdhsa_next_free_sgpr`` Required GFX6-GFX10 Maximum SGPR number explicitly referenced, plus one. 12325 Used to calculate GRANULATED_WAVEFRONT_SGPR_COUNT in 12326 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 12327 ``.amdhsa_accum_offset`` Required GFX90A, Offset of a first AccVGPR in the unified register file. 12328 GFX940 Used to calculate ACCUM_OFFSET in 12329 :ref:`amdgpu-amdhsa-compute_pgm_rsrc3-gfx90a-table`. 12330 ``.amdhsa_reserve_vcc`` 1 GFX6-GFX10 Whether the kernel may use the special VCC SGPR. 12331 Used to calculate GRANULATED_WAVEFRONT_SGPR_COUNT in 12332 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 12333 ``.amdhsa_reserve_flat_scratch`` 1 GFX7-GFX10 Whether the kernel may use flat instructions to access 12334 (except scratch memory. Used to calculate 12335 GFX940) GRANULATED_WAVEFRONT_SGPR_COUNT in 12336 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 12337 ``.amdhsa_reserve_xnack_mask`` Target GFX8-GFX10 Whether the kernel may trigger XNACK replay. 12338 Feature Used to calculate GRANULATED_WAVEFRONT_SGPR_COUNT in 12339 Specific :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 12340 (xnack) 12341 ``.amdhsa_float_round_mode_32`` 0 GFX6-GFX10 Controls FLOAT_ROUND_MODE_32 in 12342 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 12343 Possible values are defined in 12344 :ref:`amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table`. 12345 ``.amdhsa_float_round_mode_16_64`` 0 GFX6-GFX10 Controls FLOAT_ROUND_MODE_16_64 in 12346 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 12347 Possible values are defined in 12348 :ref:`amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table`. 12349 ``.amdhsa_float_denorm_mode_32`` 0 GFX6-GFX10 Controls FLOAT_DENORM_MODE_32 in 12350 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 12351 Possible values are defined in 12352 :ref:`amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table`. 12353 ``.amdhsa_float_denorm_mode_16_64`` 3 GFX6-GFX10 Controls FLOAT_DENORM_MODE_16_64 in 12354 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 12355 Possible values are defined in 12356 :ref:`amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table`. 12357 ``.amdhsa_dx10_clamp`` 1 GFX6-GFX10 Controls ENABLE_DX10_CLAMP in 12358 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 12359 ``.amdhsa_ieee_mode`` 1 GFX6-GFX10 Controls ENABLE_IEEE_MODE in 12360 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 12361 ``.amdhsa_fp16_overflow`` 0 GFX9-GFX10 Controls FP16_OVFL in 12362 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 12363 ``.amdhsa_tg_split`` Target GFX90A, Controls TG_SPLIT in 12364 Feature GFX940 :ref:`amdgpu-amdhsa-compute_pgm_rsrc3-gfx90a-table`. 12365 Specific 12366 (tgsplit) 12367 ``.amdhsa_workgroup_processor_mode`` Target GFX10 Controls ENABLE_WGP_MODE in 12368 Feature :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 12369 Specific 12370 (cumode) 12371 ``.amdhsa_memory_ordered`` 1 GFX10 Controls MEM_ORDERED in 12372 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 12373 ``.amdhsa_forward_progress`` 0 GFX10 Controls FWD_PROGRESS in 12374 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 12375 ``.amdhsa_exception_fp_ieee_invalid_op`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_INVALID_OPERATION in 12376 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 12377 ``.amdhsa_exception_fp_denorm_src`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_FP_DENORMAL_SOURCE in 12378 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 12379 ``.amdhsa_exception_fp_ieee_div_zero`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_DIVISION_BY_ZERO in 12380 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 12381 ``.amdhsa_exception_fp_ieee_overflow`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_OVERFLOW in 12382 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 12383 ``.amdhsa_exception_fp_ieee_underflow`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_UNDERFLOW in 12384 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 12385 ``.amdhsa_exception_fp_ieee_inexact`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_INEXACT in 12386 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 12387 ``.amdhsa_exception_int_div_zero`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_INT_DIVIDE_BY_ZERO in 12388 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 12389 ======================================================== =================== ============ =================== 12390 12391.amdgpu_metadata 12392++++++++++++++++ 12393 12394Optional directive which declares the contents of the ``NT_AMDGPU_METADATA`` 12395note record (see :ref:`amdgpu-elf-note-records-table-v3-onwards`). 12396 12397The contents must be in the [YAML]_ markup format, with the same structure and 12398semantics described in :ref:`amdgpu-amdhsa-code-object-metadata-v3`, 12399:ref:`amdgpu-amdhsa-code-object-metadata-v4` or 12400:ref:`amdgpu-amdhsa-code-object-metadata-v5`. 12401 12402This directive is terminated by an ``.end_amdgpu_metadata`` directive. 12403 12404.. _amdgpu-amdhsa-assembler-example-v3-onwards: 12405 12406Code Object V3 and Above Example Source Code 12407~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 12408 12409Here is an example of a minimal assembly source file, defining one HSA kernel: 12410 12411.. code:: 12412 :number-lines: 12413 12414 .amdgcn_target "amdgcn-amd-amdhsa--gfx900+xnack" // optional 12415 12416 .text 12417 .globl hello_world 12418 .p2align 8 12419 .type hello_world,@function 12420 hello_world: 12421 s_load_dwordx2 s[0:1], s[0:1] 0x0 12422 v_mov_b32 v0, 3.14159 12423 s_waitcnt lgkmcnt(0) 12424 v_mov_b32 v1, s0 12425 v_mov_b32 v2, s1 12426 flat_store_dword v[1:2], v0 12427 s_endpgm 12428 .Lfunc_end0: 12429 .size hello_world, .Lfunc_end0-hello_world 12430 12431 .rodata 12432 .p2align 6 12433 .amdhsa_kernel hello_world 12434 .amdhsa_user_sgpr_kernarg_segment_ptr 1 12435 .amdhsa_next_free_vgpr .amdgcn.next_free_vgpr 12436 .amdhsa_next_free_sgpr .amdgcn.next_free_sgpr 12437 .end_amdhsa_kernel 12438 12439 .amdgpu_metadata 12440 --- 12441 amdhsa.version: 12442 - 1 12443 - 0 12444 amdhsa.kernels: 12445 - .name: hello_world 12446 .symbol: hello_world.kd 12447 .kernarg_segment_size: 48 12448 .group_segment_fixed_size: 0 12449 .private_segment_fixed_size: 0 12450 .kernarg_segment_align: 4 12451 .wavefront_size: 64 12452 .sgpr_count: 2 12453 .vgpr_count: 3 12454 .max_flat_workgroup_size: 256 12455 .args: 12456 - .size: 8 12457 .offset: 0 12458 .value_kind: global_buffer 12459 .address_space: global 12460 .actual_access: write_only 12461 //... 12462 .end_amdgpu_metadata 12463 12464This kernel is equivalent to the following HIP program: 12465 12466.. code:: 12467 :number-lines: 12468 12469 __global__ void hello_world(float *p) { 12470 *p = 3.14159f; 12471 } 12472 12473If an assembly source file contains multiple kernels and/or functions, the 12474:ref:`amdgpu-amdhsa-assembler-symbol-next_free_vgpr` and 12475:ref:`amdgpu-amdhsa-assembler-symbol-next_free_sgpr` symbols may be reset using 12476the ``.set <symbol>, <expression>`` directive. For example, in the case of two 12477kernels, where ``function1`` is only called from ``kernel1`` it is sufficient 12478to group the function with the kernel that calls it and reset the symbols 12479between the two connected components: 12480 12481.. code:: 12482 :number-lines: 12483 12484 .amdgcn_target "amdgcn-amd-amdhsa--gfx900+xnack" // optional 12485 12486 // gpr tracking symbols are implicitly set to zero 12487 12488 .text 12489 .globl kern0 12490 .p2align 8 12491 .type kern0,@function 12492 kern0: 12493 // ... 12494 s_endpgm 12495 .Lkern0_end: 12496 .size kern0, .Lkern0_end-kern0 12497 12498 .rodata 12499 .p2align 6 12500 .amdhsa_kernel kern0 12501 // ... 12502 .amdhsa_next_free_vgpr .amdgcn.next_free_vgpr 12503 .amdhsa_next_free_sgpr .amdgcn.next_free_sgpr 12504 .end_amdhsa_kernel 12505 12506 // reset symbols to begin tracking usage in func1 and kern1 12507 .set .amdgcn.next_free_vgpr, 0 12508 .set .amdgcn.next_free_sgpr, 0 12509 12510 .text 12511 .hidden func1 12512 .global func1 12513 .p2align 2 12514 .type func1,@function 12515 func1: 12516 // ... 12517 s_setpc_b64 s[30:31] 12518 .Lfunc1_end: 12519 .size func1, .Lfunc1_end-func1 12520 12521 .globl kern1 12522 .p2align 8 12523 .type kern1,@function 12524 kern1: 12525 // ... 12526 s_getpc_b64 s[4:5] 12527 s_add_u32 s4, s4, func1@rel32@lo+4 12528 s_addc_u32 s5, s5, func1@rel32@lo+4 12529 s_swappc_b64 s[30:31], s[4:5] 12530 // ... 12531 s_endpgm 12532 .Lkern1_end: 12533 .size kern1, .Lkern1_end-kern1 12534 12535 .rodata 12536 .p2align 6 12537 .amdhsa_kernel kern1 12538 // ... 12539 .amdhsa_next_free_vgpr .amdgcn.next_free_vgpr 12540 .amdhsa_next_free_sgpr .amdgcn.next_free_sgpr 12541 .end_amdhsa_kernel 12542 12543These symbols cannot identify connected components in order to automatically 12544track the usage for each kernel. However, in some cases careful organization of 12545the kernels and functions in the source file means there is minimal additional 12546effort required to accurately calculate GPR usage. 12547 12548Additional Documentation 12549======================== 12550 12551.. [AMD-GCN-GFX6] `AMD Southern Islands Series ISA <http://developer.amd.com/wordpress/media/2012/12/AMD_Southern_Islands_Instruction_Set_Architecture.pdf>`__ 12552.. [AMD-GCN-GFX7] `AMD Sea Islands Series ISA <http://developer.amd.com/wordpress/media/2013/07/AMD_Sea_Islands_Instruction_Set_Architecture.pdf>`_ 12553.. [AMD-GCN-GFX8] `AMD GCN3 Instruction Set Architecture <http://amd-dev.wpengine.netdna-cdn.com/wordpress/media/2013/12/AMD_GCN3_Instruction_Set_Architecture_rev1.1.pdf>`__ 12554.. [AMD-GCN-GFX900-GFX904-VEGA] `AMD Vega Instruction Set Architecture <http://developer.amd.com/wordpress/media/2013/12/Vega_Shader_ISA_28July2017.pdf>`__ 12555.. [AMD-GCN-GFX906-VEGA7NM] `AMD Vega 7nm Instruction Set Architecture <https://gpuopen.com/wp-content/uploads/2019/11/Vega_7nm_Shader_ISA_26November2019.pdf>`__ 12556.. [AMD-GCN-GFX908-CDNA1] `AMD Instinct MI100 Instruction Set Architecture <https://developer.amd.com/wp-content/resources/CDNA1_Shader_ISA_14December2020.pdf>`__ 12557.. [AMD-GCN-GFX10-RDNA1] `AMD RDNA 1.0 Instruction Set Architecture <https://gpuopen.com/wp-content/uploads/2019/08/RDNA_Shader_ISA_5August2019.pdf>`__ 12558.. [AMD-GCN-GFX10-RDNA2] `AMD RDNA 2 Instruction Set Architecture <https://developer.amd.com/wp-content/resources/RDNA2_Shader_ISA_November2020.pdf>`__ 12559.. [AMD-RADEON-HD-2000-3000] `AMD R6xx shader ISA <http://developer.amd.com/wordpress/media/2012/10/R600_Instruction_Set_Architecture.pdf>`__ 12560.. [AMD-RADEON-HD-4000] `AMD R7xx shader ISA <http://developer.amd.com/wordpress/media/2012/10/R700-Family_Instruction_Set_Architecture.pdf>`__ 12561.. [AMD-RADEON-HD-5000] `AMD Evergreen shader ISA <http://developer.amd.com/wordpress/media/2012/10/AMD_Evergreen-Family_Instruction_Set_Architecture.pdf>`__ 12562.. [AMD-RADEON-HD-6000] `AMD Cayman/Trinity shader ISA <http://developer.amd.com/wordpress/media/2012/10/AMD_HD_6900_Series_Instruction_Set_Architecture.pdf>`__ 12563.. [AMD-ROCm] `AMD ROCm™ Platform <https://rocmdocs.amd.com/>`__ 12564.. [AMD-ROCm-github] `AMD ROCm™ github <http://github.com/RadeonOpenCompute>`__ 12565.. [AMD-ROCm-Release-Notes] `AMD ROCm Release Notes <https://github.com/RadeonOpenCompute/ROCm>`__ 12566.. [CLANG-ATTR] `Attributes in Clang <https://clang.llvm.org/docs/AttributeReference.html>`__ 12567.. [DWARF] `DWARF Debugging Information Format <http://dwarfstd.org/>`__ 12568.. [ELF] `Executable and Linkable Format (ELF) <http://www.sco.com/developers/gabi/>`__ 12569.. [HRF] `Heterogeneous-race-free Memory Models <http://benedictgaster.org/wp-content/uploads/2014/01/asplos269-FINAL.pdf>`__ 12570.. [HSA] `Heterogeneous System Architecture (HSA) Foundation <http://www.hsafoundation.com/>`__ 12571.. [MsgPack] `Message Pack <http://www.msgpack.org/>`__ 12572.. [OpenCL] `The OpenCL Specification Version 2.0 <http://www.khronos.org/registry/cl/specs/opencl-2.0.pdf>`__ 12573.. [SEMVER] `Semantic Versioning <https://semver.org/>`__ 12574.. [YAML] `YAML Ain't Markup Language (YAML™) Version 1.2 <http://www.yaml.org/spec/1.2/spec.html>`__ 12575