1============================= 2User Guide for AMDGPU Backend 3============================= 4 5.. contents:: 6 :local: 7 8.. toctree:: 9 :hidden: 10 11 AMDGPU/AMDGPUAsmGFX7 12 AMDGPU/AMDGPUAsmGFX8 13 AMDGPU/AMDGPUAsmGFX9 14 AMDGPU/AMDGPUAsmGFX900 15 AMDGPU/AMDGPUAsmGFX904 16 AMDGPU/AMDGPUAsmGFX906 17 AMDGPU/AMDGPUAsmGFX908 18 AMDGPU/AMDGPUAsmGFX90a 19 AMDGPU/AMDGPUAsmGFX10 20 AMDGPU/AMDGPUAsmGFX1011 21 AMDGPUModifierSyntax 22 AMDGPUOperandSyntax 23 AMDGPUInstructionSyntax 24 AMDGPUInstructionNotation 25 AMDGPUDwarfExtensionsForHeterogeneousDebugging 26 AMDGPUDwarfExtensionAllowLocationDescriptionOnTheDwarfExpressionStack/AMDGPUDwarfExtensionAllowLocationDescriptionOnTheDwarfExpressionStack 27 28Introduction 29============ 30 31The AMDGPU backend provides ISA code generation for AMD GPUs, starting with the 32R600 family up until the current GCN families. It lives in the 33``llvm/lib/Target/AMDGPU`` directory. 34 35LLVM 36==== 37 38.. _amdgpu-target-triples: 39 40Target Triples 41-------------- 42 43Use the Clang option ``-target <Architecture>-<Vendor>-<OS>-<Environment>`` 44to specify the target triple: 45 46 .. table:: AMDGPU Architectures 47 :name: amdgpu-architecture-table 48 49 ============ ============================================================== 50 Architecture Description 51 ============ ============================================================== 52 ``r600`` AMD GPUs HD2XXX-HD6XXX for graphics and compute shaders. 53 ``amdgcn`` AMD GPUs GCN GFX6 onwards for graphics and compute shaders. 54 ============ ============================================================== 55 56 .. table:: AMDGPU Vendors 57 :name: amdgpu-vendor-table 58 59 ============ ============================================================== 60 Vendor Description 61 ============ ============================================================== 62 ``amd`` Can be used for all AMD GPU usage. 63 ``mesa3d`` Can be used if the OS is ``mesa3d``. 64 ============ ============================================================== 65 66 .. table:: AMDGPU Operating Systems 67 :name: amdgpu-os 68 69 ============== ============================================================ 70 OS Description 71 ============== ============================================================ 72 *<empty>* Defaults to the *unknown* OS. 73 ``amdhsa`` Compute kernels executed on HSA [HSA]_ compatible runtimes 74 such as: 75 76 - AMD's ROCm™ runtime [AMD-ROCm]_ using the *rocm-amdhsa* 77 loader on Linux. See *AMD ROCm Platform Release Notes* 78 [AMD-ROCm-Release-Notes]_ for supported hardware and 79 software. 80 - AMD's PAL runtime using the *pal-amdhsa* loader on 81 Windows. 82 83 ``amdpal`` Graphic shaders and compute kernels executed on AMD's PAL 84 runtime using the *pal-amdpal* loader on Windows and Linux 85 Pro. 86 ``mesa3d`` Graphic shaders and compute kernels executed on AMD's Mesa 87 3D runtime using the *mesa-mesa3d* loader on Linux. 88 ============== ============================================================ 89 90 .. table:: AMDGPU Environments 91 :name: amdgpu-environment-table 92 93 ============ ============================================================== 94 Environment Description 95 ============ ============================================================== 96 *<empty>* Default. 97 ============ ============================================================== 98 99.. _amdgpu-processors: 100 101Processors 102---------- 103 104Use the Clang options ``-mcpu=<target-id>`` or ``--offload-arch=<target-id>`` to 105specify the AMDGPU processor together with optional target features. See 106:ref:`amdgpu-target-id` and :ref:`amdgpu-target-features` for AMD GPU target 107specific information. 108 109Every processor supports every OS ABI (see :ref:`amdgpu-os`) with the following exceptions: 110 111* ``amdhsa`` is not supported in ``r600`` architecture (see :ref:`amdgpu-architecture-table`). 112 113 114 .. table:: AMDGPU Processors 115 :name: amdgpu-processor-table 116 117 =========== =============== ============ ===== ================= =============== =============== ====================== 118 Processor Alternative Target dGPU/ Target Target OS Support Example 119 Processor Triple APU Features Properties *(see* Products 120 Architecture Supported `amdgpu-os`_ 121 *and 122 corresponding 123 runtime release 124 notes for 125 current 126 information and 127 level of 128 support)* 129 =========== =============== ============ ===== ================= =============== =============== ====================== 130 **Radeon HD 2000/3000 Series (R600)** [AMD-RADEON-HD-2000-3000]_ 131 ----------------------------------------------------------------------------------------------------------------------- 132 ``r600`` ``r600`` dGPU - Does not 133 support 134 generic 135 address 136 space 137 ``r630`` ``r600`` dGPU - Does not 138 support 139 generic 140 address 141 space 142 ``rs880`` ``r600`` dGPU - Does not 143 support 144 generic 145 address 146 space 147 ``rv670`` ``r600`` dGPU - Does not 148 support 149 generic 150 address 151 space 152 **Radeon HD 4000 Series (R700)** [AMD-RADEON-HD-4000]_ 153 ----------------------------------------------------------------------------------------------------------------------- 154 ``rv710`` ``r600`` dGPU - Does not 155 support 156 generic 157 address 158 space 159 ``rv730`` ``r600`` dGPU - Does not 160 support 161 generic 162 address 163 space 164 ``rv770`` ``r600`` dGPU - Does not 165 support 166 generic 167 address 168 space 169 **Radeon HD 5000 Series (Evergreen)** [AMD-RADEON-HD-5000]_ 170 ----------------------------------------------------------------------------------------------------------------------- 171 ``cedar`` ``r600`` dGPU - Does not 172 support 173 generic 174 address 175 space 176 ``cypress`` ``r600`` dGPU - Does not 177 support 178 generic 179 address 180 space 181 ``juniper`` ``r600`` dGPU - Does not 182 support 183 generic 184 address 185 space 186 ``redwood`` ``r600`` dGPU - Does not 187 support 188 generic 189 address 190 space 191 ``sumo`` ``r600`` dGPU - Does not 192 support 193 generic 194 address 195 space 196 **Radeon HD 6000 Series (Northern Islands)** [AMD-RADEON-HD-6000]_ 197 ----------------------------------------------------------------------------------------------------------------------- 198 ``barts`` ``r600`` dGPU - Does not 199 support 200 generic 201 address 202 space 203 ``caicos`` ``r600`` dGPU - Does not 204 support 205 generic 206 address 207 space 208 ``cayman`` ``r600`` dGPU - Does not 209 support 210 generic 211 address 212 space 213 ``turks`` ``r600`` dGPU - Does not 214 support 215 generic 216 address 217 space 218 **GCN GFX6 (Southern Islands (SI))** [AMD-GCN-GFX6]_ 219 ----------------------------------------------------------------------------------------------------------------------- 220 ``gfx600`` - ``tahiti`` ``amdgcn`` dGPU - Does not - *pal-amdpal* 221 support 222 generic 223 address 224 space 225 ``gfx601`` - ``pitcairn`` ``amdgcn`` dGPU - Does not - *pal-amdpal* 226 - ``verde`` support 227 generic 228 address 229 space 230 ``gfx602`` - ``hainan`` ``amdgcn`` dGPU - Does not - *pal-amdpal* 231 - ``oland`` support 232 generic 233 address 234 space 235 **GCN GFX7 (Sea Islands (CI))** [AMD-GCN-GFX7]_ 236 ----------------------------------------------------------------------------------------------------------------------- 237 ``gfx700`` - ``kaveri`` ``amdgcn`` APU - Offset - *rocm-amdhsa* - A6-7000 238 flat - *pal-amdhsa* - A6 Pro-7050B 239 scratch - *pal-amdpal* - A8-7100 240 - A8 Pro-7150B 241 - A10-7300 242 - A10 Pro-7350B 243 - FX-7500 244 - A8-7200P 245 - A10-7400P 246 - FX-7600P 247 ``gfx701`` - ``hawaii`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - FirePro W8100 248 flat - *pal-amdhsa* - FirePro W9100 249 scratch - *pal-amdpal* - FirePro S9150 250 - FirePro S9170 251 ``gfx702`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - Radeon R9 290 252 flat - *pal-amdhsa* - Radeon R9 290x 253 scratch - *pal-amdpal* - Radeon R390 254 - Radeon R390x 255 ``gfx703`` - ``kabini`` ``amdgcn`` APU - Offset - *pal-amdhsa* - E1-2100 256 - ``mullins`` flat - *pal-amdpal* - E1-2200 257 scratch - E1-2500 258 - E2-3000 259 - E2-3800 260 - A4-5000 261 - A4-5100 262 - A6-5200 263 - A4 Pro-3340B 264 ``gfx704`` - ``bonaire`` ``amdgcn`` dGPU - Offset - *pal-amdhsa* - Radeon HD 7790 265 flat - *pal-amdpal* - Radeon HD 8770 266 scratch - R7 260 267 - R7 260X 268 ``gfx705`` ``amdgcn`` APU - Offset - *pal-amdhsa* *TBA* 269 flat - *pal-amdpal* 270 scratch .. TODO:: 271 272 Add product 273 names. 274 275 **GCN GFX8 (Volcanic Islands (VI))** [AMD-GCN-GFX8]_ 276 ----------------------------------------------------------------------------------------------------------------------- 277 ``gfx801`` - ``carrizo`` ``amdgcn`` APU - xnack - Offset - *rocm-amdhsa* - A6-8500P 278 flat - *pal-amdhsa* - Pro A6-8500B 279 scratch - *pal-amdpal* - A8-8600P 280 - Pro A8-8600B 281 - FX-8800P 282 - Pro A12-8800B 283 - A10-8700P 284 - Pro A10-8700B 285 - A10-8780P 286 - A10-9600P 287 - A10-9630P 288 - A12-9700P 289 - A12-9730P 290 - FX-9800P 291 - FX-9830P 292 - E2-9010 293 - A6-9210 294 - A9-9410 295 ``gfx802`` - ``iceland`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - Radeon R9 285 296 - ``tonga`` flat - *pal-amdhsa* - Radeon R9 380 297 scratch - *pal-amdpal* - Radeon R9 385 298 ``gfx803`` - ``fiji`` ``amdgcn`` dGPU - *rocm-amdhsa* - Radeon R9 Nano 299 - *pal-amdhsa* - Radeon R9 Fury 300 - *pal-amdpal* - Radeon R9 FuryX 301 - Radeon Pro Duo 302 - FirePro S9300x2 303 - Radeon Instinct MI8 304 \ - ``polaris10`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - Radeon RX 470 305 flat - *pal-amdhsa* - Radeon RX 480 306 scratch - *pal-amdpal* - Radeon Instinct MI6 307 \ - ``polaris11`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - Radeon RX 460 308 flat - *pal-amdhsa* 309 scratch - *pal-amdpal* 310 ``gfx805`` - ``tongapro`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - FirePro S7150 311 flat - *pal-amdhsa* - FirePro S7100 312 scratch - *pal-amdpal* - FirePro W7100 313 - Mobile FirePro 314 M7170 315 ``gfx810`` - ``stoney`` ``amdgcn`` APU - xnack - Offset - *rocm-amdhsa* *TBA* 316 flat - *pal-amdhsa* 317 scratch - *pal-amdpal* .. TODO:: 318 319 Add product 320 names. 321 322 **GCN GFX9 (Vega)** [AMD-GCN-GFX900-GFX904-VEGA]_ [AMD-GCN-GFX906-VEGA7NM]_ [AMD-GCN-GFX908-CDNA1]_ 323 ----------------------------------------------------------------------------------------------------------------------- 324 ``gfx900`` ``amdgcn`` dGPU - xnack - Absolute - *rocm-amdhsa* - Radeon Vega 325 flat - *pal-amdhsa* Frontier Edition 326 scratch - *pal-amdpal* - Radeon RX Vega 56 327 - Radeon RX Vega 64 328 - Radeon RX Vega 64 329 Liquid 330 - Radeon Instinct MI25 331 ``gfx902`` ``amdgcn`` APU - xnack - Absolute - *rocm-amdhsa* - Ryzen 3 2200G 332 flat - *pal-amdhsa* - Ryzen 5 2400G 333 scratch - *pal-amdpal* 334 ``gfx904`` ``amdgcn`` dGPU - xnack - *rocm-amdhsa* *TBA* 335 - *pal-amdhsa* 336 - *pal-amdpal* .. TODO:: 337 338 Add product 339 names. 340 341 ``gfx906`` ``amdgcn`` dGPU - sramecc - Absolute - *rocm-amdhsa* - Radeon Instinct MI50 342 - xnack flat - *pal-amdhsa* - Radeon Instinct MI60 343 scratch - *pal-amdpal* - Radeon VII 344 - Radeon Pro VII 345 ``gfx908`` ``amdgcn`` dGPU - sramecc - *rocm-amdhsa* - AMD Instinct MI100 Accelerator 346 - xnack - Absolute 347 flat 348 scratch 349 ``gfx909`` ``amdgcn`` APU - xnack - Absolute - *pal-amdpal* *TBA* 350 flat 351 scratch .. TODO:: 352 353 Add product 354 names. 355 356 ``gfx90a`` ``amdgcn`` dGPU - sramecc - Absolute - *rocm-amdhsa* *TBA* 357 - tgsplit flat 358 - xnack scratch .. TODO:: 359 - Packed 360 work-item Add product 361 IDs names. 362 363 ``gfx90c`` ``amdgcn`` APU - xnack - Absolute - *pal-amdpal* - Ryzen 7 4700G 364 flat - Ryzen 7 4700GE 365 scratch - Ryzen 5 4600G 366 - Ryzen 5 4600GE 367 - Ryzen 3 4300G 368 - Ryzen 3 4300GE 369 - Ryzen Pro 4000G 370 - Ryzen 7 Pro 4700G 371 - Ryzen 7 Pro 4750GE 372 - Ryzen 5 Pro 4650G 373 - Ryzen 5 Pro 4650GE 374 - Ryzen 3 Pro 4350G 375 - Ryzen 3 Pro 4350GE 376 377 ``gfx940`` ``amdgcn`` dGPU - sramecc - Architected *TBA* 378 - tgsplit flat 379 - xnack scratch .. TODO:: 380 - Packed 381 work-item Add product 382 IDs names. 383 384 **GCN GFX10.1 (RDNA 1)** [AMD-GCN-GFX10-RDNA1]_ 385 ----------------------------------------------------------------------------------------------------------------------- 386 ``gfx1010`` ``amdgcn`` dGPU - cumode - Absolute - *rocm-amdhsa* - Radeon RX 5700 387 - wavefrontsize64 flat - *pal-amdhsa* - Radeon RX 5700 XT 388 - xnack scratch - *pal-amdpal* - Radeon Pro 5600 XT 389 - Radeon Pro 5600M 390 ``gfx1011`` ``amdgcn`` dGPU - cumode - *rocm-amdhsa* - Radeon Pro V520 391 - wavefrontsize64 - Absolute - *pal-amdhsa* 392 - xnack flat - *pal-amdpal* 393 scratch 394 ``gfx1012`` ``amdgcn`` dGPU - cumode - Absolute - *rocm-amdhsa* - Radeon RX 5500 395 - wavefrontsize64 flat - *pal-amdhsa* - Radeon RX 5500 XT 396 - xnack scratch - *pal-amdpal* 397 ``gfx1013`` ``amdgcn`` APU - cumode - Absolute - *rocm-amdhsa* *TBA* 398 - wavefrontsize64 flat - *pal-amdhsa* 399 - xnack scratch - *pal-amdpal* .. TODO:: 400 401 Add product 402 names. 403 404 **GCN GFX10.3 (RDNA 2)** [AMD-GCN-GFX10-RDNA2]_ 405 ----------------------------------------------------------------------------------------------------------------------- 406 ``gfx1030`` ``amdgcn`` dGPU - cumode - Absolute - *rocm-amdhsa* - Radeon RX 6800 407 - wavefrontsize64 flat - *pal-amdhsa* - Radeon RX 6800 XT 408 scratch - *pal-amdpal* - Radeon RX 6900 XT 409 ``gfx1031`` ``amdgcn`` dGPU - cumode - Absolute - *rocm-amdhsa* - Radeon RX 6700 XT 410 - wavefrontsize64 flat - *pal-amdhsa* 411 scratch - *pal-amdpal* 412 ``gfx1032`` ``amdgcn`` dGPU - cumode - Absolute - *rocm-amdhsa* *TBA* 413 - wavefrontsize64 flat - *pal-amdhsa* 414 scratch - *pal-amdpal* .. TODO:: 415 416 Add product 417 names. 418 419 ``gfx1033`` ``amdgcn`` APU - cumode - Absolute - *pal-amdpal* *TBA* 420 - wavefrontsize64 flat 421 scratch .. TODO:: 422 423 Add product 424 names. 425 ``gfx1034`` ``amdgcn`` dGPU - cumode - Absolute - *pal-amdpal* *TBA* 426 - wavefrontsize64 flat 427 scratch .. TODO:: 428 429 Add product 430 names. 431 432 ``gfx1035`` ``amdgcn`` APU - cumode - Absolute - *pal-amdpal* *TBA* 433 - wavefrontsize64 flat 434 scratch .. TODO:: 435 Add product 436 names. 437 438 ``gfx1036`` ``amdgcn`` APU - cumode - Absolute - *pal-amdpal* *TBA* 439 - wavefrontsize64 flat 440 scratch .. TODO:: 441 442 Add product 443 names. 444 445 =========== =============== ============ ===== ================= =============== =============== ====================== 446 447.. _amdgpu-target-features: 448 449Target Features 450--------------- 451 452Target features control how code is generated to support certain 453processor specific features. Not all target features are supported by 454all processors. The runtime must ensure that the features supported by 455the device used to execute the code match the features enabled when 456generating the code. A mismatch of features may result in incorrect 457execution, or a reduction in performance. 458 459The target features supported by each processor is listed in 460:ref:`amdgpu-processor-table`. 461 462Target features are controlled by exactly one of the following Clang 463options: 464 465``-mcpu=<target-id>`` or ``--offload-arch=<target-id>`` 466 467 The ``-mcpu`` and ``--offload-arch`` can specify the target feature as 468 optional components of the target ID. If omitted, the target feature has the 469 ``any`` value. See :ref:`amdgpu-target-id`. 470 471``-m[no-]<target-feature>`` 472 473 Target features not specified by the target ID are specified using a 474 separate option. These target features can have an ``on`` or ``off`` 475 value. ``on`` is specified by omitting the ``no-`` prefix, and 476 ``off`` is specified by including the ``no-`` prefix. The default 477 if not specified is ``off``. 478 479For example: 480 481``-mcpu=gfx908:xnack+`` 482 Enable the ``xnack`` feature. 483``-mcpu=gfx908:xnack-`` 484 Disable the ``xnack`` feature. 485``-mcumode`` 486 Enable the ``cumode`` feature. 487``-mno-cumode`` 488 Disable the ``cumode`` feature. 489 490 .. table:: AMDGPU Target Features 491 :name: amdgpu-target-features-table 492 493 =============== ============================ ================================================== 494 Target Feature Clang Option to Control Description 495 Name 496 =============== ============================ ================================================== 497 cumode - ``-m[no-]cumode`` Control the wavefront execution mode used 498 when generating code for kernels. When disabled 499 native WGP wavefront execution mode is used, 500 when enabled CU wavefront execution mode is used 501 (see :ref:`amdgpu-amdhsa-memory-model`). 502 503 sramecc - ``-mcpu`` If specified, generate code that can only be 504 - ``--offload-arch`` loaded and executed in a process that has a 505 matching setting for SRAMECC. 506 507 If not specified for code object V2 to V3, generate 508 code that can be loaded and executed in a process 509 with SRAMECC enabled. 510 511 If not specified for code object V4 or above, generate 512 code that can be loaded and executed in a process 513 with either setting of SRAMECC. 514 515 tgsplit ``-m[no-]tgsplit`` Enable/disable generating code that assumes 516 work-groups are launched in threadgroup split mode. 517 When enabled the waves of a work-group may be 518 launched in different CUs. 519 520 wavefrontsize64 - ``-m[no-]wavefrontsize64`` Control the wavefront size used when 521 generating code for kernels. When disabled 522 native wavefront size 32 is used, when enabled 523 wavefront size 64 is used. 524 525 xnack - ``-mcpu`` If specified, generate code that can only be 526 - ``--offload-arch`` loaded and executed in a process that has a 527 matching setting for XNACK replay. 528 529 If not specified for code object V2 to V3, generate 530 code that can be loaded and executed in a process 531 with XNACK replay enabled. 532 533 If not specified for code object V4 or above, generate 534 code that can be loaded and executed in a process 535 with either setting of XNACK replay. 536 537 XNACK replay can be used for demand paging and 538 page migration. If enabled in the device, then if 539 a page fault occurs the code may execute 540 incorrectly unless generated with XNACK replay 541 enabled, or generated for code object V4 or above without 542 specifying XNACK replay. Executing code that was 543 generated with XNACK replay enabled, or generated 544 for code object V4 or above without specifying XNACK replay, 545 on a device that does not have XNACK replay 546 enabled will execute correctly but may be less 547 performant than code generated for XNACK replay 548 disabled. 549 =============== ============================ ================================================== 550 551.. _amdgpu-target-id: 552 553Target ID 554--------- 555 556AMDGPU supports target IDs. See `Clang Offload Bundler 557<https://clang.llvm.org/docs/ClangOffloadBundler.html>`_ for a general 558description. The AMDGPU target specific information is: 559 560**processor** 561 Is an AMDGPU processor or alternative processor name specified in 562 :ref:`amdgpu-processor-table`. The non-canonical form target ID allows both 563 the primary processor and alternative processor names. The canonical form 564 target ID only allow the primary processor name. 565 566**target-feature** 567 Is a target feature name specified in :ref:`amdgpu-target-features-table` that 568 is supported by the processor. The target features supported by each processor 569 is specified in :ref:`amdgpu-processor-table`. Those that can be specified in 570 a target ID are marked as being controlled by ``-mcpu`` and 571 ``--offload-arch``. Each target feature must appear at most once in a target 572 ID. The non-canonical form target ID allows the target features to be 573 specified in any order. The canonical form target ID requires the target 574 features to be specified in alphabetic order. 575 576.. _amdgpu-target-id-v2-v3: 577 578Code Object V2 to V3 Target ID 579~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 580 581The target ID syntax for code object V2 to V3 is the same as defined in `Clang 582Offload Bundler <https://clang.llvm.org/docs/ClangOffloadBundler.html>`_ except 583when used in the :ref:`amdgpu-assembler-directive-amdgcn-target` assembler 584directive and the bundle entry ID. In those cases it has the following BNF 585syntax: 586 587.. code:: 588 589 <target-id> ::== <processor> ( "+" <target-feature> )* 590 591Where a target feature is omitted if *Off* and present if *On* or *Any*. 592 593.. note:: 594 595 The code object V2 to V3 cannot represent *Any* and treats it the same as 596 *On*. 597 598.. _amdgpu-embedding-bundled-objects: 599 600Embedding Bundled Code Objects 601------------------------------ 602 603AMDGPU supports the HIP and OpenMP languages that perform code object embedding 604as described in `Clang Offload Bundler 605<https://clang.llvm.org/docs/ClangOffloadBundler.html>`_. 606 607.. note:: 608 609 The target ID syntax used for code object V2 to V3 for a bundle entry ID 610 differs from that used elsewhere. See :ref:`amdgpu-target-id-v2-v3`. 611 612.. _amdgpu-address-spaces: 613 614Address Spaces 615-------------- 616 617The AMDGPU architecture supports a number of memory address spaces. The address 618space names use the OpenCL standard names, with some additions. 619 620The AMDGPU address spaces correspond to target architecture specific LLVM 621address space numbers used in LLVM IR. 622 623The AMDGPU address spaces are described in 624:ref:`amdgpu-address-spaces-table`. Only 64-bit process address spaces are 625supported for the ``amdgcn`` target. 626 627 .. table:: AMDGPU Address Spaces 628 :name: amdgpu-address-spaces-table 629 630 ================================= =============== =========== ================ ======= ============================ 631 .. 64-Bit Process Address Space 632 --------------------------------- --------------- ----------- ---------------- ------------------------------------ 633 Address Space Name LLVM IR Address HSA Segment Hardware Address NULL Value 634 Space Number Name Name Size 635 ================================= =============== =========== ================ ======= ============================ 636 Generic 0 flat flat 64 0x0000000000000000 637 Global 1 global global 64 0x0000000000000000 638 Region 2 N/A GDS 32 *not implemented for AMDHSA* 639 Local 3 group LDS 32 0xFFFFFFFF 640 Constant 4 constant *same as global* 64 0x0000000000000000 641 Private 5 private scratch 32 0xFFFFFFFF 642 Constant 32-bit 6 *TODO* 0x00000000 643 Buffer Fat Pointer (experimental) 7 *TODO* 644 ================================= =============== =========== ================ ======= ============================ 645 646**Generic** 647 The generic address space is supported unless the *Target Properties* column 648 of :ref:`amdgpu-processor-table` specifies *Does not support generic address 649 space*. 650 651 The generic address space uses the hardware flat address support for two fixed 652 ranges of virtual addresses (the private and local apertures), that are 653 outside the range of addressable global memory, to map from a flat address to 654 a private or local address. This uses FLAT instructions that can take a flat 655 address and access global, private (scratch), and group (LDS) memory depending 656 on if the address is within one of the aperture ranges. 657 658 Flat access to scratch requires hardware aperture setup and setup in the 659 kernel prologue (see :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`). Flat 660 access to LDS requires hardware aperture setup and M0 (GFX7-GFX8) register 661 setup (see :ref:`amdgpu-amdhsa-kernel-prolog-m0`). 662 663 To convert between a private or group address space address (termed a segment 664 address) and a flat address the base address of the corresponding aperture 665 can be used. For GFX7-GFX8 these are available in the 666 :ref:`amdgpu-amdhsa-hsa-aql-queue` the address of which can be obtained with 667 Queue Ptr SGPR (see :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). For 668 GFX9-GFX10 the aperture base addresses are directly available as inline 669 constant registers ``SRC_SHARED_BASE/LIMIT`` and ``SRC_PRIVATE_BASE/LIMIT``. 670 In 64-bit address mode the aperture sizes are 2^32 bytes and the base is 671 aligned to 2^32 which makes it easier to convert from flat to segment or 672 segment to flat. 673 674 A global address space address has the same value when used as a flat address 675 so no conversion is needed. 676 677**Global and Constant** 678 The global and constant address spaces both use global virtual addresses, 679 which are the same virtual address space used by the CPU. However, some 680 virtual addresses may only be accessible to the CPU, some only accessible 681 by the GPU, and some by both. 682 683 Using the constant address space indicates that the data will not change 684 during the execution of the kernel. This allows scalar read instructions to 685 be used. As the constant address space could only be modified on the host 686 side, a generic pointer loaded from the constant address space is safe to be 687 assumed as a global pointer since only the device global memory is visible 688 and managed on the host side. The vector and scalar L1 caches are invalidated 689 of volatile data before each kernel dispatch execution to allow constant 690 memory to change values between kernel dispatches. 691 692**Region** 693 The region address space uses the hardware Global Data Store (GDS). All 694 wavefronts executing on the same device will access the same memory for any 695 given region address. However, the same region address accessed by wavefronts 696 executing on different devices will access different memory. It is higher 697 performance than global memory. It is allocated by the runtime. The data 698 store (DS) instructions can be used to access it. 699 700**Local** 701 The local address space uses the hardware Local Data Store (LDS) which is 702 automatically allocated when the hardware creates the wavefronts of a 703 work-group, and freed when all the wavefronts of a work-group have 704 terminated. All wavefronts belonging to the same work-group will access the 705 same memory for any given local address. However, the same local address 706 accessed by wavefronts belonging to different work-groups will access 707 different memory. It is higher performance than global memory. The data store 708 (DS) instructions can be used to access it. 709 710**Private** 711 The private address space uses the hardware scratch memory support which 712 automatically allocates memory when it creates a wavefront and frees it when 713 a wavefronts terminates. The memory accessed by a lane of a wavefront for any 714 given private address will be different to the memory accessed by another lane 715 of the same or different wavefront for the same private address. 716 717 If a kernel dispatch uses scratch, then the hardware allocates memory from a 718 pool of backing memory allocated by the runtime for each wavefront. The lanes 719 of the wavefront access this using dword (4 byte) interleaving. The mapping 720 used from private address to backing memory address is: 721 722 ``wavefront-scratch-base + 723 ((private-address / 4) * wavefront-size * 4) + 724 (wavefront-lane-id * 4) + (private-address % 4)`` 725 726 If each lane of a wavefront accesses the same private address, the 727 interleaving results in adjacent dwords being accessed and hence requires 728 fewer cache lines to be fetched. 729 730 There are different ways that the wavefront scratch base address is 731 determined by a wavefront (see 732 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 733 734 Scratch memory can be accessed in an interleaved manner using buffer 735 instructions with the scratch buffer descriptor and per wavefront scratch 736 offset, by the scratch instructions, or by flat instructions. Multi-dword 737 access is not supported except by flat and scratch instructions in 738 GFX9-GFX10. 739 740**Constant 32-bit** 741 *TODO* 742 743**Buffer Fat Pointer** 744 The buffer fat pointer is an experimental address space that is currently 745 unsupported in the backend. It exposes a non-integral pointer that is in 746 the future intended to support the modelling of 128-bit buffer descriptors 747 plus a 32-bit offset into the buffer (in total encapsulating a 160-bit 748 *pointer*), allowing normal LLVM load/store/atomic operations to be used to 749 model the buffer descriptors used heavily in graphics workloads targeting 750 the backend. 751 752.. _amdgpu-memory-scopes: 753 754Memory Scopes 755------------- 756 757This section provides LLVM memory synchronization scopes supported by the AMDGPU 758backend memory model when the target triple OS is ``amdhsa`` (see 759:ref:`amdgpu-amdhsa-memory-model` and :ref:`amdgpu-target-triples`). 760 761The memory model supported is based on the HSA memory model [HSA]_ which is 762based in turn on HRF-indirect with scope inclusion [HRF]_. The happens-before 763relation is transitive over the synchronizes-with relation independent of scope 764and synchronizes-with allows the memory scope instances to be inclusive (see 765table :ref:`amdgpu-amdhsa-llvm-sync-scopes-table`). 766 767This is different to the OpenCL [OpenCL]_ memory model which does not have scope 768inclusion and requires the memory scopes to exactly match. However, this 769is conservatively correct for OpenCL. 770 771 .. table:: AMDHSA LLVM Sync Scopes 772 :name: amdgpu-amdhsa-llvm-sync-scopes-table 773 774 ======================= =================================================== 775 LLVM Sync Scope Description 776 ======================= =================================================== 777 *none* The default: ``system``. 778 779 Synchronizes with, and participates in modification 780 and seq_cst total orderings with, other operations 781 (except image operations) for all address spaces 782 (except private, or generic that accesses private) 783 provided the other operation's sync scope is: 784 785 - ``system``. 786 - ``agent`` and executed by a thread on the same 787 agent. 788 - ``workgroup`` and executed by a thread in the 789 same work-group. 790 - ``wavefront`` and executed by a thread in the 791 same wavefront. 792 793 ``agent`` Synchronizes with, and participates in modification 794 and seq_cst total orderings with, other operations 795 (except image operations) for all address spaces 796 (except private, or generic that accesses private) 797 provided the other operation's sync scope is: 798 799 - ``system`` or ``agent`` and executed by a thread 800 on the same agent. 801 - ``workgroup`` and executed by a thread in the 802 same work-group. 803 - ``wavefront`` and executed by a thread in the 804 same wavefront. 805 806 ``workgroup`` Synchronizes with, and participates in modification 807 and seq_cst total orderings with, other operations 808 (except image operations) for all address spaces 809 (except private, or generic that accesses private) 810 provided the other operation's sync scope is: 811 812 - ``system``, ``agent`` or ``workgroup`` and 813 executed by a thread in the same work-group. 814 - ``wavefront`` and executed by a thread in the 815 same wavefront. 816 817 ``wavefront`` Synchronizes with, and participates in modification 818 and seq_cst total orderings with, other operations 819 (except image operations) for all address spaces 820 (except private, or generic that accesses private) 821 provided the other operation's sync scope is: 822 823 - ``system``, ``agent``, ``workgroup`` or 824 ``wavefront`` and executed by a thread in the 825 same wavefront. 826 827 ``singlethread`` Only synchronizes with and participates in 828 modification and seq_cst total orderings with, 829 other operations (except image operations) running 830 in the same thread for all address spaces (for 831 example, in signal handlers). 832 833 ``one-as`` Same as ``system`` but only synchronizes with other 834 operations within the same address space. 835 836 ``agent-one-as`` Same as ``agent`` but only synchronizes with other 837 operations within the same address space. 838 839 ``workgroup-one-as`` Same as ``workgroup`` but only synchronizes with 840 other operations within the same address space. 841 842 ``wavefront-one-as`` Same as ``wavefront`` but only synchronizes with 843 other operations within the same address space. 844 845 ``singlethread-one-as`` Same as ``singlethread`` but only synchronizes with 846 other operations within the same address space. 847 ======================= =================================================== 848 849LLVM IR Intrinsics 850------------------ 851 852The AMDGPU backend implements the following LLVM IR intrinsics. 853 854*This section is WIP.* 855 856.. TODO:: 857 858 List AMDGPU intrinsics. 859 860LLVM IR Attributes 861------------------ 862 863The AMDGPU backend supports the following LLVM IR attributes. 864 865 .. table:: AMDGPU LLVM IR Attributes 866 :name: amdgpu-llvm-ir-attributes-table 867 868 ======================================= ========================================================== 869 LLVM Attribute Description 870 ======================================= ========================================================== 871 "amdgpu-flat-work-group-size"="min,max" Specify the minimum and maximum flat work group sizes that 872 will be specified when the kernel is dispatched. Generated 873 by the ``amdgpu_flat_work_group_size`` CLANG attribute [CLANG-ATTR]_. 874 The implied default value is 1,1024. 875 876 "amdgpu-implicitarg-num-bytes"="n" Number of kernel argument bytes to add to the kernel 877 argument block size for the implicit arguments. This 878 varies by OS and language (for OpenCL see 879 :ref:`opencl-kernel-implicit-arguments-appended-for-amdhsa-os-table`). 880 "amdgpu-num-sgpr"="n" Specifies the number of SGPRs to use. Generated by 881 the ``amdgpu_num_sgpr`` CLANG attribute [CLANG-ATTR]_. 882 "amdgpu-num-vgpr"="n" Specifies the number of VGPRs to use. Generated by the 883 ``amdgpu_num_vgpr`` CLANG attribute [CLANG-ATTR]_. 884 "amdgpu-waves-per-eu"="m,n" Specify the minimum and maximum number of waves per 885 execution unit. Generated by the ``amdgpu_waves_per_eu`` 886 CLANG attribute [CLANG-ATTR]_. This is an optimization hint, 887 and the backend may not be able to satisfy the request. If 888 the specified range is incompatible with the function's 889 "amdgpu-flat-work-group-size" value, the implied occupancy 890 bounds by the workgroup size takes precedence. 891 892 "amdgpu-ieee" true/false. Specify whether the function expects the IEEE field of the 893 mode register to be set on entry. Overrides the default for 894 the calling convention. 895 "amdgpu-dx10-clamp" true/false. Specify whether the function expects the DX10_CLAMP field of 896 the mode register to be set on entry. Overrides the default 897 for the calling convention. 898 899 "amdgpu-no-workitem-id-x" Indicates the function does not depend on the value of the 900 llvm.amdgcn.workitem.id.x intrinsic. If a function is marked with this 901 attribute, or reached through a call site marked with this attribute, 902 the value returned by the intrinsic is undefined. The backend can 903 generally infer this during code generation, so typically there is no 904 benefit to frontends marking functions with this. 905 906 "amdgpu-no-workitem-id-y" The same as amdgpu-no-workitem-id-x, except for the 907 llvm.amdgcn.workitem.id.y intrinsic. 908 909 "amdgpu-no-workitem-id-z" The same as amdgpu-no-workitem-id-x, except for the 910 llvm.amdgcn.workitem.id.z intrinsic. 911 912 "amdgpu-no-workgroup-id-x" The same as amdgpu-no-workitem-id-x, except for the 913 llvm.amdgcn.workgroup.id.x intrinsic. 914 915 "amdgpu-no-workgroup-id-y" The same as amdgpu-no-workitem-id-x, except for the 916 llvm.amdgcn.workgroup.id.y intrinsic. 917 918 "amdgpu-no-workgroup-id-z" The same as amdgpu-no-workitem-id-x, except for the 919 llvm.amdgcn.workgroup.id.z intrinsic. 920 921 "amdgpu-no-dispatch-ptr" The same as amdgpu-no-workitem-id-x, except for the 922 llvm.amdgcn.dispatch.ptr intrinsic. 923 924 "amdgpu-no-implicitarg-ptr" The same as amdgpu-no-workitem-id-x, except for the 925 llvm.amdgcn.implicitarg.ptr intrinsic. 926 927 "amdgpu-no-dispatch-id" The same as amdgpu-no-workitem-id-x, except for the 928 llvm.amdgcn.dispatch.id intrinsic. 929 930 "amdgpu-no-queue-ptr" Similar to amdgpu-no-workitem-id-x, except for the 931 llvm.amdgcn.queue.ptr intrinsic. Note that unlike the other ABI hint 932 attributes, the queue pointer may be required in situations where the 933 intrinsic call does not directly appear in the program. Some subtargets 934 require the queue pointer for to handle some addrspacecasts, as well 935 as the llvm.amdgcn.is.shared, llvm.amdgcn.is.private, llvm.trap, and 936 llvm.debug intrinsics. 937 938 "amdgpu-no-hostcall-ptr" Similar to amdgpu-no-implicitarg-ptr, except specific to the implicit 939 kernel argument that holds the pointer to the hostcall buffer. If this 940 attribute is absent, then the amdgpu-no-implicitarg-ptr is also removed. 941 942 "amdgpu-no-heap-ptr" Similar to amdgpu-no-implicitarg-ptr, except specific to the implicit 943 kernel argument that holds the pointer to an initialized memory buffer 944 that conforms to the requirements of the malloc/free device library V1 945 version implementation. If this attribute is absent, then the 946 amdgpu-no-implicitarg-ptr is also removed. 947 948 ======================================= ========================================================== 949 950.. _amdgpu-elf-code-object: 951 952ELF Code Object 953=============== 954 955The AMDGPU backend generates a standard ELF [ELF]_ relocatable code object that 956can be linked by ``lld`` to produce a standard ELF shared code object which can 957be loaded and executed on an AMDGPU target. 958 959.. _amdgpu-elf-header: 960 961Header 962------ 963 964The AMDGPU backend uses the following ELF header: 965 966 .. table:: AMDGPU ELF Header 967 :name: amdgpu-elf-header-table 968 969 ========================== =============================== 970 Field Value 971 ========================== =============================== 972 ``e_ident[EI_CLASS]`` ``ELFCLASS64`` 973 ``e_ident[EI_DATA]`` ``ELFDATA2LSB`` 974 ``e_ident[EI_OSABI]`` - ``ELFOSABI_NONE`` 975 - ``ELFOSABI_AMDGPU_HSA`` 976 - ``ELFOSABI_AMDGPU_PAL`` 977 - ``ELFOSABI_AMDGPU_MESA3D`` 978 ``e_ident[EI_ABIVERSION]`` - ``ELFABIVERSION_AMDGPU_HSA_V2`` 979 - ``ELFABIVERSION_AMDGPU_HSA_V3`` 980 - ``ELFABIVERSION_AMDGPU_HSA_V4`` 981 - ``ELFABIVERSION_AMDGPU_HSA_V5`` 982 - ``ELFABIVERSION_AMDGPU_PAL`` 983 - ``ELFABIVERSION_AMDGPU_MESA3D`` 984 ``e_type`` - ``ET_REL`` 985 - ``ET_DYN`` 986 ``e_machine`` ``EM_AMDGPU`` 987 ``e_entry`` 0 988 ``e_flags`` See :ref:`amdgpu-elf-header-e_flags-v2-table`, 989 :ref:`amdgpu-elf-header-e_flags-table-v3`, 990 and :ref:`amdgpu-elf-header-e_flags-table-v4-onwards` 991 ========================== =============================== 992 993.. 994 995 .. table:: AMDGPU ELF Header Enumeration Values 996 :name: amdgpu-elf-header-enumeration-values-table 997 998 =============================== ===== 999 Name Value 1000 =============================== ===== 1001 ``EM_AMDGPU`` 224 1002 ``ELFOSABI_NONE`` 0 1003 ``ELFOSABI_AMDGPU_HSA`` 64 1004 ``ELFOSABI_AMDGPU_PAL`` 65 1005 ``ELFOSABI_AMDGPU_MESA3D`` 66 1006 ``ELFABIVERSION_AMDGPU_HSA_V2`` 0 1007 ``ELFABIVERSION_AMDGPU_HSA_V3`` 1 1008 ``ELFABIVERSION_AMDGPU_HSA_V4`` 2 1009 ``ELFABIVERSION_AMDGPU_HSA_V5`` 3 1010 ``ELFABIVERSION_AMDGPU_PAL`` 0 1011 ``ELFABIVERSION_AMDGPU_MESA3D`` 0 1012 =============================== ===== 1013 1014``e_ident[EI_CLASS]`` 1015 The ELF class is: 1016 1017 * ``ELFCLASS32`` for ``r600`` architecture. 1018 1019 * ``ELFCLASS64`` for ``amdgcn`` architecture which only supports 64-bit 1020 process address space applications. 1021 1022``e_ident[EI_DATA]`` 1023 All AMDGPU targets use ``ELFDATA2LSB`` for little-endian byte ordering. 1024 1025``e_ident[EI_OSABI]`` 1026 One of the following AMDGPU target architecture specific OS ABIs 1027 (see :ref:`amdgpu-os`): 1028 1029 * ``ELFOSABI_NONE`` for *unknown* OS. 1030 1031 * ``ELFOSABI_AMDGPU_HSA`` for ``amdhsa`` OS. 1032 1033 * ``ELFOSABI_AMDGPU_PAL`` for ``amdpal`` OS. 1034 1035 * ``ELFOSABI_AMDGPU_MESA3D`` for ``mesa3D`` OS. 1036 1037``e_ident[EI_ABIVERSION]`` 1038 The ABI version of the AMDGPU target architecture specific OS ABI to which the code 1039 object conforms: 1040 1041 * ``ELFABIVERSION_AMDGPU_HSA_V2`` is used to specify the version of AMD HSA 1042 runtime ABI for code object V2. Specify using the Clang option 1043 ``-mcode-object-version=2``. 1044 1045 * ``ELFABIVERSION_AMDGPU_HSA_V3`` is used to specify the version of AMD HSA 1046 runtime ABI for code object V3. Specify using the Clang option 1047 ``-mcode-object-version=3``. 1048 1049 * ``ELFABIVERSION_AMDGPU_HSA_V4`` is used to specify the version of AMD HSA 1050 runtime ABI for code object V4. Specify using the Clang option 1051 ``-mcode-object-version=4``. This is the default code object 1052 version if not specified. 1053 1054 * ``ELFABIVERSION_AMDGPU_HSA_V5`` is used to specify the version of AMD HSA 1055 runtime ABI for code object V5. Specify using the Clang option 1056 ``-mcode-object-version=5``. 1057 1058 * ``ELFABIVERSION_AMDGPU_PAL`` is used to specify the version of AMD PAL 1059 runtime ABI. 1060 1061 * ``ELFABIVERSION_AMDGPU_MESA3D`` is used to specify the version of AMD MESA 1062 3D runtime ABI. 1063 1064``e_type`` 1065 Can be one of the following values: 1066 1067 1068 ``ET_REL`` 1069 The type produced by the AMDGPU backend compiler as it is relocatable code 1070 object. 1071 1072 ``ET_DYN`` 1073 The type produced by the linker as it is a shared code object. 1074 1075 The AMD HSA runtime loader requires a ``ET_DYN`` code object. 1076 1077``e_machine`` 1078 The value ``EM_AMDGPU`` is used for the machine for all processors supported 1079 by the ``r600`` and ``amdgcn`` architectures (see 1080 :ref:`amdgpu-processor-table`). The specific processor is specified in the 1081 ``NT_AMD_HSA_ISA_VERSION`` note record for code object V2 (see 1082 :ref:`amdgpu-note-records-v2`) and in the ``EF_AMDGPU_MACH`` bit field of the 1083 ``e_flags`` for code object V3 and above (see 1084 :ref:`amdgpu-elf-header-e_flags-table-v3` and 1085 :ref:`amdgpu-elf-header-e_flags-table-v4-onwards`). 1086 1087``e_entry`` 1088 The entry point is 0 as the entry points for individual kernels must be 1089 selected in order to invoke them through AQL packets. 1090 1091``e_flags`` 1092 The AMDGPU backend uses the following ELF header flags: 1093 1094 .. table:: AMDGPU ELF Header ``e_flags`` for Code Object V2 1095 :name: amdgpu-elf-header-e_flags-v2-table 1096 1097 ===================================== ===== ============================= 1098 Name Value Description 1099 ===================================== ===== ============================= 1100 ``EF_AMDGPU_FEATURE_XNACK_V2`` 0x01 Indicates if the ``xnack`` 1101 target feature is 1102 enabled for all code 1103 contained in the code object. 1104 If the processor 1105 does not support the 1106 ``xnack`` target 1107 feature then must 1108 be 0. 1109 See 1110 :ref:`amdgpu-target-features`. 1111 ``EF_AMDGPU_FEATURE_TRAP_HANDLER_V2`` 0x02 Indicates if the trap 1112 handler is enabled for all 1113 code contained in the code 1114 object. If the processor 1115 does not support a trap 1116 handler then must be 0. 1117 See 1118 :ref:`amdgpu-target-features`. 1119 ===================================== ===== ============================= 1120 1121 .. table:: AMDGPU ELF Header ``e_flags`` for Code Object V3 1122 :name: amdgpu-elf-header-e_flags-table-v3 1123 1124 ================================= ===== ============================= 1125 Name Value Description 1126 ================================= ===== ============================= 1127 ``EF_AMDGPU_MACH`` 0x0ff AMDGPU processor selection 1128 mask for 1129 ``EF_AMDGPU_MACH_xxx`` values 1130 defined in 1131 :ref:`amdgpu-ef-amdgpu-mach-table`. 1132 ``EF_AMDGPU_FEATURE_XNACK_V3`` 0x100 Indicates if the ``xnack`` 1133 target feature is 1134 enabled for all code 1135 contained in the code object. 1136 If the processor 1137 does not support the 1138 ``xnack`` target 1139 feature then must 1140 be 0. 1141 See 1142 :ref:`amdgpu-target-features`. 1143 ``EF_AMDGPU_FEATURE_SRAMECC_V3`` 0x200 Indicates if the ``sramecc`` 1144 target feature is 1145 enabled for all code 1146 contained in the code object. 1147 If the processor 1148 does not support the 1149 ``sramecc`` target 1150 feature then must 1151 be 0. 1152 See 1153 :ref:`amdgpu-target-features`. 1154 ================================= ===== ============================= 1155 1156 .. table:: AMDGPU ELF Header ``e_flags`` for Code Object V4 and After 1157 :name: amdgpu-elf-header-e_flags-table-v4-onwards 1158 1159 ============================================ ===== =================================== 1160 Name Value Description 1161 ============================================ ===== =================================== 1162 ``EF_AMDGPU_MACH`` 0x0ff AMDGPU processor selection 1163 mask for 1164 ``EF_AMDGPU_MACH_xxx`` values 1165 defined in 1166 :ref:`amdgpu-ef-amdgpu-mach-table`. 1167 ``EF_AMDGPU_FEATURE_XNACK_V4`` 0x300 XNACK selection mask for 1168 ``EF_AMDGPU_FEATURE_XNACK_*_V4`` 1169 values. 1170 ``EF_AMDGPU_FEATURE_XNACK_UNSUPPORTED_V4`` 0x000 XNACK unsuppored. 1171 ``EF_AMDGPU_FEATURE_XNACK_ANY_V4`` 0x100 XNACK can have any value. 1172 ``EF_AMDGPU_FEATURE_XNACK_OFF_V4`` 0x200 XNACK disabled. 1173 ``EF_AMDGPU_FEATURE_XNACK_ON_V4`` 0x300 XNACK enabled. 1174 ``EF_AMDGPU_FEATURE_SRAMECC_V4`` 0xc00 SRAMECC selection mask for 1175 ``EF_AMDGPU_FEATURE_SRAMECC_*_V4`` 1176 values. 1177 ``EF_AMDGPU_FEATURE_SRAMECC_UNSUPPORTED_V4`` 0x000 SRAMECC unsuppored. 1178 ``EF_AMDGPU_FEATURE_SRAMECC_ANY_V4`` 0x400 SRAMECC can have any value. 1179 ``EF_AMDGPU_FEATURE_SRAMECC_OFF_V4`` 0x800 SRAMECC disabled, 1180 ``EF_AMDGPU_FEATURE_SRAMECC_ON_V4`` 0xc00 SRAMECC enabled. 1181 ============================================ ===== =================================== 1182 1183 .. table:: AMDGPU ``EF_AMDGPU_MACH`` Values 1184 :name: amdgpu-ef-amdgpu-mach-table 1185 1186 ==================================== ========== ============================= 1187 Name Value Description (see 1188 :ref:`amdgpu-processor-table`) 1189 ==================================== ========== ============================= 1190 ``EF_AMDGPU_MACH_NONE`` 0x000 *not specified* 1191 ``EF_AMDGPU_MACH_R600_R600`` 0x001 ``r600`` 1192 ``EF_AMDGPU_MACH_R600_R630`` 0x002 ``r630`` 1193 ``EF_AMDGPU_MACH_R600_RS880`` 0x003 ``rs880`` 1194 ``EF_AMDGPU_MACH_R600_RV670`` 0x004 ``rv670`` 1195 ``EF_AMDGPU_MACH_R600_RV710`` 0x005 ``rv710`` 1196 ``EF_AMDGPU_MACH_R600_RV730`` 0x006 ``rv730`` 1197 ``EF_AMDGPU_MACH_R600_RV770`` 0x007 ``rv770`` 1198 ``EF_AMDGPU_MACH_R600_CEDAR`` 0x008 ``cedar`` 1199 ``EF_AMDGPU_MACH_R600_CYPRESS`` 0x009 ``cypress`` 1200 ``EF_AMDGPU_MACH_R600_JUNIPER`` 0x00a ``juniper`` 1201 ``EF_AMDGPU_MACH_R600_REDWOOD`` 0x00b ``redwood`` 1202 ``EF_AMDGPU_MACH_R600_SUMO`` 0x00c ``sumo`` 1203 ``EF_AMDGPU_MACH_R600_BARTS`` 0x00d ``barts`` 1204 ``EF_AMDGPU_MACH_R600_CAICOS`` 0x00e ``caicos`` 1205 ``EF_AMDGPU_MACH_R600_CAYMAN`` 0x00f ``cayman`` 1206 ``EF_AMDGPU_MACH_R600_TURKS`` 0x010 ``turks`` 1207 *reserved* 0x011 - Reserved for ``r600`` 1208 0x01f architecture processors. 1209 ``EF_AMDGPU_MACH_AMDGCN_GFX600`` 0x020 ``gfx600`` 1210 ``EF_AMDGPU_MACH_AMDGCN_GFX601`` 0x021 ``gfx601`` 1211 ``EF_AMDGPU_MACH_AMDGCN_GFX700`` 0x022 ``gfx700`` 1212 ``EF_AMDGPU_MACH_AMDGCN_GFX701`` 0x023 ``gfx701`` 1213 ``EF_AMDGPU_MACH_AMDGCN_GFX702`` 0x024 ``gfx702`` 1214 ``EF_AMDGPU_MACH_AMDGCN_GFX703`` 0x025 ``gfx703`` 1215 ``EF_AMDGPU_MACH_AMDGCN_GFX704`` 0x026 ``gfx704`` 1216 *reserved* 0x027 Reserved. 1217 ``EF_AMDGPU_MACH_AMDGCN_GFX801`` 0x028 ``gfx801`` 1218 ``EF_AMDGPU_MACH_AMDGCN_GFX802`` 0x029 ``gfx802`` 1219 ``EF_AMDGPU_MACH_AMDGCN_GFX803`` 0x02a ``gfx803`` 1220 ``EF_AMDGPU_MACH_AMDGCN_GFX810`` 0x02b ``gfx810`` 1221 ``EF_AMDGPU_MACH_AMDGCN_GFX900`` 0x02c ``gfx900`` 1222 ``EF_AMDGPU_MACH_AMDGCN_GFX902`` 0x02d ``gfx902`` 1223 ``EF_AMDGPU_MACH_AMDGCN_GFX904`` 0x02e ``gfx904`` 1224 ``EF_AMDGPU_MACH_AMDGCN_GFX906`` 0x02f ``gfx906`` 1225 ``EF_AMDGPU_MACH_AMDGCN_GFX908`` 0x030 ``gfx908`` 1226 ``EF_AMDGPU_MACH_AMDGCN_GFX909`` 0x031 ``gfx909`` 1227 ``EF_AMDGPU_MACH_AMDGCN_GFX90C`` 0x032 ``gfx90c`` 1228 ``EF_AMDGPU_MACH_AMDGCN_GFX1010`` 0x033 ``gfx1010`` 1229 ``EF_AMDGPU_MACH_AMDGCN_GFX1011`` 0x034 ``gfx1011`` 1230 ``EF_AMDGPU_MACH_AMDGCN_GFX1012`` 0x035 ``gfx1012`` 1231 ``EF_AMDGPU_MACH_AMDGCN_GFX1030`` 0x036 ``gfx1030`` 1232 ``EF_AMDGPU_MACH_AMDGCN_GFX1031`` 0x037 ``gfx1031`` 1233 ``EF_AMDGPU_MACH_AMDGCN_GFX1032`` 0x038 ``gfx1032`` 1234 ``EF_AMDGPU_MACH_AMDGCN_GFX1033`` 0x039 ``gfx1033`` 1235 ``EF_AMDGPU_MACH_AMDGCN_GFX602`` 0x03a ``gfx602`` 1236 ``EF_AMDGPU_MACH_AMDGCN_GFX705`` 0x03b ``gfx705`` 1237 ``EF_AMDGPU_MACH_AMDGCN_GFX805`` 0x03c ``gfx805`` 1238 ``EF_AMDGPU_MACH_AMDGCN_GFX1035`` 0x03d ``gfx1035`` 1239 ``EF_AMDGPU_MACH_AMDGCN_GFX1034`` 0x03e ``gfx1034`` 1240 ``EF_AMDGPU_MACH_AMDGCN_GFX90A`` 0x03f ``gfx90a`` 1241 ``EF_AMDGPU_MACH_AMDGCN_GFX940`` 0x040 ``gfx940`` 1242 *reserved* 0x041 Reserved. 1243 ``EF_AMDGPU_MACH_AMDGCN_GFX1013`` 0x042 ``gfx1013`` 1244 *reserved* 0x043 Reserved. 1245 *reserved* 0x044 Reserved. 1246 ``EF_AMDGPU_MACH_AMDGCN_GFX1036`` 0x045 ``gfx1036`` 1247 ==================================== ========== ============================= 1248 1249Sections 1250-------- 1251 1252An AMDGPU target ELF code object has the standard ELF sections which include: 1253 1254 .. table:: AMDGPU ELF Sections 1255 :name: amdgpu-elf-sections-table 1256 1257 ================== ================ ================================= 1258 Name Type Attributes 1259 ================== ================ ================================= 1260 ``.bss`` ``SHT_NOBITS`` ``SHF_ALLOC`` + ``SHF_WRITE`` 1261 ``.data`` ``SHT_PROGBITS`` ``SHF_ALLOC`` + ``SHF_WRITE`` 1262 ``.debug_``\ *\** ``SHT_PROGBITS`` *none* 1263 ``.dynamic`` ``SHT_DYNAMIC`` ``SHF_ALLOC`` 1264 ``.dynstr`` ``SHT_PROGBITS`` ``SHF_ALLOC`` 1265 ``.dynsym`` ``SHT_PROGBITS`` ``SHF_ALLOC`` 1266 ``.got`` ``SHT_PROGBITS`` ``SHF_ALLOC`` + ``SHF_WRITE`` 1267 ``.hash`` ``SHT_HASH`` ``SHF_ALLOC`` 1268 ``.note`` ``SHT_NOTE`` *none* 1269 ``.rela``\ *name* ``SHT_RELA`` *none* 1270 ``.rela.dyn`` ``SHT_RELA`` *none* 1271 ``.rodata`` ``SHT_PROGBITS`` ``SHF_ALLOC`` 1272 ``.shstrtab`` ``SHT_STRTAB`` *none* 1273 ``.strtab`` ``SHT_STRTAB`` *none* 1274 ``.symtab`` ``SHT_SYMTAB`` *none* 1275 ``.text`` ``SHT_PROGBITS`` ``SHF_ALLOC`` + ``SHF_EXECINSTR`` 1276 ================== ================ ================================= 1277 1278These sections have their standard meanings (see [ELF]_) and are only generated 1279if needed. 1280 1281``.debug``\ *\** 1282 The standard DWARF sections. See :ref:`amdgpu-dwarf-debug-information` for 1283 information on the DWARF produced by the AMDGPU backend. 1284 1285``.dynamic``, ``.dynstr``, ``.dynsym``, ``.hash`` 1286 The standard sections used by a dynamic loader. 1287 1288``.note`` 1289 See :ref:`amdgpu-note-records` for the note records supported by the AMDGPU 1290 backend. 1291 1292``.rela``\ *name*, ``.rela.dyn`` 1293 For relocatable code objects, *name* is the name of the section that the 1294 relocation records apply. For example, ``.rela.text`` is the section name for 1295 relocation records associated with the ``.text`` section. 1296 1297 For linked shared code objects, ``.rela.dyn`` contains all the relocation 1298 records from each of the relocatable code object's ``.rela``\ *name* sections. 1299 1300 See :ref:`amdgpu-relocation-records` for the relocation records supported by 1301 the AMDGPU backend. 1302 1303``.text`` 1304 The executable machine code for the kernels and functions they call. Generated 1305 as position independent code. See :ref:`amdgpu-code-conventions` for 1306 information on conventions used in the isa generation. 1307 1308.. _amdgpu-note-records: 1309 1310Note Records 1311------------ 1312 1313The AMDGPU backend code object contains ELF note records in the ``.note`` 1314section. The set of generated notes and their semantics depend on the code 1315object version; see :ref:`amdgpu-note-records-v2` and 1316:ref:`amdgpu-note-records-v3-onwards`. 1317 1318As required by ``ELFCLASS32`` and ``ELFCLASS64``, minimal zero-byte padding 1319must be generated after the ``name`` field to ensure the ``desc`` field is 4 1320byte aligned. In addition, minimal zero-byte padding must be generated to 1321ensure the ``desc`` field size is a multiple of 4 bytes. The ``sh_addralign`` 1322field of the ``.note`` section must be at least 4 to indicate at least 8 byte 1323alignment. 1324 1325.. _amdgpu-note-records-v2: 1326 1327Code Object V2 Note Records 1328~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1329 1330.. warning:: 1331 Code object V2 is not the default code object version emitted by 1332 this version of LLVM. 1333 1334The AMDGPU backend code object uses the following ELF note record in the 1335``.note`` section when compiling for code object V2. 1336 1337The note record vendor field is "AMD". 1338 1339Additional note records may be present, but any which are not documented here 1340are deprecated and should not be used. 1341 1342 .. table:: AMDGPU Code Object V2 ELF Note Records 1343 :name: amdgpu-elf-note-records-v2-table 1344 1345 ===== ===================================== ====================================== 1346 Name Type Description 1347 ===== ===================================== ====================================== 1348 "AMD" ``NT_AMD_HSA_CODE_OBJECT_VERSION`` Code object version. 1349 "AMD" ``NT_AMD_HSA_HSAIL`` HSAIL properties generated by the HSAIL 1350 Finalizer and not the LLVM compiler. 1351 "AMD" ``NT_AMD_HSA_ISA_VERSION`` Target ISA version. 1352 "AMD" ``NT_AMD_HSA_METADATA`` Metadata null terminated string in 1353 YAML [YAML]_ textual format. 1354 "AMD" ``NT_AMD_HSA_ISA_NAME`` Target ISA name. 1355 ===== ===================================== ====================================== 1356 1357.. 1358 1359 .. table:: AMDGPU Code Object V2 ELF Note Record Enumeration Values 1360 :name: amdgpu-elf-note-record-enumeration-values-v2-table 1361 1362 ===================================== ===== 1363 Name Value 1364 ===================================== ===== 1365 ``NT_AMD_HSA_CODE_OBJECT_VERSION`` 1 1366 ``NT_AMD_HSA_HSAIL`` 2 1367 ``NT_AMD_HSA_ISA_VERSION`` 3 1368 *reserved* 4-9 1369 ``NT_AMD_HSA_METADATA`` 10 1370 ``NT_AMD_HSA_ISA_NAME`` 11 1371 ===================================== ===== 1372 1373``NT_AMD_HSA_CODE_OBJECT_VERSION`` 1374 Specifies the code object version number. The description field has the 1375 following layout: 1376 1377 .. code:: c 1378 1379 struct amdgpu_hsa_note_code_object_version_s { 1380 uint32_t major_version; 1381 uint32_t minor_version; 1382 }; 1383 1384 The ``major_version`` has a value less than or equal to 2. 1385 1386``NT_AMD_HSA_HSAIL`` 1387 Specifies the HSAIL properties used by the HSAIL Finalizer. The description 1388 field has the following layout: 1389 1390 .. code:: c 1391 1392 struct amdgpu_hsa_note_hsail_s { 1393 uint32_t hsail_major_version; 1394 uint32_t hsail_minor_version; 1395 uint8_t profile; 1396 uint8_t machine_model; 1397 uint8_t default_float_round; 1398 }; 1399 1400``NT_AMD_HSA_ISA_VERSION`` 1401 Specifies the target ISA version. The description field has the following layout: 1402 1403 .. code:: c 1404 1405 struct amdgpu_hsa_note_isa_s { 1406 uint16_t vendor_name_size; 1407 uint16_t architecture_name_size; 1408 uint32_t major; 1409 uint32_t minor; 1410 uint32_t stepping; 1411 char vendor_and_architecture_name[1]; 1412 }; 1413 1414 ``vendor_name_size`` and ``architecture_name_size`` are the length of the 1415 vendor and architecture names respectively, including the NUL character. 1416 1417 ``vendor_and_architecture_name`` contains the NUL terminates string for the 1418 vendor, immediately followed by the NUL terminated string for the 1419 architecture. 1420 1421 This note record is used by the HSA runtime loader. 1422 1423 Code object V2 only supports a limited number of processors and has fixed 1424 settings for target features. See 1425 :ref:`amdgpu-elf-note-record-supported_processors-v2-table` for a list of 1426 processors and the corresponding target ID. In the table the note record ISA 1427 name is a concatenation of the vendor name, architecture name, major, minor, 1428 and stepping separated by a ":". 1429 1430 The target ID column shows the processor name and fixed target features used 1431 by the LLVM compiler. The LLVM compiler does not generate a 1432 ``NT_AMD_HSA_HSAIL`` note record. 1433 1434 A code object generated by the Finalizer also uses code object V2 and always 1435 generates a ``NT_AMD_HSA_HSAIL`` note record. The processor name and 1436 ``sramecc`` target feature is as shown in 1437 :ref:`amdgpu-elf-note-record-supported_processors-v2-table` but the ``xnack`` 1438 target feature is specified by the ``EF_AMDGPU_FEATURE_XNACK_V2`` ``e_flags`` 1439 bit. 1440 1441``NT_AMD_HSA_ISA_NAME`` 1442 Specifies the target ISA name as a non-NUL terminated string. 1443 1444 This note record is not used by the HSA runtime loader. 1445 1446 See the ``NT_AMD_HSA_ISA_VERSION`` note record description of the code object 1447 V2's limited support of processors and fixed settings for target features. 1448 1449 See :ref:`amdgpu-elf-note-record-supported_processors-v2-table` for a mapping 1450 from the string to the corresponding target ID. If the ``xnack`` target 1451 feature is supported and enabled, the string produced by the LLVM compiler 1452 will may have a ``+xnack`` appended. The Finlizer did not do the appending and 1453 instead used the ``EF_AMDGPU_FEATURE_XNACK_V2`` ``e_flags`` bit. 1454 1455``NT_AMD_HSA_METADATA`` 1456 Specifies extensible metadata associated with the code objects executed on HSA 1457 [HSA]_ compatible runtimes (see :ref:`amdgpu-os`). It is required when the 1458 target triple OS is ``amdhsa`` (see :ref:`amdgpu-target-triples`). See 1459 :ref:`amdgpu-amdhsa-code-object-metadata-v2` for the syntax of the code object 1460 metadata string. 1461 1462 .. table:: AMDGPU Code Object V2 Supported Processors and Fixed Target Feature Settings 1463 :name: amdgpu-elf-note-record-supported_processors-v2-table 1464 1465 ===================== ========================== 1466 Note Record ISA Name Target ID 1467 ===================== ========================== 1468 ``AMD:AMDGPU:6:0:0`` ``gfx600`` 1469 ``AMD:AMDGPU:6:0:1`` ``gfx601`` 1470 ``AMD:AMDGPU:6:0:2`` ``gfx602`` 1471 ``AMD:AMDGPU:7:0:0`` ``gfx700`` 1472 ``AMD:AMDGPU:7:0:1`` ``gfx701`` 1473 ``AMD:AMDGPU:7:0:2`` ``gfx702`` 1474 ``AMD:AMDGPU:7:0:3`` ``gfx703`` 1475 ``AMD:AMDGPU:7:0:4`` ``gfx704`` 1476 ``AMD:AMDGPU:7:0:5`` ``gfx705`` 1477 ``AMD:AMDGPU:8:0:0`` ``gfx802`` 1478 ``AMD:AMDGPU:8:0:1`` ``gfx801:xnack+`` 1479 ``AMD:AMDGPU:8:0:2`` ``gfx802`` 1480 ``AMD:AMDGPU:8:0:3`` ``gfx803`` 1481 ``AMD:AMDGPU:8:0:4`` ``gfx803`` 1482 ``AMD:AMDGPU:8:0:5`` ``gfx805`` 1483 ``AMD:AMDGPU:8:1:0`` ``gfx810:xnack+`` 1484 ``AMD:AMDGPU:9:0:0`` ``gfx900:xnack-`` 1485 ``AMD:AMDGPU:9:0:1`` ``gfx900:xnack+`` 1486 ``AMD:AMDGPU:9:0:2`` ``gfx902:xnack-`` 1487 ``AMD:AMDGPU:9:0:3`` ``gfx902:xnack+`` 1488 ``AMD:AMDGPU:9:0:4`` ``gfx904:xnack-`` 1489 ``AMD:AMDGPU:9:0:5`` ``gfx904:xnack+`` 1490 ``AMD:AMDGPU:9:0:6`` ``gfx906:sramecc-:xnack-`` 1491 ``AMD:AMDGPU:9:0:7`` ``gfx906:sramecc-:xnack+`` 1492 ``AMD:AMDGPU:9:0:12`` ``gfx90c:xnack-`` 1493 ===================== ========================== 1494 1495.. _amdgpu-note-records-v3-onwards: 1496 1497Code Object V3 and Above Note Records 1498~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1499 1500The AMDGPU backend code object uses the following ELF note record in the 1501``.note`` section when compiling for code object V3 and above. 1502 1503The note record vendor field is "AMDGPU". 1504 1505Additional note records may be present, but any which are not documented here 1506are deprecated and should not be used. 1507 1508 .. table:: AMDGPU Code Object V3 and Above ELF Note Records 1509 :name: amdgpu-elf-note-records-table-v3-onwards 1510 1511 ======== ============================== ====================================== 1512 Name Type Description 1513 ======== ============================== ====================================== 1514 "AMDGPU" ``NT_AMDGPU_METADATA`` Metadata in Message Pack [MsgPack]_ 1515 binary format. 1516 ======== ============================== ====================================== 1517 1518.. 1519 1520 .. table:: AMDGPU Code Object V3 and Above ELF Note Record Enumeration Values 1521 :name: amdgpu-elf-note-record-enumeration-values-table-v3-onwards 1522 1523 ============================== ===== 1524 Name Value 1525 ============================== ===== 1526 *reserved* 0-31 1527 ``NT_AMDGPU_METADATA`` 32 1528 ============================== ===== 1529 1530``NT_AMDGPU_METADATA`` 1531 Specifies extensible metadata associated with an AMDGPU code object. It is 1532 encoded as a map in the Message Pack [MsgPack]_ binary data format. See 1533 :ref:`amdgpu-amdhsa-code-object-metadata-v3`, 1534 :ref:`amdgpu-amdhsa-code-object-metadata-v4` and 1535 :ref:`amdgpu-amdhsa-code-object-metadata-v5` for the map keys defined for the 1536 ``amdhsa`` OS. 1537 1538.. _amdgpu-symbols: 1539 1540Symbols 1541------- 1542 1543Symbols include the following: 1544 1545 .. table:: AMDGPU ELF Symbols 1546 :name: amdgpu-elf-symbols-table 1547 1548 ===================== ================== ================ ================== 1549 Name Type Section Description 1550 ===================== ================== ================ ================== 1551 *link-name* ``STT_OBJECT`` - ``.data`` Global variable 1552 - ``.rodata`` 1553 - ``.bss`` 1554 *link-name*\ ``.kd`` ``STT_OBJECT`` - ``.rodata`` Kernel descriptor 1555 *link-name* ``STT_FUNC`` - ``.text`` Kernel entry point 1556 *link-name* ``STT_OBJECT`` - SHN_AMDGPU_LDS Global variable in LDS 1557 ===================== ================== ================ ================== 1558 1559Global variable 1560 Global variables both used and defined by the compilation unit. 1561 1562 If the symbol is defined in the compilation unit then it is allocated in the 1563 appropriate section according to if it has initialized data or is readonly. 1564 1565 If the symbol is external then its section is ``STN_UNDEF`` and the loader 1566 will resolve relocations using the definition provided by another code object 1567 or explicitly defined by the runtime. 1568 1569 If the symbol resides in local/group memory (LDS) then its section is the 1570 special processor specific section name ``SHN_AMDGPU_LDS``, and the 1571 ``st_value`` field describes alignment requirements as it does for common 1572 symbols. 1573 1574 .. TODO:: 1575 1576 Add description of linked shared object symbols. Seems undefined symbols 1577 are marked as STT_NOTYPE. 1578 1579Kernel descriptor 1580 Every HSA kernel has an associated kernel descriptor. It is the address of the 1581 kernel descriptor that is used in the AQL dispatch packet used to invoke the 1582 kernel, not the kernel entry point. The layout of the HSA kernel descriptor is 1583 defined in :ref:`amdgpu-amdhsa-kernel-descriptor`. 1584 1585Kernel entry point 1586 Every HSA kernel also has a symbol for its machine code entry point. 1587 1588.. _amdgpu-relocation-records: 1589 1590Relocation Records 1591------------------ 1592 1593AMDGPU backend generates ``Elf64_Rela`` relocation records. Supported 1594relocatable fields are: 1595 1596``word32`` 1597 This specifies a 32-bit field occupying 4 bytes with arbitrary byte 1598 alignment. These values use the same byte order as other word values in the 1599 AMDGPU architecture. 1600 1601``word64`` 1602 This specifies a 64-bit field occupying 8 bytes with arbitrary byte 1603 alignment. These values use the same byte order as other word values in the 1604 AMDGPU architecture. 1605 1606Following notations are used for specifying relocation calculations: 1607 1608**A** 1609 Represents the addend used to compute the value of the relocatable field. 1610 1611**G** 1612 Represents the offset into the global offset table at which the relocation 1613 entry's symbol will reside during execution. 1614 1615**GOT** 1616 Represents the address of the global offset table. 1617 1618**P** 1619 Represents the place (section offset for ``et_rel`` or address for ``et_dyn``) 1620 of the storage unit being relocated (computed using ``r_offset``). 1621 1622**S** 1623 Represents the value of the symbol whose index resides in the relocation 1624 entry. Relocations not using this must specify a symbol index of 1625 ``STN_UNDEF``. 1626 1627**B** 1628 Represents the base address of a loaded executable or shared object which is 1629 the difference between the ELF address and the actual load address. 1630 Relocations using this are only valid in executable or shared objects. 1631 1632The following relocation types are supported: 1633 1634 .. table:: AMDGPU ELF Relocation Records 1635 :name: amdgpu-elf-relocation-records-table 1636 1637 ========================== ======= ===== ========== ============================== 1638 Relocation Type Kind Value Field Calculation 1639 ========================== ======= ===== ========== ============================== 1640 ``R_AMDGPU_NONE`` 0 *none* *none* 1641 ``R_AMDGPU_ABS32_LO`` Static, 1 ``word32`` (S + A) & 0xFFFFFFFF 1642 Dynamic 1643 ``R_AMDGPU_ABS32_HI`` Static, 2 ``word32`` (S + A) >> 32 1644 Dynamic 1645 ``R_AMDGPU_ABS64`` Static, 3 ``word64`` S + A 1646 Dynamic 1647 ``R_AMDGPU_REL32`` Static 4 ``word32`` S + A - P 1648 ``R_AMDGPU_REL64`` Static 5 ``word64`` S + A - P 1649 ``R_AMDGPU_ABS32`` Static, 6 ``word32`` S + A 1650 Dynamic 1651 ``R_AMDGPU_GOTPCREL`` Static 7 ``word32`` G + GOT + A - P 1652 ``R_AMDGPU_GOTPCREL32_LO`` Static 8 ``word32`` (G + GOT + A - P) & 0xFFFFFFFF 1653 ``R_AMDGPU_GOTPCREL32_HI`` Static 9 ``word32`` (G + GOT + A - P) >> 32 1654 ``R_AMDGPU_REL32_LO`` Static 10 ``word32`` (S + A - P) & 0xFFFFFFFF 1655 ``R_AMDGPU_REL32_HI`` Static 11 ``word32`` (S + A - P) >> 32 1656 *reserved* 12 1657 ``R_AMDGPU_RELATIVE64`` Dynamic 13 ``word64`` B + A 1658 ``R_AMDGPU_REL16`` Static 14 ``word16`` ((S + A - P) - 4) / 4 1659 ========================== ======= ===== ========== ============================== 1660 1661``R_AMDGPU_ABS32_LO`` and ``R_AMDGPU_ABS32_HI`` are only supported by 1662the ``mesa3d`` OS, which does not support ``R_AMDGPU_ABS64``. 1663 1664There is no current OS loader support for 32-bit programs and so 1665``R_AMDGPU_ABS32`` is not used. 1666 1667.. _amdgpu-loaded-code-object-path-uniform-resource-identifier: 1668 1669Loaded Code Object Path Uniform Resource Identifier (URI) 1670--------------------------------------------------------- 1671 1672The AMD GPU code object loader represents the path of the ELF shared object from 1673which the code object was loaded as a textual Uniform Resource Identifier (URI). 1674Note that the code object is the in memory loaded relocated form of the ELF 1675shared object. Multiple code objects may be loaded at different memory 1676addresses in the same process from the same ELF shared object. 1677 1678The loaded code object path URI syntax is defined by the following BNF syntax: 1679 1680.. code:: 1681 1682 code_object_uri ::== file_uri | memory_uri 1683 file_uri ::== "file://" file_path [ range_specifier ] 1684 memory_uri ::== "memory://" process_id range_specifier 1685 range_specifier ::== [ "#" | "?" ] "offset=" number "&" "size=" number 1686 file_path ::== URI_ENCODED_OS_FILE_PATH 1687 process_id ::== DECIMAL_NUMBER 1688 number ::== HEX_NUMBER | DECIMAL_NUMBER | OCTAL_NUMBER 1689 1690**number** 1691 Is a C integral literal where hexadecimal values are prefixed by "0x" or "0X", 1692 and octal values by "0". 1693 1694**file_path** 1695 Is the file's path specified as a URI encoded UTF-8 string. In URI encoding, 1696 every character that is not in the regular expression ``[a-zA-Z0-9/_.~-]`` is 1697 encoded as two uppercase hexadecimal digits proceeded by "%". Directories in 1698 the path are separated by "/". 1699 1700**offset** 1701 Is a 0-based byte offset to the start of the code object. For a file URI, it 1702 is from the start of the file specified by the ``file_path``, and if omitted 1703 defaults to 0. For a memory URI, it is the memory address and is required. 1704 1705**size** 1706 Is the number of bytes in the code object. For a file URI, if omitted it 1707 defaults to the size of the file. It is required for a memory URI. 1708 1709**process_id** 1710 Is the identity of the process owning the memory. For Linux it is the C 1711 unsigned integral decimal literal for the process ID (PID). 1712 1713For example: 1714 1715.. code:: 1716 1717 file:///dir1/dir2/file1 1718 file:///dir3/dir4/file2#offset=0x2000&size=3000 1719 memory://1234#offset=0x20000&size=3000 1720 1721.. _amdgpu-dwarf-debug-information: 1722 1723DWARF Debug Information 1724======================= 1725 1726.. warning:: 1727 1728 This section describes **provisional support** for AMDGPU DWARF [DWARF]_ that 1729 is not currently fully implemented and is subject to change. 1730 1731AMDGPU generates DWARF [DWARF]_ debugging information ELF sections (see 1732:ref:`amdgpu-elf-code-object`) which contain information that maps the code 1733object executable code and data to the source language constructs. It can be 1734used by tools such as debuggers and profilers. It uses features defined in 1735:doc:`AMDGPUDwarfExtensionsForHeterogeneousDebugging` that are made available in 1736DWARF Version 4 and DWARF Version 5 as an LLVM vendor extension. 1737 1738This section defines the AMDGPU target architecture specific DWARF mappings. 1739 1740.. _amdgpu-dwarf-register-identifier: 1741 1742Register Identifier 1743------------------- 1744 1745This section defines the AMDGPU target architecture register numbers used in 1746DWARF operation expressions (see DWARF Version 5 section 2.5 and 1747:ref:`amdgpu-dwarf-operation-expressions`) and Call Frame Information 1748instructions (see DWARF Version 5 section 6.4 and 1749:ref:`amdgpu-dwarf-call-frame-information`). 1750 1751A single code object can contain code for kernels that have different wavefront 1752sizes. The vector registers and some scalar registers are based on the wavefront 1753size. AMDGPU defines distinct DWARF registers for each wavefront size. This 1754simplifies the consumer of the DWARF so that each register has a fixed size, 1755rather than being dynamic according to the wavefront size mode. Similarly, 1756distinct DWARF registers are defined for those registers that vary in size 1757according to the process address size. This allows a consumer to treat a 1758specific AMDGPU processor as a single architecture regardless of how it is 1759configured at run time. The compiler explicitly specifies the DWARF registers 1760that match the mode in which the code it is generating will be executed. 1761 1762DWARF registers are encoded as numbers, which are mapped to architecture 1763registers. The mapping for AMDGPU is defined in 1764:ref:`amdgpu-dwarf-register-mapping-table`. All AMDGPU targets use the same 1765mapping. 1766 1767.. table:: AMDGPU DWARF Register Mapping 1768 :name: amdgpu-dwarf-register-mapping-table 1769 1770 ============== ================= ======== ================================== 1771 DWARF Register AMDGPU Register Bit Size Description 1772 ============== ================= ======== ================================== 1773 0 PC_32 32 Program Counter (PC) when 1774 executing in a 32-bit process 1775 address space. Used in the CFI to 1776 describe the PC of the calling 1777 frame. 1778 1 EXEC_MASK_32 32 Execution Mask Register when 1779 executing in wavefront 32 mode. 1780 2-15 *Reserved* *Reserved for highly accessed 1781 registers using DWARF shortcut.* 1782 16 PC_64 64 Program Counter (PC) when 1783 executing in a 64-bit process 1784 address space. Used in the CFI to 1785 describe the PC of the calling 1786 frame. 1787 17 EXEC_MASK_64 64 Execution Mask Register when 1788 executing in wavefront 64 mode. 1789 18-31 *Reserved* *Reserved for highly accessed 1790 registers using DWARF shortcut.* 1791 32-95 SGPR0-SGPR63 32 Scalar General Purpose 1792 Registers. 1793 96-127 *Reserved* *Reserved for frequently accessed 1794 registers using DWARF 1-byte ULEB.* 1795 128 STATUS 32 Status Register. 1796 129-511 *Reserved* *Reserved for future Scalar 1797 Architectural Registers.* 1798 512 VCC_32 32 Vector Condition Code Register 1799 when executing in wavefront 32 1800 mode. 1801 513-767 *Reserved* *Reserved for future Vector 1802 Architectural Registers when 1803 executing in wavefront 32 mode.* 1804 768 VCC_64 64 Vector Condition Code Register 1805 when executing in wavefront 64 1806 mode. 1807 769-1023 *Reserved* *Reserved for future Vector 1808 Architectural Registers when 1809 executing in wavefront 64 mode.* 1810 1024-1087 *Reserved* *Reserved for padding.* 1811 1088-1129 SGPR64-SGPR105 32 Scalar General Purpose Registers. 1812 1130-1535 *Reserved* *Reserved for future Scalar 1813 General Purpose Registers.* 1814 1536-1791 VGPR0-VGPR255 32*32 Vector General Purpose Registers 1815 when executing in wavefront 32 1816 mode. 1817 1792-2047 *Reserved* *Reserved for future Vector 1818 General Purpose Registers when 1819 executing in wavefront 32 mode.* 1820 2048-2303 AGPR0-AGPR255 32*32 Vector Accumulation Registers 1821 when executing in wavefront 32 1822 mode. 1823 2304-2559 *Reserved* *Reserved for future Vector 1824 Accumulation Registers when 1825 executing in wavefront 32 mode.* 1826 2560-2815 VGPR0-VGPR255 64*32 Vector General Purpose Registers 1827 when executing in wavefront 64 1828 mode. 1829 2816-3071 *Reserved* *Reserved for future Vector 1830 General Purpose Registers when 1831 executing in wavefront 64 mode.* 1832 3072-3327 AGPR0-AGPR255 64*32 Vector Accumulation Registers 1833 when executing in wavefront 64 1834 mode. 1835 3328-3583 *Reserved* *Reserved for future Vector 1836 Accumulation Registers when 1837 executing in wavefront 64 mode.* 1838 ============== ================= ======== ================================== 1839 1840The vector registers are represented as the full size for the wavefront. They 1841are organized as consecutive dwords (32-bits), one per lane, with the dword at 1842the least significant bit position corresponding to lane 0 and so forth. DWARF 1843location expressions involving the ``DW_OP_LLVM_offset`` and 1844``DW_OP_LLVM_push_lane`` operations are used to select the part of the vector 1845register corresponding to the lane that is executing the current thread of 1846execution in languages that are implemented using a SIMD or SIMT execution 1847model. 1848 1849If the wavefront size is 32 lanes then the wavefront 32 mode register 1850definitions are used. If the wavefront size is 64 lanes then the wavefront 64 1851mode register definitions are used. Some AMDGPU targets support executing in 1852both wavefront 32 and wavefront 64 mode. The register definitions corresponding 1853to the wavefront mode of the generated code will be used. 1854 1855If code is generated to execute in a 32-bit process address space, then the 185632-bit process address space register definitions are used. If code is generated 1857to execute in a 64-bit process address space, then the 64-bit process address 1858space register definitions are used. The ``amdgcn`` target only supports the 185964-bit process address space. 1860 1861.. _amdgpu-dwarf-address-class-identifier: 1862 1863Address Class Identifier 1864------------------------ 1865 1866The DWARF address class represents the source language memory space. See DWARF 1867Version 5 section 2.12 which is updated by the *DWARF Extensions For 1868Heterogeneous Debugging* section :ref:`amdgpu-dwarf-segment_addresses`. 1869 1870The DWARF address class mapping used for AMDGPU is defined in 1871:ref:`amdgpu-dwarf-address-class-mapping-table`. 1872 1873.. table:: AMDGPU DWARF Address Class Mapping 1874 :name: amdgpu-dwarf-address-class-mapping-table 1875 1876 ========================= ====== ================= 1877 DWARF AMDGPU 1878 -------------------------------- ----------------- 1879 Address Class Name Value Address Space 1880 ========================= ====== ================= 1881 ``DW_ADDR_none`` 0x0000 Generic (Flat) 1882 ``DW_ADDR_LLVM_global`` 0x0001 Global 1883 ``DW_ADDR_LLVM_constant`` 0x0002 Global 1884 ``DW_ADDR_LLVM_group`` 0x0003 Local (group/LDS) 1885 ``DW_ADDR_LLVM_private`` 0x0004 Private (Scratch) 1886 ``DW_ADDR_AMDGPU_region`` 0x8000 Region (GDS) 1887 ========================= ====== ================= 1888 1889The DWARF address class values defined in the *DWARF Extensions For 1890Heterogeneous Debugging* section :ref:`amdgpu-dwarf-segment_addresses` are used. 1891 1892In addition, ``DW_ADDR_AMDGPU_region`` is encoded as a vendor extension. This is 1893available for use for the AMD extension for access to the hardware GDS memory 1894which is scratchpad memory allocated per device. 1895 1896For AMDGPU if no ``DW_AT_address_class`` attribute is present, then the default 1897address class of ``DW_ADDR_none`` is used. 1898 1899See :ref:`amdgpu-dwarf-address-space-identifier` for information on the AMDGPU 1900mapping of DWARF address classes to DWARF address spaces, including address size 1901and NULL value. 1902 1903.. _amdgpu-dwarf-address-space-identifier: 1904 1905Address Space Identifier 1906------------------------ 1907 1908DWARF address spaces correspond to target architecture specific linear 1909addressable memory areas. See DWARF Version 5 section 2.12 and *DWARF Extensions 1910For Heterogeneous Debugging* section :ref:`amdgpu-dwarf-segment_addresses`. 1911 1912The DWARF address space mapping used for AMDGPU is defined in 1913:ref:`amdgpu-dwarf-address-space-mapping-table`. 1914 1915.. table:: AMDGPU DWARF Address Space Mapping 1916 :name: amdgpu-dwarf-address-space-mapping-table 1917 1918 ======================================= ===== ======= ======== ================= ======================= 1919 DWARF AMDGPU Notes 1920 --------------------------------------- ----- ---------------- ----------------- ----------------------- 1921 Address Space Name Value Address Bit Size Address Space 1922 --------------------------------------- ----- ------- -------- ----------------- ----------------------- 1923 .. 64-bit 32-bit 1924 process process 1925 address address 1926 space space 1927 ======================================= ===== ======= ======== ================= ======================= 1928 ``DW_ASPACE_none`` 0x00 64 32 Global *default address space* 1929 ``DW_ASPACE_AMDGPU_generic`` 0x01 64 32 Generic (Flat) 1930 ``DW_ASPACE_AMDGPU_region`` 0x02 32 32 Region (GDS) 1931 ``DW_ASPACE_AMDGPU_local`` 0x03 32 32 Local (group/LDS) 1932 *Reserved* 0x04 1933 ``DW_ASPACE_AMDGPU_private_lane`` 0x05 32 32 Private (Scratch) *focused lane* 1934 ``DW_ASPACE_AMDGPU_private_wave`` 0x06 32 32 Private (Scratch) *unswizzled wavefront* 1935 ======================================= ===== ======= ======== ================= ======================= 1936 1937See :ref:`amdgpu-address-spaces` for information on the AMDGPU address spaces 1938including address size and NULL value. 1939 1940The ``DW_ASPACE_none`` address space is the default target architecture address 1941space used in DWARF operations that do not specify an address space. It 1942therefore has to map to the global address space so that the ``DW_OP_addr*`` and 1943related operations can refer to addresses in the program code. 1944 1945The ``DW_ASPACE_AMDGPU_generic`` address space allows location expressions to 1946specify the flat address space. If the address corresponds to an address in the 1947local address space, then it corresponds to the wavefront that is executing the 1948focused thread of execution. If the address corresponds to an address in the 1949private address space, then it corresponds to the lane that is executing the 1950focused thread of execution for languages that are implemented using a SIMD or 1951SIMT execution model. 1952 1953.. note:: 1954 1955 CUDA-like languages such as HIP that do not have address spaces in the 1956 language type system, but do allow variables to be allocated in different 1957 address spaces, need to explicitly specify the ``DW_ASPACE_AMDGPU_generic`` 1958 address space in the DWARF expression operations as the default address space 1959 is the global address space. 1960 1961The ``DW_ASPACE_AMDGPU_local`` address space allows location expressions to 1962specify the local address space corresponding to the wavefront that is executing 1963the focused thread of execution. 1964 1965The ``DW_ASPACE_AMDGPU_private_lane`` address space allows location expressions 1966to specify the private address space corresponding to the lane that is executing 1967the focused thread of execution for languages that are implemented using a SIMD 1968or SIMT execution model. 1969 1970The ``DW_ASPACE_AMDGPU_private_wave`` address space allows location expressions 1971to specify the unswizzled private address space corresponding to the wavefront 1972that is executing the focused thread of execution. The wavefront view of private 1973memory is the per wavefront unswizzled backing memory layout defined in 1974:ref:`amdgpu-address-spaces`, such that address 0 corresponds to the first 1975location for the backing memory of the wavefront (namely the address is not 1976offset by ``wavefront-scratch-base``). The following formula can be used to 1977convert from a ``DW_ASPACE_AMDGPU_private_lane`` address to a 1978``DW_ASPACE_AMDGPU_private_wave`` address: 1979 1980:: 1981 1982 private-address-wavefront = 1983 ((private-address-lane / 4) * wavefront-size * 4) + 1984 (wavefront-lane-id * 4) + (private-address-lane % 4) 1985 1986If the ``DW_ASPACE_AMDGPU_private_lane`` address is dword aligned, and the start 1987of the dwords for each lane starting with lane 0 is required, then this 1988simplifies to: 1989 1990:: 1991 1992 private-address-wavefront = 1993 private-address-lane * wavefront-size 1994 1995A compiler can use the ``DW_ASPACE_AMDGPU_private_wave`` address space to read a 1996complete spilled vector register back into a complete vector register in the 1997CFI. The frame pointer can be a private lane address which is dword aligned, 1998which can be shifted to multiply by the wavefront size, and then used to form a 1999private wavefront address that gives a location for a contiguous set of dwords, 2000one per lane, where the vector register dwords are spilled. The compiler knows 2001the wavefront size since it generates the code. Note that the type of the 2002address may have to be converted as the size of a 2003``DW_ASPACE_AMDGPU_private_lane`` address may be smaller than the size of a 2004``DW_ASPACE_AMDGPU_private_wave`` address. 2005 2006.. _amdgpu-dwarf-lane-identifier: 2007 2008Lane identifier 2009--------------- 2010 2011DWARF lane identifies specify a target architecture lane position for hardware 2012that executes in a SIMD or SIMT manner, and on which a source language maps its 2013threads of execution onto those lanes. The DWARF lane identifier is pushed by 2014the ``DW_OP_LLVM_push_lane`` DWARF expression operation. See DWARF Version 5 2015section 2.5 which is updated by *DWARF Extensions For Heterogeneous Debugging* 2016section :ref:`amdgpu-dwarf-operation-expressions`. 2017 2018For AMDGPU, the lane identifier corresponds to the hardware lane ID of a 2019wavefront. It is numbered from 0 to the wavefront size minus 1. 2020 2021Operation Expressions 2022--------------------- 2023 2024DWARF expressions are used to compute program values and the locations of 2025program objects. See DWARF Version 5 section 2.5 and 2026:ref:`amdgpu-dwarf-operation-expressions`. 2027 2028DWARF location descriptions describe how to access storage which includes memory 2029and registers. When accessing storage on AMDGPU, bytes are ordered with least 2030significant bytes first, and bits are ordered within bytes with least 2031significant bits first. 2032 2033For AMDGPU CFI expressions, ``DW_OP_LLVM_select_bit_piece`` is used to describe 2034unwinding vector registers that are spilled under the execution mask to memory: 2035the zero-single location description is the vector register, and the one-single 2036location description is the spilled memory location description. The 2037``DW_OP_LLVM_form_aspace_address`` is used to specify the address space of the 2038memory location description. 2039 2040In AMDGPU expressions, ``DW_OP_LLVM_select_bit_piece`` is used by the 2041``DW_AT_LLVM_lane_pc`` attribute expression where divergent control flow is 2042controlled by the execution mask. An undefined location description together 2043with ``DW_OP_LLVM_extend`` is used to indicate the lane was not active on entry 2044to the subprogram. See :ref:`amdgpu-dwarf-dw-at-llvm-lane-pc` for an example. 2045 2046Debugger Information Entry Attributes 2047------------------------------------- 2048 2049This section describes how certain debugger information entry attributes are 2050used by AMDGPU. See the sections in DWARF Version 5 section 3.3.5 and 3.1.1 2051which are updated by *DWARF Extensions For Heterogeneous Debugging* section 2052:ref:`amdgpu-dwarf-low-level-information` and 2053:ref:`amdgpu-dwarf-full-and-partial-compilation-unit-entries`. 2054 2055.. _amdgpu-dwarf-dw-at-llvm-lane-pc: 2056 2057``DW_AT_LLVM_lane_pc`` 2058~~~~~~~~~~~~~~~~~~~~~~ 2059 2060For AMDGPU, the ``DW_AT_LLVM_lane_pc`` attribute is used to specify the program 2061location of the separate lanes of a SIMT thread. 2062 2063If the lane is an active lane then this will be the same as the current program 2064location. 2065 2066If the lane is inactive, but was active on entry to the subprogram, then this is 2067the program location in the subprogram at which execution of the lane is 2068conceptual positioned. 2069 2070If the lane was not active on entry to the subprogram, then this will be the 2071undefined location. A client debugger can check if the lane is part of a valid 2072work-group by checking that the lane is in the range of the associated 2073work-group within the grid, accounting for partial work-groups. If it is not, 2074then the debugger can omit any information for the lane. Otherwise, the debugger 2075may repeatedly unwind the stack and inspect the ``DW_AT_LLVM_lane_pc`` of the 2076calling subprogram until it finds a non-undefined location. Conceptually the 2077lane only has the call frames that it has a non-undefined 2078``DW_AT_LLVM_lane_pc``. 2079 2080The following example illustrates how the AMDGPU backend can generate a DWARF 2081location list expression for the nested ``IF/THEN/ELSE`` structures of the 2082following subprogram pseudo code for a target with 64 lanes per wavefront. 2083 2084.. code:: 2085 :number-lines: 2086 2087 SUBPROGRAM X 2088 BEGIN 2089 a; 2090 IF (c1) THEN 2091 b; 2092 IF (c2) THEN 2093 c; 2094 ELSE 2095 d; 2096 ENDIF 2097 e; 2098 ELSE 2099 f; 2100 ENDIF 2101 g; 2102 END 2103 2104The AMDGPU backend may generate the following pseudo LLVM MIR to manipulate the 2105execution mask (``EXEC``) to linearize the control flow. The condition is 2106evaluated to make a mask of the lanes for which the condition evaluates to true. 2107First the ``THEN`` region is executed by setting the ``EXEC`` mask to the 2108logical ``AND`` of the current ``EXEC`` mask with the condition mask. Then the 2109``ELSE`` region is executed by negating the ``EXEC`` mask and logical ``AND`` of 2110the saved ``EXEC`` mask at the start of the region. After the ``IF/THEN/ELSE`` 2111region the ``EXEC`` mask is restored to the value it had at the beginning of the 2112region. This is shown below. Other approaches are possible, but the basic 2113concept is the same. 2114 2115.. code:: 2116 :number-lines: 2117 2118 $lex_start: 2119 a; 2120 %1 = EXEC 2121 %2 = c1 2122 $lex_1_start: 2123 EXEC = %1 & %2 2124 $if_1_then: 2125 b; 2126 %3 = EXEC 2127 %4 = c2 2128 $lex_1_1_start: 2129 EXEC = %3 & %4 2130 $lex_1_1_then: 2131 c; 2132 EXEC = ~EXEC & %3 2133 $lex_1_1_else: 2134 d; 2135 EXEC = %3 2136 $lex_1_1_end: 2137 e; 2138 EXEC = ~EXEC & %1 2139 $lex_1_else: 2140 f; 2141 EXEC = %1 2142 $lex_1_end: 2143 g; 2144 $lex_end: 2145 2146To create the DWARF location list expression that defines the location 2147description of a vector of lane program locations, the LLVM MIR ``DBG_VALUE`` 2148pseudo instruction can be used to annotate the linearized control flow. This can 2149be done by defining an artificial variable for the lane PC. The DWARF location 2150list expression created for it is used as the value of the 2151``DW_AT_LLVM_lane_pc`` attribute on the subprogram's debugger information entry. 2152 2153A DWARF procedure is defined for each well nested structured control flow region 2154which provides the conceptual lane program location for a lane if it is not 2155active (namely it is divergent). The DWARF operation expression for each region 2156conceptually inherits the value of the immediately enclosing region and modifies 2157it according to the semantics of the region. 2158 2159For an ``IF/THEN/ELSE`` region the divergent program location is at the start of 2160the region for the ``THEN`` region since it is executed first. For the ``ELSE`` 2161region the divergent program location is at the end of the ``IF/THEN/ELSE`` 2162region since the ``THEN`` region has completed. 2163 2164The lane PC artificial variable is assigned at each region transition. It uses 2165the immediately enclosing region's DWARF procedure to compute the program 2166location for each lane assuming they are divergent, and then modifies the result 2167by inserting the current program location for each lane that the ``EXEC`` mask 2168indicates is active. 2169 2170By having separate DWARF procedures for each region, they can be reused to 2171define the value for any nested region. This reduces the total size of the DWARF 2172operation expressions. 2173 2174The following provides an example using pseudo LLVM MIR. 2175 2176.. code:: 2177 :number-lines: 2178 2179 $lex_start: 2180 DEFINE_DWARF %__uint_64 = DW_TAG_base_type[ 2181 DW_AT_name = "__uint64"; 2182 DW_AT_byte_size = 8; 2183 DW_AT_encoding = DW_ATE_unsigned; 2184 ]; 2185 DEFINE_DWARF %__active_lane_pc = DW_TAG_dwarf_procedure[ 2186 DW_AT_name = "__active_lane_pc"; 2187 DW_AT_location = [ 2188 DW_OP_regx PC; 2189 DW_OP_LLVM_extend 64, 64; 2190 DW_OP_regval_type EXEC, %uint_64; 2191 DW_OP_LLVM_select_bit_piece 64, 64; 2192 ]; 2193 ]; 2194 DEFINE_DWARF %__divergent_lane_pc = DW_TAG_dwarf_procedure[ 2195 DW_AT_name = "__divergent_lane_pc"; 2196 DW_AT_location = [ 2197 DW_OP_LLVM_undefined; 2198 DW_OP_LLVM_extend 64, 64; 2199 ]; 2200 ]; 2201 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 2202 DW_OP_call_ref %__divergent_lane_pc; 2203 DW_OP_call_ref %__active_lane_pc; 2204 ]; 2205 a; 2206 %1 = EXEC; 2207 DBG_VALUE %1, $noreg, %__lex_1_save_exec; 2208 %2 = c1; 2209 $lex_1_start: 2210 EXEC = %1 & %2; 2211 $lex_1_then: 2212 DEFINE_DWARF %__divergent_lane_pc_1_then = DW_TAG_dwarf_procedure[ 2213 DW_AT_name = "__divergent_lane_pc_1_then"; 2214 DW_AT_location = DIExpression[ 2215 DW_OP_call_ref %__divergent_lane_pc; 2216 DW_OP_addrx &lex_1_start; 2217 DW_OP_stack_value; 2218 DW_OP_LLVM_extend 64, 64; 2219 DW_OP_call_ref %__lex_1_save_exec; 2220 DW_OP_deref_type 64, %__uint_64; 2221 DW_OP_LLVM_select_bit_piece 64, 64; 2222 ]; 2223 ]; 2224 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 2225 DW_OP_call_ref %__divergent_lane_pc_1_then; 2226 DW_OP_call_ref %__active_lane_pc; 2227 ]; 2228 b; 2229 %3 = EXEC; 2230 DBG_VALUE %3, %__lex_1_1_save_exec; 2231 %4 = c2; 2232 $lex_1_1_start: 2233 EXEC = %3 & %4; 2234 $lex_1_1_then: 2235 DEFINE_DWARF %__divergent_lane_pc_1_1_then = DW_TAG_dwarf_procedure[ 2236 DW_AT_name = "__divergent_lane_pc_1_1_then"; 2237 DW_AT_location = DIExpression[ 2238 DW_OP_call_ref %__divergent_lane_pc_1_then; 2239 DW_OP_addrx &lex_1_1_start; 2240 DW_OP_stack_value; 2241 DW_OP_LLVM_extend 64, 64; 2242 DW_OP_call_ref %__lex_1_1_save_exec; 2243 DW_OP_deref_type 64, %__uint_64; 2244 DW_OP_LLVM_select_bit_piece 64, 64; 2245 ]; 2246 ]; 2247 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 2248 DW_OP_call_ref %__divergent_lane_pc_1_1_then; 2249 DW_OP_call_ref %__active_lane_pc; 2250 ]; 2251 c; 2252 EXEC = ~EXEC & %3; 2253 $lex_1_1_else: 2254 DEFINE_DWARF %__divergent_lane_pc_1_1_else = DW_TAG_dwarf_procedure[ 2255 DW_AT_name = "__divergent_lane_pc_1_1_else"; 2256 DW_AT_location = DIExpression[ 2257 DW_OP_call_ref %__divergent_lane_pc_1_then; 2258 DW_OP_addrx &lex_1_1_end; 2259 DW_OP_stack_value; 2260 DW_OP_LLVM_extend 64, 64; 2261 DW_OP_call_ref %__lex_1_1_save_exec; 2262 DW_OP_deref_type 64, %__uint_64; 2263 DW_OP_LLVM_select_bit_piece 64, 64; 2264 ]; 2265 ]; 2266 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 2267 DW_OP_call_ref %__divergent_lane_pc_1_1_else; 2268 DW_OP_call_ref %__active_lane_pc; 2269 ]; 2270 d; 2271 EXEC = %3; 2272 $lex_1_1_end: 2273 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 2274 DW_OP_call_ref %__divergent_lane_pc; 2275 DW_OP_call_ref %__active_lane_pc; 2276 ]; 2277 e; 2278 EXEC = ~EXEC & %1; 2279 $lex_1_else: 2280 DEFINE_DWARF %__divergent_lane_pc_1_else = DW_TAG_dwarf_procedure[ 2281 DW_AT_name = "__divergent_lane_pc_1_else"; 2282 DW_AT_location = DIExpression[ 2283 DW_OP_call_ref %__divergent_lane_pc; 2284 DW_OP_addrx &lex_1_end; 2285 DW_OP_stack_value; 2286 DW_OP_LLVM_extend 64, 64; 2287 DW_OP_call_ref %__lex_1_save_exec; 2288 DW_OP_deref_type 64, %__uint_64; 2289 DW_OP_LLVM_select_bit_piece 64, 64; 2290 ]; 2291 ]; 2292 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 2293 DW_OP_call_ref %__divergent_lane_pc_1_else; 2294 DW_OP_call_ref %__active_lane_pc; 2295 ]; 2296 f; 2297 EXEC = %1; 2298 $lex_1_end: 2299 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc DIExpression[ 2300 DW_OP_call_ref %__divergent_lane_pc; 2301 DW_OP_call_ref %__active_lane_pc; 2302 ]; 2303 g; 2304 $lex_end: 2305 2306The DWARF procedure ``%__active_lane_pc`` is used to update the lane pc elements 2307that are active, with the current program location. 2308 2309Artificial variables %__lex_1_save_exec and %__lex_1_1_save_exec are created for 2310the execution masks saved on entry to a region. Using the ``DBG_VALUE`` pseudo 2311instruction, location list entries will be created that describe where the 2312artificial variables are allocated at any given program location. The compiler 2313may allocate them to registers or spill them to memory. 2314 2315The DWARF procedures for each region use the values of the saved execution mask 2316artificial variables to only update the lanes that are active on entry to the 2317region. All other lanes retain the value of the enclosing region where they were 2318last active. If they were not active on entry to the subprogram, then will have 2319the undefined location description. 2320 2321Other structured control flow regions can be handled similarly. For example, 2322loops would set the divergent program location for the region at the end of the 2323loop. Any lanes active will be in the loop, and any lanes not active must have 2324exited the loop. 2325 2326An ``IF/THEN/ELSEIF/ELSEIF/...`` region can be treated as a nest of 2327``IF/THEN/ELSE`` regions. 2328 2329The DWARF procedures can use the active lane artificial variable described in 2330:ref:`amdgpu-dwarf-amdgpu-dw-at-llvm-active-lane` rather than the actual 2331``EXEC`` mask in order to support whole or quad wavefront mode. 2332 2333.. _amdgpu-dwarf-amdgpu-dw-at-llvm-active-lane: 2334 2335``DW_AT_LLVM_active_lane`` 2336~~~~~~~~~~~~~~~~~~~~~~~~~~ 2337 2338The ``DW_AT_LLVM_active_lane`` attribute on a subprogram debugger information 2339entry is used to specify the lanes that are conceptually active for a SIMT 2340thread. 2341 2342The execution mask may be modified to implement whole or quad wavefront mode 2343operations. For example, all lanes may need to temporarily be made active to 2344execute a whole wavefront operation. Such regions would save the ``EXEC`` mask, 2345update it to enable the necessary lanes, perform the operations, and then 2346restore the ``EXEC`` mask from the saved value. While executing the whole 2347wavefront region, the conceptual execution mask is the saved value, not the 2348``EXEC`` value. 2349 2350This is handled by defining an artificial variable for the active lane mask. The 2351active lane mask artificial variable would be the actual ``EXEC`` mask for 2352normal regions, and the saved execution mask for regions where the mask is 2353temporarily updated. The location list expression created for this artificial 2354variable is used to define the value of the ``DW_AT_LLVM_active_lane`` 2355attribute. 2356 2357``DW_AT_LLVM_augmentation`` 2358~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2359 2360For AMDGPU, the ``DW_AT_LLVM_augmentation`` attribute of a compilation unit 2361debugger information entry has the following value for the augmentation string: 2362 2363:: 2364 2365 [amdgpu:v0.0] 2366 2367The "vX.Y" specifies the major X and minor Y version number of the AMDGPU 2368extensions used in the DWARF of the compilation unit. The version number 2369conforms to [SEMVER]_. 2370 2371Call Frame Information 2372---------------------- 2373 2374DWARF Call Frame Information (CFI) describes how a consumer can virtually 2375*unwind* call frames in a running process or core dump. See DWARF Version 5 2376section 6.4 and :ref:`amdgpu-dwarf-call-frame-information`. 2377 2378For AMDGPU, the Common Information Entry (CIE) fields have the following values: 2379 23801. ``augmentation`` string contains the following null-terminated UTF-8 string: 2381 2382 :: 2383 2384 [amd:v0.0] 2385 2386 The ``vX.Y`` specifies the major X and minor Y version number of the AMDGPU 2387 extensions used in this CIE or to the FDEs that use it. The version number 2388 conforms to [SEMVER]_. 2389 23902. ``address_size`` for the ``Global`` address space is defined in 2391 :ref:`amdgpu-dwarf-address-space-identifier`. 2392 23933. ``segment_selector_size`` is 0 as AMDGPU does not use a segment selector. 2394 23954. ``code_alignment_factor`` is 4 bytes. 2396 2397 .. TODO:: 2398 2399 Add to :ref:`amdgpu-processor-table` table. 2400 24015. ``data_alignment_factor`` is 4 bytes. 2402 2403 .. TODO:: 2404 2405 Add to :ref:`amdgpu-processor-table` table. 2406 24076. ``return_address_register`` is ``PC_32`` for 32-bit processes and ``PC_64`` 2408 for 64-bit processes defined in :ref:`amdgpu-dwarf-register-identifier`. 2409 24107. ``initial_instructions`` Since a subprogram X with fewer registers can be 2411 called from subprogram Y that has more allocated, X will not change any of 2412 the extra registers as it cannot access them. Therefore, the default rule 2413 for all columns is ``same value``. 2414 2415For AMDGPU the register number follows the numbering defined in 2416:ref:`amdgpu-dwarf-register-identifier`. 2417 2418For AMDGPU the instructions are variable size. A consumer can subtract 1 from 2419the return address to get the address of a byte within the call site 2420instructions. See DWARF Version 5 section 6.4.4. 2421 2422Accelerated Access 2423------------------ 2424 2425See DWARF Version 5 section 6.1. 2426 2427Lookup By Name Section Header 2428~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2429 2430See DWARF Version 5 section 6.1.1.4.1 and :ref:`amdgpu-dwarf-lookup-by-name`. 2431 2432For AMDGPU the lookup by name section header table: 2433 2434``augmentation_string_size`` (uword) 2435 2436 Set to the length of the ``augmentation_string`` value which is always a 2437 multiple of 4. 2438 2439``augmentation_string`` (sequence of UTF-8 characters) 2440 2441 Contains the following UTF-8 string null padded to a multiple of 4 bytes: 2442 2443 :: 2444 2445 [amdgpu:v0.0] 2446 2447 The "vX.Y" specifies the major X and minor Y version number of the AMDGPU 2448 extensions used in the DWARF of this index. The version number conforms to 2449 [SEMVER]_. 2450 2451 .. note:: 2452 2453 This is different to the DWARF Version 5 definition that requires the first 2454 4 characters to be the vendor ID. But this is consistent with the other 2455 augmentation strings and does allow multiple vendor contributions. However, 2456 backwards compatibility may be more desirable. 2457 2458Lookup By Address Section Header 2459~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2460 2461See DWARF Version 5 section 6.1.2. 2462 2463For AMDGPU the lookup by address section header table: 2464 2465``address_size`` (ubyte) 2466 2467 Match the address size for the ``Global`` address space defined in 2468 :ref:`amdgpu-dwarf-address-space-identifier`. 2469 2470``segment_selector_size`` (ubyte) 2471 2472 AMDGPU does not use a segment selector so this is 0. The entries in the 2473 ``.debug_aranges`` do not have a segment selector. 2474 2475Line Number Information 2476----------------------- 2477 2478See DWARF Version 5 section 6.2 and :ref:`amdgpu-dwarf-line-number-information`. 2479 2480AMDGPU does not use the ``isa`` state machine registers and always sets it to 0. 2481The instruction set must be obtained from the ELF file header ``e_flags`` field 2482in the ``EF_AMDGPU_MACH`` bit position (see :ref:`ELF Header 2483<amdgpu-elf-header>`). See DWARF Version 5 section 6.2.2. 2484 2485.. TODO:: 2486 2487 Should the ``isa`` state machine register be used to indicate if the code is 2488 in wavefront32 or wavefront64 mode? Or used to specify the architecture ISA? 2489 2490For AMDGPU the line number program header fields have the following values (see 2491DWARF Version 5 section 6.2.4): 2492 2493``address_size`` (ubyte) 2494 Matches the address size for the ``Global`` address space defined in 2495 :ref:`amdgpu-dwarf-address-space-identifier`. 2496 2497``segment_selector_size`` (ubyte) 2498 AMDGPU does not use a segment selector so this is 0. 2499 2500``minimum_instruction_length`` (ubyte) 2501 For GFX9-GFX10 this is 4. 2502 2503``maximum_operations_per_instruction`` (ubyte) 2504 For GFX9-GFX10 this is 1. 2505 2506Source text for online-compiled programs (for example, those compiled by the 2507OpenCL language runtime) may be embedded into the DWARF Version 5 line table. 2508See DWARF Version 5 section 6.2.4.1 which is updated by *DWARF Extensions For 2509Heterogeneous Debugging* section :ref:`DW_LNCT_LLVM_source 2510<amdgpu-dwarf-line-number-information-dw-lnct-llvm-source>`. 2511 2512The Clang option used to control source embedding in AMDGPU is defined in 2513:ref:`amdgpu-clang-debug-options-table`. 2514 2515 .. table:: AMDGPU Clang Debug Options 2516 :name: amdgpu-clang-debug-options-table 2517 2518 ==================== ================================================== 2519 Debug Flag Description 2520 ==================== ================================================== 2521 -g[no-]embed-source Enable/disable embedding source text in DWARF 2522 debug sections. Useful for environments where 2523 source cannot be written to disk, such as 2524 when performing online compilation. 2525 ==================== ================================================== 2526 2527For example: 2528 2529``-gembed-source`` 2530 Enable the embedded source. 2531 2532``-gno-embed-source`` 2533 Disable the embedded source. 2534 253532-Bit and 64-Bit DWARF Formats 2536------------------------------- 2537 2538See DWARF Version 5 section 7.4 and 2539:ref:`amdgpu-dwarf-32-bit-and-64-bit-dwarf-formats`. 2540 2541For AMDGPU: 2542 2543* For the ``amdgcn`` target architecture only the 64-bit process address space 2544 is supported. 2545 2546* The producer can generate either 32-bit or 64-bit DWARF format. LLVM generates 2547 the 32-bit DWARF format. 2548 2549Unit Headers 2550------------ 2551 2552For AMDGPU the following values apply for each of the unit headers described in 2553DWARF Version 5 sections 7.5.1.1, 7.5.1.2, and 7.5.1.3: 2554 2555``address_size`` (ubyte) 2556 Matches the address size for the ``Global`` address space defined in 2557 :ref:`amdgpu-dwarf-address-space-identifier`. 2558 2559.. _amdgpu-code-conventions: 2560 2561Code Conventions 2562================ 2563 2564This section provides code conventions used for each supported target triple OS 2565(see :ref:`amdgpu-target-triples`). 2566 2567AMDHSA 2568------ 2569 2570This section provides code conventions used when the target triple OS is 2571``amdhsa`` (see :ref:`amdgpu-target-triples`). 2572 2573.. _amdgpu-amdhsa-code-object-metadata: 2574 2575Code Object Metadata 2576~~~~~~~~~~~~~~~~~~~~ 2577 2578The code object metadata specifies extensible metadata associated with the code 2579objects executed on HSA [HSA]_ compatible runtimes (see :ref:`amdgpu-os`). The 2580encoding and semantics of this metadata depends on the code object version; see 2581:ref:`amdgpu-amdhsa-code-object-metadata-v2`, 2582:ref:`amdgpu-amdhsa-code-object-metadata-v3`, 2583:ref:`amdgpu-amdhsa-code-object-metadata-v4` and 2584:ref:`amdgpu-amdhsa-code-object-metadata-v5`. 2585 2586Code object metadata is specified in a note record (see 2587:ref:`amdgpu-note-records`) and is required when the target triple OS is 2588``amdhsa`` (see :ref:`amdgpu-target-triples`). It must contain the minimum 2589information necessary to support the HSA compatible runtime kernel queries. For 2590example, the segment sizes needed in a dispatch packet. In addition, a 2591high-level language runtime may require other information to be included. For 2592example, the AMD OpenCL runtime records kernel argument information. 2593 2594.. _amdgpu-amdhsa-code-object-metadata-v2: 2595 2596Code Object V2 Metadata 2597+++++++++++++++++++++++ 2598 2599.. warning:: 2600 Code object V2 is not the default code object version emitted by this version 2601 of LLVM. 2602 2603Code object V2 metadata is specified by the ``NT_AMD_HSA_METADATA`` note record 2604(see :ref:`amdgpu-note-records-v2`). 2605 2606The metadata is specified as a YAML formatted string (see [YAML]_ and 2607:doc:`YamlIO`). 2608 2609.. TODO:: 2610 2611 Is the string null terminated? It probably should not if YAML allows it to 2612 contain null characters, otherwise it should be. 2613 2614The metadata is represented as a single YAML document comprised of the mapping 2615defined in table :ref:`amdgpu-amdhsa-code-object-metadata-map-v2-table` and 2616referenced tables. 2617 2618For boolean values, the string values of ``false`` and ``true`` are used for 2619false and true respectively. 2620 2621Additional information can be added to the mappings. To avoid conflicts, any 2622non-AMD key names should be prefixed by "*vendor-name*.". 2623 2624 .. table:: AMDHSA Code Object V2 Metadata Map 2625 :name: amdgpu-amdhsa-code-object-metadata-map-v2-table 2626 2627 ========== ============== ========= ======================================= 2628 String Key Value Type Required? Description 2629 ========== ============== ========= ======================================= 2630 "Version" sequence of Required - The first integer is the major 2631 2 integers version. Currently 1. 2632 - The second integer is the minor 2633 version. Currently 0. 2634 "Printf" sequence of Each string is encoded information 2635 strings about a printf function call. The 2636 encoded information is organized as 2637 fields separated by colon (':'): 2638 2639 ``ID:N:S[0]:S[1]:...:S[N-1]:FormatString`` 2640 2641 where: 2642 2643 ``ID`` 2644 A 32-bit integer as a unique id for 2645 each printf function call 2646 2647 ``N`` 2648 A 32-bit integer equal to the number 2649 of arguments of printf function call 2650 minus 1 2651 2652 ``S[i]`` (where i = 0, 1, ... , N-1) 2653 32-bit integers for the size in bytes 2654 of the i-th FormatString argument of 2655 the printf function call 2656 2657 FormatString 2658 The format string passed to the 2659 printf function call. 2660 "Kernels" sequence of Required Sequence of the mappings for each 2661 mapping kernel in the code object. See 2662 :ref:`amdgpu-amdhsa-code-object-kernel-metadata-map-v2-table` 2663 for the definition of the mapping. 2664 ========== ============== ========= ======================================= 2665 2666.. 2667 2668 .. table:: AMDHSA Code Object V2 Kernel Metadata Map 2669 :name: amdgpu-amdhsa-code-object-kernel-metadata-map-v2-table 2670 2671 ================= ============== ========= ================================ 2672 String Key Value Type Required? Description 2673 ================= ============== ========= ================================ 2674 "Name" string Required Source name of the kernel. 2675 "SymbolName" string Required Name of the kernel 2676 descriptor ELF symbol. 2677 "Language" string Source language of the kernel. 2678 Values include: 2679 2680 - "OpenCL C" 2681 - "OpenCL C++" 2682 - "HCC" 2683 - "OpenMP" 2684 2685 "LanguageVersion" sequence of - The first integer is the major 2686 2 integers version. 2687 - The second integer is the 2688 minor version. 2689 "Attrs" mapping Mapping of kernel attributes. 2690 See 2691 :ref:`amdgpu-amdhsa-code-object-kernel-attribute-metadata-map-v2-table` 2692 for the mapping definition. 2693 "Args" sequence of Sequence of mappings of the 2694 mapping kernel arguments. See 2695 :ref:`amdgpu-amdhsa-code-object-kernel-argument-metadata-map-v2-table` 2696 for the definition of the mapping. 2697 "CodeProps" mapping Mapping of properties related to 2698 the kernel code. See 2699 :ref:`amdgpu-amdhsa-code-object-kernel-code-properties-metadata-map-v2-table` 2700 for the mapping definition. 2701 ================= ============== ========= ================================ 2702 2703.. 2704 2705 .. table:: AMDHSA Code Object V2 Kernel Attribute Metadata Map 2706 :name: amdgpu-amdhsa-code-object-kernel-attribute-metadata-map-v2-table 2707 2708 =================== ============== ========= ============================== 2709 String Key Value Type Required? Description 2710 =================== ============== ========= ============================== 2711 "ReqdWorkGroupSize" sequence of If not 0, 0, 0 then all values 2712 3 integers must be >=1 and the dispatch 2713 work-group size X, Y, Z must 2714 correspond to the specified 2715 values. Defaults to 0, 0, 0. 2716 2717 Corresponds to the OpenCL 2718 ``reqd_work_group_size`` 2719 attribute. 2720 "WorkGroupSizeHint" sequence of The dispatch work-group size 2721 3 integers X, Y, Z is likely to be the 2722 specified values. 2723 2724 Corresponds to the OpenCL 2725 ``work_group_size_hint`` 2726 attribute. 2727 "VecTypeHint" string The name of a scalar or vector 2728 type. 2729 2730 Corresponds to the OpenCL 2731 ``vec_type_hint`` attribute. 2732 2733 "RuntimeHandle" string The external symbol name 2734 associated with a kernel. 2735 OpenCL runtime allocates a 2736 global buffer for the symbol 2737 and saves the kernel's address 2738 to it, which is used for 2739 device side enqueueing. Only 2740 available for device side 2741 enqueued kernels. 2742 =================== ============== ========= ============================== 2743 2744.. 2745 2746 .. table:: AMDHSA Code Object V2 Kernel Argument Metadata Map 2747 :name: amdgpu-amdhsa-code-object-kernel-argument-metadata-map-v2-table 2748 2749 ================= ============== ========= ================================ 2750 String Key Value Type Required? Description 2751 ================= ============== ========= ================================ 2752 "Name" string Kernel argument name. 2753 "TypeName" string Kernel argument type name. 2754 "Size" integer Required Kernel argument size in bytes. 2755 "Align" integer Required Kernel argument alignment in 2756 bytes. Must be a power of two. 2757 "ValueKind" string Required Kernel argument kind that 2758 specifies how to set up the 2759 corresponding argument. 2760 Values include: 2761 2762 "ByValue" 2763 The argument is copied 2764 directly into the kernarg. 2765 2766 "GlobalBuffer" 2767 A global address space pointer 2768 to the buffer data is passed 2769 in the kernarg. 2770 2771 "DynamicSharedPointer" 2772 A group address space pointer 2773 to dynamically allocated LDS 2774 is passed in the kernarg. 2775 2776 "Sampler" 2777 A global address space 2778 pointer to a S# is passed in 2779 the kernarg. 2780 2781 "Image" 2782 A global address space 2783 pointer to a T# is passed in 2784 the kernarg. 2785 2786 "Pipe" 2787 A global address space pointer 2788 to an OpenCL pipe is passed in 2789 the kernarg. 2790 2791 "Queue" 2792 A global address space pointer 2793 to an OpenCL device enqueue 2794 queue is passed in the 2795 kernarg. 2796 2797 "HiddenGlobalOffsetX" 2798 The OpenCL grid dispatch 2799 global offset for the X 2800 dimension is passed in the 2801 kernarg. 2802 2803 "HiddenGlobalOffsetY" 2804 The OpenCL grid dispatch 2805 global offset for the Y 2806 dimension is passed in the 2807 kernarg. 2808 2809 "HiddenGlobalOffsetZ" 2810 The OpenCL grid dispatch 2811 global offset for the Z 2812 dimension is passed in the 2813 kernarg. 2814 2815 "HiddenNone" 2816 An argument that is not used 2817 by the kernel. Space needs to 2818 be left for it, but it does 2819 not need to be set up. 2820 2821 "HiddenPrintfBuffer" 2822 A global address space pointer 2823 to the runtime printf buffer 2824 is passed in kernarg. 2825 2826 "HiddenHostcallBuffer" 2827 A global address space pointer 2828 to the runtime hostcall buffer 2829 is passed in kernarg. 2830 2831 "HiddenDefaultQueue" 2832 A global address space pointer 2833 to the OpenCL device enqueue 2834 queue that should be used by 2835 the kernel by default is 2836 passed in the kernarg. 2837 2838 "HiddenCompletionAction" 2839 A global address space pointer 2840 to help link enqueued kernels into 2841 the ancestor tree for determining 2842 when the parent kernel has finished. 2843 2844 "HiddenMultiGridSyncArg" 2845 A global address space pointer for 2846 multi-grid synchronization is 2847 passed in the kernarg. 2848 2849 "ValueType" string Unused and deprecated. This should no longer 2850 be emitted, but is accepted for compatibility. 2851 2852 2853 "PointeeAlign" integer Alignment in bytes of pointee 2854 type for pointer type kernel 2855 argument. Must be a power 2856 of 2. Only present if 2857 "ValueKind" is 2858 "DynamicSharedPointer". 2859 "AddrSpaceQual" string Kernel argument address space 2860 qualifier. Only present if 2861 "ValueKind" is "GlobalBuffer" or 2862 "DynamicSharedPointer". Values 2863 are: 2864 2865 - "Private" 2866 - "Global" 2867 - "Constant" 2868 - "Local" 2869 - "Generic" 2870 - "Region" 2871 2872 .. TODO:: 2873 2874 Is GlobalBuffer only Global 2875 or Constant? Is 2876 DynamicSharedPointer always 2877 Local? Can HCC allow Generic? 2878 How can Private or Region 2879 ever happen? 2880 2881 "AccQual" string Kernel argument access 2882 qualifier. Only present if 2883 "ValueKind" is "Image" or 2884 "Pipe". Values 2885 are: 2886 2887 - "ReadOnly" 2888 - "WriteOnly" 2889 - "ReadWrite" 2890 2891 .. TODO:: 2892 2893 Does this apply to 2894 GlobalBuffer? 2895 2896 "ActualAccQual" string The actual memory accesses 2897 performed by the kernel on the 2898 kernel argument. Only present if 2899 "ValueKind" is "GlobalBuffer", 2900 "Image", or "Pipe". This may be 2901 more restrictive than indicated 2902 by "AccQual" to reflect what the 2903 kernel actual does. If not 2904 present then the runtime must 2905 assume what is implied by 2906 "AccQual" and "IsConst". Values 2907 are: 2908 2909 - "ReadOnly" 2910 - "WriteOnly" 2911 - "ReadWrite" 2912 2913 "IsConst" boolean Indicates if the kernel argument 2914 is const qualified. Only present 2915 if "ValueKind" is 2916 "GlobalBuffer". 2917 2918 "IsRestrict" boolean Indicates if the kernel argument 2919 is restrict qualified. Only 2920 present if "ValueKind" is 2921 "GlobalBuffer". 2922 2923 "IsVolatile" boolean Indicates if the kernel argument 2924 is volatile qualified. Only 2925 present if "ValueKind" is 2926 "GlobalBuffer". 2927 2928 "IsPipe" boolean Indicates if the kernel argument 2929 is pipe qualified. Only present 2930 if "ValueKind" is "Pipe". 2931 2932 .. TODO:: 2933 2934 Can GlobalBuffer be pipe 2935 qualified? 2936 2937 ================= ============== ========= ================================ 2938 2939.. 2940 2941 .. table:: AMDHSA Code Object V2 Kernel Code Properties Metadata Map 2942 :name: amdgpu-amdhsa-code-object-kernel-code-properties-metadata-map-v2-table 2943 2944 ============================ ============== ========= ===================== 2945 String Key Value Type Required? Description 2946 ============================ ============== ========= ===================== 2947 "KernargSegmentSize" integer Required The size in bytes of 2948 the kernarg segment 2949 that holds the values 2950 of the arguments to 2951 the kernel. 2952 "GroupSegmentFixedSize" integer Required The amount of group 2953 segment memory 2954 required by a 2955 work-group in 2956 bytes. This does not 2957 include any 2958 dynamically allocated 2959 group segment memory 2960 that may be added 2961 when the kernel is 2962 dispatched. 2963 "PrivateSegmentFixedSize" integer Required The amount of fixed 2964 private address space 2965 memory required for a 2966 work-item in 2967 bytes. If the kernel 2968 uses a dynamic call 2969 stack then additional 2970 space must be added 2971 to this value for the 2972 call stack. 2973 "KernargSegmentAlign" integer Required The maximum byte 2974 alignment of 2975 arguments in the 2976 kernarg segment. Must 2977 be a power of 2. 2978 "WavefrontSize" integer Required Wavefront size. Must 2979 be a power of 2. 2980 "NumSGPRs" integer Required Number of scalar 2981 registers used by a 2982 wavefront for 2983 GFX6-GFX10. This 2984 includes the special 2985 SGPRs for VCC, Flat 2986 Scratch (GFX7-GFX10) 2987 and XNACK (for 2988 GFX8-GFX10). It does 2989 not include the 16 2990 SGPR added if a trap 2991 handler is 2992 enabled. It is not 2993 rounded up to the 2994 allocation 2995 granularity. 2996 "NumVGPRs" integer Required Number of vector 2997 registers used by 2998 each work-item for 2999 GFX6-GFX10 3000 "MaxFlatWorkGroupSize" integer Required Maximum flat 3001 work-group size 3002 supported by the 3003 kernel in work-items. 3004 Must be >=1 and 3005 consistent with 3006 ReqdWorkGroupSize if 3007 not 0, 0, 0. 3008 "NumSpilledSGPRs" integer Number of stores from 3009 a scalar register to 3010 a register allocator 3011 created spill 3012 location. 3013 "NumSpilledVGPRs" integer Number of stores from 3014 a vector register to 3015 a register allocator 3016 created spill 3017 location. 3018 ============================ ============== ========= ===================== 3019 3020.. _amdgpu-amdhsa-code-object-metadata-v3: 3021 3022Code Object V3 Metadata 3023+++++++++++++++++++++++ 3024 3025.. warning:: 3026 Code object V3 is not the default code object version emitted by this version 3027 of LLVM. 3028 3029Code object V3 and above metadata is specified by the ``NT_AMDGPU_METADATA`` note 3030record (see :ref:`amdgpu-note-records-v3-onwards`). 3031 3032The metadata is represented as Message Pack formatted binary data (see 3033[MsgPack]_). The top level is a Message Pack map that includes the 3034keys defined in table 3035:ref:`amdgpu-amdhsa-code-object-metadata-map-table-v3` and referenced 3036tables. 3037 3038Additional information can be added to the maps. To avoid conflicts, 3039any key names should be prefixed by "*vendor-name*." where 3040``vendor-name`` can be the name of the vendor and specific vendor 3041tool that generates the information. The prefix is abbreviated to 3042simply "." when it appears within a map that has been added by the 3043same *vendor-name*. 3044 3045 .. table:: AMDHSA Code Object V3 Metadata Map 3046 :name: amdgpu-amdhsa-code-object-metadata-map-table-v3 3047 3048 ================= ============== ========= ======================================= 3049 String Key Value Type Required? Description 3050 ================= ============== ========= ======================================= 3051 "amdhsa.version" sequence of Required - The first integer is the major 3052 2 integers version. Currently 1. 3053 - The second integer is the minor 3054 version. Currently 0. 3055 "amdhsa.printf" sequence of Each string is encoded information 3056 strings about a printf function call. The 3057 encoded information is organized as 3058 fields separated by colon (':'): 3059 3060 ``ID:N:S[0]:S[1]:...:S[N-1]:FormatString`` 3061 3062 where: 3063 3064 ``ID`` 3065 A 32-bit integer as a unique id for 3066 each printf function call 3067 3068 ``N`` 3069 A 32-bit integer equal to the number 3070 of arguments of printf function call 3071 minus 1 3072 3073 ``S[i]`` (where i = 0, 1, ... , N-1) 3074 32-bit integers for the size in bytes 3075 of the i-th FormatString argument of 3076 the printf function call 3077 3078 FormatString 3079 The format string passed to the 3080 printf function call. 3081 "amdhsa.kernels" sequence of Required Sequence of the maps for each 3082 map kernel in the code object. See 3083 :ref:`amdgpu-amdhsa-code-object-kernel-metadata-map-table-v3` 3084 for the definition of the keys included 3085 in that map. 3086 ================= ============== ========= ======================================= 3087 3088.. 3089 3090 .. table:: AMDHSA Code Object V3 Kernel Metadata Map 3091 :name: amdgpu-amdhsa-code-object-kernel-metadata-map-table-v3 3092 3093 =================================== ============== ========= ================================ 3094 String Key Value Type Required? Description 3095 =================================== ============== ========= ================================ 3096 ".name" string Required Source name of the kernel. 3097 ".symbol" string Required Name of the kernel 3098 descriptor ELF symbol. 3099 ".language" string Source language of the kernel. 3100 Values include: 3101 3102 - "OpenCL C" 3103 - "OpenCL C++" 3104 - "HCC" 3105 - "HIP" 3106 - "OpenMP" 3107 - "Assembler" 3108 3109 ".language_version" sequence of - The first integer is the major 3110 2 integers version. 3111 - The second integer is the 3112 minor version. 3113 ".args" sequence of Sequence of maps of the 3114 map kernel arguments. See 3115 :ref:`amdgpu-amdhsa-code-object-kernel-argument-metadata-map-table-v3` 3116 for the definition of the keys 3117 included in that map. 3118 ".reqd_workgroup_size" sequence of If not 0, 0, 0 then all values 3119 3 integers must be >=1 and the dispatch 3120 work-group size X, Y, Z must 3121 correspond to the specified 3122 values. Defaults to 0, 0, 0. 3123 3124 Corresponds to the OpenCL 3125 ``reqd_work_group_size`` 3126 attribute. 3127 ".workgroup_size_hint" sequence of The dispatch work-group size 3128 3 integers X, Y, Z is likely to be the 3129 specified values. 3130 3131 Corresponds to the OpenCL 3132 ``work_group_size_hint`` 3133 attribute. 3134 ".vec_type_hint" string The name of a scalar or vector 3135 type. 3136 3137 Corresponds to the OpenCL 3138 ``vec_type_hint`` attribute. 3139 3140 ".device_enqueue_symbol" string The external symbol name 3141 associated with a kernel. 3142 OpenCL runtime allocates a 3143 global buffer for the symbol 3144 and saves the kernel's address 3145 to it, which is used for 3146 device side enqueueing. Only 3147 available for device side 3148 enqueued kernels. 3149 ".kernarg_segment_size" integer Required The size in bytes of 3150 the kernarg segment 3151 that holds the values 3152 of the arguments to 3153 the kernel. 3154 ".group_segment_fixed_size" integer Required The amount of group 3155 segment memory 3156 required by a 3157 work-group in 3158 bytes. This does not 3159 include any 3160 dynamically allocated 3161 group segment memory 3162 that may be added 3163 when the kernel is 3164 dispatched. 3165 ".private_segment_fixed_size" integer Required The amount of fixed 3166 private address space 3167 memory required for a 3168 work-item in 3169 bytes. If the kernel 3170 uses a dynamic call 3171 stack then additional 3172 space must be added 3173 to this value for the 3174 call stack. 3175 ".kernarg_segment_align" integer Required The maximum byte 3176 alignment of 3177 arguments in the 3178 kernarg segment. Must 3179 be a power of 2. 3180 ".wavefront_size" integer Required Wavefront size. Must 3181 be a power of 2. 3182 ".sgpr_count" integer Required Number of scalar 3183 registers required by a 3184 wavefront for 3185 GFX6-GFX9. A register 3186 is required if it is 3187 used explicitly, or 3188 if a higher numbered 3189 register is used 3190 explicitly. This 3191 includes the special 3192 SGPRs for VCC, Flat 3193 Scratch (GFX7-GFX9) 3194 and XNACK (for 3195 GFX8-GFX9). It does 3196 not include the 16 3197 SGPR added if a trap 3198 handler is 3199 enabled. It is not 3200 rounded up to the 3201 allocation 3202 granularity. 3203 ".vgpr_count" integer Required Number of vector 3204 registers required by 3205 each work-item for 3206 GFX6-GFX9. A register 3207 is required if it is 3208 used explicitly, or 3209 if a higher numbered 3210 register is used 3211 explicitly. 3212 ".agpr_count" integer Required Number of accumulator 3213 registers required by 3214 each work-item for 3215 GFX90A, GFX908. 3216 ".max_flat_workgroup_size" integer Required Maximum flat 3217 work-group size 3218 supported by the 3219 kernel in work-items. 3220 Must be >=1 and 3221 consistent with 3222 ReqdWorkGroupSize if 3223 not 0, 0, 0. 3224 ".sgpr_spill_count" integer Number of stores from 3225 a scalar register to 3226 a register allocator 3227 created spill 3228 location. 3229 ".vgpr_spill_count" integer Number of stores from 3230 a vector register to 3231 a register allocator 3232 created spill 3233 location. 3234 ".kind" string The kind of the kernel 3235 with the following 3236 values: 3237 3238 "normal" 3239 Regular kernels. 3240 3241 "init" 3242 These kernels must be 3243 invoked after loading 3244 the containing code 3245 object and must 3246 complete before any 3247 normal and fini 3248 kernels in the same 3249 code object are 3250 invoked. 3251 3252 "fini" 3253 These kernels must be 3254 invoked before 3255 unloading the 3256 containing code object 3257 and after all init and 3258 normal kernels in the 3259 same code object have 3260 been invoked and 3261 completed. 3262 3263 If omitted, "normal" is 3264 assumed. 3265 =================================== ============== ========= ================================ 3266 3267.. 3268 3269 .. table:: AMDHSA Code Object V3 Kernel Argument Metadata Map 3270 :name: amdgpu-amdhsa-code-object-kernel-argument-metadata-map-table-v3 3271 3272 ====================== ============== ========= ================================ 3273 String Key Value Type Required? Description 3274 ====================== ============== ========= ================================ 3275 ".name" string Kernel argument name. 3276 ".type_name" string Kernel argument type name. 3277 ".size" integer Required Kernel argument size in bytes. 3278 ".offset" integer Required Kernel argument offset in 3279 bytes. The offset must be a 3280 multiple of the alignment 3281 required by the argument. 3282 ".value_kind" string Required Kernel argument kind that 3283 specifies how to set up the 3284 corresponding argument. 3285 Values include: 3286 3287 "by_value" 3288 The argument is copied 3289 directly into the kernarg. 3290 3291 "global_buffer" 3292 A global address space pointer 3293 to the buffer data is passed 3294 in the kernarg. 3295 3296 "dynamic_shared_pointer" 3297 A group address space pointer 3298 to dynamically allocated LDS 3299 is passed in the kernarg. 3300 3301 "sampler" 3302 A global address space 3303 pointer to a S# is passed in 3304 the kernarg. 3305 3306 "image" 3307 A global address space 3308 pointer to a T# is passed in 3309 the kernarg. 3310 3311 "pipe" 3312 A global address space pointer 3313 to an OpenCL pipe is passed in 3314 the kernarg. 3315 3316 "queue" 3317 A global address space pointer 3318 to an OpenCL device enqueue 3319 queue is passed in the 3320 kernarg. 3321 3322 "hidden_global_offset_x" 3323 The OpenCL grid dispatch 3324 global offset for the X 3325 dimension is passed in the 3326 kernarg. 3327 3328 "hidden_global_offset_y" 3329 The OpenCL grid dispatch 3330 global offset for the Y 3331 dimension is passed in the 3332 kernarg. 3333 3334 "hidden_global_offset_z" 3335 The OpenCL grid dispatch 3336 global offset for the Z 3337 dimension is passed in the 3338 kernarg. 3339 3340 "hidden_none" 3341 An argument that is not used 3342 by the kernel. Space needs to 3343 be left for it, but it does 3344 not need to be set up. 3345 3346 "hidden_printf_buffer" 3347 A global address space pointer 3348 to the runtime printf buffer 3349 is passed in kernarg. 3350 3351 "hidden_hostcall_buffer" 3352 A global address space pointer 3353 to the runtime hostcall buffer 3354 is passed in kernarg. 3355 3356 "hidden_default_queue" 3357 A global address space pointer 3358 to the OpenCL device enqueue 3359 queue that should be used by 3360 the kernel by default is 3361 passed in the kernarg. 3362 3363 "hidden_completion_action" 3364 A global address space pointer 3365 to help link enqueued kernels into 3366 the ancestor tree for determining 3367 when the parent kernel has finished. 3368 3369 "hidden_multigrid_sync_arg" 3370 A global address space pointer for 3371 multi-grid synchronization is 3372 passed in the kernarg. 3373 3374 ".value_type" string Unused and deprecated. This should no longer 3375 be emitted, but is accepted for compatibility. 3376 3377 ".pointee_align" integer Alignment in bytes of pointee 3378 type for pointer type kernel 3379 argument. Must be a power 3380 of 2. Only present if 3381 ".value_kind" is 3382 "dynamic_shared_pointer". 3383 ".address_space" string Kernel argument address space 3384 qualifier. Only present if 3385 ".value_kind" is "global_buffer" or 3386 "dynamic_shared_pointer". Values 3387 are: 3388 3389 - "private" 3390 - "global" 3391 - "constant" 3392 - "local" 3393 - "generic" 3394 - "region" 3395 3396 .. TODO:: 3397 3398 Is "global_buffer" only "global" 3399 or "constant"? Is 3400 "dynamic_shared_pointer" always 3401 "local"? Can HCC allow "generic"? 3402 How can "private" or "region" 3403 ever happen? 3404 3405 ".access" string Kernel argument access 3406 qualifier. Only present if 3407 ".value_kind" is "image" or 3408 "pipe". Values 3409 are: 3410 3411 - "read_only" 3412 - "write_only" 3413 - "read_write" 3414 3415 .. TODO:: 3416 3417 Does this apply to 3418 "global_buffer"? 3419 3420 ".actual_access" string The actual memory accesses 3421 performed by the kernel on the 3422 kernel argument. Only present if 3423 ".value_kind" is "global_buffer", 3424 "image", or "pipe". This may be 3425 more restrictive than indicated 3426 by ".access" to reflect what the 3427 kernel actual does. If not 3428 present then the runtime must 3429 assume what is implied by 3430 ".access" and ".is_const" . Values 3431 are: 3432 3433 - "read_only" 3434 - "write_only" 3435 - "read_write" 3436 3437 ".is_const" boolean Indicates if the kernel argument 3438 is const qualified. Only present 3439 if ".value_kind" is 3440 "global_buffer". 3441 3442 ".is_restrict" boolean Indicates if the kernel argument 3443 is restrict qualified. Only 3444 present if ".value_kind" is 3445 "global_buffer". 3446 3447 ".is_volatile" boolean Indicates if the kernel argument 3448 is volatile qualified. Only 3449 present if ".value_kind" is 3450 "global_buffer". 3451 3452 ".is_pipe" boolean Indicates if the kernel argument 3453 is pipe qualified. Only present 3454 if ".value_kind" is "pipe". 3455 3456 .. TODO:: 3457 3458 Can "global_buffer" be pipe 3459 qualified? 3460 3461 ====================== ============== ========= ================================ 3462 3463.. _amdgpu-amdhsa-code-object-metadata-v4: 3464 3465Code Object V4 Metadata 3466+++++++++++++++++++++++ 3467 3468Code object V4 metadata is the same as 3469:ref:`amdgpu-amdhsa-code-object-metadata-v3` with the changes and additions 3470defined in table :ref:`amdgpu-amdhsa-code-object-metadata-map-table-v4`. 3471 3472 .. table:: AMDHSA Code Object V4 Metadata Map Changes 3473 :name: amdgpu-amdhsa-code-object-metadata-map-table-v4 3474 3475 ================= ============== ========= ======================================= 3476 String Key Value Type Required? Description 3477 ================= ============== ========= ======================================= 3478 "amdhsa.version" sequence of Required - The first integer is the major 3479 2 integers version. Currently 1. 3480 - The second integer is the minor 3481 version. Currently 1. 3482 "amdhsa.target" string Required The target name of the code using the syntax: 3483 3484 .. code:: 3485 3486 <target-triple> [ "-" <target-id> ] 3487 3488 A canonical target ID must be 3489 used. See :ref:`amdgpu-target-triples` 3490 and :ref:`amdgpu-target-id`. 3491 ================= ============== ========= ======================================= 3492 3493.. _amdgpu-amdhsa-code-object-metadata-v5: 3494 3495Code Object V5 Metadata 3496+++++++++++++++++++++++ 3497 3498.. warning:: 3499 Code object V5 is not the default code object version emitted by this version 3500 of LLVM. 3501 3502 3503Code object V5 metadata is the same as 3504:ref:`amdgpu-amdhsa-code-object-metadata-v4` with the changes defined in table 3505:ref:`amdgpu-amdhsa-code-object-metadata-map-table-v5` and table 3506:ref:`amdgpu-amdhsa-code-object-kernel-argument-metadata-map-table-v5`. 3507 3508 .. table:: AMDHSA Code Object V5 Metadata Map Changes 3509 :name: amdgpu-amdhsa-code-object-metadata-map-table-v5 3510 3511 ================= ============== ========= ======================================= 3512 String Key Value Type Required? Description 3513 ================= ============== ========= ======================================= 3514 "amdhsa.version" sequence of Required - The first integer is the major 3515 2 integers version. Currently 1. 3516 - The second integer is the minor 3517 version. Currently 2. 3518 ================= ============== ========= ======================================= 3519 3520.. 3521 3522 .. table:: AMDHSA Code Object V5 Kernel Argument Metadata Map Additions and Changes 3523 :name: amdgpu-amdhsa-code-object-kernel-argument-metadata-map-table-v5 3524 3525 ====================== ============== ========= ================================ 3526 String Key Value Type Required? Description 3527 ====================== ============== ========= ================================ 3528 ".value_kind" string Required Kernel argument kind that 3529 specifies how to set up the 3530 corresponding argument. 3531 Values include: 3532 the same as code object V3 metadata 3533 (see :ref:`amdgpu-amdhsa-code-object-kernel-argument-metadata-map-table-v3`) 3534 with the following additions: 3535 3536 "hidden_block_count_x" 3537 The grid dispatch work-group count for the X dimension 3538 is passed in the kernarg. Some languages, such as OpenCL, 3539 support a last work-group in each dimension being partial. 3540 This count only includes the non-partial work-group count. 3541 This is not the same as the value in the AQL dispatch packet, 3542 which has the grid size in work-items. 3543 3544 "hidden_block_count_y" 3545 The grid dispatch work-group count for the Y dimension 3546 is passed in the kernarg. Some languages, such as OpenCL, 3547 support a last work-group in each dimension being partial. 3548 This count only includes the non-partial work-group count. 3549 This is not the same as the value in the AQL dispatch packet, 3550 which has the grid size in work-items. If the grid dimensionality 3551 is 1, then must be 1. 3552 3553 "hidden_block_count_z" 3554 The grid dispatch work-group count for the Z dimension 3555 is passed in the kernarg. Some languages, such as OpenCL, 3556 support a last work-group in each dimension being partial. 3557 This count only includes the non-partial work-group count. 3558 This is not the same as the value in the AQL dispatch packet, 3559 which has the grid size in work-items. If the grid dimensionality 3560 is 1 or 2, then must be 1. 3561 3562 "hidden_group_size_x" 3563 The grid dispatch work-group size for the X dimension is 3564 passed in the kernarg. This size only applies to the 3565 non-partial work-groups. This is the same value as the AQL 3566 dispatch packet work-group size. 3567 3568 "hidden_group_size_y" 3569 The grid dispatch work-group size for the Y dimension is 3570 passed in the kernarg. This size only applies to the 3571 non-partial work-groups. This is the same value as the AQL 3572 dispatch packet work-group size. If the grid dimensionality 3573 is 1, then must be 1. 3574 3575 "hidden_group_size_z" 3576 The grid dispatch work-group size for the Z dimension is 3577 passed in the kernarg. This size only applies to the 3578 non-partial work-groups. This is the same value as the AQL 3579 dispatch packet work-group size. If the grid dimensionality 3580 is 1 or 2, then must be 1. 3581 3582 "hidden_remainder_x" 3583 The grid dispatch work group size of the the partial work group 3584 of the X dimension, if it exists. Must be zero if a partial 3585 work group does not exist in the X dimension. 3586 3587 "hidden_remainder_y" 3588 The grid dispatch work group size of the the partial work group 3589 of the Y dimension, if it exists. Must be zero if a partial 3590 work group does not exist in the Y dimension. 3591 3592 "hidden_remainder_z" 3593 The grid dispatch work group size of the the partial work group 3594 of the Z dimension, if it exists. Must be zero if a partial 3595 work group does not exist in the Z dimension. 3596 3597 "hidden_grid_dims" 3598 The grid dispatch dimensionality. This is the same value 3599 as the AQL dispatch packet dimensionality. Must be a value 3600 between 1 and 3. 3601 3602 "hidden_heap_v1" 3603 A global address space pointer to an initialized memory 3604 buffer that conforms to the requirements of the malloc/free 3605 device library V1 version implementation. 3606 3607 "hidden_private_base" 3608 The high 32 bits of the flat addressing private aperture base. 3609 Only used by GFX8 to allow conversion between private segment 3610 and flat addresses. See :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`. 3611 3612 "hidden_shared_base" 3613 The high 32 bits of the flat addressing shared aperture base. 3614 Only used by GFX8 to allow conversion between shared segment 3615 and flat addresses. See :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`. 3616 3617 "hidden_queue_ptr" 3618 A global memory address space pointer to the ROCm runtime 3619 ``struct amd_queue_t`` structure for the HSA queue of the 3620 associated dispatch AQL packet. It is only required for pre-GFX9 3621 devices for the trap handler ABI (see :ref:`amdgpu-amdhsa-trap-handler-abi`). 3622 3623 ====================== ============== ========= ================================ 3624 3625.. 3626 3627Kernel Dispatch 3628~~~~~~~~~~~~~~~ 3629 3630The HSA architected queuing language (AQL) defines a user space memory interface 3631that can be used to control the dispatch of kernels, in an agent independent 3632way. An agent can have zero or more AQL queues created for it using an HSA 3633compatible runtime (see :ref:`amdgpu-os`), in which AQL packets (all of which 3634are 64 bytes) can be placed. See the *HSA Platform System Architecture 3635Specification* [HSA]_ for the AQL queue mechanics and packet layouts. 3636 3637The packet processor of a kernel agent is responsible for detecting and 3638dispatching HSA kernels from the AQL queues associated with it. For AMD GPUs the 3639packet processor is implemented by the hardware command processor (CP), 3640asynchronous dispatch controller (ADC) and shader processor input controller 3641(SPI). 3642 3643An HSA compatible runtime can be used to allocate an AQL queue object. It uses 3644the kernel mode driver to initialize and register the AQL queue with CP. 3645 3646To dispatch a kernel the following actions are performed. This can occur in the 3647CPU host program, or from an HSA kernel executing on a GPU. 3648 36491. A pointer to an AQL queue for the kernel agent on which the kernel is to be 3650 executed is obtained. 36512. A pointer to the kernel descriptor (see 3652 :ref:`amdgpu-amdhsa-kernel-descriptor`) of the kernel to execute is obtained. 3653 It must be for a kernel that is contained in a code object that was loaded 3654 by an HSA compatible runtime on the kernel agent with which the AQL queue is 3655 associated. 36563. Space is allocated for the kernel arguments using the HSA compatible runtime 3657 allocator for a memory region with the kernarg property for the kernel agent 3658 that will execute the kernel. It must be at least 16-byte aligned. 36594. Kernel argument values are assigned to the kernel argument memory 3660 allocation. The layout is defined in the *HSA Programmer's Language 3661 Reference* [HSA]_. For AMDGPU the kernel execution directly accesses the 3662 kernel argument memory in the same way constant memory is accessed. (Note 3663 that the HSA specification allows an implementation to copy the kernel 3664 argument contents to another location that is accessed by the kernel.) 36655. An AQL kernel dispatch packet is created on the AQL queue. The HSA compatible 3666 runtime api uses 64-bit atomic operations to reserve space in the AQL queue 3667 for the packet. The packet must be set up, and the final write must use an 3668 atomic store release to set the packet kind to ensure the packet contents are 3669 visible to the kernel agent. AQL defines a doorbell signal mechanism to 3670 notify the kernel agent that the AQL queue has been updated. These rules, and 3671 the layout of the AQL queue and kernel dispatch packet is defined in the *HSA 3672 System Architecture Specification* [HSA]_. 36736. A kernel dispatch packet includes information about the actual dispatch, 3674 such as grid and work-group size, together with information from the code 3675 object about the kernel, such as segment sizes. The HSA compatible runtime 3676 queries on the kernel symbol can be used to obtain the code object values 3677 which are recorded in the :ref:`amdgpu-amdhsa-code-object-metadata`. 36787. CP executes micro-code and is responsible for detecting and setting up the 3679 GPU to execute the wavefronts of a kernel dispatch. 36808. CP ensures that when the a wavefront starts executing the kernel machine 3681 code, the scalar general purpose registers (SGPR) and vector general purpose 3682 registers (VGPR) are set up as required by the machine code. The required 3683 setup is defined in the :ref:`amdgpu-amdhsa-kernel-descriptor`. The initial 3684 register state is defined in 3685 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`. 36869. The prolog of the kernel machine code (see 3687 :ref:`amdgpu-amdhsa-kernel-prolog`) sets up the machine state as necessary 3688 before continuing executing the machine code that corresponds to the kernel. 368910. When the kernel dispatch has completed execution, CP signals the completion 3690 signal specified in the kernel dispatch packet if not 0. 3691 3692.. _amdgpu-amdhsa-memory-spaces: 3693 3694Memory Spaces 3695~~~~~~~~~~~~~ 3696 3697The memory space properties are: 3698 3699 .. table:: AMDHSA Memory Spaces 3700 :name: amdgpu-amdhsa-memory-spaces-table 3701 3702 ================= =========== ======== ======= ================== 3703 Memory Space Name HSA Segment Hardware Address NULL Value 3704 Name Name Size 3705 ================= =========== ======== ======= ================== 3706 Private private scratch 32 0x00000000 3707 Local group LDS 32 0xFFFFFFFF 3708 Global global global 64 0x0000000000000000 3709 Constant constant *same as 64 0x0000000000000000 3710 global* 3711 Generic flat flat 64 0x0000000000000000 3712 Region N/A GDS 32 *not implemented 3713 for AMDHSA* 3714 ================= =========== ======== ======= ================== 3715 3716The global and constant memory spaces both use global virtual addresses, which 3717are the same virtual address space used by the CPU. However, some virtual 3718addresses may only be accessible to the CPU, some only accessible by the GPU, 3719and some by both. 3720 3721Using the constant memory space indicates that the data will not change during 3722the execution of the kernel. This allows scalar read instructions to be 3723used. The vector and scalar L1 caches are invalidated of volatile data before 3724each kernel dispatch execution to allow constant memory to change values between 3725kernel dispatches. 3726 3727The local memory space uses the hardware Local Data Store (LDS) which is 3728automatically allocated when the hardware creates work-groups of wavefronts, and 3729freed when all the wavefronts of a work-group have terminated. The data store 3730(DS) instructions can be used to access it. 3731 3732The private memory space uses the hardware scratch memory support. If the kernel 3733uses scratch, then the hardware allocates memory that is accessed using 3734wavefront lane dword (4 byte) interleaving. The mapping used from private 3735address to physical address is: 3736 3737 ``wavefront-scratch-base + 3738 (private-address * wavefront-size * 4) + 3739 (wavefront-lane-id * 4)`` 3740 3741There are different ways that the wavefront scratch base address is determined 3742by a wavefront (see :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). This 3743memory can be accessed in an interleaved manner using buffer instruction with 3744the scratch buffer descriptor and per wavefront scratch offset, by the scratch 3745instructions, or by flat instructions. If each lane of a wavefront accesses the 3746same private address, the interleaving results in adjacent dwords being accessed 3747and hence requires fewer cache lines to be fetched. Multi-dword access is not 3748supported except by flat and scratch instructions in GFX9-GFX10. 3749 3750The generic address space uses the hardware flat address support available in 3751GFX7-GFX10. This uses two fixed ranges of virtual addresses (the private and 3752local apertures), that are outside the range of addressible global memory, to 3753map from a flat address to a private or local address. 3754 3755FLAT instructions can take a flat address and access global, private (scratch) 3756and group (LDS) memory depending on if the address is within one of the 3757aperture ranges. Flat access to scratch requires hardware aperture setup and 3758setup in the kernel prologue (see 3759:ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`). Flat access to LDS requires 3760hardware aperture setup and M0 (GFX7-GFX8) register setup (see 3761:ref:`amdgpu-amdhsa-kernel-prolog-m0`). 3762 3763To convert between a segment address and a flat address the base address of the 3764apertures address can be used. For GFX7-GFX8 these are available in the 3765:ref:`amdgpu-amdhsa-hsa-aql-queue` the address of which can be obtained with 3766Queue Ptr SGPR (see :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). For 3767GFX9-GFX10 the aperture base addresses are directly available as inline constant 3768registers ``SRC_SHARED_BASE/LIMIT`` and ``SRC_PRIVATE_BASE/LIMIT``. In 64 bit 3769address mode the aperture sizes are 2^32 bytes and the base is aligned to 2^32 3770which makes it easier to convert from flat to segment or segment to flat. 3771 3772Image and Samplers 3773~~~~~~~~~~~~~~~~~~ 3774 3775Image and sample handles created by an HSA compatible runtime (see 3776:ref:`amdgpu-os`) are 64-bit addresses of a hardware 32-byte V# and 48 byte S# 3777object respectively. In order to support the HSA ``query_sampler`` operations 3778two extra dwords are used to store the HSA BRIG enumeration values for the 3779queries that are not trivially deducible from the S# representation. 3780 3781HSA Signals 3782~~~~~~~~~~~ 3783 3784HSA signal handles created by an HSA compatible runtime (see :ref:`amdgpu-os`) 3785are 64-bit addresses of a structure allocated in memory accessible from both the 3786CPU and GPU. The structure is defined by the runtime and subject to change 3787between releases. For example, see [AMD-ROCm-github]_. 3788 3789.. _amdgpu-amdhsa-hsa-aql-queue: 3790 3791HSA AQL Queue 3792~~~~~~~~~~~~~ 3793 3794The HSA AQL queue structure is defined by an HSA compatible runtime (see 3795:ref:`amdgpu-os`) and subject to change between releases. For example, see 3796[AMD-ROCm-github]_. For some processors it contains fields needed to implement 3797certain language features such as the flat address aperture bases. It also 3798contains fields used by CP such as managing the allocation of scratch memory. 3799 3800.. _amdgpu-amdhsa-kernel-descriptor: 3801 3802Kernel Descriptor 3803~~~~~~~~~~~~~~~~~ 3804 3805A kernel descriptor consists of the information needed by CP to initiate the 3806execution of a kernel, including the entry point address of the machine code 3807that implements the kernel. 3808 3809Code Object V3 Kernel Descriptor 3810++++++++++++++++++++++++++++++++ 3811 3812CP microcode requires the Kernel descriptor to be allocated on 64-byte 3813alignment. 3814 3815The fields used by CP for code objects before V3 also match those specified in 3816:ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 3817 3818 .. table:: Code Object V3 Kernel Descriptor 3819 :name: amdgpu-amdhsa-kernel-descriptor-v3-table 3820 3821 ======= ======= =============================== ============================ 3822 Bits Size Field Name Description 3823 ======= ======= =============================== ============================ 3824 31:0 4 bytes GROUP_SEGMENT_FIXED_SIZE The amount of fixed local 3825 address space memory 3826 required for a work-group 3827 in bytes. This does not 3828 include any dynamically 3829 allocated local address 3830 space memory that may be 3831 added when the kernel is 3832 dispatched. 3833 63:32 4 bytes PRIVATE_SEGMENT_FIXED_SIZE The amount of fixed 3834 private address space 3835 memory required for a 3836 work-item in bytes. 3837 Additional space may need to 3838 be added to this value if 3839 the call stack has 3840 non-inlined function calls. 3841 95:64 4 bytes KERNARG_SIZE The size of the kernarg 3842 memory pointed to by the 3843 AQL dispatch packet. The 3844 kernarg memory is used to 3845 pass arguments to the 3846 kernel. 3847 3848 * If the kernarg pointer in 3849 the dispatch packet is NULL 3850 then there are no kernel 3851 arguments. 3852 * If the kernarg pointer in 3853 the dispatch packet is 3854 not NULL and this value 3855 is 0 then the kernarg 3856 memory size is 3857 unspecified. 3858 * If the kernarg pointer in 3859 the dispatch packet is 3860 not NULL and this value 3861 is not 0 then the value 3862 specifies the kernarg 3863 memory size in bytes. It 3864 is recommended to provide 3865 a value as it may be used 3866 by CP to optimize making 3867 the kernarg memory 3868 visible to the kernel 3869 code. 3870 3871 127:96 4 bytes Reserved, must be 0. 3872 191:128 8 bytes KERNEL_CODE_ENTRY_BYTE_OFFSET Byte offset (possibly 3873 negative) from base 3874 address of kernel 3875 descriptor to kernel's 3876 entry point instruction 3877 which must be 256 byte 3878 aligned. 3879 351:272 20 Reserved, must be 0. 3880 bytes 3881 383:352 4 bytes COMPUTE_PGM_RSRC3 GFX6-GFX9 3882 Reserved, must be 0. 3883 GFX90A, GFX940 3884 Compute Shader (CS) 3885 program settings used by 3886 CP to set up 3887 ``COMPUTE_PGM_RSRC3`` 3888 configuration 3889 register. See 3890 :ref:`amdgpu-amdhsa-compute_pgm_rsrc3-gfx90a-table`. 3891 GFX10 3892 Compute Shader (CS) 3893 program settings used by 3894 CP to set up 3895 ``COMPUTE_PGM_RSRC3`` 3896 configuration 3897 register. See 3898 :ref:`amdgpu-amdhsa-compute_pgm_rsrc3-gfx10-table`. 3899 415:384 4 bytes COMPUTE_PGM_RSRC1 Compute Shader (CS) 3900 program settings used by 3901 CP to set up 3902 ``COMPUTE_PGM_RSRC1`` 3903 configuration 3904 register. See 3905 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 3906 447:416 4 bytes COMPUTE_PGM_RSRC2 Compute Shader (CS) 3907 program settings used by 3908 CP to set up 3909 ``COMPUTE_PGM_RSRC2`` 3910 configuration 3911 register. See 3912 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 3913 458:448 7 bits *See separate bits below.* Enable the setup of the 3914 SGPR user data registers 3915 (see 3916 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 3917 3918 The total number of SGPR 3919 user data registers 3920 requested must not exceed 3921 16 and match value in 3922 ``compute_pgm_rsrc2.user_sgpr.user_sgpr_count``. 3923 Any requests beyond 16 3924 will be ignored. 3925 >448 1 bit ENABLE_SGPR_PRIVATE_SEGMENT If the *Target Properties* 3926 _BUFFER column of 3927 :ref:`amdgpu-processor-table` 3928 specifies *Architected flat 3929 scratch* then not supported 3930 and must be 0, 3931 >449 1 bit ENABLE_SGPR_DISPATCH_PTR 3932 >450 1 bit ENABLE_SGPR_QUEUE_PTR 3933 >451 1 bit ENABLE_SGPR_KERNARG_SEGMENT_PTR 3934 >452 1 bit ENABLE_SGPR_DISPATCH_ID 3935 >453 1 bit ENABLE_SGPR_FLAT_SCRATCH_INIT If the *Target Properties* 3936 column of 3937 :ref:`amdgpu-processor-table` 3938 specifies *Architected flat 3939 scratch* then not supported 3940 and must be 0, 3941 >454 1 bit ENABLE_SGPR_PRIVATE_SEGMENT 3942 _SIZE 3943 457:455 3 bits Reserved, must be 0. 3944 458 1 bit ENABLE_WAVEFRONT_SIZE32 GFX6-GFX9 3945 Reserved, must be 0. 3946 GFX10 3947 - If 0 execute in 3948 wavefront size 64 mode. 3949 - If 1 execute in 3950 native wavefront size 3951 32 mode. 3952 463:459 1 bit Reserved, must be 0. 3953 464 1 bit RESERVED_464 Deprecated, must be 0. 3954 467:465 3 bits Reserved, must be 0. 3955 468 1 bit RESERVED_468 Deprecated, must be 0. 3956 469:471 3 bits Reserved, must be 0. 3957 511:472 5 bytes Reserved, must be 0. 3958 512 **Total size 64 bytes.** 3959 ======= ==================================================================== 3960 3961.. 3962 3963 .. table:: compute_pgm_rsrc1 for GFX6-GFX10 3964 :name: amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table 3965 3966 ======= ======= =============================== =========================================================================== 3967 Bits Size Field Name Description 3968 ======= ======= =============================== =========================================================================== 3969 5:0 6 bits GRANULATED_WORKITEM_VGPR_COUNT Number of vector register 3970 blocks used by each work-item; 3971 granularity is device 3972 specific: 3973 3974 GFX6-GFX9 3975 - vgprs_used 0..256 3976 - max(0, ceil(vgprs_used / 4) - 1) 3977 GFX90A, GFX940 3978 - vgprs_used 0..512 3979 - vgprs_used = align(arch_vgprs, 4) 3980 + acc_vgprs 3981 - max(0, ceil(vgprs_used / 8) - 1) 3982 GFX10 (wavefront size 64) 3983 - max_vgpr 1..256 3984 - max(0, ceil(vgprs_used / 4) - 1) 3985 GFX10 (wavefront size 32) 3986 - max_vgpr 1..256 3987 - max(0, ceil(vgprs_used / 8) - 1) 3988 3989 Where vgprs_used is defined 3990 as the highest VGPR number 3991 explicitly referenced plus 3992 one. 3993 3994 Used by CP to set up 3995 ``COMPUTE_PGM_RSRC1.VGPRS``. 3996 3997 The 3998 :ref:`amdgpu-assembler` 3999 calculates this 4000 automatically for the 4001 selected processor from 4002 values provided to the 4003 `.amdhsa_kernel` directive 4004 by the 4005 `.amdhsa_next_free_vgpr` 4006 nested directive (see 4007 :ref:`amdhsa-kernel-directives-table`). 4008 9:6 4 bits GRANULATED_WAVEFRONT_SGPR_COUNT Number of scalar register 4009 blocks used by a wavefront; 4010 granularity is device 4011 specific: 4012 4013 GFX6-GFX8 4014 - sgprs_used 0..112 4015 - max(0, ceil(sgprs_used / 8) - 1) 4016 GFX9 4017 - sgprs_used 0..112 4018 - 2 * max(0, ceil(sgprs_used / 16) - 1) 4019 GFX10 4020 Reserved, must be 0. 4021 (128 SGPRs always 4022 allocated.) 4023 4024 Where sgprs_used is 4025 defined as the highest 4026 SGPR number explicitly 4027 referenced plus one, plus 4028 a target specific number 4029 of additional special 4030 SGPRs for VCC, 4031 FLAT_SCRATCH (GFX7+) and 4032 XNACK_MASK (GFX8+), and 4033 any additional 4034 target specific 4035 limitations. It does not 4036 include the 16 SGPRs added 4037 if a trap handler is 4038 enabled. 4039 4040 The target specific 4041 limitations and special 4042 SGPR layout are defined in 4043 the hardware 4044 documentation, which can 4045 be found in the 4046 :ref:`amdgpu-processors` 4047 table. 4048 4049 Used by CP to set up 4050 ``COMPUTE_PGM_RSRC1.SGPRS``. 4051 4052 The 4053 :ref:`amdgpu-assembler` 4054 calculates this 4055 automatically for the 4056 selected processor from 4057 values provided to the 4058 `.amdhsa_kernel` directive 4059 by the 4060 `.amdhsa_next_free_sgpr` 4061 and `.amdhsa_reserve_*` 4062 nested directives (see 4063 :ref:`amdhsa-kernel-directives-table`). 4064 11:10 2 bits PRIORITY Must be 0. 4065 4066 Start executing wavefront 4067 at the specified priority. 4068 4069 CP is responsible for 4070 filling in 4071 ``COMPUTE_PGM_RSRC1.PRIORITY``. 4072 13:12 2 bits FLOAT_ROUND_MODE_32 Wavefront starts execution 4073 with specified rounding 4074 mode for single (32 4075 bit) floating point 4076 precision floating point 4077 operations. 4078 4079 Floating point rounding 4080 mode values are defined in 4081 :ref:`amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table`. 4082 4083 Used by CP to set up 4084 ``COMPUTE_PGM_RSRC1.FLOAT_MODE``. 4085 15:14 2 bits FLOAT_ROUND_MODE_16_64 Wavefront starts execution 4086 with specified rounding 4087 denorm mode for half/double (16 4088 and 64-bit) floating point 4089 precision floating point 4090 operations. 4091 4092 Floating point rounding 4093 mode values are defined in 4094 :ref:`amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table`. 4095 4096 Used by CP to set up 4097 ``COMPUTE_PGM_RSRC1.FLOAT_MODE``. 4098 17:16 2 bits FLOAT_DENORM_MODE_32 Wavefront starts execution 4099 with specified denorm mode 4100 for single (32 4101 bit) floating point 4102 precision floating point 4103 operations. 4104 4105 Floating point denorm mode 4106 values are defined in 4107 :ref:`amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table`. 4108 4109 Used by CP to set up 4110 ``COMPUTE_PGM_RSRC1.FLOAT_MODE``. 4111 19:18 2 bits FLOAT_DENORM_MODE_16_64 Wavefront starts execution 4112 with specified denorm mode 4113 for half/double (16 4114 and 64-bit) floating point 4115 precision floating point 4116 operations. 4117 4118 Floating point denorm mode 4119 values are defined in 4120 :ref:`amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table`. 4121 4122 Used by CP to set up 4123 ``COMPUTE_PGM_RSRC1.FLOAT_MODE``. 4124 20 1 bit PRIV Must be 0. 4125 4126 Start executing wavefront 4127 in privilege trap handler 4128 mode. 4129 4130 CP is responsible for 4131 filling in 4132 ``COMPUTE_PGM_RSRC1.PRIV``. 4133 21 1 bit ENABLE_DX10_CLAMP Wavefront starts execution 4134 with DX10 clamp mode 4135 enabled. Used by the vector 4136 ALU to force DX10 style 4137 treatment of NaN's (when 4138 set, clamp NaN to zero, 4139 otherwise pass NaN 4140 through). 4141 4142 Used by CP to set up 4143 ``COMPUTE_PGM_RSRC1.DX10_CLAMP``. 4144 22 1 bit DEBUG_MODE Must be 0. 4145 4146 Start executing wavefront 4147 in single step mode. 4148 4149 CP is responsible for 4150 filling in 4151 ``COMPUTE_PGM_RSRC1.DEBUG_MODE``. 4152 23 1 bit ENABLE_IEEE_MODE Wavefront starts execution 4153 with IEEE mode 4154 enabled. Floating point 4155 opcodes that support 4156 exception flag gathering 4157 will quiet and propagate 4158 signaling-NaN inputs per 4159 IEEE 754-2008. Min_dx10 and 4160 max_dx10 become IEEE 4161 754-2008 compliant due to 4162 signaling-NaN propagation 4163 and quieting. 4164 4165 Used by CP to set up 4166 ``COMPUTE_PGM_RSRC1.IEEE_MODE``. 4167 24 1 bit BULKY Must be 0. 4168 4169 Only one work-group allowed 4170 to execute on a compute 4171 unit. 4172 4173 CP is responsible for 4174 filling in 4175 ``COMPUTE_PGM_RSRC1.BULKY``. 4176 25 1 bit CDBG_USER Must be 0. 4177 4178 Flag that can be used to 4179 control debugging code. 4180 4181 CP is responsible for 4182 filling in 4183 ``COMPUTE_PGM_RSRC1.CDBG_USER``. 4184 26 1 bit FP16_OVFL GFX6-GFX8 4185 Reserved, must be 0. 4186 GFX9-GFX10 4187 Wavefront starts execution 4188 with specified fp16 overflow 4189 mode. 4190 4191 - If 0, fp16 overflow generates 4192 +/-INF values. 4193 - If 1, fp16 overflow that is the 4194 result of an +/-INF input value 4195 or divide by 0 produces a +/-INF, 4196 otherwise clamps computed 4197 overflow to +/-MAX_FP16 as 4198 appropriate. 4199 4200 Used by CP to set up 4201 ``COMPUTE_PGM_RSRC1.FP16_OVFL``. 4202 28:27 2 bits Reserved, must be 0. 4203 29 1 bit WGP_MODE GFX6-GFX9 4204 Reserved, must be 0. 4205 GFX10 4206 - If 0 execute work-groups in 4207 CU wavefront execution mode. 4208 - If 1 execute work-groups on 4209 in WGP wavefront execution mode. 4210 4211 See :ref:`amdgpu-amdhsa-memory-model`. 4212 4213 Used by CP to set up 4214 ``COMPUTE_PGM_RSRC1.WGP_MODE``. 4215 30 1 bit MEM_ORDERED GFX6-GFX9 4216 Reserved, must be 0. 4217 GFX10 4218 Controls the behavior of the 4219 s_waitcnt's vmcnt and vscnt 4220 counters. 4221 4222 - If 0 vmcnt reports completion 4223 of load and atomic with return 4224 out of order with sample 4225 instructions, and the vscnt 4226 reports the completion of 4227 store and atomic without 4228 return in order. 4229 - If 1 vmcnt reports completion 4230 of load, atomic with return 4231 and sample instructions in 4232 order, and the vscnt reports 4233 the completion of store and 4234 atomic without return in order. 4235 4236 Used by CP to set up 4237 ``COMPUTE_PGM_RSRC1.MEM_ORDERED``. 4238 31 1 bit FWD_PROGRESS GFX6-GFX9 4239 Reserved, must be 0. 4240 GFX10 4241 - If 0 execute SIMD wavefronts 4242 using oldest first policy. 4243 - If 1 execute SIMD wavefronts to 4244 ensure wavefronts will make some 4245 forward progress. 4246 4247 Used by CP to set up 4248 ``COMPUTE_PGM_RSRC1.FWD_PROGRESS``. 4249 32 **Total size 4 bytes** 4250 ======= =================================================================================================================== 4251 4252.. 4253 4254 .. table:: compute_pgm_rsrc2 for GFX6-GFX10 4255 :name: amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table 4256 4257 ======= ======= =============================== =========================================================================== 4258 Bits Size Field Name Description 4259 ======= ======= =============================== =========================================================================== 4260 0 1 bit ENABLE_PRIVATE_SEGMENT * Enable the setup of the 4261 private segment. 4262 * If the *Target Properties* 4263 column of 4264 :ref:`amdgpu-processor-table` 4265 does not specify 4266 *Architected flat 4267 scratch* then enable the 4268 setup of the SGPR 4269 wavefront scratch offset 4270 system register (see 4271 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4272 * If the *Target Properties* 4273 column of 4274 :ref:`amdgpu-processor-table` 4275 specifies *Architected 4276 flat scratch* then enable 4277 the setup of the 4278 FLAT_SCRATCH register 4279 pair (see 4280 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4281 4282 Used by CP to set up 4283 ``COMPUTE_PGM_RSRC2.SCRATCH_EN``. 4284 5:1 5 bits USER_SGPR_COUNT The total number of SGPR 4285 user data 4286 registers requested. This 4287 number must be greater than 4288 or equal to the number of user 4289 data registers enabled. 4290 4291 Used by CP to set up 4292 ``COMPUTE_PGM_RSRC2.USER_SGPR``. 4293 6 1 bit ENABLE_TRAP_HANDLER Must be 0. 4294 4295 This bit represents 4296 ``COMPUTE_PGM_RSRC2.TRAP_PRESENT``, 4297 which is set by the CP if 4298 the runtime has installed a 4299 trap handler. 4300 7 1 bit ENABLE_SGPR_WORKGROUP_ID_X Enable the setup of the 4301 system SGPR register for 4302 the work-group id in the X 4303 dimension (see 4304 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4305 4306 Used by CP to set up 4307 ``COMPUTE_PGM_RSRC2.TGID_X_EN``. 4308 8 1 bit ENABLE_SGPR_WORKGROUP_ID_Y Enable the setup of the 4309 system SGPR register for 4310 the work-group id in the Y 4311 dimension (see 4312 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4313 4314 Used by CP to set up 4315 ``COMPUTE_PGM_RSRC2.TGID_Y_EN``. 4316 9 1 bit ENABLE_SGPR_WORKGROUP_ID_Z Enable the setup of the 4317 system SGPR register for 4318 the work-group id in the Z 4319 dimension (see 4320 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4321 4322 Used by CP to set up 4323 ``COMPUTE_PGM_RSRC2.TGID_Z_EN``. 4324 10 1 bit ENABLE_SGPR_WORKGROUP_INFO Enable the setup of the 4325 system SGPR register for 4326 work-group information (see 4327 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4328 4329 Used by CP to set up 4330 ``COMPUTE_PGM_RSRC2.TGID_SIZE_EN``. 4331 12:11 2 bits ENABLE_VGPR_WORKITEM_ID Enable the setup of the 4332 VGPR system registers used 4333 for the work-item ID. 4334 :ref:`amdgpu-amdhsa-system-vgpr-work-item-id-enumeration-values-table` 4335 defines the values. 4336 4337 Used by CP to set up 4338 ``COMPUTE_PGM_RSRC2.TIDIG_CMP_CNT``. 4339 13 1 bit ENABLE_EXCEPTION_ADDRESS_WATCH Must be 0. 4340 4341 Wavefront starts execution 4342 with address watch 4343 exceptions enabled which 4344 are generated when L1 has 4345 witnessed a thread access 4346 an *address of 4347 interest*. 4348 4349 CP is responsible for 4350 filling in the address 4351 watch bit in 4352 ``COMPUTE_PGM_RSRC2.EXCP_EN_MSB`` 4353 according to what the 4354 runtime requests. 4355 14 1 bit ENABLE_EXCEPTION_MEMORY Must be 0. 4356 4357 Wavefront starts execution 4358 with memory violation 4359 exceptions exceptions 4360 enabled which are generated 4361 when a memory violation has 4362 occurred for this wavefront from 4363 L1 or LDS 4364 (write-to-read-only-memory, 4365 mis-aligned atomic, LDS 4366 address out of range, 4367 illegal address, etc.). 4368 4369 CP sets the memory 4370 violation bit in 4371 ``COMPUTE_PGM_RSRC2.EXCP_EN_MSB`` 4372 according to what the 4373 runtime requests. 4374 23:15 9 bits GRANULATED_LDS_SIZE Must be 0. 4375 4376 CP uses the rounded value 4377 from the dispatch packet, 4378 not this value, as the 4379 dispatch may contain 4380 dynamically allocated group 4381 segment memory. CP writes 4382 directly to 4383 ``COMPUTE_PGM_RSRC2.LDS_SIZE``. 4384 4385 Amount of group segment 4386 (LDS) to allocate for each 4387 work-group. Granularity is 4388 device specific: 4389 4390 GFX6 4391 roundup(lds-size / (64 * 4)) 4392 GFX7-GFX10 4393 roundup(lds-size / (128 * 4)) 4394 4395 24 1 bit ENABLE_EXCEPTION_IEEE_754_FP Wavefront starts execution 4396 _INVALID_OPERATION with specified exceptions 4397 enabled. 4398 4399 Used by CP to set up 4400 ``COMPUTE_PGM_RSRC2.EXCP_EN`` 4401 (set from bits 0..6). 4402 4403 IEEE 754 FP Invalid 4404 Operation 4405 25 1 bit ENABLE_EXCEPTION_FP_DENORMAL FP Denormal one or more 4406 _SOURCE input operands is a 4407 denormal number 4408 26 1 bit ENABLE_EXCEPTION_IEEE_754_FP IEEE 754 FP Division by 4409 _DIVISION_BY_ZERO Zero 4410 27 1 bit ENABLE_EXCEPTION_IEEE_754_FP IEEE 754 FP FP Overflow 4411 _OVERFLOW 4412 28 1 bit ENABLE_EXCEPTION_IEEE_754_FP IEEE 754 FP Underflow 4413 _UNDERFLOW 4414 29 1 bit ENABLE_EXCEPTION_IEEE_754_FP IEEE 754 FP Inexact 4415 _INEXACT 4416 30 1 bit ENABLE_EXCEPTION_INT_DIVIDE_BY Integer Division by Zero 4417 _ZERO (rcp_iflag_f32 instruction 4418 only) 4419 31 1 bit Reserved, must be 0. 4420 32 **Total size 4 bytes.** 4421 ======= =================================================================================================================== 4422 4423.. 4424 4425 .. table:: compute_pgm_rsrc3 for GFX90A, GFX940 4426 :name: amdgpu-amdhsa-compute_pgm_rsrc3-gfx90a-table 4427 4428 ======= ======= =============================== =========================================================================== 4429 Bits Size Field Name Description 4430 ======= ======= =============================== =========================================================================== 4431 5:0 6 bits ACCUM_OFFSET Offset of a first AccVGPR in the unified register file. Granularity 4. 4432 Value 0-63. 0 - accum-offset = 4, 1 - accum-offset = 8, ..., 4433 63 - accum-offset = 256. 4434 6:15 10 Reserved, must be 0. 4435 bits 4436 16 1 bit TG_SPLIT - If 0 the waves of a work-group are 4437 launched in the same CU. 4438 - If 1 the waves of a work-group can be 4439 launched in different CUs. The waves 4440 cannot use S_BARRIER or LDS. 4441 17:31 15 Reserved, must be 0. 4442 bits 4443 32 **Total size 4 bytes.** 4444 ======= =================================================================================================================== 4445 4446.. 4447 4448 .. table:: compute_pgm_rsrc3 for GFX10 4449 :name: amdgpu-amdhsa-compute_pgm_rsrc3-gfx10-table 4450 4451 ======= ======= =============================== =========================================================================== 4452 Bits Size Field Name Description 4453 ======= ======= =============================== =========================================================================== 4454 3:0 4 bits SHARED_VGPR_COUNT Number of shared VGPR blocks when executing in subvector mode. For 4455 wavefront size 64 the value is 0-15, representing 0-120 VGPRs (granularity 4456 of 8), such that (compute_pgm_rsrc1.vgprs +1)*4 + shared_vgpr_count*8 does 4457 not exceed 256. For wavefront size 32 shared_vgpr_count must be 0. 4458 31:4 28 Reserved, must be 0. 4459 bits 4460 32 **Total size 4 bytes.** 4461 ======= =================================================================================================================== 4462 4463.. 4464 4465 .. table:: Floating Point Rounding Mode Enumeration Values 4466 :name: amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table 4467 4468 ====================================== ===== ============================== 4469 Enumeration Name Value Description 4470 ====================================== ===== ============================== 4471 FLOAT_ROUND_MODE_NEAR_EVEN 0 Round Ties To Even 4472 FLOAT_ROUND_MODE_PLUS_INFINITY 1 Round Toward +infinity 4473 FLOAT_ROUND_MODE_MINUS_INFINITY 2 Round Toward -infinity 4474 FLOAT_ROUND_MODE_ZERO 3 Round Toward 0 4475 ====================================== ===== ============================== 4476 4477.. 4478 4479 .. table:: Floating Point Denorm Mode Enumeration Values 4480 :name: amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table 4481 4482 ====================================== ===== ============================== 4483 Enumeration Name Value Description 4484 ====================================== ===== ============================== 4485 FLOAT_DENORM_MODE_FLUSH_SRC_DST 0 Flush Source and Destination 4486 Denorms 4487 FLOAT_DENORM_MODE_FLUSH_DST 1 Flush Output Denorms 4488 FLOAT_DENORM_MODE_FLUSH_SRC 2 Flush Source Denorms 4489 FLOAT_DENORM_MODE_FLUSH_NONE 3 No Flush 4490 ====================================== ===== ============================== 4491 4492.. 4493 4494 .. table:: System VGPR Work-Item ID Enumeration Values 4495 :name: amdgpu-amdhsa-system-vgpr-work-item-id-enumeration-values-table 4496 4497 ======================================== ===== ============================ 4498 Enumeration Name Value Description 4499 ======================================== ===== ============================ 4500 SYSTEM_VGPR_WORKITEM_ID_X 0 Set work-item X dimension 4501 ID. 4502 SYSTEM_VGPR_WORKITEM_ID_X_Y 1 Set work-item X and Y 4503 dimensions ID. 4504 SYSTEM_VGPR_WORKITEM_ID_X_Y_Z 2 Set work-item X, Y and Z 4505 dimensions ID. 4506 SYSTEM_VGPR_WORKITEM_ID_UNDEFINED 3 Undefined. 4507 ======================================== ===== ============================ 4508 4509.. _amdgpu-amdhsa-initial-kernel-execution-state: 4510 4511Initial Kernel Execution State 4512~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 4513 4514This section defines the register state that will be set up by the packet 4515processor prior to the start of execution of every wavefront. This is limited by 4516the constraints of the hardware controllers of CP/ADC/SPI. 4517 4518The order of the SGPR registers is defined, but the compiler can specify which 4519ones are actually setup in the kernel descriptor using the ``enable_sgpr_*`` bit 4520fields (see :ref:`amdgpu-amdhsa-kernel-descriptor`). The register numbers used 4521for enabled registers are dense starting at SGPR0: the first enabled register is 4522SGPR0, the next enabled register is SGPR1 etc.; disabled registers do not have 4523an SGPR number. 4524 4525The initial SGPRs comprise up to 16 User SRGPs that are set by CP and apply to 4526all wavefronts of the grid. It is possible to specify more than 16 User SGPRs 4527using the ``enable_sgpr_*`` bit fields, in which case only the first 16 are 4528actually initialized. These are then immediately followed by the System SGPRs 4529that are set up by ADC/SPI and can have different values for each wavefront of 4530the grid dispatch. 4531 4532SGPR register initial state is defined in 4533:ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 4534 4535 .. table:: SGPR Register Set Up Order 4536 :name: amdgpu-amdhsa-sgpr-register-set-up-order-table 4537 4538 ========== ========================== ====== ============================== 4539 SGPR Order Name Number Description 4540 (kernel descriptor enable of 4541 field) SGPRs 4542 ========== ========================== ====== ============================== 4543 First Private Segment Buffer 4 See 4544 (enable_sgpr_private :ref:`amdgpu-amdhsa-kernel-prolog-private-segment-buffer`. 4545 _segment_buffer) 4546 then Dispatch Ptr 2 64-bit address of AQL dispatch 4547 (enable_sgpr_dispatch_ptr) packet for kernel dispatch 4548 actually executing. 4549 then Queue Ptr 2 64-bit address of amd_queue_t 4550 (enable_sgpr_queue_ptr) object for AQL queue on which 4551 the dispatch packet was 4552 queued. 4553 then Kernarg Segment Ptr 2 64-bit address of Kernarg 4554 (enable_sgpr_kernarg segment. This is directly 4555 _segment_ptr) copied from the 4556 kernarg_address in the kernel 4557 dispatch packet. 4558 4559 Having CP load it once avoids 4560 loading it at the beginning of 4561 every wavefront. 4562 then Dispatch Id 2 64-bit Dispatch ID of the 4563 (enable_sgpr_dispatch_id) dispatch packet being 4564 executed. 4565 then Flat Scratch Init 2 See 4566 (enable_sgpr_flat_scratch :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`. 4567 _init) 4568 then Private Segment Size 1 The 32-bit byte size of a 4569 (enable_sgpr_private single work-item's memory 4570 _segment_size) allocation. This is the 4571 value from the kernel 4572 dispatch packet Private 4573 Segment Byte Size rounded up 4574 by CP to a multiple of 4575 DWORD. 4576 4577 Having CP load it once avoids 4578 loading it at the beginning of 4579 every wavefront. 4580 4581 This is not used for 4582 GFX7-GFX8 since it is the same 4583 value as the second SGPR of 4584 Flat Scratch Init. However, it 4585 may be needed for GFX9-GFX10 which 4586 changes the meaning of the 4587 Flat Scratch Init value. 4588 then Work-Group Id X 1 32-bit work-group id in X 4589 (enable_sgpr_workgroup_id dimension of grid for 4590 _X) wavefront. 4591 then Work-Group Id Y 1 32-bit work-group id in Y 4592 (enable_sgpr_workgroup_id dimension of grid for 4593 _Y) wavefront. 4594 then Work-Group Id Z 1 32-bit work-group id in Z 4595 (enable_sgpr_workgroup_id dimension of grid for 4596 _Z) wavefront. 4597 then Work-Group Info 1 {first_wavefront, 14'b0000, 4598 (enable_sgpr_workgroup ordered_append_term[10:0], 4599 _info) threadgroup_size_in_wavefronts[5:0]} 4600 then Scratch Wavefront Offset 1 See 4601 (enable_sgpr_private :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`. 4602 _segment_wavefront_offset) and 4603 :ref:`amdgpu-amdhsa-kernel-prolog-private-segment-buffer`. 4604 ========== ========================== ====== ============================== 4605 4606The order of the VGPR registers is defined, but the compiler can specify which 4607ones are actually setup in the kernel descriptor using the ``enable_vgpr*`` bit 4608fields (see :ref:`amdgpu-amdhsa-kernel-descriptor`). The register numbers used 4609for enabled registers are dense starting at VGPR0: the first enabled register is 4610VGPR0, the next enabled register is VGPR1 etc.; disabled registers do not have a 4611VGPR number. 4612 4613There are different methods used for the VGPR initial state: 4614 4615* Unless the *Target Properties* column of :ref:`amdgpu-processor-table` 4616 specifies otherwise, a separate VGPR register is used per work-item ID. The 4617 VGPR register initial state for this method is defined in 4618 :ref:`amdgpu-amdhsa-vgpr-register-set-up-order-for-unpacked-work-item-id-method-table`. 4619* If *Target Properties* column of :ref:`amdgpu-processor-table` 4620 specifies *Packed work-item IDs*, the initial value of VGPR0 register is used 4621 for all work-item IDs. The register layout for this method is defined in 4622 :ref:`amdgpu-amdhsa-register-layout-for-packed-work-item-id-method-table`. 4623 4624 .. table:: VGPR Register Set Up Order for Unpacked Work-Item ID Method 4625 :name: amdgpu-amdhsa-vgpr-register-set-up-order-for-unpacked-work-item-id-method-table 4626 4627 ========== ========================== ====== ============================== 4628 VGPR Order Name Number Description 4629 (kernel descriptor enable of 4630 field) VGPRs 4631 ========== ========================== ====== ============================== 4632 First Work-Item Id X 1 32-bit work-item id in X 4633 (Always initialized) dimension of work-group for 4634 wavefront lane. 4635 then Work-Item Id Y 1 32-bit work-item id in Y 4636 (enable_vgpr_workitem_id dimension of work-group for 4637 > 0) wavefront lane. 4638 then Work-Item Id Z 1 32-bit work-item id in Z 4639 (enable_vgpr_workitem_id dimension of work-group for 4640 > 1) wavefront lane. 4641 ========== ========================== ====== ============================== 4642 4643.. 4644 4645 .. table:: Register Layout for Packed Work-Item ID Method 4646 :name: amdgpu-amdhsa-register-layout-for-packed-work-item-id-method-table 4647 4648 ======= ======= ================ ========================================= 4649 Bits Size Field Name Description 4650 ======= ======= ================ ========================================= 4651 0:9 10 bits Work-Item Id X Work-item id in X 4652 dimension of work-group for 4653 wavefront lane. 4654 4655 Always initialized. 4656 4657 10:19 10 bits Work-Item Id Y Work-item id in Y 4658 dimension of work-group for 4659 wavefront lane. 4660 4661 Initialized if enable_vgpr_workitem_id > 4662 0, otherwise set to 0. 4663 20:29 10 bits Work-Item Id Z Work-item id in Z 4664 dimension of work-group for 4665 wavefront lane. 4666 4667 Initialized if enable_vgpr_workitem_id > 4668 1, otherwise set to 0. 4669 30:31 2 bits Reserved, set to 0. 4670 ======= ======= ================ ========================================= 4671 4672The setting of registers is done by GPU CP/ADC/SPI hardware as follows: 4673 46741. SGPRs before the Work-Group Ids are set by CP using the 16 User Data 4675 registers. 46762. Work-group Id registers X, Y, Z are set by ADC which supports any 4677 combination including none. 46783. Scratch Wavefront Offset is set by SPI in a per wavefront basis which is why 4679 its value cannot be included with the flat scratch init value which is per 4680 queue (see :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`). 46814. The VGPRs are set by SPI which only supports specifying either (X), (X, Y) 4682 or (X, Y, Z). 46835. Flat Scratch register pair initialization is described in 4684 :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`. 4685 4686The global segment can be accessed either using buffer instructions (GFX6 which 4687has V# 64-bit address support), flat instructions (GFX7-GFX10), or global 4688instructions (GFX9-GFX10). 4689 4690If buffer operations are used, then the compiler can generate a V# with the 4691following properties: 4692 4693* base address of 0 4694* no swizzle 4695* ATC: 1 if IOMMU present (such as APU) 4696* ptr64: 1 4697* MTYPE set to support memory coherence that matches the runtime (such as CC for 4698 APU and NC for dGPU). 4699 4700.. _amdgpu-amdhsa-kernel-prolog: 4701 4702Kernel Prolog 4703~~~~~~~~~~~~~ 4704 4705The compiler performs initialization in the kernel prologue depending on the 4706target and information about things like stack usage in the kernel and called 4707functions. Some of this initialization requires the compiler to request certain 4708User and System SGPRs be present in the 4709:ref:`amdgpu-amdhsa-initial-kernel-execution-state` via the 4710:ref:`amdgpu-amdhsa-kernel-descriptor`. 4711 4712.. _amdgpu-amdhsa-kernel-prolog-cfi: 4713 4714CFI 4715+++ 4716 47171. The CFI return address is undefined. 4718 47192. The CFI CFA is defined using an expression which evaluates to a location 4720 description that comprises one memory location description for the 4721 ``DW_ASPACE_AMDGPU_private_lane`` address space address ``0``. 4722 4723.. _amdgpu-amdhsa-kernel-prolog-m0: 4724 4725M0 4726++ 4727 4728GFX6-GFX8 4729 The M0 register must be initialized with a value at least the total LDS size 4730 if the kernel may access LDS via DS or flat operations. Total LDS size is 4731 available in dispatch packet. For M0, it is also possible to use maximum 4732 possible value of LDS for given target (0x7FFF for GFX6 and 0xFFFF for 4733 GFX7-GFX8). 4734GFX9-GFX10 4735 The M0 register is not used for range checking LDS accesses and so does not 4736 need to be initialized in the prolog. 4737 4738.. _amdgpu-amdhsa-kernel-prolog-stack-pointer: 4739 4740Stack Pointer 4741+++++++++++++ 4742 4743If the kernel has function calls it must set up the ABI stack pointer described 4744in :ref:`amdgpu-amdhsa-function-call-convention-non-kernel-functions` by setting 4745SGPR32 to the unswizzled scratch offset of the address past the last local 4746allocation. 4747 4748.. _amdgpu-amdhsa-kernel-prolog-frame-pointer: 4749 4750Frame Pointer 4751+++++++++++++ 4752 4753If the kernel needs a frame pointer for the reasons defined in 4754``SIFrameLowering`` then SGPR33 is used and is always set to ``0`` in the 4755kernel prolog. If a frame pointer is not required then all uses of the frame 4756pointer are replaced with immediate ``0`` offsets. 4757 4758.. _amdgpu-amdhsa-kernel-prolog-flat-scratch: 4759 4760Flat Scratch 4761++++++++++++ 4762 4763There are different methods used for initializing flat scratch: 4764 4765* If the *Target Properties* column of :ref:`amdgpu-processor-table` 4766 specifies *Does not support generic address space*: 4767 4768 Flat scratch is not supported and there is no flat scratch register pair. 4769 4770* If the *Target Properties* column of :ref:`amdgpu-processor-table` 4771 specifies *Offset flat scratch*: 4772 4773 If the kernel or any function it calls may use flat operations to access 4774 scratch memory, the prolog code must set up the FLAT_SCRATCH register pair 4775 (FLAT_SCRATCH_LO/FLAT_SCRATCH_HI). Initialization uses Flat Scratch Init and 4776 Scratch Wavefront Offset SGPR registers (see 4777 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`): 4778 4779 1. The low word of Flat Scratch Init is the 32-bit byte offset from 4780 ``SH_HIDDEN_PRIVATE_BASE_VIMID`` to the base of scratch backing memory 4781 being managed by SPI for the queue executing the kernel dispatch. This is 4782 the same value used in the Scratch Segment Buffer V# base address. 4783 4784 CP obtains this from the runtime. (The Scratch Segment Buffer base address 4785 is ``SH_HIDDEN_PRIVATE_BASE_VIMID`` plus this offset.) 4786 4787 The prolog must add the value of Scratch Wavefront Offset to get the 4788 wavefront's byte scratch backing memory offset from 4789 ``SH_HIDDEN_PRIVATE_BASE_VIMID``. 4790 4791 The Scratch Wavefront Offset must also be used as an offset with Private 4792 segment address when using the Scratch Segment Buffer. 4793 4794 Since FLAT_SCRATCH_LO is in units of 256 bytes, the offset must be right 4795 shifted by 8 before moving into FLAT_SCRATCH_HI. 4796 4797 FLAT_SCRATCH_HI corresponds to SGPRn-4 on GFX7, and SGPRn-6 on GFX8 (where 4798 SGPRn is the highest numbered SGPR allocated to the wavefront). 4799 FLAT_SCRATCH_HI is multiplied by 256 (as it is in units of 256 bytes) and 4800 added to ``SH_HIDDEN_PRIVATE_BASE_VIMID`` to calculate the per wavefront 4801 FLAT SCRATCH BASE in flat memory instructions that access the scratch 4802 aperture. 4803 2. The second word of Flat Scratch Init is 32-bit byte size of a single 4804 work-items scratch memory usage. 4805 4806 CP obtains this from the runtime, and it is always a multiple of DWORD. CP 4807 checks that the value in the kernel dispatch packet Private Segment Byte 4808 Size is not larger and requests the runtime to increase the queue's scratch 4809 size if necessary. 4810 4811 CP directly loads from the kernel dispatch packet Private Segment Byte Size 4812 field and rounds up to a multiple of DWORD. Having CP load it once avoids 4813 loading it at the beginning of every wavefront. 4814 4815 The kernel prolog code must move it to FLAT_SCRATCH_LO which is SGPRn-3 on 4816 GFX7 and SGPRn-5 on GFX8. FLAT_SCRATCH_LO is used as the FLAT SCRATCH SIZE 4817 in flat memory instructions. 4818 4819* If the *Target Properties* column of :ref:`amdgpu-processor-table` 4820 specifies *Absolute flat scratch*: 4821 4822 If the kernel or any function it calls may use flat operations to access 4823 scratch memory, the prolog code must set up the FLAT_SCRATCH register pair 4824 (FLAT_SCRATCH_LO/FLAT_SCRATCH_HI which are in SGPRn-4/SGPRn-3). Initialization 4825 uses Flat Scratch Init and Scratch Wavefront Offset SGPR registers (see 4826 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`): 4827 4828 The Flat Scratch Init is the 64-bit address of the base of scratch backing 4829 memory being managed by SPI for the queue executing the kernel dispatch. 4830 4831 CP obtains this from the runtime. 4832 4833 The kernel prolog must add the value of the wave's Scratch Wavefront Offset 4834 and move the result as a 64-bit value to the FLAT_SCRATCH SGPR register pair 4835 which is SGPRn-6 and SGPRn-5. It is used as the FLAT SCRATCH BASE in flat 4836 memory instructions. 4837 4838 The Scratch Wavefront Offset must also be used as an offset with Private 4839 segment address when using the Scratch Segment Buffer (see 4840 :ref:`amdgpu-amdhsa-kernel-prolog-private-segment-buffer`). 4841 4842* If the *Target Properties* column of :ref:`amdgpu-processor-table` 4843 specifies *Architected flat scratch*: 4844 4845 If ENABLE_PRIVATE_SEGMENT is enabled in 4846 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table` then the FLAT_SCRATCH 4847 register pair will be initialized to the 64-bit address of the base of scratch 4848 backing memory being managed by SPI for the queue executing the kernel 4849 dispatch plus the value of the wave's Scratch Wavefront Offset for use as the 4850 flat scratch base in flat memory instructions. 4851 4852.. _amdgpu-amdhsa-kernel-prolog-private-segment-buffer: 4853 4854Private Segment Buffer 4855++++++++++++++++++++++ 4856 4857If the *Target Properties* column of :ref:`amdgpu-processor-table` specifies 4858*Architected flat scratch* then a Private Segment Buffer is not supported. 4859Instead the flat SCRATCH instructions are used. 4860 4861Otherwise, Private Segment Buffer SGPR register is used to initialize 4 SGPRs 4862that are used as a V# to access scratch. CP uses the value provided by the 4863runtime. It is used, together with Scratch Wavefront Offset as an offset, to 4864access the private memory space using a segment address. See 4865:ref:`amdgpu-amdhsa-initial-kernel-execution-state`. 4866 4867The scratch V# is a four-aligned SGPR and always selected for the kernel as 4868follows: 4869 4870 - If it is known during instruction selection that there is stack usage, 4871 SGPR0-3 is reserved for use as the scratch V#. Stack usage is assumed if 4872 optimizations are disabled (``-O0``), if stack objects already exist (for 4873 locals, etc.), or if there are any function calls. 4874 4875 - Otherwise, four high numbered SGPRs beginning at a four-aligned SGPR index 4876 are reserved for the tentative scratch V#. These will be used if it is 4877 determined that spilling is needed. 4878 4879 - If no use is made of the tentative scratch V#, then it is unreserved, 4880 and the register count is determined ignoring it. 4881 - If use is made of the tentative scratch V#, then its register numbers 4882 are shifted to the first four-aligned SGPR index after the highest one 4883 allocated by the register allocator, and all uses are updated. The 4884 register count includes them in the shifted location. 4885 - In either case, if the processor has the SGPR allocation bug, the 4886 tentative allocation is not shifted or unreserved in order to ensure 4887 the register count is higher to workaround the bug. 4888 4889 .. note:: 4890 4891 This approach of using a tentative scratch V# and shifting the register 4892 numbers if used avoids having to perform register allocation a second 4893 time if the tentative V# is eliminated. This is more efficient and 4894 avoids the problem that the second register allocation may perform 4895 spilling which will fail as there is no longer a scratch V#. 4896 4897When the kernel prolog code is being emitted it is known whether the scratch V# 4898described above is actually used. If it is, the prolog code must set it up by 4899copying the Private Segment Buffer to the scratch V# registers and then adding 4900the Private Segment Wavefront Offset to the queue base address in the V#. The 4901result is a V# with a base address pointing to the beginning of the wavefront 4902scratch backing memory. 4903 4904The Private Segment Buffer is always requested, but the Private Segment 4905Wavefront Offset is only requested if it is used (see 4906:ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4907 4908.. _amdgpu-amdhsa-memory-model: 4909 4910Memory Model 4911~~~~~~~~~~~~ 4912 4913This section describes the mapping of the LLVM memory model onto AMDGPU machine 4914code (see :ref:`memmodel`). 4915 4916The AMDGPU backend supports the memory synchronization scopes specified in 4917:ref:`amdgpu-memory-scopes`. 4918 4919The code sequences used to implement the memory model specify the order of 4920instructions that a single thread must execute. The ``s_waitcnt`` and cache 4921management instructions such as ``buffer_wbinvl1_vol`` are defined with respect 4922to other memory instructions executed by the same thread. This allows them to be 4923moved earlier or later which can allow them to be combined with other instances 4924of the same instruction, or hoisted/sunk out of loops to improve performance. 4925Only the instructions related to the memory model are given; additional 4926``s_waitcnt`` instructions are required to ensure registers are defined before 4927being used. These may be able to be combined with the memory model ``s_waitcnt`` 4928instructions as described above. 4929 4930The AMDGPU backend supports the following memory models: 4931 4932 HSA Memory Model [HSA]_ 4933 The HSA memory model uses a single happens-before relation for all address 4934 spaces (see :ref:`amdgpu-address-spaces`). 4935 OpenCL Memory Model [OpenCL]_ 4936 The OpenCL memory model which has separate happens-before relations for the 4937 global and local address spaces. Only a fence specifying both global and 4938 local address space, and seq_cst instructions join the relationships. Since 4939 the LLVM ``memfence`` instruction does not allow an address space to be 4940 specified the OpenCL fence has to conservatively assume both local and 4941 global address space was specified. However, optimizations can often be 4942 done to eliminate the additional ``s_waitcnt`` instructions when there are 4943 no intervening memory instructions which access the corresponding address 4944 space. The code sequences in the table indicate what can be omitted for the 4945 OpenCL memory. The target triple environment is used to determine if the 4946 source language is OpenCL (see :ref:`amdgpu-opencl`). 4947 4948``ds/flat_load/store/atomic`` instructions to local memory are termed LDS 4949operations. 4950 4951``buffer/global/flat_load/store/atomic`` instructions to global memory are 4952termed vector memory operations. 4953 4954Private address space uses ``buffer_load/store`` using the scratch V# 4955(GFX6-GFX8), or ``scratch_load/store`` (GFX9-GFX10). Since only a single thread 4956is accessing the memory, atomic memory orderings are not meaningful, and all 4957accesses are treated as non-atomic. 4958 4959Constant address space uses ``buffer/global_load`` instructions (or equivalent 4960scalar memory instructions). Since the constant address space contents do not 4961change during the execution of a kernel dispatch it is not legal to perform 4962stores, and atomic memory orderings are not meaningful, and all accesses are 4963treated as non-atomic. 4964 4965A memory synchronization scope wider than work-group is not meaningful for the 4966group (LDS) address space and is treated as work-group. 4967 4968The memory model does not support the region address space which is treated as 4969non-atomic. 4970 4971Acquire memory ordering is not meaningful on store atomic instructions and is 4972treated as non-atomic. 4973 4974Release memory ordering is not meaningful on load atomic instructions and is 4975treated a non-atomic. 4976 4977Acquire-release memory ordering is not meaningful on load or store atomic 4978instructions and is treated as acquire and release respectively. 4979 4980The memory order also adds the single thread optimization constraints defined in 4981table 4982:ref:`amdgpu-amdhsa-memory-model-single-thread-optimization-constraints-table`. 4983 4984 .. table:: AMDHSA Memory Model Single Thread Optimization Constraints 4985 :name: amdgpu-amdhsa-memory-model-single-thread-optimization-constraints-table 4986 4987 ============ ============================================================== 4988 LLVM Memory Optimization Constraints 4989 Ordering 4990 ============ ============================================================== 4991 unordered *none* 4992 monotonic *none* 4993 acquire - If a load atomic/atomicrmw then no following load/load 4994 atomic/store/store atomic/atomicrmw/fence instruction can be 4995 moved before the acquire. 4996 - If a fence then same as load atomic, plus no preceding 4997 associated fence-paired-atomic can be moved after the fence. 4998 release - If a store atomic/atomicrmw then no preceding load/load 4999 atomic/store/store atomic/atomicrmw/fence instruction can be 5000 moved after the release. 5001 - If a fence then same as store atomic, plus no following 5002 associated fence-paired-atomic can be moved before the 5003 fence. 5004 acq_rel Same constraints as both acquire and release. 5005 seq_cst - If a load atomic then same constraints as acquire, plus no 5006 preceding sequentially consistent load atomic/store 5007 atomic/atomicrmw/fence instruction can be moved after the 5008 seq_cst. 5009 - If a store atomic then the same constraints as release, plus 5010 no following sequentially consistent load atomic/store 5011 atomic/atomicrmw/fence instruction can be moved before the 5012 seq_cst. 5013 - If an atomicrmw/fence then same constraints as acq_rel. 5014 ============ ============================================================== 5015 5016The code sequences used to implement the memory model are defined in the 5017following sections: 5018 5019* :ref:`amdgpu-amdhsa-memory-model-gfx6-gfx9` 5020* :ref:`amdgpu-amdhsa-memory-model-gfx90a` 5021* :ref:`amdgpu-amdhsa-memory-model-gfx10` 5022 5023.. _amdgpu-amdhsa-memory-model-gfx6-gfx9: 5024 5025Memory Model GFX6-GFX9 5026++++++++++++++++++++++ 5027 5028For GFX6-GFX9: 5029 5030* Each agent has multiple shader arrays (SA). 5031* Each SA has multiple compute units (CU). 5032* Each CU has multiple SIMDs that execute wavefronts. 5033* The wavefronts for a single work-group are executed in the same CU but may be 5034 executed by different SIMDs. 5035* Each CU has a single LDS memory shared by the wavefronts of the work-groups 5036 executing on it. 5037* All LDS operations of a CU are performed as wavefront wide operations in a 5038 global order and involve no caching. Completion is reported to a wavefront in 5039 execution order. 5040* The LDS memory has multiple request queues shared by the SIMDs of a 5041 CU. Therefore, the LDS operations performed by different wavefronts of a 5042 work-group can be reordered relative to each other, which can result in 5043 reordering the visibility of vector memory operations with respect to LDS 5044 operations of other wavefronts in the same work-group. A ``s_waitcnt 5045 lgkmcnt(0)`` is required to ensure synchronization between LDS operations and 5046 vector memory operations between wavefronts of a work-group, but not between 5047 operations performed by the same wavefront. 5048* The vector memory operations are performed as wavefront wide operations and 5049 completion is reported to a wavefront in execution order. The exception is 5050 that for GFX7-GFX9 ``flat_load/store/atomic`` instructions can report out of 5051 vector memory order if they access LDS memory, and out of LDS operation order 5052 if they access global memory. 5053* The vector memory operations access a single vector L1 cache shared by all 5054 SIMDs a CU. Therefore, no special action is required for coherence between the 5055 lanes of a single wavefront, or for coherence between wavefronts in the same 5056 work-group. A ``buffer_wbinvl1_vol`` is required for coherence between 5057 wavefronts executing in different work-groups as they may be executing on 5058 different CUs. 5059* The scalar memory operations access a scalar L1 cache shared by all wavefronts 5060 on a group of CUs. The scalar and vector L1 caches are not coherent. However, 5061 scalar operations are used in a restricted way so do not impact the memory 5062 model. See :ref:`amdgpu-amdhsa-memory-spaces`. 5063* The vector and scalar memory operations use an L2 cache shared by all CUs on 5064 the same agent. 5065* The L2 cache has independent channels to service disjoint ranges of virtual 5066 addresses. 5067* Each CU has a separate request queue per channel. Therefore, the vector and 5068 scalar memory operations performed by wavefronts executing in different 5069 work-groups (which may be executing on different CUs) of an agent can be 5070 reordered relative to each other. A ``s_waitcnt vmcnt(0)`` is required to 5071 ensure synchronization between vector memory operations of different CUs. It 5072 ensures a previous vector memory operation has completed before executing a 5073 subsequent vector memory or LDS operation and so can be used to meet the 5074 requirements of acquire and release. 5075* The L2 cache can be kept coherent with other agents on some targets, or ranges 5076 of virtual addresses can be set up to bypass it to ensure system coherence. 5077 5078Scalar memory operations are only used to access memory that is proven to not 5079change during the execution of the kernel dispatch. This includes constant 5080address space and global address space for program scope ``const`` variables. 5081Therefore, the kernel machine code does not have to maintain the scalar cache to 5082ensure it is coherent with the vector caches. The scalar and vector caches are 5083invalidated between kernel dispatches by CP since constant address space data 5084may change between kernel dispatch executions. See 5085:ref:`amdgpu-amdhsa-memory-spaces`. 5086 5087The one exception is if scalar writes are used to spill SGPR registers. In this 5088case the AMDGPU backend ensures the memory location used to spill is never 5089accessed by vector memory operations at the same time. If scalar writes are used 5090then a ``s_dcache_wb`` is inserted before the ``s_endpgm`` and before a function 5091return since the locations may be used for vector memory instructions by a 5092future wavefront that uses the same scratch area, or a function call that 5093creates a frame at the same address, respectively. There is no need for a 5094``s_dcache_inv`` as all scalar writes are write-before-read in the same thread. 5095 5096For kernarg backing memory: 5097 5098* CP invalidates the L1 cache at the start of each kernel dispatch. 5099* On dGPU the kernarg backing memory is allocated in host memory accessed as 5100 MTYPE UC (uncached) to avoid needing to invalidate the L2 cache. This also 5101 causes it to be treated as non-volatile and so is not invalidated by 5102 ``*_vol``. 5103* On APU the kernarg backing memory it is accessed as MTYPE CC (cache coherent) 5104 and so the L2 cache will be coherent with the CPU and other agents. 5105 5106Scratch backing memory (which is used for the private address space) is accessed 5107with MTYPE NC_NV (non-coherent non-volatile). Since the private address space is 5108only accessed by a single thread, and is always write-before-read, there is 5109never a need to invalidate these entries from the L1 cache. Hence all cache 5110invalidates are done as ``*_vol`` to only invalidate the volatile cache lines. 5111 5112The code sequences used to implement the memory model for GFX6-GFX9 are defined 5113in table :ref:`amdgpu-amdhsa-memory-model-code-sequences-gfx6-gfx9-table`. 5114 5115 .. table:: AMDHSA Memory Model Code Sequences GFX6-GFX9 5116 :name: amdgpu-amdhsa-memory-model-code-sequences-gfx6-gfx9-table 5117 5118 ============ ============ ============== ========== ================================ 5119 LLVM Instr LLVM Memory LLVM Memory AMDGPU AMDGPU Machine Code 5120 Ordering Sync Scope Address GFX6-GFX9 5121 Space 5122 ============ ============ ============== ========== ================================ 5123 **Non-Atomic** 5124 ------------------------------------------------------------------------------------ 5125 load *none* *none* - global - !volatile & !nontemporal 5126 - generic 5127 - private 1. buffer/global/flat_load 5128 - constant 5129 - !volatile & nontemporal 5130 5131 1. buffer/global/flat_load 5132 glc=1 slc=1 5133 5134 - volatile 5135 5136 1. buffer/global/flat_load 5137 glc=1 5138 2. s_waitcnt vmcnt(0) 5139 5140 - Must happen before 5141 any following volatile 5142 global/generic 5143 load/store. 5144 - Ensures that 5145 volatile 5146 operations to 5147 different 5148 addresses will not 5149 be reordered by 5150 hardware. 5151 5152 load *none* *none* - local 1. ds_load 5153 store *none* *none* - global - !volatile & !nontemporal 5154 - generic 5155 - private 1. buffer/global/flat_store 5156 - constant 5157 - !volatile & nontemporal 5158 5159 1. buffer/global/flat_store 5160 glc=1 slc=1 5161 5162 - volatile 5163 5164 1. buffer/global/flat_store 5165 2. s_waitcnt vmcnt(0) 5166 5167 - Must happen before 5168 any following volatile 5169 global/generic 5170 load/store. 5171 - Ensures that 5172 volatile 5173 operations to 5174 different 5175 addresses will not 5176 be reordered by 5177 hardware. 5178 5179 store *none* *none* - local 1. ds_store 5180 **Unordered Atomic** 5181 ------------------------------------------------------------------------------------ 5182 load atomic unordered *any* *any* *Same as non-atomic*. 5183 store atomic unordered *any* *any* *Same as non-atomic*. 5184 atomicrmw unordered *any* *any* *Same as monotonic atomic*. 5185 **Monotonic Atomic** 5186 ------------------------------------------------------------------------------------ 5187 load atomic monotonic - singlethread - global 1. buffer/global/ds/flat_load 5188 - wavefront - local 5189 - workgroup - generic 5190 load atomic monotonic - agent - global 1. buffer/global/flat_load 5191 - system - generic glc=1 5192 store atomic monotonic - singlethread - global 1. buffer/global/flat_store 5193 - wavefront - generic 5194 - workgroup 5195 - agent 5196 - system 5197 store atomic monotonic - singlethread - local 1. ds_store 5198 - wavefront 5199 - workgroup 5200 atomicrmw monotonic - singlethread - global 1. buffer/global/flat_atomic 5201 - wavefront - generic 5202 - workgroup 5203 - agent 5204 - system 5205 atomicrmw monotonic - singlethread - local 1. ds_atomic 5206 - wavefront 5207 - workgroup 5208 **Acquire Atomic** 5209 ------------------------------------------------------------------------------------ 5210 load atomic acquire - singlethread - global 1. buffer/global/ds/flat_load 5211 - wavefront - local 5212 - generic 5213 load atomic acquire - workgroup - global 1. buffer/global_load 5214 load atomic acquire - workgroup - local 1. ds/flat_load 5215 - generic 2. s_waitcnt lgkmcnt(0) 5216 5217 - If OpenCL, omit. 5218 - Must happen before 5219 any following 5220 global/generic 5221 load/load 5222 atomic/store/store 5223 atomic/atomicrmw. 5224 - Ensures any 5225 following global 5226 data read is no 5227 older than a local load 5228 atomic value being 5229 acquired. 5230 5231 load atomic acquire - agent - global 1. buffer/global_load 5232 - system glc=1 5233 2. s_waitcnt vmcnt(0) 5234 5235 - Must happen before 5236 following 5237 buffer_wbinvl1_vol. 5238 - Ensures the load 5239 has completed 5240 before invalidating 5241 the cache. 5242 5243 3. buffer_wbinvl1_vol 5244 5245 - Must happen before 5246 any following 5247 global/generic 5248 load/load 5249 atomic/atomicrmw. 5250 - Ensures that 5251 following 5252 loads will not see 5253 stale global data. 5254 5255 load atomic acquire - agent - generic 1. flat_load glc=1 5256 - system 2. s_waitcnt vmcnt(0) & 5257 lgkmcnt(0) 5258 5259 - If OpenCL omit 5260 lgkmcnt(0). 5261 - Must happen before 5262 following 5263 buffer_wbinvl1_vol. 5264 - Ensures the flat_load 5265 has completed 5266 before invalidating 5267 the cache. 5268 5269 3. buffer_wbinvl1_vol 5270 5271 - Must happen before 5272 any following 5273 global/generic 5274 load/load 5275 atomic/atomicrmw. 5276 - Ensures that 5277 following loads 5278 will not see stale 5279 global data. 5280 5281 atomicrmw acquire - singlethread - global 1. buffer/global/ds/flat_atomic 5282 - wavefront - local 5283 - generic 5284 atomicrmw acquire - workgroup - global 1. buffer/global_atomic 5285 atomicrmw acquire - workgroup - local 1. ds/flat_atomic 5286 - generic 2. s_waitcnt lgkmcnt(0) 5287 5288 - If OpenCL, omit. 5289 - Must happen before 5290 any following 5291 global/generic 5292 load/load 5293 atomic/store/store 5294 atomic/atomicrmw. 5295 - Ensures any 5296 following global 5297 data read is no 5298 older than a local 5299 atomicrmw value 5300 being acquired. 5301 5302 atomicrmw acquire - agent - global 1. buffer/global_atomic 5303 - system 2. s_waitcnt vmcnt(0) 5304 5305 - Must happen before 5306 following 5307 buffer_wbinvl1_vol. 5308 - Ensures the 5309 atomicrmw has 5310 completed before 5311 invalidating the 5312 cache. 5313 5314 3. buffer_wbinvl1_vol 5315 5316 - Must happen before 5317 any following 5318 global/generic 5319 load/load 5320 atomic/atomicrmw. 5321 - Ensures that 5322 following loads 5323 will not see stale 5324 global data. 5325 5326 atomicrmw acquire - agent - generic 1. flat_atomic 5327 - system 2. s_waitcnt vmcnt(0) & 5328 lgkmcnt(0) 5329 5330 - If OpenCL, omit 5331 lgkmcnt(0). 5332 - Must happen before 5333 following 5334 buffer_wbinvl1_vol. 5335 - Ensures the 5336 atomicrmw has 5337 completed before 5338 invalidating the 5339 cache. 5340 5341 3. buffer_wbinvl1_vol 5342 5343 - Must happen before 5344 any following 5345 global/generic 5346 load/load 5347 atomic/atomicrmw. 5348 - Ensures that 5349 following loads 5350 will not see stale 5351 global data. 5352 5353 fence acquire - singlethread *none* *none* 5354 - wavefront 5355 fence acquire - workgroup *none* 1. s_waitcnt lgkmcnt(0) 5356 5357 - If OpenCL and 5358 address space is 5359 not generic, omit. 5360 - However, since LLVM 5361 currently has no 5362 address space on 5363 the fence need to 5364 conservatively 5365 always generate. If 5366 fence had an 5367 address space then 5368 set to address 5369 space of OpenCL 5370 fence flag, or to 5371 generic if both 5372 local and global 5373 flags are 5374 specified. 5375 - Must happen after 5376 any preceding 5377 local/generic load 5378 atomic/atomicrmw 5379 with an equal or 5380 wider sync scope 5381 and memory ordering 5382 stronger than 5383 unordered (this is 5384 termed the 5385 fence-paired-atomic). 5386 - Must happen before 5387 any following 5388 global/generic 5389 load/load 5390 atomic/store/store 5391 atomic/atomicrmw. 5392 - Ensures any 5393 following global 5394 data read is no 5395 older than the 5396 value read by the 5397 fence-paired-atomic. 5398 5399 fence acquire - agent *none* 1. s_waitcnt lgkmcnt(0) & 5400 - system vmcnt(0) 5401 5402 - If OpenCL and 5403 address space is 5404 not generic, omit 5405 lgkmcnt(0). 5406 - However, since LLVM 5407 currently has no 5408 address space on 5409 the fence need to 5410 conservatively 5411 always generate 5412 (see comment for 5413 previous fence). 5414 - Could be split into 5415 separate s_waitcnt 5416 vmcnt(0) and 5417 s_waitcnt 5418 lgkmcnt(0) to allow 5419 them to be 5420 independently moved 5421 according to the 5422 following rules. 5423 - s_waitcnt vmcnt(0) 5424 must happen after 5425 any preceding 5426 global/generic load 5427 atomic/atomicrmw 5428 with an equal or 5429 wider sync scope 5430 and memory ordering 5431 stronger than 5432 unordered (this is 5433 termed the 5434 fence-paired-atomic). 5435 - s_waitcnt lgkmcnt(0) 5436 must happen after 5437 any preceding 5438 local/generic load 5439 atomic/atomicrmw 5440 with an equal or 5441 wider sync scope 5442 and memory ordering 5443 stronger than 5444 unordered (this is 5445 termed the 5446 fence-paired-atomic). 5447 - Must happen before 5448 the following 5449 buffer_wbinvl1_vol. 5450 - Ensures that the 5451 fence-paired atomic 5452 has completed 5453 before invalidating 5454 the 5455 cache. Therefore 5456 any following 5457 locations read must 5458 be no older than 5459 the value read by 5460 the 5461 fence-paired-atomic. 5462 5463 2. buffer_wbinvl1_vol 5464 5465 - Must happen before any 5466 following global/generic 5467 load/load 5468 atomic/store/store 5469 atomic/atomicrmw. 5470 - Ensures that 5471 following loads 5472 will not see stale 5473 global data. 5474 5475 **Release Atomic** 5476 ------------------------------------------------------------------------------------ 5477 store atomic release - singlethread - global 1. buffer/global/ds/flat_store 5478 - wavefront - local 5479 - generic 5480 store atomic release - workgroup - global 1. s_waitcnt lgkmcnt(0) 5481 - generic 5482 - If OpenCL, omit. 5483 - Must happen after 5484 any preceding 5485 local/generic 5486 load/store/load 5487 atomic/store 5488 atomic/atomicrmw. 5489 - Must happen before 5490 the following 5491 store. 5492 - Ensures that all 5493 memory operations 5494 to local have 5495 completed before 5496 performing the 5497 store that is being 5498 released. 5499 5500 2. buffer/global/flat_store 5501 store atomic release - workgroup - local 1. ds_store 5502 store atomic release - agent - global 1. s_waitcnt lgkmcnt(0) & 5503 - system - generic vmcnt(0) 5504 5505 - If OpenCL and 5506 address space is 5507 not generic, omit 5508 lgkmcnt(0). 5509 - Could be split into 5510 separate s_waitcnt 5511 vmcnt(0) and 5512 s_waitcnt 5513 lgkmcnt(0) to allow 5514 them to be 5515 independently moved 5516 according to the 5517 following rules. 5518 - s_waitcnt vmcnt(0) 5519 must happen after 5520 any preceding 5521 global/generic 5522 load/store/load 5523 atomic/store 5524 atomic/atomicrmw. 5525 - s_waitcnt lgkmcnt(0) 5526 must happen after 5527 any preceding 5528 local/generic 5529 load/store/load 5530 atomic/store 5531 atomic/atomicrmw. 5532 - Must happen before 5533 the following 5534 store. 5535 - Ensures that all 5536 memory operations 5537 to memory have 5538 completed before 5539 performing the 5540 store that is being 5541 released. 5542 5543 2. buffer/global/flat_store 5544 atomicrmw release - singlethread - global 1. buffer/global/ds/flat_atomic 5545 - wavefront - local 5546 - generic 5547 atomicrmw release - workgroup - global 1. s_waitcnt lgkmcnt(0) 5548 - generic 5549 - If OpenCL, omit. 5550 - Must happen after 5551 any preceding 5552 local/generic 5553 load/store/load 5554 atomic/store 5555 atomic/atomicrmw. 5556 - Must happen before 5557 the following 5558 atomicrmw. 5559 - Ensures that all 5560 memory operations 5561 to local have 5562 completed before 5563 performing the 5564 atomicrmw that is 5565 being released. 5566 5567 2. buffer/global/flat_atomic 5568 atomicrmw release - workgroup - local 1. ds_atomic 5569 atomicrmw release - agent - global 1. s_waitcnt lgkmcnt(0) & 5570 - system - generic vmcnt(0) 5571 5572 - If OpenCL, omit 5573 lgkmcnt(0). 5574 - Could be split into 5575 separate s_waitcnt 5576 vmcnt(0) and 5577 s_waitcnt 5578 lgkmcnt(0) to allow 5579 them to be 5580 independently moved 5581 according to the 5582 following rules. 5583 - s_waitcnt vmcnt(0) 5584 must happen after 5585 any preceding 5586 global/generic 5587 load/store/load 5588 atomic/store 5589 atomic/atomicrmw. 5590 - s_waitcnt lgkmcnt(0) 5591 must happen after 5592 any preceding 5593 local/generic 5594 load/store/load 5595 atomic/store 5596 atomic/atomicrmw. 5597 - Must happen before 5598 the following 5599 atomicrmw. 5600 - Ensures that all 5601 memory operations 5602 to global and local 5603 have completed 5604 before performing 5605 the atomicrmw that 5606 is being released. 5607 5608 2. buffer/global/flat_atomic 5609 fence release - singlethread *none* *none* 5610 - wavefront 5611 fence release - workgroup *none* 1. s_waitcnt lgkmcnt(0) 5612 5613 - If OpenCL and 5614 address space is 5615 not generic, omit. 5616 - However, since LLVM 5617 currently has no 5618 address space on 5619 the fence need to 5620 conservatively 5621 always generate. If 5622 fence had an 5623 address space then 5624 set to address 5625 space of OpenCL 5626 fence flag, or to 5627 generic if both 5628 local and global 5629 flags are 5630 specified. 5631 - Must happen after 5632 any preceding 5633 local/generic 5634 load/load 5635 atomic/store/store 5636 atomic/atomicrmw. 5637 - Must happen before 5638 any following store 5639 atomic/atomicrmw 5640 with an equal or 5641 wider sync scope 5642 and memory ordering 5643 stronger than 5644 unordered (this is 5645 termed the 5646 fence-paired-atomic). 5647 - Ensures that all 5648 memory operations 5649 to local have 5650 completed before 5651 performing the 5652 following 5653 fence-paired-atomic. 5654 5655 fence release - agent *none* 1. s_waitcnt lgkmcnt(0) & 5656 - system vmcnt(0) 5657 5658 - If OpenCL and 5659 address space is 5660 not generic, omit 5661 lgkmcnt(0). 5662 - If OpenCL and 5663 address space is 5664 local, omit 5665 vmcnt(0). 5666 - However, since LLVM 5667 currently has no 5668 address space on 5669 the fence need to 5670 conservatively 5671 always generate. If 5672 fence had an 5673 address space then 5674 set to address 5675 space of OpenCL 5676 fence flag, or to 5677 generic if both 5678 local and global 5679 flags are 5680 specified. 5681 - Could be split into 5682 separate s_waitcnt 5683 vmcnt(0) and 5684 s_waitcnt 5685 lgkmcnt(0) to allow 5686 them to be 5687 independently moved 5688 according to the 5689 following rules. 5690 - s_waitcnt vmcnt(0) 5691 must happen after 5692 any preceding 5693 global/generic 5694 load/store/load 5695 atomic/store 5696 atomic/atomicrmw. 5697 - s_waitcnt lgkmcnt(0) 5698 must happen after 5699 any preceding 5700 local/generic 5701 load/store/load 5702 atomic/store 5703 atomic/atomicrmw. 5704 - Must happen before 5705 any following store 5706 atomic/atomicrmw 5707 with an equal or 5708 wider sync scope 5709 and memory ordering 5710 stronger than 5711 unordered (this is 5712 termed the 5713 fence-paired-atomic). 5714 - Ensures that all 5715 memory operations 5716 have 5717 completed before 5718 performing the 5719 following 5720 fence-paired-atomic. 5721 5722 **Acquire-Release Atomic** 5723 ------------------------------------------------------------------------------------ 5724 atomicrmw acq_rel - singlethread - global 1. buffer/global/ds/flat_atomic 5725 - wavefront - local 5726 - generic 5727 atomicrmw acq_rel - workgroup - global 1. s_waitcnt lgkmcnt(0) 5728 5729 - If OpenCL, omit. 5730 - Must happen after 5731 any preceding 5732 local/generic 5733 load/store/load 5734 atomic/store 5735 atomic/atomicrmw. 5736 - Must happen before 5737 the following 5738 atomicrmw. 5739 - Ensures that all 5740 memory operations 5741 to local have 5742 completed before 5743 performing the 5744 atomicrmw that is 5745 being released. 5746 5747 2. buffer/global_atomic 5748 5749 atomicrmw acq_rel - workgroup - local 1. ds_atomic 5750 2. s_waitcnt lgkmcnt(0) 5751 5752 - If OpenCL, omit. 5753 - Must happen before 5754 any following 5755 global/generic 5756 load/load 5757 atomic/store/store 5758 atomic/atomicrmw. 5759 - Ensures any 5760 following global 5761 data read is no 5762 older than the local load 5763 atomic value being 5764 acquired. 5765 5766 atomicrmw acq_rel - workgroup - generic 1. s_waitcnt lgkmcnt(0) 5767 5768 - If OpenCL, omit. 5769 - Must happen after 5770 any preceding 5771 local/generic 5772 load/store/load 5773 atomic/store 5774 atomic/atomicrmw. 5775 - Must happen before 5776 the following 5777 atomicrmw. 5778 - Ensures that all 5779 memory operations 5780 to local have 5781 completed before 5782 performing the 5783 atomicrmw that is 5784 being released. 5785 5786 2. flat_atomic 5787 3. s_waitcnt lgkmcnt(0) 5788 5789 - If OpenCL, omit. 5790 - Must happen before 5791 any following 5792 global/generic 5793 load/load 5794 atomic/store/store 5795 atomic/atomicrmw. 5796 - Ensures any 5797 following global 5798 data read is no 5799 older than a local load 5800 atomic value being 5801 acquired. 5802 5803 atomicrmw acq_rel - agent - global 1. s_waitcnt lgkmcnt(0) & 5804 - system vmcnt(0) 5805 5806 - If OpenCL, omit 5807 lgkmcnt(0). 5808 - Could be split into 5809 separate s_waitcnt 5810 vmcnt(0) and 5811 s_waitcnt 5812 lgkmcnt(0) to allow 5813 them to be 5814 independently moved 5815 according to the 5816 following rules. 5817 - s_waitcnt vmcnt(0) 5818 must happen after 5819 any preceding 5820 global/generic 5821 load/store/load 5822 atomic/store 5823 atomic/atomicrmw. 5824 - s_waitcnt lgkmcnt(0) 5825 must happen after 5826 any preceding 5827 local/generic 5828 load/store/load 5829 atomic/store 5830 atomic/atomicrmw. 5831 - Must happen before 5832 the following 5833 atomicrmw. 5834 - Ensures that all 5835 memory operations 5836 to global have 5837 completed before 5838 performing the 5839 atomicrmw that is 5840 being released. 5841 5842 2. buffer/global_atomic 5843 3. s_waitcnt vmcnt(0) 5844 5845 - Must happen before 5846 following 5847 buffer_wbinvl1_vol. 5848 - Ensures the 5849 atomicrmw has 5850 completed before 5851 invalidating the 5852 cache. 5853 5854 4. buffer_wbinvl1_vol 5855 5856 - Must happen before 5857 any following 5858 global/generic 5859 load/load 5860 atomic/atomicrmw. 5861 - Ensures that 5862 following loads 5863 will not see stale 5864 global data. 5865 5866 atomicrmw acq_rel - agent - generic 1. s_waitcnt lgkmcnt(0) & 5867 - system vmcnt(0) 5868 5869 - If OpenCL, omit 5870 lgkmcnt(0). 5871 - Could be split into 5872 separate s_waitcnt 5873 vmcnt(0) and 5874 s_waitcnt 5875 lgkmcnt(0) to allow 5876 them to be 5877 independently moved 5878 according to the 5879 following rules. 5880 - s_waitcnt vmcnt(0) 5881 must happen after 5882 any preceding 5883 global/generic 5884 load/store/load 5885 atomic/store 5886 atomic/atomicrmw. 5887 - s_waitcnt lgkmcnt(0) 5888 must happen after 5889 any preceding 5890 local/generic 5891 load/store/load 5892 atomic/store 5893 atomic/atomicrmw. 5894 - Must happen before 5895 the following 5896 atomicrmw. 5897 - Ensures that all 5898 memory operations 5899 to global have 5900 completed before 5901 performing the 5902 atomicrmw that is 5903 being released. 5904 5905 2. flat_atomic 5906 3. s_waitcnt vmcnt(0) & 5907 lgkmcnt(0) 5908 5909 - If OpenCL, omit 5910 lgkmcnt(0). 5911 - Must happen before 5912 following 5913 buffer_wbinvl1_vol. 5914 - Ensures the 5915 atomicrmw has 5916 completed before 5917 invalidating the 5918 cache. 5919 5920 4. buffer_wbinvl1_vol 5921 5922 - Must happen before 5923 any following 5924 global/generic 5925 load/load 5926 atomic/atomicrmw. 5927 - Ensures that 5928 following loads 5929 will not see stale 5930 global data. 5931 5932 fence acq_rel - singlethread *none* *none* 5933 - wavefront 5934 fence acq_rel - workgroup *none* 1. s_waitcnt lgkmcnt(0) 5935 5936 - If OpenCL and 5937 address space is 5938 not generic, omit. 5939 - However, 5940 since LLVM 5941 currently has no 5942 address space on 5943 the fence need to 5944 conservatively 5945 always generate 5946 (see comment for 5947 previous fence). 5948 - Must happen after 5949 any preceding 5950 local/generic 5951 load/load 5952 atomic/store/store 5953 atomic/atomicrmw. 5954 - Must happen before 5955 any following 5956 global/generic 5957 load/load 5958 atomic/store/store 5959 atomic/atomicrmw. 5960 - Ensures that all 5961 memory operations 5962 to local have 5963 completed before 5964 performing any 5965 following global 5966 memory operations. 5967 - Ensures that the 5968 preceding 5969 local/generic load 5970 atomic/atomicrmw 5971 with an equal or 5972 wider sync scope 5973 and memory ordering 5974 stronger than 5975 unordered (this is 5976 termed the 5977 acquire-fence-paired-atomic) 5978 has completed 5979 before following 5980 global memory 5981 operations. This 5982 satisfies the 5983 requirements of 5984 acquire. 5985 - Ensures that all 5986 previous memory 5987 operations have 5988 completed before a 5989 following 5990 local/generic store 5991 atomic/atomicrmw 5992 with an equal or 5993 wider sync scope 5994 and memory ordering 5995 stronger than 5996 unordered (this is 5997 termed the 5998 release-fence-paired-atomic). 5999 This satisfies the 6000 requirements of 6001 release. 6002 6003 fence acq_rel - agent *none* 1. s_waitcnt lgkmcnt(0) & 6004 - system vmcnt(0) 6005 6006 - If OpenCL and 6007 address space is 6008 not generic, omit 6009 lgkmcnt(0). 6010 - However, since LLVM 6011 currently has no 6012 address space on 6013 the fence need to 6014 conservatively 6015 always generate 6016 (see comment for 6017 previous fence). 6018 - Could be split into 6019 separate s_waitcnt 6020 vmcnt(0) and 6021 s_waitcnt 6022 lgkmcnt(0) to allow 6023 them to be 6024 independently moved 6025 according to the 6026 following rules. 6027 - s_waitcnt vmcnt(0) 6028 must happen after 6029 any preceding 6030 global/generic 6031 load/store/load 6032 atomic/store 6033 atomic/atomicrmw. 6034 - s_waitcnt lgkmcnt(0) 6035 must happen after 6036 any preceding 6037 local/generic 6038 load/store/load 6039 atomic/store 6040 atomic/atomicrmw. 6041 - Must happen before 6042 the following 6043 buffer_wbinvl1_vol. 6044 - Ensures that the 6045 preceding 6046 global/local/generic 6047 load 6048 atomic/atomicrmw 6049 with an equal or 6050 wider sync scope 6051 and memory ordering 6052 stronger than 6053 unordered (this is 6054 termed the 6055 acquire-fence-paired-atomic) 6056 has completed 6057 before invalidating 6058 the cache. This 6059 satisfies the 6060 requirements of 6061 acquire. 6062 - Ensures that all 6063 previous memory 6064 operations have 6065 completed before a 6066 following 6067 global/local/generic 6068 store 6069 atomic/atomicrmw 6070 with an equal or 6071 wider sync scope 6072 and memory ordering 6073 stronger than 6074 unordered (this is 6075 termed the 6076 release-fence-paired-atomic). 6077 This satisfies the 6078 requirements of 6079 release. 6080 6081 2. buffer_wbinvl1_vol 6082 6083 - Must happen before 6084 any following 6085 global/generic 6086 load/load 6087 atomic/store/store 6088 atomic/atomicrmw. 6089 - Ensures that 6090 following loads 6091 will not see stale 6092 global data. This 6093 satisfies the 6094 requirements of 6095 acquire. 6096 6097 **Sequential Consistent Atomic** 6098 ------------------------------------------------------------------------------------ 6099 load atomic seq_cst - singlethread - global *Same as corresponding 6100 - wavefront - local load atomic acquire, 6101 - generic except must generate 6102 all instructions even 6103 for OpenCL.* 6104 load atomic seq_cst - workgroup - global 1. s_waitcnt lgkmcnt(0) 6105 - generic 6106 6107 - Must 6108 happen after 6109 preceding 6110 local/generic load 6111 atomic/store 6112 atomic/atomicrmw 6113 with memory 6114 ordering of seq_cst 6115 and with equal or 6116 wider sync scope. 6117 (Note that seq_cst 6118 fences have their 6119 own s_waitcnt 6120 lgkmcnt(0) and so do 6121 not need to be 6122 considered.) 6123 - Ensures any 6124 preceding 6125 sequential 6126 consistent local 6127 memory instructions 6128 have completed 6129 before executing 6130 this sequentially 6131 consistent 6132 instruction. This 6133 prevents reordering 6134 a seq_cst store 6135 followed by a 6136 seq_cst load. (Note 6137 that seq_cst is 6138 stronger than 6139 acquire/release as 6140 the reordering of 6141 load acquire 6142 followed by a store 6143 release is 6144 prevented by the 6145 s_waitcnt of 6146 the release, but 6147 there is nothing 6148 preventing a store 6149 release followed by 6150 load acquire from 6151 completing out of 6152 order. The s_waitcnt 6153 could be placed after 6154 seq_store or before 6155 the seq_load. We 6156 choose the load to 6157 make the s_waitcnt be 6158 as late as possible 6159 so that the store 6160 may have already 6161 completed.) 6162 6163 2. *Following 6164 instructions same as 6165 corresponding load 6166 atomic acquire, 6167 except must generate 6168 all instructions even 6169 for OpenCL.* 6170 load atomic seq_cst - workgroup - local *Same as corresponding 6171 load atomic acquire, 6172 except must generate 6173 all instructions even 6174 for OpenCL.* 6175 6176 load atomic seq_cst - agent - global 1. s_waitcnt lgkmcnt(0) & 6177 - system - generic vmcnt(0) 6178 6179 - Could be split into 6180 separate s_waitcnt 6181 vmcnt(0) 6182 and s_waitcnt 6183 lgkmcnt(0) to allow 6184 them to be 6185 independently moved 6186 according to the 6187 following rules. 6188 - s_waitcnt lgkmcnt(0) 6189 must happen after 6190 preceding 6191 global/generic load 6192 atomic/store 6193 atomic/atomicrmw 6194 with memory 6195 ordering of seq_cst 6196 and with equal or 6197 wider sync scope. 6198 (Note that seq_cst 6199 fences have their 6200 own s_waitcnt 6201 lgkmcnt(0) and so do 6202 not need to be 6203 considered.) 6204 - s_waitcnt vmcnt(0) 6205 must happen after 6206 preceding 6207 global/generic load 6208 atomic/store 6209 atomic/atomicrmw 6210 with memory 6211 ordering of seq_cst 6212 and with equal or 6213 wider sync scope. 6214 (Note that seq_cst 6215 fences have their 6216 own s_waitcnt 6217 vmcnt(0) and so do 6218 not need to be 6219 considered.) 6220 - Ensures any 6221 preceding 6222 sequential 6223 consistent global 6224 memory instructions 6225 have completed 6226 before executing 6227 this sequentially 6228 consistent 6229 instruction. This 6230 prevents reordering 6231 a seq_cst store 6232 followed by a 6233 seq_cst load. (Note 6234 that seq_cst is 6235 stronger than 6236 acquire/release as 6237 the reordering of 6238 load acquire 6239 followed by a store 6240 release is 6241 prevented by the 6242 s_waitcnt of 6243 the release, but 6244 there is nothing 6245 preventing a store 6246 release followed by 6247 load acquire from 6248 completing out of 6249 order. The s_waitcnt 6250 could be placed after 6251 seq_store or before 6252 the seq_load. We 6253 choose the load to 6254 make the s_waitcnt be 6255 as late as possible 6256 so that the store 6257 may have already 6258 completed.) 6259 6260 2. *Following 6261 instructions same as 6262 corresponding load 6263 atomic acquire, 6264 except must generate 6265 all instructions even 6266 for OpenCL.* 6267 store atomic seq_cst - singlethread - global *Same as corresponding 6268 - wavefront - local store atomic release, 6269 - workgroup - generic except must generate 6270 - agent all instructions even 6271 - system for OpenCL.* 6272 atomicrmw seq_cst - singlethread - global *Same as corresponding 6273 - wavefront - local atomicrmw acq_rel, 6274 - workgroup - generic except must generate 6275 - agent all instructions even 6276 - system for OpenCL.* 6277 fence seq_cst - singlethread *none* *Same as corresponding 6278 - wavefront fence acq_rel, 6279 - workgroup except must generate 6280 - agent all instructions even 6281 - system for OpenCL.* 6282 ============ ============ ============== ========== ================================ 6283 6284.. _amdgpu-amdhsa-memory-model-gfx90a: 6285 6286Memory Model GFX90A 6287+++++++++++++++++++ 6288 6289For GFX90A: 6290 6291* Each agent has multiple shader arrays (SA). 6292* Each SA has multiple compute units (CU). 6293* Each CU has multiple SIMDs that execute wavefronts. 6294* The wavefronts for a single work-group are executed in the same CU but may be 6295 executed by different SIMDs. The exception is when in tgsplit execution mode 6296 when the wavefronts may be executed by different SIMDs in different CUs. 6297* Each CU has a single LDS memory shared by the wavefronts of the work-groups 6298 executing on it. The exception is when in tgsplit execution mode when no LDS 6299 is allocated as wavefronts of the same work-group can be in different CUs. 6300* All LDS operations of a CU are performed as wavefront wide operations in a 6301 global order and involve no caching. Completion is reported to a wavefront in 6302 execution order. 6303* The LDS memory has multiple request queues shared by the SIMDs of a 6304 CU. Therefore, the LDS operations performed by different wavefronts of a 6305 work-group can be reordered relative to each other, which can result in 6306 reordering the visibility of vector memory operations with respect to LDS 6307 operations of other wavefronts in the same work-group. A ``s_waitcnt 6308 lgkmcnt(0)`` is required to ensure synchronization between LDS operations and 6309 vector memory operations between wavefronts of a work-group, but not between 6310 operations performed by the same wavefront. 6311* The vector memory operations are performed as wavefront wide operations and 6312 completion is reported to a wavefront in execution order. The exception is 6313 that ``flat_load/store/atomic`` instructions can report out of vector memory 6314 order if they access LDS memory, and out of LDS operation order if they access 6315 global memory. 6316* The vector memory operations access a single vector L1 cache shared by all 6317 SIMDs a CU. Therefore: 6318 6319 * No special action is required for coherence between the lanes of a single 6320 wavefront. 6321 6322 * No special action is required for coherence between wavefronts in the same 6323 work-group since they execute on the same CU. The exception is when in 6324 tgsplit execution mode as wavefronts of the same work-group can be in 6325 different CUs and so a ``buffer_wbinvl1_vol`` is required as described in 6326 the following item. 6327 6328 * A ``buffer_wbinvl1_vol`` is required for coherence between wavefronts 6329 executing in different work-groups as they may be executing on different 6330 CUs. 6331 6332* The scalar memory operations access a scalar L1 cache shared by all wavefronts 6333 on a group of CUs. The scalar and vector L1 caches are not coherent. However, 6334 scalar operations are used in a restricted way so do not impact the memory 6335 model. See :ref:`amdgpu-amdhsa-memory-spaces`. 6336* The vector and scalar memory operations use an L2 cache shared by all CUs on 6337 the same agent. 6338 6339 * The L2 cache has independent channels to service disjoint ranges of virtual 6340 addresses. 6341 * Each CU has a separate request queue per channel. Therefore, the vector and 6342 scalar memory operations performed by wavefronts executing in different 6343 work-groups (which may be executing on different CUs), or the same 6344 work-group if executing in tgsplit mode, of an agent can be reordered 6345 relative to each other. A ``s_waitcnt vmcnt(0)`` is required to ensure 6346 synchronization between vector memory operations of different CUs. It 6347 ensures a previous vector memory operation has completed before executing a 6348 subsequent vector memory or LDS operation and so can be used to meet the 6349 requirements of acquire and release. 6350 * The L2 cache of one agent can be kept coherent with other agents by: 6351 using the MTYPE RW (read-write) or MTYPE CC (cache-coherent) with the PTE 6352 C-bit for memory local to the L2; and using the MTYPE NC (non-coherent) with 6353 the PTE C-bit set or MTYPE UC (uncached) for memory not local to the L2. 6354 6355 * Any local memory cache lines will be automatically invalidated by writes 6356 from CUs associated with other L2 caches, or writes from the CPU, due to 6357 the cache probe caused by coherent requests. Coherent requests are caused 6358 by GPU accesses to pages with the PTE C-bit set, by CPU accesses over 6359 XGMI, and by PCIe requests that are configured to be coherent requests. 6360 * XGMI accesses from the CPU to local memory may be cached on the CPU. 6361 Subsequent access from the GPU will automatically invalidate or writeback 6362 the CPU cache due to the L2 probe filter and and the PTE C-bit being set. 6363 * Since all work-groups on the same agent share the same L2, no L2 6364 invalidation or writeback is required for coherence. 6365 * To ensure coherence of local and remote memory writes of work-groups in 6366 different agents a ``buffer_wbl2`` is required. It will writeback dirty L2 6367 cache lines of MTYPE RW (used for local coarse grain memory) and MTYPE NC 6368 ()used for remote coarse grain memory). Note that MTYPE CC (used for local 6369 fine grain memory) causes write through to DRAM, and MTYPE UC (used for 6370 remote fine grain memory) bypasses the L2, so both will never result in 6371 dirty L2 cache lines. 6372 * To ensure coherence of local and remote memory reads of work-groups in 6373 different agents a ``buffer_invl2`` is required. It will invalidate L2 6374 cache lines with MTYPE NC (used for remote coarse grain memory). Note that 6375 MTYPE CC (used for local fine grain memory) and MTYPE RW (used for local 6376 coarse memory) cause local reads to be invalidated by remote writes with 6377 with the PTE C-bit so these cache lines are not invalidated. Note that 6378 MTYPE UC (used for remote fine grain memory) bypasses the L2, so will 6379 never result in L2 cache lines that need to be invalidated. 6380 6381 * PCIe access from the GPU to the CPU memory is kept coherent by using the 6382 MTYPE UC (uncached) which bypasses the L2. 6383 6384Scalar memory operations are only used to access memory that is proven to not 6385change during the execution of the kernel dispatch. This includes constant 6386address space and global address space for program scope ``const`` variables. 6387Therefore, the kernel machine code does not have to maintain the scalar cache to 6388ensure it is coherent with the vector caches. The scalar and vector caches are 6389invalidated between kernel dispatches by CP since constant address space data 6390may change between kernel dispatch executions. See 6391:ref:`amdgpu-amdhsa-memory-spaces`. 6392 6393The one exception is if scalar writes are used to spill SGPR registers. In this 6394case the AMDGPU backend ensures the memory location used to spill is never 6395accessed by vector memory operations at the same time. If scalar writes are used 6396then a ``s_dcache_wb`` is inserted before the ``s_endpgm`` and before a function 6397return since the locations may be used for vector memory instructions by a 6398future wavefront that uses the same scratch area, or a function call that 6399creates a frame at the same address, respectively. There is no need for a 6400``s_dcache_inv`` as all scalar writes are write-before-read in the same thread. 6401 6402For kernarg backing memory: 6403 6404* CP invalidates the L1 cache at the start of each kernel dispatch. 6405* On dGPU over XGMI or PCIe the kernarg backing memory is allocated in host 6406 memory accessed as MTYPE UC (uncached) to avoid needing to invalidate the L2 6407 cache. This also causes it to be treated as non-volatile and so is not 6408 invalidated by ``*_vol``. 6409* On APU the kernarg backing memory is accessed as MTYPE CC (cache coherent) and 6410 so the L2 cache will be coherent with the CPU and other agents. 6411 6412Scratch backing memory (which is used for the private address space) is accessed 6413with MTYPE NC_NV (non-coherent non-volatile). Since the private address space is 6414only accessed by a single thread, and is always write-before-read, there is 6415never a need to invalidate these entries from the L1 cache. Hence all cache 6416invalidates are done as ``*_vol`` to only invalidate the volatile cache lines. 6417 6418The code sequences used to implement the memory model for GFX90A are defined 6419in table :ref:`amdgpu-amdhsa-memory-model-code-sequences-gfx90a-table`. 6420 6421 .. table:: AMDHSA Memory Model Code Sequences GFX90A 6422 :name: amdgpu-amdhsa-memory-model-code-sequences-gfx90a-table 6423 6424 ============ ============ ============== ========== ================================ 6425 LLVM Instr LLVM Memory LLVM Memory AMDGPU AMDGPU Machine Code 6426 Ordering Sync Scope Address GFX90A 6427 Space 6428 ============ ============ ============== ========== ================================ 6429 **Non-Atomic** 6430 ------------------------------------------------------------------------------------ 6431 load *none* *none* - global - !volatile & !nontemporal 6432 - generic 6433 - private 1. buffer/global/flat_load 6434 - constant 6435 - !volatile & nontemporal 6436 6437 1. buffer/global/flat_load 6438 glc=1 slc=1 6439 6440 - volatile 6441 6442 1. buffer/global/flat_load 6443 glc=1 6444 2. s_waitcnt vmcnt(0) 6445 6446 - Must happen before 6447 any following volatile 6448 global/generic 6449 load/store. 6450 - Ensures that 6451 volatile 6452 operations to 6453 different 6454 addresses will not 6455 be reordered by 6456 hardware. 6457 6458 load *none* *none* - local 1. ds_load 6459 store *none* *none* - global - !volatile & !nontemporal 6460 - generic 6461 - private 1. buffer/global/flat_store 6462 - constant 6463 - !volatile & nontemporal 6464 6465 1. buffer/global/flat_store 6466 glc=1 slc=1 6467 6468 - volatile 6469 6470 1. buffer/global/flat_store 6471 2. s_waitcnt vmcnt(0) 6472 6473 - Must happen before 6474 any following volatile 6475 global/generic 6476 load/store. 6477 - Ensures that 6478 volatile 6479 operations to 6480 different 6481 addresses will not 6482 be reordered by 6483 hardware. 6484 6485 store *none* *none* - local 1. ds_store 6486 **Unordered Atomic** 6487 ------------------------------------------------------------------------------------ 6488 load atomic unordered *any* *any* *Same as non-atomic*. 6489 store atomic unordered *any* *any* *Same as non-atomic*. 6490 atomicrmw unordered *any* *any* *Same as monotonic atomic*. 6491 **Monotonic Atomic** 6492 ------------------------------------------------------------------------------------ 6493 load atomic monotonic - singlethread - global 1. buffer/global/flat_load 6494 - wavefront - generic 6495 load atomic monotonic - workgroup - global 1. buffer/global/flat_load 6496 - generic glc=1 6497 6498 - If not TgSplit execution 6499 mode, omit glc=1. 6500 6501 load atomic monotonic - singlethread - local *If TgSplit execution mode, 6502 - wavefront local address space cannot 6503 - workgroup be used.* 6504 6505 1. ds_load 6506 load atomic monotonic - agent - global 1. buffer/global/flat_load 6507 - generic glc=1 6508 load atomic monotonic - system - global 1. buffer/global/flat_load 6509 - generic glc=1 6510 store atomic monotonic - singlethread - global 1. buffer/global/flat_store 6511 - wavefront - generic 6512 - workgroup 6513 - agent 6514 store atomic monotonic - system - global 1. buffer/global/flat_store 6515 - generic 6516 store atomic monotonic - singlethread - local *If TgSplit execution mode, 6517 - wavefront local address space cannot 6518 - workgroup be used.* 6519 6520 1. ds_store 6521 atomicrmw monotonic - singlethread - global 1. buffer/global/flat_atomic 6522 - wavefront - generic 6523 - workgroup 6524 - agent 6525 atomicrmw monotonic - system - global 1. buffer/global/flat_atomic 6526 - generic 6527 atomicrmw monotonic - singlethread - local *If TgSplit execution mode, 6528 - wavefront local address space cannot 6529 - workgroup be used.* 6530 6531 1. ds_atomic 6532 **Acquire Atomic** 6533 ------------------------------------------------------------------------------------ 6534 load atomic acquire - singlethread - global 1. buffer/global/ds/flat_load 6535 - wavefront - local 6536 - generic 6537 load atomic acquire - workgroup - global 1. buffer/global_load glc=1 6538 6539 - If not TgSplit execution 6540 mode, omit glc=1. 6541 6542 2. s_waitcnt vmcnt(0) 6543 6544 - If not TgSplit execution 6545 mode, omit. 6546 - Must happen before the 6547 following buffer_wbinvl1_vol. 6548 6549 3. buffer_wbinvl1_vol 6550 6551 - If not TgSplit execution 6552 mode, omit. 6553 - Must happen before 6554 any following 6555 global/generic 6556 load/load 6557 atomic/store/store 6558 atomic/atomicrmw. 6559 - Ensures that 6560 following 6561 loads will not see 6562 stale data. 6563 6564 load atomic acquire - workgroup - local *If TgSplit execution mode, 6565 local address space cannot 6566 be used.* 6567 6568 1. ds_load 6569 2. s_waitcnt lgkmcnt(0) 6570 6571 - If OpenCL, omit. 6572 - Must happen before 6573 any following 6574 global/generic 6575 load/load 6576 atomic/store/store 6577 atomic/atomicrmw. 6578 - Ensures any 6579 following global 6580 data read is no 6581 older than the local load 6582 atomic value being 6583 acquired. 6584 6585 load atomic acquire - workgroup - generic 1. flat_load glc=1 6586 6587 - If not TgSplit execution 6588 mode, omit glc=1. 6589 6590 2. s_waitcnt lgkm/vmcnt(0) 6591 6592 - Use lgkmcnt(0) if not 6593 TgSplit execution mode 6594 and vmcnt(0) if TgSplit 6595 execution mode. 6596 - If OpenCL, omit lgkmcnt(0). 6597 - Must happen before 6598 the following 6599 buffer_wbinvl1_vol and any 6600 following global/generic 6601 load/load 6602 atomic/store/store 6603 atomic/atomicrmw. 6604 - Ensures any 6605 following global 6606 data read is no 6607 older than a local load 6608 atomic value being 6609 acquired. 6610 6611 3. buffer_wbinvl1_vol 6612 6613 - If not TgSplit execution 6614 mode, omit. 6615 - Ensures that 6616 following 6617 loads will not see 6618 stale data. 6619 6620 load atomic acquire - agent - global 1. buffer/global_load 6621 glc=1 6622 2. s_waitcnt vmcnt(0) 6623 6624 - Must happen before 6625 following 6626 buffer_wbinvl1_vol. 6627 - Ensures the load 6628 has completed 6629 before invalidating 6630 the cache. 6631 6632 3. buffer_wbinvl1_vol 6633 6634 - Must happen before 6635 any following 6636 global/generic 6637 load/load 6638 atomic/atomicrmw. 6639 - Ensures that 6640 following 6641 loads will not see 6642 stale global data. 6643 6644 load atomic acquire - system - global 1. buffer/global/flat_load 6645 glc=1 6646 2. s_waitcnt vmcnt(0) 6647 6648 - Must happen before 6649 following buffer_invl2 and 6650 buffer_wbinvl1_vol. 6651 - Ensures the load 6652 has completed 6653 before invalidating 6654 the cache. 6655 6656 3. buffer_invl2; 6657 buffer_wbinvl1_vol 6658 6659 - Must happen before 6660 any following 6661 global/generic 6662 load/load 6663 atomic/atomicrmw. 6664 - Ensures that 6665 following 6666 loads will not see 6667 stale L1 global data, 6668 nor see stale L2 MTYPE 6669 NC global data. 6670 MTYPE RW and CC memory will 6671 never be stale in L2 due to 6672 the memory probes. 6673 6674 load atomic acquire - agent - generic 1. flat_load glc=1 6675 2. s_waitcnt vmcnt(0) & 6676 lgkmcnt(0) 6677 6678 - If TgSplit execution mode, 6679 omit lgkmcnt(0). 6680 - If OpenCL omit 6681 lgkmcnt(0). 6682 - Must happen before 6683 following 6684 buffer_wbinvl1_vol. 6685 - Ensures the flat_load 6686 has completed 6687 before invalidating 6688 the cache. 6689 6690 3. buffer_wbinvl1_vol 6691 6692 - Must happen before 6693 any following 6694 global/generic 6695 load/load 6696 atomic/atomicrmw. 6697 - Ensures that 6698 following loads 6699 will not see stale 6700 global data. 6701 6702 load atomic acquire - system - generic 1. flat_load glc=1 6703 2. s_waitcnt vmcnt(0) & 6704 lgkmcnt(0) 6705 6706 - If TgSplit execution mode, 6707 omit lgkmcnt(0). 6708 - If OpenCL omit 6709 lgkmcnt(0). 6710 - Must happen before 6711 following 6712 buffer_invl2 and 6713 buffer_wbinvl1_vol. 6714 - Ensures the flat_load 6715 has completed 6716 before invalidating 6717 the caches. 6718 6719 3. buffer_invl2; 6720 buffer_wbinvl1_vol 6721 6722 - Must happen before 6723 any following 6724 global/generic 6725 load/load 6726 atomic/atomicrmw. 6727 - Ensures that 6728 following 6729 loads will not see 6730 stale L1 global data, 6731 nor see stale L2 MTYPE 6732 NC global data. 6733 MTYPE RW and CC memory will 6734 never be stale in L2 due to 6735 the memory probes. 6736 6737 atomicrmw acquire - singlethread - global 1. buffer/global/flat_atomic 6738 - wavefront - generic 6739 atomicrmw acquire - singlethread - local *If TgSplit execution mode, 6740 - wavefront local address space cannot 6741 be used.* 6742 6743 1. ds_atomic 6744 atomicrmw acquire - workgroup - global 1. buffer/global_atomic 6745 2. s_waitcnt vmcnt(0) 6746 6747 - If not TgSplit execution 6748 mode, omit. 6749 - Must happen before the 6750 following buffer_wbinvl1_vol. 6751 - Ensures the atomicrmw 6752 has completed 6753 before invalidating 6754 the cache. 6755 6756 3. buffer_wbinvl1_vol 6757 6758 - If not TgSplit execution 6759 mode, omit. 6760 - Must happen before 6761 any following 6762 global/generic 6763 load/load 6764 atomic/atomicrmw. 6765 - Ensures that 6766 following loads 6767 will not see stale 6768 global data. 6769 6770 atomicrmw acquire - workgroup - local *If TgSplit execution mode, 6771 local address space cannot 6772 be used.* 6773 6774 1. ds_atomic 6775 2. s_waitcnt lgkmcnt(0) 6776 6777 - If OpenCL, omit. 6778 - Must happen before 6779 any following 6780 global/generic 6781 load/load 6782 atomic/store/store 6783 atomic/atomicrmw. 6784 - Ensures any 6785 following global 6786 data read is no 6787 older than the local 6788 atomicrmw value 6789 being acquired. 6790 6791 atomicrmw acquire - workgroup - generic 1. flat_atomic 6792 2. s_waitcnt lgkm/vmcnt(0) 6793 6794 - Use lgkmcnt(0) if not 6795 TgSplit execution mode 6796 and vmcnt(0) if TgSplit 6797 execution mode. 6798 - If OpenCL, omit lgkmcnt(0). 6799 - Must happen before 6800 the following 6801 buffer_wbinvl1_vol and 6802 any following 6803 global/generic 6804 load/load 6805 atomic/store/store 6806 atomic/atomicrmw. 6807 - Ensures any 6808 following global 6809 data read is no 6810 older than a local 6811 atomicrmw value 6812 being acquired. 6813 6814 3. buffer_wbinvl1_vol 6815 6816 - If not TgSplit execution 6817 mode, omit. 6818 - Ensures that 6819 following 6820 loads will not see 6821 stale data. 6822 6823 atomicrmw acquire - agent - global 1. buffer/global_atomic 6824 2. s_waitcnt vmcnt(0) 6825 6826 - Must happen before 6827 following 6828 buffer_wbinvl1_vol. 6829 - Ensures the 6830 atomicrmw has 6831 completed before 6832 invalidating the 6833 cache. 6834 6835 3. buffer_wbinvl1_vol 6836 6837 - Must happen before 6838 any following 6839 global/generic 6840 load/load 6841 atomic/atomicrmw. 6842 - Ensures that 6843 following loads 6844 will not see stale 6845 global data. 6846 6847 atomicrmw acquire - system - global 1. buffer/global_atomic 6848 2. s_waitcnt vmcnt(0) 6849 6850 - Must happen before 6851 following buffer_invl2 and 6852 buffer_wbinvl1_vol. 6853 - Ensures the 6854 atomicrmw has 6855 completed before 6856 invalidating the 6857 caches. 6858 6859 3. buffer_invl2; 6860 buffer_wbinvl1_vol 6861 6862 - Must happen before 6863 any following 6864 global/generic 6865 load/load 6866 atomic/atomicrmw. 6867 - Ensures that 6868 following 6869 loads will not see 6870 stale L1 global data, 6871 nor see stale L2 MTYPE 6872 NC global data. 6873 MTYPE RW and CC memory will 6874 never be stale in L2 due to 6875 the memory probes. 6876 6877 atomicrmw acquire - agent - generic 1. flat_atomic 6878 2. s_waitcnt vmcnt(0) & 6879 lgkmcnt(0) 6880 6881 - If TgSplit execution mode, 6882 omit lgkmcnt(0). 6883 - If OpenCL, omit 6884 lgkmcnt(0). 6885 - Must happen before 6886 following 6887 buffer_wbinvl1_vol. 6888 - Ensures the 6889 atomicrmw has 6890 completed before 6891 invalidating the 6892 cache. 6893 6894 3. buffer_wbinvl1_vol 6895 6896 - Must happen before 6897 any following 6898 global/generic 6899 load/load 6900 atomic/atomicrmw. 6901 - Ensures that 6902 following loads 6903 will not see stale 6904 global data. 6905 6906 atomicrmw acquire - system - generic 1. flat_atomic 6907 2. s_waitcnt vmcnt(0) & 6908 lgkmcnt(0) 6909 6910 - If TgSplit execution mode, 6911 omit lgkmcnt(0). 6912 - If OpenCL, omit 6913 lgkmcnt(0). 6914 - Must happen before 6915 following 6916 buffer_invl2 and 6917 buffer_wbinvl1_vol. 6918 - Ensures the 6919 atomicrmw has 6920 completed before 6921 invalidating the 6922 caches. 6923 6924 3. buffer_invl2; 6925 buffer_wbinvl1_vol 6926 6927 - Must happen before 6928 any following 6929 global/generic 6930 load/load 6931 atomic/atomicrmw. 6932 - Ensures that 6933 following 6934 loads will not see 6935 stale L1 global data, 6936 nor see stale L2 MTYPE 6937 NC global data. 6938 MTYPE RW and CC memory will 6939 never be stale in L2 due to 6940 the memory probes. 6941 6942 fence acquire - singlethread *none* *none* 6943 - wavefront 6944 fence acquire - workgroup *none* 1. s_waitcnt lgkm/vmcnt(0) 6945 6946 - Use lgkmcnt(0) if not 6947 TgSplit execution mode 6948 and vmcnt(0) if TgSplit 6949 execution mode. 6950 - If OpenCL and 6951 address space is 6952 not generic, omit 6953 lgkmcnt(0). 6954 - If OpenCL and 6955 address space is 6956 local, omit 6957 vmcnt(0). 6958 - However, since LLVM 6959 currently has no 6960 address space on 6961 the fence need to 6962 conservatively 6963 always generate. If 6964 fence had an 6965 address space then 6966 set to address 6967 space of OpenCL 6968 fence flag, or to 6969 generic if both 6970 local and global 6971 flags are 6972 specified. 6973 - s_waitcnt vmcnt(0) 6974 must happen after 6975 any preceding 6976 global/generic load 6977 atomic/ 6978 atomicrmw 6979 with an equal or 6980 wider sync scope 6981 and memory ordering 6982 stronger than 6983 unordered (this is 6984 termed the 6985 fence-paired-atomic). 6986 - s_waitcnt lgkmcnt(0) 6987 must happen after 6988 any preceding 6989 local/generic load 6990 atomic/atomicrmw 6991 with an equal or 6992 wider sync scope 6993 and memory ordering 6994 stronger than 6995 unordered (this is 6996 termed the 6997 fence-paired-atomic). 6998 - Must happen before 6999 the following 7000 buffer_wbinvl1_vol and 7001 any following 7002 global/generic 7003 load/load 7004 atomic/store/store 7005 atomic/atomicrmw. 7006 - Ensures any 7007 following global 7008 data read is no 7009 older than the 7010 value read by the 7011 fence-paired-atomic. 7012 7013 2. buffer_wbinvl1_vol 7014 7015 - If not TgSplit execution 7016 mode, omit. 7017 - Ensures that 7018 following 7019 loads will not see 7020 stale data. 7021 7022 fence acquire - agent *none* 1. s_waitcnt lgkmcnt(0) & 7023 vmcnt(0) 7024 7025 - If TgSplit execution mode, 7026 omit lgkmcnt(0). 7027 - If OpenCL and 7028 address space is 7029 not generic, omit 7030 lgkmcnt(0). 7031 - However, since LLVM 7032 currently has no 7033 address space on 7034 the fence need to 7035 conservatively 7036 always generate 7037 (see comment for 7038 previous fence). 7039 - Could be split into 7040 separate s_waitcnt 7041 vmcnt(0) and 7042 s_waitcnt 7043 lgkmcnt(0) to allow 7044 them to be 7045 independently moved 7046 according to the 7047 following rules. 7048 - s_waitcnt vmcnt(0) 7049 must happen after 7050 any preceding 7051 global/generic load 7052 atomic/atomicrmw 7053 with an equal or 7054 wider sync scope 7055 and memory ordering 7056 stronger than 7057 unordered (this is 7058 termed the 7059 fence-paired-atomic). 7060 - s_waitcnt lgkmcnt(0) 7061 must happen after 7062 any preceding 7063 local/generic load 7064 atomic/atomicrmw 7065 with an equal or 7066 wider sync scope 7067 and memory ordering 7068 stronger than 7069 unordered (this is 7070 termed the 7071 fence-paired-atomic). 7072 - Must happen before 7073 the following 7074 buffer_wbinvl1_vol. 7075 - Ensures that the 7076 fence-paired atomic 7077 has completed 7078 before invalidating 7079 the 7080 cache. Therefore 7081 any following 7082 locations read must 7083 be no older than 7084 the value read by 7085 the 7086 fence-paired-atomic. 7087 7088 2. buffer_wbinvl1_vol 7089 7090 - Must happen before any 7091 following global/generic 7092 load/load 7093 atomic/store/store 7094 atomic/atomicrmw. 7095 - Ensures that 7096 following loads 7097 will not see stale 7098 global data. 7099 7100 fence acquire - system *none* 1. s_waitcnt lgkmcnt(0) & 7101 vmcnt(0) 7102 7103 - If TgSplit execution mode, 7104 omit lgkmcnt(0). 7105 - If OpenCL and 7106 address space is 7107 not generic, omit 7108 lgkmcnt(0). 7109 - However, since LLVM 7110 currently has no 7111 address space on 7112 the fence need to 7113 conservatively 7114 always generate 7115 (see comment for 7116 previous fence). 7117 - Could be split into 7118 separate s_waitcnt 7119 vmcnt(0) and 7120 s_waitcnt 7121 lgkmcnt(0) to allow 7122 them to be 7123 independently moved 7124 according to the 7125 following rules. 7126 - s_waitcnt vmcnt(0) 7127 must happen after 7128 any preceding 7129 global/generic load 7130 atomic/atomicrmw 7131 with an equal or 7132 wider sync scope 7133 and memory ordering 7134 stronger than 7135 unordered (this is 7136 termed the 7137 fence-paired-atomic). 7138 - s_waitcnt lgkmcnt(0) 7139 must happen after 7140 any preceding 7141 local/generic load 7142 atomic/atomicrmw 7143 with an equal or 7144 wider sync scope 7145 and memory ordering 7146 stronger than 7147 unordered (this is 7148 termed the 7149 fence-paired-atomic). 7150 - Must happen before 7151 the following buffer_invl2 and 7152 buffer_wbinvl1_vol. 7153 - Ensures that the 7154 fence-paired atomic 7155 has completed 7156 before invalidating 7157 the 7158 cache. Therefore 7159 any following 7160 locations read must 7161 be no older than 7162 the value read by 7163 the 7164 fence-paired-atomic. 7165 7166 2. buffer_invl2; 7167 buffer_wbinvl1_vol 7168 7169 - Must happen before any 7170 following global/generic 7171 load/load 7172 atomic/store/store 7173 atomic/atomicrmw. 7174 - Ensures that 7175 following 7176 loads will not see 7177 stale L1 global data, 7178 nor see stale L2 MTYPE 7179 NC global data. 7180 MTYPE RW and CC memory will 7181 never be stale in L2 due to 7182 the memory probes. 7183 **Release Atomic** 7184 ------------------------------------------------------------------------------------ 7185 store atomic release - singlethread - global 1. buffer/global/flat_store 7186 - wavefront - generic 7187 store atomic release - singlethread - local *If TgSplit execution mode, 7188 - wavefront local address space cannot 7189 be used.* 7190 7191 1. ds_store 7192 store atomic release - workgroup - global 1. s_waitcnt lgkm/vmcnt(0) 7193 - generic 7194 - Use lgkmcnt(0) if not 7195 TgSplit execution mode 7196 and vmcnt(0) if TgSplit 7197 execution mode. 7198 - If OpenCL, omit lgkmcnt(0). 7199 - s_waitcnt vmcnt(0) 7200 must happen after 7201 any preceding 7202 global/generic load/store/ 7203 load atomic/store atomic/ 7204 atomicrmw. 7205 - s_waitcnt lgkmcnt(0) 7206 must happen after 7207 any preceding 7208 local/generic 7209 load/store/load 7210 atomic/store 7211 atomic/atomicrmw. 7212 - Must happen before 7213 the following 7214 store. 7215 - Ensures that all 7216 memory operations 7217 have 7218 completed before 7219 performing the 7220 store that is being 7221 released. 7222 7223 2. buffer/global/flat_store 7224 store atomic release - workgroup - local *If TgSplit execution mode, 7225 local address space cannot 7226 be used.* 7227 7228 1. ds_store 7229 store atomic release - agent - global 1. s_waitcnt lgkmcnt(0) & 7230 - generic vmcnt(0) 7231 7232 - If TgSplit execution mode, 7233 omit lgkmcnt(0). 7234 - If OpenCL and 7235 address space is 7236 not generic, omit 7237 lgkmcnt(0). 7238 - Could be split into 7239 separate s_waitcnt 7240 vmcnt(0) and 7241 s_waitcnt 7242 lgkmcnt(0) to allow 7243 them to be 7244 independently moved 7245 according to the 7246 following rules. 7247 - s_waitcnt vmcnt(0) 7248 must happen after 7249 any preceding 7250 global/generic 7251 load/store/load 7252 atomic/store 7253 atomic/atomicrmw. 7254 - s_waitcnt lgkmcnt(0) 7255 must happen after 7256 any preceding 7257 local/generic 7258 load/store/load 7259 atomic/store 7260 atomic/atomicrmw. 7261 - Must happen before 7262 the following 7263 store. 7264 - Ensures that all 7265 memory operations 7266 to memory have 7267 completed before 7268 performing the 7269 store that is being 7270 released. 7271 7272 2. buffer/global/flat_store 7273 store atomic release - system - global 1. buffer_wbl2 7274 - generic 7275 - Must happen before 7276 following s_waitcnt. 7277 - Performs L2 writeback to 7278 ensure previous 7279 global/generic 7280 store/atomicrmw are 7281 visible at system scope. 7282 7283 2. s_waitcnt lgkmcnt(0) & 7284 vmcnt(0) 7285 7286 - If TgSplit execution mode, 7287 omit lgkmcnt(0). 7288 - If OpenCL and 7289 address space is 7290 not generic, omit 7291 lgkmcnt(0). 7292 - Could be split into 7293 separate s_waitcnt 7294 vmcnt(0) and 7295 s_waitcnt 7296 lgkmcnt(0) to allow 7297 them to be 7298 independently moved 7299 according to the 7300 following rules. 7301 - s_waitcnt vmcnt(0) 7302 must happen after any 7303 preceding 7304 global/generic 7305 load/store/load 7306 atomic/store 7307 atomic/atomicrmw. 7308 - s_waitcnt lgkmcnt(0) 7309 must happen after any 7310 preceding 7311 local/generic 7312 load/store/load 7313 atomic/store 7314 atomic/atomicrmw. 7315 - Must happen before 7316 the following 7317 store. 7318 - Ensures that all 7319 memory operations 7320 to memory and the L2 7321 writeback have 7322 completed before 7323 performing the 7324 store that is being 7325 released. 7326 7327 3. buffer/global/flat_store 7328 atomicrmw release - singlethread - global 1. buffer/global/flat_atomic 7329 - wavefront - generic 7330 atomicrmw release - singlethread - local *If TgSplit execution mode, 7331 - wavefront local address space cannot 7332 be used.* 7333 7334 1. ds_atomic 7335 atomicrmw release - workgroup - global 1. s_waitcnt lgkm/vmcnt(0) 7336 - generic 7337 - Use lgkmcnt(0) if not 7338 TgSplit execution mode 7339 and vmcnt(0) if TgSplit 7340 execution mode. 7341 - If OpenCL, omit 7342 lgkmcnt(0). 7343 - s_waitcnt vmcnt(0) 7344 must happen after 7345 any preceding 7346 global/generic load/store/ 7347 load atomic/store atomic/ 7348 atomicrmw. 7349 - s_waitcnt lgkmcnt(0) 7350 must happen after 7351 any preceding 7352 local/generic 7353 load/store/load 7354 atomic/store 7355 atomic/atomicrmw. 7356 - Must happen before 7357 the following 7358 atomicrmw. 7359 - Ensures that all 7360 memory operations 7361 have 7362 completed before 7363 performing the 7364 atomicrmw that is 7365 being released. 7366 7367 2. buffer/global/flat_atomic 7368 atomicrmw release - workgroup - local *If TgSplit execution mode, 7369 local address space cannot 7370 be used.* 7371 7372 1. ds_atomic 7373 atomicrmw release - agent - global 1. s_waitcnt lgkmcnt(0) & 7374 - generic vmcnt(0) 7375 7376 - If TgSplit execution mode, 7377 omit lgkmcnt(0). 7378 - If OpenCL, omit 7379 lgkmcnt(0). 7380 - Could be split into 7381 separate s_waitcnt 7382 vmcnt(0) and 7383 s_waitcnt 7384 lgkmcnt(0) to allow 7385 them to be 7386 independently moved 7387 according to the 7388 following rules. 7389 - s_waitcnt vmcnt(0) 7390 must happen after 7391 any preceding 7392 global/generic 7393 load/store/load 7394 atomic/store 7395 atomic/atomicrmw. 7396 - s_waitcnt lgkmcnt(0) 7397 must happen after 7398 any preceding 7399 local/generic 7400 load/store/load 7401 atomic/store 7402 atomic/atomicrmw. 7403 - Must happen before 7404 the following 7405 atomicrmw. 7406 - Ensures that all 7407 memory operations 7408 to global and local 7409 have completed 7410 before performing 7411 the atomicrmw that 7412 is being released. 7413 7414 2. buffer/global/flat_atomic 7415 atomicrmw release - system - global 1. buffer_wbl2 7416 - generic 7417 - Must happen before 7418 following s_waitcnt. 7419 - Performs L2 writeback to 7420 ensure previous 7421 global/generic 7422 store/atomicrmw are 7423 visible at system scope. 7424 7425 2. s_waitcnt lgkmcnt(0) & 7426 vmcnt(0) 7427 7428 - If TgSplit execution mode, 7429 omit lgkmcnt(0). 7430 - If OpenCL, omit 7431 lgkmcnt(0). 7432 - Could be split into 7433 separate s_waitcnt 7434 vmcnt(0) and 7435 s_waitcnt 7436 lgkmcnt(0) to allow 7437 them to be 7438 independently moved 7439 according to the 7440 following rules. 7441 - s_waitcnt vmcnt(0) 7442 must happen after 7443 any preceding 7444 global/generic 7445 load/store/load 7446 atomic/store 7447 atomic/atomicrmw. 7448 - s_waitcnt lgkmcnt(0) 7449 must happen after 7450 any preceding 7451 local/generic 7452 load/store/load 7453 atomic/store 7454 atomic/atomicrmw. 7455 - Must happen before 7456 the following 7457 atomicrmw. 7458 - Ensures that all 7459 memory operations 7460 to memory and the L2 7461 writeback have 7462 completed before 7463 performing the 7464 store that is being 7465 released. 7466 7467 3. buffer/global/flat_atomic 7468 fence release - singlethread *none* *none* 7469 - wavefront 7470 fence release - workgroup *none* 1. s_waitcnt lgkm/vmcnt(0) 7471 7472 - Use lgkmcnt(0) if not 7473 TgSplit execution mode 7474 and vmcnt(0) if TgSplit 7475 execution mode. 7476 - If OpenCL and 7477 address space is 7478 not generic, omit 7479 lgkmcnt(0). 7480 - If OpenCL and 7481 address space is 7482 local, omit 7483 vmcnt(0). 7484 - However, since LLVM 7485 currently has no 7486 address space on 7487 the fence need to 7488 conservatively 7489 always generate. If 7490 fence had an 7491 address space then 7492 set to address 7493 space of OpenCL 7494 fence flag, or to 7495 generic if both 7496 local and global 7497 flags are 7498 specified. 7499 - s_waitcnt vmcnt(0) 7500 must happen after 7501 any preceding 7502 global/generic 7503 load/store/ 7504 load atomic/store atomic/ 7505 atomicrmw. 7506 - s_waitcnt lgkmcnt(0) 7507 must happen after 7508 any preceding 7509 local/generic 7510 load/load 7511 atomic/store/store 7512 atomic/atomicrmw. 7513 - Must happen before 7514 any following store 7515 atomic/atomicrmw 7516 with an equal or 7517 wider sync scope 7518 and memory ordering 7519 stronger than 7520 unordered (this is 7521 termed the 7522 fence-paired-atomic). 7523 - Ensures that all 7524 memory operations 7525 have 7526 completed before 7527 performing the 7528 following 7529 fence-paired-atomic. 7530 7531 fence release - agent *none* 1. s_waitcnt lgkmcnt(0) & 7532 vmcnt(0) 7533 7534 - If TgSplit execution mode, 7535 omit lgkmcnt(0). 7536 - If OpenCL and 7537 address space is 7538 not generic, omit 7539 lgkmcnt(0). 7540 - If OpenCL and 7541 address space is 7542 local, omit 7543 vmcnt(0). 7544 - However, since LLVM 7545 currently has no 7546 address space on 7547 the fence need to 7548 conservatively 7549 always generate. If 7550 fence had an 7551 address space then 7552 set to address 7553 space of OpenCL 7554 fence flag, or to 7555 generic if both 7556 local and global 7557 flags are 7558 specified. 7559 - Could be split into 7560 separate s_waitcnt 7561 vmcnt(0) and 7562 s_waitcnt 7563 lgkmcnt(0) to allow 7564 them to be 7565 independently moved 7566 according to the 7567 following rules. 7568 - s_waitcnt vmcnt(0) 7569 must happen after 7570 any preceding 7571 global/generic 7572 load/store/load 7573 atomic/store 7574 atomic/atomicrmw. 7575 - s_waitcnt lgkmcnt(0) 7576 must happen after 7577 any preceding 7578 local/generic 7579 load/store/load 7580 atomic/store 7581 atomic/atomicrmw. 7582 - Must happen before 7583 any following store 7584 atomic/atomicrmw 7585 with an equal or 7586 wider sync scope 7587 and memory ordering 7588 stronger than 7589 unordered (this is 7590 termed the 7591 fence-paired-atomic). 7592 - Ensures that all 7593 memory operations 7594 have 7595 completed before 7596 performing the 7597 following 7598 fence-paired-atomic. 7599 7600 fence release - system *none* 1. buffer_wbl2 7601 7602 - If OpenCL and 7603 address space is 7604 local, omit. 7605 - Must happen before 7606 following s_waitcnt. 7607 - Performs L2 writeback to 7608 ensure previous 7609 global/generic 7610 store/atomicrmw are 7611 visible at system scope. 7612 7613 2. s_waitcnt lgkmcnt(0) & 7614 vmcnt(0) 7615 7616 - If TgSplit execution mode, 7617 omit lgkmcnt(0). 7618 - If OpenCL and 7619 address space is 7620 not generic, omit 7621 lgkmcnt(0). 7622 - If OpenCL and 7623 address space is 7624 local, omit 7625 vmcnt(0). 7626 - However, since LLVM 7627 currently has no 7628 address space on 7629 the fence need to 7630 conservatively 7631 always generate. If 7632 fence had an 7633 address space then 7634 set to address 7635 space of OpenCL 7636 fence flag, or to 7637 generic if both 7638 local and global 7639 flags are 7640 specified. 7641 - Could be split into 7642 separate s_waitcnt 7643 vmcnt(0) and 7644 s_waitcnt 7645 lgkmcnt(0) to allow 7646 them to be 7647 independently moved 7648 according to the 7649 following rules. 7650 - s_waitcnt vmcnt(0) 7651 must happen after 7652 any preceding 7653 global/generic 7654 load/store/load 7655 atomic/store 7656 atomic/atomicrmw. 7657 - s_waitcnt lgkmcnt(0) 7658 must happen after 7659 any preceding 7660 local/generic 7661 load/store/load 7662 atomic/store 7663 atomic/atomicrmw. 7664 - Must happen before 7665 any following store 7666 atomic/atomicrmw 7667 with an equal or 7668 wider sync scope 7669 and memory ordering 7670 stronger than 7671 unordered (this is 7672 termed the 7673 fence-paired-atomic). 7674 - Ensures that all 7675 memory operations 7676 have 7677 completed before 7678 performing the 7679 following 7680 fence-paired-atomic. 7681 7682 **Acquire-Release Atomic** 7683 ------------------------------------------------------------------------------------ 7684 atomicrmw acq_rel - singlethread - global 1. buffer/global/flat_atomic 7685 - wavefront - generic 7686 atomicrmw acq_rel - singlethread - local *If TgSplit execution mode, 7687 - wavefront local address space cannot 7688 be used.* 7689 7690 1. ds_atomic 7691 atomicrmw acq_rel - workgroup - global 1. s_waitcnt lgkm/vmcnt(0) 7692 7693 - Use lgkmcnt(0) if not 7694 TgSplit execution mode 7695 and vmcnt(0) if TgSplit 7696 execution mode. 7697 - If OpenCL, omit 7698 lgkmcnt(0). 7699 - Must happen after 7700 any preceding 7701 local/generic 7702 load/store/load 7703 atomic/store 7704 atomic/atomicrmw. 7705 - s_waitcnt vmcnt(0) 7706 must happen after 7707 any preceding 7708 global/generic load/store/ 7709 load atomic/store atomic/ 7710 atomicrmw. 7711 - s_waitcnt lgkmcnt(0) 7712 must happen after 7713 any preceding 7714 local/generic 7715 load/store/load 7716 atomic/store 7717 atomic/atomicrmw. 7718 - Must happen before 7719 the following 7720 atomicrmw. 7721 - Ensures that all 7722 memory operations 7723 have 7724 completed before 7725 performing the 7726 atomicrmw that is 7727 being released. 7728 7729 2. buffer/global_atomic 7730 3. s_waitcnt vmcnt(0) 7731 7732 - If not TgSplit execution 7733 mode, omit. 7734 - Must happen before 7735 the following 7736 buffer_wbinvl1_vol. 7737 - Ensures any 7738 following global 7739 data read is no 7740 older than the 7741 atomicrmw value 7742 being acquired. 7743 7744 4. buffer_wbinvl1_vol 7745 7746 - If not TgSplit execution 7747 mode, omit. 7748 - Ensures that 7749 following 7750 loads will not see 7751 stale data. 7752 7753 atomicrmw acq_rel - workgroup - local *If TgSplit execution mode, 7754 local address space cannot 7755 be used.* 7756 7757 1. ds_atomic 7758 2. s_waitcnt lgkmcnt(0) 7759 7760 - If OpenCL, omit. 7761 - Must happen before 7762 any following 7763 global/generic 7764 load/load 7765 atomic/store/store 7766 atomic/atomicrmw. 7767 - Ensures any 7768 following global 7769 data read is no 7770 older than the local load 7771 atomic value being 7772 acquired. 7773 7774 atomicrmw acq_rel - workgroup - generic 1. s_waitcnt lgkm/vmcnt(0) 7775 7776 - Use lgkmcnt(0) if not 7777 TgSplit execution mode 7778 and vmcnt(0) if TgSplit 7779 execution mode. 7780 - If OpenCL, omit 7781 lgkmcnt(0). 7782 - s_waitcnt vmcnt(0) 7783 must happen after 7784 any preceding 7785 global/generic load/store/ 7786 load atomic/store atomic/ 7787 atomicrmw. 7788 - s_waitcnt lgkmcnt(0) 7789 must happen after 7790 any preceding 7791 local/generic 7792 load/store/load 7793 atomic/store 7794 atomic/atomicrmw. 7795 - Must happen before 7796 the following 7797 atomicrmw. 7798 - Ensures that all 7799 memory operations 7800 have 7801 completed before 7802 performing the 7803 atomicrmw that is 7804 being released. 7805 7806 2. flat_atomic 7807 3. s_waitcnt lgkmcnt(0) & 7808 vmcnt(0) 7809 7810 - If not TgSplit execution 7811 mode, omit vmcnt(0). 7812 - If OpenCL, omit 7813 lgkmcnt(0). 7814 - Must happen before 7815 the following 7816 buffer_wbinvl1_vol and 7817 any following 7818 global/generic 7819 load/load 7820 atomic/store/store 7821 atomic/atomicrmw. 7822 - Ensures any 7823 following global 7824 data read is no 7825 older than a local load 7826 atomic value being 7827 acquired. 7828 7829 3. buffer_wbinvl1_vol 7830 7831 - If not TgSplit execution 7832 mode, omit. 7833 - Ensures that 7834 following 7835 loads will not see 7836 stale data. 7837 7838 atomicrmw acq_rel - agent - global 1. s_waitcnt lgkmcnt(0) & 7839 vmcnt(0) 7840 7841 - If TgSplit execution mode, 7842 omit lgkmcnt(0). 7843 - If OpenCL, omit 7844 lgkmcnt(0). 7845 - Could be split into 7846 separate s_waitcnt 7847 vmcnt(0) and 7848 s_waitcnt 7849 lgkmcnt(0) to allow 7850 them to be 7851 independently moved 7852 according to the 7853 following rules. 7854 - s_waitcnt vmcnt(0) 7855 must happen after 7856 any preceding 7857 global/generic 7858 load/store/load 7859 atomic/store 7860 atomic/atomicrmw. 7861 - s_waitcnt lgkmcnt(0) 7862 must happen after 7863 any preceding 7864 local/generic 7865 load/store/load 7866 atomic/store 7867 atomic/atomicrmw. 7868 - Must happen before 7869 the following 7870 atomicrmw. 7871 - Ensures that all 7872 memory operations 7873 to global have 7874 completed before 7875 performing the 7876 atomicrmw that is 7877 being released. 7878 7879 2. buffer/global_atomic 7880 3. s_waitcnt vmcnt(0) 7881 7882 - Must happen before 7883 following 7884 buffer_wbinvl1_vol. 7885 - Ensures the 7886 atomicrmw has 7887 completed before 7888 invalidating the 7889 cache. 7890 7891 4. buffer_wbinvl1_vol 7892 7893 - Must happen before 7894 any following 7895 global/generic 7896 load/load 7897 atomic/atomicrmw. 7898 - Ensures that 7899 following loads 7900 will not see stale 7901 global data. 7902 7903 atomicrmw acq_rel - system - global 1. buffer_wbl2 7904 7905 - Must happen before 7906 following s_waitcnt. 7907 - Performs L2 writeback to 7908 ensure previous 7909 global/generic 7910 store/atomicrmw are 7911 visible at system scope. 7912 7913 2. s_waitcnt lgkmcnt(0) & 7914 vmcnt(0) 7915 7916 - If TgSplit execution mode, 7917 omit lgkmcnt(0). 7918 - If OpenCL, omit 7919 lgkmcnt(0). 7920 - Could be split into 7921 separate s_waitcnt 7922 vmcnt(0) and 7923 s_waitcnt 7924 lgkmcnt(0) to allow 7925 them to be 7926 independently moved 7927 according to the 7928 following rules. 7929 - s_waitcnt vmcnt(0) 7930 must happen after 7931 any preceding 7932 global/generic 7933 load/store/load 7934 atomic/store 7935 atomic/atomicrmw. 7936 - s_waitcnt lgkmcnt(0) 7937 must happen after 7938 any preceding 7939 local/generic 7940 load/store/load 7941 atomic/store 7942 atomic/atomicrmw. 7943 - Must happen before 7944 the following 7945 atomicrmw. 7946 - Ensures that all 7947 memory operations 7948 to global and L2 writeback 7949 have completed before 7950 performing the 7951 atomicrmw that is 7952 being released. 7953 7954 3. buffer/global_atomic 7955 4. s_waitcnt vmcnt(0) 7956 7957 - Must happen before 7958 following buffer_invl2 and 7959 buffer_wbinvl1_vol. 7960 - Ensures the 7961 atomicrmw has 7962 completed before 7963 invalidating the 7964 caches. 7965 7966 5. buffer_invl2; 7967 buffer_wbinvl1_vol 7968 7969 - Must happen before 7970 any following 7971 global/generic 7972 load/load 7973 atomic/atomicrmw. 7974 - Ensures that 7975 following 7976 loads will not see 7977 stale L1 global data, 7978 nor see stale L2 MTYPE 7979 NC global data. 7980 MTYPE RW and CC memory will 7981 never be stale in L2 due to 7982 the memory probes. 7983 7984 atomicrmw acq_rel - agent - generic 1. s_waitcnt lgkmcnt(0) & 7985 vmcnt(0) 7986 7987 - If TgSplit execution mode, 7988 omit lgkmcnt(0). 7989 - If OpenCL, omit 7990 lgkmcnt(0). 7991 - Could be split into 7992 separate s_waitcnt 7993 vmcnt(0) and 7994 s_waitcnt 7995 lgkmcnt(0) to allow 7996 them to be 7997 independently moved 7998 according to the 7999 following rules. 8000 - s_waitcnt vmcnt(0) 8001 must happen after 8002 any preceding 8003 global/generic 8004 load/store/load 8005 atomic/store 8006 atomic/atomicrmw. 8007 - s_waitcnt lgkmcnt(0) 8008 must happen after 8009 any preceding 8010 local/generic 8011 load/store/load 8012 atomic/store 8013 atomic/atomicrmw. 8014 - Must happen before 8015 the following 8016 atomicrmw. 8017 - Ensures that all 8018 memory operations 8019 to global have 8020 completed before 8021 performing the 8022 atomicrmw that is 8023 being released. 8024 8025 2. flat_atomic 8026 3. s_waitcnt vmcnt(0) & 8027 lgkmcnt(0) 8028 8029 - If TgSplit execution mode, 8030 omit lgkmcnt(0). 8031 - If OpenCL, omit 8032 lgkmcnt(0). 8033 - Must happen before 8034 following 8035 buffer_wbinvl1_vol. 8036 - Ensures the 8037 atomicrmw has 8038 completed before 8039 invalidating the 8040 cache. 8041 8042 4. buffer_wbinvl1_vol 8043 8044 - Must happen before 8045 any following 8046 global/generic 8047 load/load 8048 atomic/atomicrmw. 8049 - Ensures that 8050 following loads 8051 will not see stale 8052 global data. 8053 8054 atomicrmw acq_rel - system - generic 1. buffer_wbl2 8055 8056 - Must happen before 8057 following s_waitcnt. 8058 - Performs L2 writeback to 8059 ensure previous 8060 global/generic 8061 store/atomicrmw are 8062 visible at system scope. 8063 8064 2. s_waitcnt lgkmcnt(0) & 8065 vmcnt(0) 8066 8067 - If TgSplit execution mode, 8068 omit lgkmcnt(0). 8069 - If OpenCL, omit 8070 lgkmcnt(0). 8071 - Could be split into 8072 separate s_waitcnt 8073 vmcnt(0) and 8074 s_waitcnt 8075 lgkmcnt(0) to allow 8076 them to be 8077 independently moved 8078 according to the 8079 following rules. 8080 - s_waitcnt vmcnt(0) 8081 must happen after 8082 any preceding 8083 global/generic 8084 load/store/load 8085 atomic/store 8086 atomic/atomicrmw. 8087 - s_waitcnt lgkmcnt(0) 8088 must happen after 8089 any preceding 8090 local/generic 8091 load/store/load 8092 atomic/store 8093 atomic/atomicrmw. 8094 - Must happen before 8095 the following 8096 atomicrmw. 8097 - Ensures that all 8098 memory operations 8099 to global and L2 writeback 8100 have completed before 8101 performing the 8102 atomicrmw that is 8103 being released. 8104 8105 3. flat_atomic 8106 4. s_waitcnt vmcnt(0) & 8107 lgkmcnt(0) 8108 8109 - If TgSplit execution mode, 8110 omit lgkmcnt(0). 8111 - If OpenCL, omit 8112 lgkmcnt(0). 8113 - Must happen before 8114 following buffer_invl2 and 8115 buffer_wbinvl1_vol. 8116 - Ensures the 8117 atomicrmw has 8118 completed before 8119 invalidating the 8120 caches. 8121 8122 5. buffer_invl2; 8123 buffer_wbinvl1_vol 8124 8125 - Must happen before 8126 any following 8127 global/generic 8128 load/load 8129 atomic/atomicrmw. 8130 - Ensures that 8131 following 8132 loads will not see 8133 stale L1 global data, 8134 nor see stale L2 MTYPE 8135 NC global data. 8136 MTYPE RW and CC memory will 8137 never be stale in L2 due to 8138 the memory probes. 8139 8140 fence acq_rel - singlethread *none* *none* 8141 - wavefront 8142 fence acq_rel - workgroup *none* 1. s_waitcnt lgkm/vmcnt(0) 8143 8144 - Use lgkmcnt(0) if not 8145 TgSplit execution mode 8146 and vmcnt(0) if TgSplit 8147 execution mode. 8148 - If OpenCL and 8149 address space is 8150 not generic, omit 8151 lgkmcnt(0). 8152 - If OpenCL and 8153 address space is 8154 local, omit 8155 vmcnt(0). 8156 - However, 8157 since LLVM 8158 currently has no 8159 address space on 8160 the fence need to 8161 conservatively 8162 always generate 8163 (see comment for 8164 previous fence). 8165 - s_waitcnt vmcnt(0) 8166 must happen after 8167 any preceding 8168 global/generic 8169 load/store/ 8170 load atomic/store atomic/ 8171 atomicrmw. 8172 - s_waitcnt lgkmcnt(0) 8173 must happen after 8174 any preceding 8175 local/generic 8176 load/load 8177 atomic/store/store 8178 atomic/atomicrmw. 8179 - Must happen before 8180 any following 8181 global/generic 8182 load/load 8183 atomic/store/store 8184 atomic/atomicrmw. 8185 - Ensures that all 8186 memory operations 8187 have 8188 completed before 8189 performing any 8190 following global 8191 memory operations. 8192 - Ensures that the 8193 preceding 8194 local/generic load 8195 atomic/atomicrmw 8196 with an equal or 8197 wider sync scope 8198 and memory ordering 8199 stronger than 8200 unordered (this is 8201 termed the 8202 acquire-fence-paired-atomic) 8203 has completed 8204 before following 8205 global memory 8206 operations. This 8207 satisfies the 8208 requirements of 8209 acquire. 8210 - Ensures that all 8211 previous memory 8212 operations have 8213 completed before a 8214 following 8215 local/generic store 8216 atomic/atomicrmw 8217 with an equal or 8218 wider sync scope 8219 and memory ordering 8220 stronger than 8221 unordered (this is 8222 termed the 8223 release-fence-paired-atomic). 8224 This satisfies the 8225 requirements of 8226 release. 8227 - Must happen before 8228 the following 8229 buffer_wbinvl1_vol. 8230 - Ensures that the 8231 acquire-fence-paired 8232 atomic has completed 8233 before invalidating 8234 the 8235 cache. Therefore 8236 any following 8237 locations read must 8238 be no older than 8239 the value read by 8240 the 8241 acquire-fence-paired-atomic. 8242 8243 2. buffer_wbinvl1_vol 8244 8245 - If not TgSplit execution 8246 mode, omit. 8247 - Ensures that 8248 following 8249 loads will not see 8250 stale data. 8251 8252 fence acq_rel - agent *none* 1. s_waitcnt lgkmcnt(0) & 8253 vmcnt(0) 8254 8255 - If TgSplit execution mode, 8256 omit lgkmcnt(0). 8257 - If OpenCL and 8258 address space is 8259 not generic, omit 8260 lgkmcnt(0). 8261 - However, since LLVM 8262 currently has no 8263 address space on 8264 the fence need to 8265 conservatively 8266 always generate 8267 (see comment for 8268 previous fence). 8269 - Could be split into 8270 separate s_waitcnt 8271 vmcnt(0) and 8272 s_waitcnt 8273 lgkmcnt(0) to allow 8274 them to be 8275 independently moved 8276 according to the 8277 following rules. 8278 - s_waitcnt vmcnt(0) 8279 must happen after 8280 any preceding 8281 global/generic 8282 load/store/load 8283 atomic/store 8284 atomic/atomicrmw. 8285 - s_waitcnt lgkmcnt(0) 8286 must happen after 8287 any preceding 8288 local/generic 8289 load/store/load 8290 atomic/store 8291 atomic/atomicrmw. 8292 - Must happen before 8293 the following 8294 buffer_wbinvl1_vol. 8295 - Ensures that the 8296 preceding 8297 global/local/generic 8298 load 8299 atomic/atomicrmw 8300 with an equal or 8301 wider sync scope 8302 and memory ordering 8303 stronger than 8304 unordered (this is 8305 termed the 8306 acquire-fence-paired-atomic) 8307 has completed 8308 before invalidating 8309 the cache. This 8310 satisfies the 8311 requirements of 8312 acquire. 8313 - Ensures that all 8314 previous memory 8315 operations have 8316 completed before a 8317 following 8318 global/local/generic 8319 store 8320 atomic/atomicrmw 8321 with an equal or 8322 wider sync scope 8323 and memory ordering 8324 stronger than 8325 unordered (this is 8326 termed the 8327 release-fence-paired-atomic). 8328 This satisfies the 8329 requirements of 8330 release. 8331 8332 2. buffer_wbinvl1_vol 8333 8334 - Must happen before 8335 any following 8336 global/generic 8337 load/load 8338 atomic/store/store 8339 atomic/atomicrmw. 8340 - Ensures that 8341 following loads 8342 will not see stale 8343 global data. This 8344 satisfies the 8345 requirements of 8346 acquire. 8347 8348 fence acq_rel - system *none* 1. buffer_wbl2 8349 8350 - If OpenCL and 8351 address space is 8352 local, omit. 8353 - Must happen before 8354 following s_waitcnt. 8355 - Performs L2 writeback to 8356 ensure previous 8357 global/generic 8358 store/atomicrmw are 8359 visible at system scope. 8360 8361 2. s_waitcnt lgkmcnt(0) & 8362 vmcnt(0) 8363 8364 - If TgSplit execution mode, 8365 omit lgkmcnt(0). 8366 - If OpenCL and 8367 address space is 8368 not generic, omit 8369 lgkmcnt(0). 8370 - However, since LLVM 8371 currently has no 8372 address space on 8373 the fence need to 8374 conservatively 8375 always generate 8376 (see comment for 8377 previous fence). 8378 - Could be split into 8379 separate s_waitcnt 8380 vmcnt(0) and 8381 s_waitcnt 8382 lgkmcnt(0) to allow 8383 them to be 8384 independently moved 8385 according to the 8386 following rules. 8387 - s_waitcnt vmcnt(0) 8388 must happen after 8389 any preceding 8390 global/generic 8391 load/store/load 8392 atomic/store 8393 atomic/atomicrmw. 8394 - s_waitcnt lgkmcnt(0) 8395 must happen after 8396 any preceding 8397 local/generic 8398 load/store/load 8399 atomic/store 8400 atomic/atomicrmw. 8401 - Must happen before 8402 the following buffer_invl2 and 8403 buffer_wbinvl1_vol. 8404 - Ensures that the 8405 preceding 8406 global/local/generic 8407 load 8408 atomic/atomicrmw 8409 with an equal or 8410 wider sync scope 8411 and memory ordering 8412 stronger than 8413 unordered (this is 8414 termed the 8415 acquire-fence-paired-atomic) 8416 has completed 8417 before invalidating 8418 the cache. This 8419 satisfies the 8420 requirements of 8421 acquire. 8422 - Ensures that all 8423 previous memory 8424 operations have 8425 completed before a 8426 following 8427 global/local/generic 8428 store 8429 atomic/atomicrmw 8430 with an equal or 8431 wider sync scope 8432 and memory ordering 8433 stronger than 8434 unordered (this is 8435 termed the 8436 release-fence-paired-atomic). 8437 This satisfies the 8438 requirements of 8439 release. 8440 8441 3. buffer_invl2; 8442 buffer_wbinvl1_vol 8443 8444 - Must happen before 8445 any following 8446 global/generic 8447 load/load 8448 atomic/store/store 8449 atomic/atomicrmw. 8450 - Ensures that 8451 following 8452 loads will not see 8453 stale L1 global data, 8454 nor see stale L2 MTYPE 8455 NC global data. 8456 MTYPE RW and CC memory will 8457 never be stale in L2 due to 8458 the memory probes. 8459 8460 **Sequential Consistent Atomic** 8461 ------------------------------------------------------------------------------------ 8462 load atomic seq_cst - singlethread - global *Same as corresponding 8463 - wavefront - local load atomic acquire, 8464 - generic except must generate 8465 all instructions even 8466 for OpenCL.* 8467 load atomic seq_cst - workgroup - global 1. s_waitcnt lgkm/vmcnt(0) 8468 - generic 8469 - Use lgkmcnt(0) if not 8470 TgSplit execution mode 8471 and vmcnt(0) if TgSplit 8472 execution mode. 8473 - s_waitcnt lgkmcnt(0) must 8474 happen after 8475 preceding 8476 local/generic load 8477 atomic/store 8478 atomic/atomicrmw 8479 with memory 8480 ordering of seq_cst 8481 and with equal or 8482 wider sync scope. 8483 (Note that seq_cst 8484 fences have their 8485 own s_waitcnt 8486 lgkmcnt(0) and so do 8487 not need to be 8488 considered.) 8489 - s_waitcnt vmcnt(0) 8490 must happen after 8491 preceding 8492 global/generic load 8493 atomic/store 8494 atomic/atomicrmw 8495 with memory 8496 ordering of seq_cst 8497 and with equal or 8498 wider sync scope. 8499 (Note that seq_cst 8500 fences have their 8501 own s_waitcnt 8502 vmcnt(0) and so do 8503 not need to be 8504 considered.) 8505 - Ensures any 8506 preceding 8507 sequential 8508 consistent global/local 8509 memory instructions 8510 have completed 8511 before executing 8512 this sequentially 8513 consistent 8514 instruction. This 8515 prevents reordering 8516 a seq_cst store 8517 followed by a 8518 seq_cst load. (Note 8519 that seq_cst is 8520 stronger than 8521 acquire/release as 8522 the reordering of 8523 load acquire 8524 followed by a store 8525 release is 8526 prevented by the 8527 s_waitcnt of 8528 the release, but 8529 there is nothing 8530 preventing a store 8531 release followed by 8532 load acquire from 8533 completing out of 8534 order. The s_waitcnt 8535 could be placed after 8536 seq_store or before 8537 the seq_load. We 8538 choose the load to 8539 make the s_waitcnt be 8540 as late as possible 8541 so that the store 8542 may have already 8543 completed.) 8544 8545 2. *Following 8546 instructions same as 8547 corresponding load 8548 atomic acquire, 8549 except must generate 8550 all instructions even 8551 for OpenCL.* 8552 load atomic seq_cst - workgroup - local *If TgSplit execution mode, 8553 local address space cannot 8554 be used.* 8555 8556 *Same as corresponding 8557 load atomic acquire, 8558 except must generate 8559 all instructions even 8560 for OpenCL.* 8561 8562 load atomic seq_cst - agent - global 1. s_waitcnt lgkmcnt(0) & 8563 - system - generic vmcnt(0) 8564 8565 - If TgSplit execution mode, 8566 omit lgkmcnt(0). 8567 - Could be split into 8568 separate s_waitcnt 8569 vmcnt(0) 8570 and s_waitcnt 8571 lgkmcnt(0) to allow 8572 them to be 8573 independently moved 8574 according to the 8575 following rules. 8576 - s_waitcnt lgkmcnt(0) 8577 must happen after 8578 preceding 8579 global/generic load 8580 atomic/store 8581 atomic/atomicrmw 8582 with memory 8583 ordering of seq_cst 8584 and with equal or 8585 wider sync scope. 8586 (Note that seq_cst 8587 fences have their 8588 own s_waitcnt 8589 lgkmcnt(0) and so do 8590 not need to be 8591 considered.) 8592 - s_waitcnt vmcnt(0) 8593 must happen after 8594 preceding 8595 global/generic load 8596 atomic/store 8597 atomic/atomicrmw 8598 with memory 8599 ordering of seq_cst 8600 and with equal or 8601 wider sync scope. 8602 (Note that seq_cst 8603 fences have their 8604 own s_waitcnt 8605 vmcnt(0) and so do 8606 not need to be 8607 considered.) 8608 - Ensures any 8609 preceding 8610 sequential 8611 consistent global 8612 memory instructions 8613 have completed 8614 before executing 8615 this sequentially 8616 consistent 8617 instruction. This 8618 prevents reordering 8619 a seq_cst store 8620 followed by a 8621 seq_cst load. (Note 8622 that seq_cst is 8623 stronger than 8624 acquire/release as 8625 the reordering of 8626 load acquire 8627 followed by a store 8628 release is 8629 prevented by the 8630 s_waitcnt of 8631 the release, but 8632 there is nothing 8633 preventing a store 8634 release followed by 8635 load acquire from 8636 completing out of 8637 order. The s_waitcnt 8638 could be placed after 8639 seq_store or before 8640 the seq_load. We 8641 choose the load to 8642 make the s_waitcnt be 8643 as late as possible 8644 so that the store 8645 may have already 8646 completed.) 8647 8648 2. *Following 8649 instructions same as 8650 corresponding load 8651 atomic acquire, 8652 except must generate 8653 all instructions even 8654 for OpenCL.* 8655 store atomic seq_cst - singlethread - global *Same as corresponding 8656 - wavefront - local store atomic release, 8657 - workgroup - generic except must generate 8658 - agent all instructions even 8659 - system for OpenCL.* 8660 atomicrmw seq_cst - singlethread - global *Same as corresponding 8661 - wavefront - local atomicrmw acq_rel, 8662 - workgroup - generic except must generate 8663 - agent all instructions even 8664 - system for OpenCL.* 8665 fence seq_cst - singlethread *none* *Same as corresponding 8666 - wavefront fence acq_rel, 8667 - workgroup except must generate 8668 - agent all instructions even 8669 - system for OpenCL.* 8670 ============ ============ ============== ========== ================================ 8671 8672.. _amdgpu-amdhsa-memory-model-gfx10: 8673 8674Memory Model GFX10 8675++++++++++++++++++ 8676 8677For GFX10: 8678 8679* Each agent has multiple shader arrays (SA). 8680* Each SA has multiple work-group processors (WGP). 8681* Each WGP has multiple compute units (CU). 8682* Each CU has multiple SIMDs that execute wavefronts. 8683* The wavefronts for a single work-group are executed in the same 8684 WGP. In CU wavefront execution mode the wavefronts may be executed by 8685 different SIMDs in the same CU. In WGP wavefront execution mode the 8686 wavefronts may be executed by different SIMDs in different CUs in the same 8687 WGP. 8688* Each WGP has a single LDS memory shared by the wavefronts of the work-groups 8689 executing on it. 8690* All LDS operations of a WGP are performed as wavefront wide operations in a 8691 global order and involve no caching. Completion is reported to a wavefront in 8692 execution order. 8693* The LDS memory has multiple request queues shared by the SIMDs of a 8694 WGP. Therefore, the LDS operations performed by different wavefronts of a 8695 work-group can be reordered relative to each other, which can result in 8696 reordering the visibility of vector memory operations with respect to LDS 8697 operations of other wavefronts in the same work-group. A ``s_waitcnt 8698 lgkmcnt(0)`` is required to ensure synchronization between LDS operations and 8699 vector memory operations between wavefronts of a work-group, but not between 8700 operations performed by the same wavefront. 8701* The vector memory operations are performed as wavefront wide operations. 8702 Completion of load/store/sample operations are reported to a wavefront in 8703 execution order of other load/store/sample operations performed by that 8704 wavefront. 8705* The vector memory operations access a vector L0 cache. There is a single L0 8706 cache per CU. Each SIMD of a CU accesses the same L0 cache. Therefore, no 8707 special action is required for coherence between the lanes of a single 8708 wavefront. However, a ``buffer_gl0_inv`` is required for coherence between 8709 wavefronts executing in the same work-group as they may be executing on SIMDs 8710 of different CUs that access different L0s. A ``buffer_gl0_inv`` is also 8711 required for coherence between wavefronts executing in different work-groups 8712 as they may be executing on different WGPs. 8713* The scalar memory operations access a scalar L0 cache shared by all wavefronts 8714 on a WGP. The scalar and vector L0 caches are not coherent. However, scalar 8715 operations are used in a restricted way so do not impact the memory model. See 8716 :ref:`amdgpu-amdhsa-memory-spaces`. 8717* The vector and scalar memory L0 caches use an L1 cache shared by all WGPs on 8718 the same SA. Therefore, no special action is required for coherence between 8719 the wavefronts of a single work-group. However, a ``buffer_gl1_inv`` is 8720 required for coherence between wavefronts executing in different work-groups 8721 as they may be executing on different SAs that access different L1s. 8722* The L1 caches have independent quadrants to service disjoint ranges of virtual 8723 addresses. 8724* Each L0 cache has a separate request queue per L1 quadrant. Therefore, the 8725 vector and scalar memory operations performed by different wavefronts, whether 8726 executing in the same or different work-groups (which may be executing on 8727 different CUs accessing different L0s), can be reordered relative to each 8728 other. A ``s_waitcnt vmcnt(0) & vscnt(0)`` is required to ensure 8729 synchronization between vector memory operations of different wavefronts. It 8730 ensures a previous vector memory operation has completed before executing a 8731 subsequent vector memory or LDS operation and so can be used to meet the 8732 requirements of acquire, release and sequential consistency. 8733* The L1 caches use an L2 cache shared by all SAs on the same agent. 8734* The L2 cache has independent channels to service disjoint ranges of virtual 8735 addresses. 8736* Each L1 quadrant of a single SA accesses a different L2 channel. Each L1 8737 quadrant has a separate request queue per L2 channel. Therefore, the vector 8738 and scalar memory operations performed by wavefronts executing in different 8739 work-groups (which may be executing on different SAs) of an agent can be 8740 reordered relative to each other. A ``s_waitcnt vmcnt(0) & vscnt(0)`` is 8741 required to ensure synchronization between vector memory operations of 8742 different SAs. It ensures a previous vector memory operation has completed 8743 before executing a subsequent vector memory and so can be used to meet the 8744 requirements of acquire, release and sequential consistency. 8745* The L2 cache can be kept coherent with other agents on some targets, or ranges 8746 of virtual addresses can be set up to bypass it to ensure system coherence. 8747* On GFX10.3 a memory attached last level (MALL) cache exists for GPU memory. 8748 The MALL cache is fully coherent with GPU memory and has no impact on system 8749 coherence. All agents (GPU and CPU) access GPU memory through the MALL cache. 8750 8751Scalar memory operations are only used to access memory that is proven to not 8752change during the execution of the kernel dispatch. This includes constant 8753address space and global address space for program scope ``const`` variables. 8754Therefore, the kernel machine code does not have to maintain the scalar cache to 8755ensure it is coherent with the vector caches. The scalar and vector caches are 8756invalidated between kernel dispatches by CP since constant address space data 8757may change between kernel dispatch executions. See 8758:ref:`amdgpu-amdhsa-memory-spaces`. 8759 8760The one exception is if scalar writes are used to spill SGPR registers. In this 8761case the AMDGPU backend ensures the memory location used to spill is never 8762accessed by vector memory operations at the same time. If scalar writes are used 8763then a ``s_dcache_wb`` is inserted before the ``s_endpgm`` and before a function 8764return since the locations may be used for vector memory instructions by a 8765future wavefront that uses the same scratch area, or a function call that 8766creates a frame at the same address, respectively. There is no need for a 8767``s_dcache_inv`` as all scalar writes are write-before-read in the same thread. 8768 8769For kernarg backing memory: 8770 8771* CP invalidates the L0 and L1 caches at the start of each kernel dispatch. 8772* On dGPU the kernarg backing memory is accessed as MTYPE UC (uncached) to avoid 8773 needing to invalidate the L2 cache. 8774* On APU the kernarg backing memory is accessed as MTYPE CC (cache coherent) and 8775 so the L2 cache will be coherent with the CPU and other agents. 8776 8777Scratch backing memory (which is used for the private address space) is accessed 8778with MTYPE NC (non-coherent). Since the private address space is only accessed 8779by a single thread, and is always write-before-read, there is never a need to 8780invalidate these entries from the L0 or L1 caches. 8781 8782Wavefronts are executed in native mode with in-order reporting of loads and 8783sample instructions. In this mode vmcnt reports completion of load, atomic with 8784return and sample instructions in order, and the vscnt reports the completion of 8785store and atomic without return in order. See ``MEM_ORDERED`` field in 8786:ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 8787 8788Wavefronts can be executed in WGP or CU wavefront execution mode: 8789 8790* In WGP wavefront execution mode the wavefronts of a work-group are executed 8791 on the SIMDs of both CUs of the WGP. Therefore, explicit management of the per 8792 CU L0 caches is required for work-group synchronization. Also accesses to L1 8793 at work-group scope need to be explicitly ordered as the accesses from 8794 different CUs are not ordered. 8795* In CU wavefront execution mode the wavefronts of a work-group are executed on 8796 the SIMDs of a single CU of the WGP. Therefore, all global memory access by 8797 the work-group access the same L0 which in turn ensures L1 accesses are 8798 ordered and so do not require explicit management of the caches for 8799 work-group synchronization. 8800 8801See ``WGP_MODE`` field in 8802:ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table` and 8803:ref:`amdgpu-target-features`. 8804 8805The code sequences used to implement the memory model for GFX10 are defined in 8806table :ref:`amdgpu-amdhsa-memory-model-code-sequences-gfx10-table`. 8807 8808 .. table:: AMDHSA Memory Model Code Sequences GFX10 8809 :name: amdgpu-amdhsa-memory-model-code-sequences-gfx10-table 8810 8811 ============ ============ ============== ========== ================================ 8812 LLVM Instr LLVM Memory LLVM Memory AMDGPU AMDGPU Machine Code 8813 Ordering Sync Scope Address GFX10 8814 Space 8815 ============ ============ ============== ========== ================================ 8816 **Non-Atomic** 8817 ------------------------------------------------------------------------------------ 8818 load *none* *none* - global - !volatile & !nontemporal 8819 - generic 8820 - private 1. buffer/global/flat_load 8821 - constant 8822 - !volatile & nontemporal 8823 8824 1. buffer/global/flat_load 8825 slc=1 8826 8827 - volatile 8828 8829 1. buffer/global/flat_load 8830 glc=1 dlc=1 8831 2. s_waitcnt vmcnt(0) 8832 8833 - Must happen before 8834 any following volatile 8835 global/generic 8836 load/store. 8837 - Ensures that 8838 volatile 8839 operations to 8840 different 8841 addresses will not 8842 be reordered by 8843 hardware. 8844 8845 load *none* *none* - local 1. ds_load 8846 store *none* *none* - global - !volatile & !nontemporal 8847 - generic 8848 - private 1. buffer/global/flat_store 8849 - constant 8850 - !volatile & nontemporal 8851 8852 1. buffer/global/flat_store 8853 glc=1 slc=1 8854 8855 - volatile 8856 8857 1. buffer/global/flat_store 8858 2. s_waitcnt vscnt(0) 8859 8860 - Must happen before 8861 any following volatile 8862 global/generic 8863 load/store. 8864 - Ensures that 8865 volatile 8866 operations to 8867 different 8868 addresses will not 8869 be reordered by 8870 hardware. 8871 8872 store *none* *none* - local 1. ds_store 8873 **Unordered Atomic** 8874 ------------------------------------------------------------------------------------ 8875 load atomic unordered *any* *any* *Same as non-atomic*. 8876 store atomic unordered *any* *any* *Same as non-atomic*. 8877 atomicrmw unordered *any* *any* *Same as monotonic atomic*. 8878 **Monotonic Atomic** 8879 ------------------------------------------------------------------------------------ 8880 load atomic monotonic - singlethread - global 1. buffer/global/flat_load 8881 - wavefront - generic 8882 load atomic monotonic - workgroup - global 1. buffer/global/flat_load 8883 - generic glc=1 8884 8885 - If CU wavefront execution 8886 mode, omit glc=1. 8887 8888 load atomic monotonic - singlethread - local 1. ds_load 8889 - wavefront 8890 - workgroup 8891 load atomic monotonic - agent - global 1. buffer/global/flat_load 8892 - system - generic glc=1 dlc=1 8893 store atomic monotonic - singlethread - global 1. buffer/global/flat_store 8894 - wavefront - generic 8895 - workgroup 8896 - agent 8897 - system 8898 store atomic monotonic - singlethread - local 1. ds_store 8899 - wavefront 8900 - workgroup 8901 atomicrmw monotonic - singlethread - global 1. buffer/global/flat_atomic 8902 - wavefront - generic 8903 - workgroup 8904 - agent 8905 - system 8906 atomicrmw monotonic - singlethread - local 1. ds_atomic 8907 - wavefront 8908 - workgroup 8909 **Acquire Atomic** 8910 ------------------------------------------------------------------------------------ 8911 load atomic acquire - singlethread - global 1. buffer/global/ds/flat_load 8912 - wavefront - local 8913 - generic 8914 load atomic acquire - workgroup - global 1. buffer/global_load glc=1 8915 8916 - If CU wavefront execution 8917 mode, omit glc=1. 8918 8919 2. s_waitcnt vmcnt(0) 8920 8921 - If CU wavefront execution 8922 mode, omit. 8923 - Must happen before 8924 the following buffer_gl0_inv 8925 and before any following 8926 global/generic 8927 load/load 8928 atomic/store/store 8929 atomic/atomicrmw. 8930 8931 3. buffer_gl0_inv 8932 8933 - If CU wavefront execution 8934 mode, omit. 8935 - Ensures that 8936 following 8937 loads will not see 8938 stale data. 8939 8940 load atomic acquire - workgroup - local 1. ds_load 8941 2. s_waitcnt lgkmcnt(0) 8942 8943 - If OpenCL, omit. 8944 - Must happen before 8945 the following buffer_gl0_inv 8946 and before any following 8947 global/generic load/load 8948 atomic/store/store 8949 atomic/atomicrmw. 8950 - Ensures any 8951 following global 8952 data read is no 8953 older than the local load 8954 atomic value being 8955 acquired. 8956 8957 3. buffer_gl0_inv 8958 8959 - If CU wavefront execution 8960 mode, omit. 8961 - If OpenCL, omit. 8962 - Ensures that 8963 following 8964 loads will not see 8965 stale data. 8966 8967 load atomic acquire - workgroup - generic 1. flat_load glc=1 8968 8969 - If CU wavefront execution 8970 mode, omit glc=1. 8971 8972 2. s_waitcnt lgkmcnt(0) & 8973 vmcnt(0) 8974 8975 - If CU wavefront execution 8976 mode, omit vmcnt(0). 8977 - If OpenCL, omit 8978 lgkmcnt(0). 8979 - Must happen before 8980 the following 8981 buffer_gl0_inv and any 8982 following global/generic 8983 load/load 8984 atomic/store/store 8985 atomic/atomicrmw. 8986 - Ensures any 8987 following global 8988 data read is no 8989 older than a local load 8990 atomic value being 8991 acquired. 8992 8993 3. buffer_gl0_inv 8994 8995 - If CU wavefront execution 8996 mode, omit. 8997 - Ensures that 8998 following 8999 loads will not see 9000 stale data. 9001 9002 load atomic acquire - agent - global 1. buffer/global_load 9003 - system glc=1 dlc=1 9004 2. s_waitcnt vmcnt(0) 9005 9006 - Must happen before 9007 following 9008 buffer_gl*_inv. 9009 - Ensures the load 9010 has completed 9011 before invalidating 9012 the caches. 9013 9014 3. buffer_gl0_inv; 9015 buffer_gl1_inv 9016 9017 - Must happen before 9018 any following 9019 global/generic 9020 load/load 9021 atomic/atomicrmw. 9022 - Ensures that 9023 following 9024 loads will not see 9025 stale global data. 9026 9027 load atomic acquire - agent - generic 1. flat_load glc=1 dlc=1 9028 - system 2. s_waitcnt vmcnt(0) & 9029 lgkmcnt(0) 9030 9031 - If OpenCL omit 9032 lgkmcnt(0). 9033 - Must happen before 9034 following 9035 buffer_gl*_invl. 9036 - Ensures the flat_load 9037 has completed 9038 before invalidating 9039 the caches. 9040 9041 3. buffer_gl0_inv; 9042 buffer_gl1_inv 9043 9044 - Must happen before 9045 any following 9046 global/generic 9047 load/load 9048 atomic/atomicrmw. 9049 - Ensures that 9050 following loads 9051 will not see stale 9052 global data. 9053 9054 atomicrmw acquire - singlethread - global 1. buffer/global/ds/flat_atomic 9055 - wavefront - local 9056 - generic 9057 atomicrmw acquire - workgroup - global 1. buffer/global_atomic 9058 2. s_waitcnt vm/vscnt(0) 9059 9060 - If CU wavefront execution 9061 mode, omit. 9062 - Use vmcnt(0) if atomic with 9063 return and vscnt(0) if 9064 atomic with no-return. 9065 - Must happen before 9066 the following buffer_gl0_inv 9067 and before any following 9068 global/generic 9069 load/load 9070 atomic/store/store 9071 atomic/atomicrmw. 9072 9073 3. buffer_gl0_inv 9074 9075 - If CU wavefront execution 9076 mode, omit. 9077 - Ensures that 9078 following 9079 loads will not see 9080 stale data. 9081 9082 atomicrmw acquire - workgroup - local 1. ds_atomic 9083 2. s_waitcnt lgkmcnt(0) 9084 9085 - If OpenCL, omit. 9086 - Must happen before 9087 the following 9088 buffer_gl0_inv. 9089 - Ensures any 9090 following global 9091 data read is no 9092 older than the local 9093 atomicrmw value 9094 being acquired. 9095 9096 3. buffer_gl0_inv 9097 9098 - If OpenCL omit. 9099 - Ensures that 9100 following 9101 loads will not see 9102 stale data. 9103 9104 atomicrmw acquire - workgroup - generic 1. flat_atomic 9105 2. s_waitcnt lgkmcnt(0) & 9106 vm/vscnt(0) 9107 9108 - If CU wavefront execution 9109 mode, omit vm/vscnt(0). 9110 - If OpenCL, omit lgkmcnt(0). 9111 - Use vmcnt(0) if atomic with 9112 return and vscnt(0) if 9113 atomic with no-return. 9114 - Must happen before 9115 the following 9116 buffer_gl0_inv. 9117 - Ensures any 9118 following global 9119 data read is no 9120 older than a local 9121 atomicrmw value 9122 being acquired. 9123 9124 3. buffer_gl0_inv 9125 9126 - If CU wavefront execution 9127 mode, omit. 9128 - Ensures that 9129 following 9130 loads will not see 9131 stale data. 9132 9133 atomicrmw acquire - agent - global 1. buffer/global_atomic 9134 - system 2. s_waitcnt vm/vscnt(0) 9135 9136 - Use vmcnt(0) if atomic with 9137 return and vscnt(0) if 9138 atomic with no-return. 9139 - Must happen before 9140 following 9141 buffer_gl*_inv. 9142 - Ensures the 9143 atomicrmw has 9144 completed before 9145 invalidating the 9146 caches. 9147 9148 3. buffer_gl0_inv; 9149 buffer_gl1_inv 9150 9151 - Must happen before 9152 any following 9153 global/generic 9154 load/load 9155 atomic/atomicrmw. 9156 - Ensures that 9157 following loads 9158 will not see stale 9159 global data. 9160 9161 atomicrmw acquire - agent - generic 1. flat_atomic 9162 - system 2. s_waitcnt vm/vscnt(0) & 9163 lgkmcnt(0) 9164 9165 - If OpenCL, omit 9166 lgkmcnt(0). 9167 - Use vmcnt(0) if atomic with 9168 return and vscnt(0) if 9169 atomic with no-return. 9170 - Must happen before 9171 following 9172 buffer_gl*_inv. 9173 - Ensures the 9174 atomicrmw has 9175 completed before 9176 invalidating the 9177 caches. 9178 9179 3. buffer_gl0_inv; 9180 buffer_gl1_inv 9181 9182 - Must happen before 9183 any following 9184 global/generic 9185 load/load 9186 atomic/atomicrmw. 9187 - Ensures that 9188 following loads 9189 will not see stale 9190 global data. 9191 9192 fence acquire - singlethread *none* *none* 9193 - wavefront 9194 fence acquire - workgroup *none* 1. s_waitcnt lgkmcnt(0) & 9195 vmcnt(0) & vscnt(0) 9196 9197 - If CU wavefront execution 9198 mode, omit vmcnt(0) and 9199 vscnt(0). 9200 - If OpenCL and 9201 address space is 9202 not generic, omit 9203 lgkmcnt(0). 9204 - If OpenCL and 9205 address space is 9206 local, omit 9207 vmcnt(0) and vscnt(0). 9208 - However, since LLVM 9209 currently has no 9210 address space on 9211 the fence need to 9212 conservatively 9213 always generate. If 9214 fence had an 9215 address space then 9216 set to address 9217 space of OpenCL 9218 fence flag, or to 9219 generic if both 9220 local and global 9221 flags are 9222 specified. 9223 - Could be split into 9224 separate s_waitcnt 9225 vmcnt(0), s_waitcnt 9226 vscnt(0) and s_waitcnt 9227 lgkmcnt(0) to allow 9228 them to be 9229 independently moved 9230 according to the 9231 following rules. 9232 - s_waitcnt vmcnt(0) 9233 must happen after 9234 any preceding 9235 global/generic load 9236 atomic/ 9237 atomicrmw-with-return-value 9238 with an equal or 9239 wider sync scope 9240 and memory ordering 9241 stronger than 9242 unordered (this is 9243 termed the 9244 fence-paired-atomic). 9245 - s_waitcnt vscnt(0) 9246 must happen after 9247 any preceding 9248 global/generic 9249 atomicrmw-no-return-value 9250 with an equal or 9251 wider sync scope 9252 and memory ordering 9253 stronger than 9254 unordered (this is 9255 termed the 9256 fence-paired-atomic). 9257 - s_waitcnt lgkmcnt(0) 9258 must happen after 9259 any preceding 9260 local/generic load 9261 atomic/atomicrmw 9262 with an equal or 9263 wider sync scope 9264 and memory ordering 9265 stronger than 9266 unordered (this is 9267 termed the 9268 fence-paired-atomic). 9269 - Must happen before 9270 the following 9271 buffer_gl0_inv. 9272 - Ensures that the 9273 fence-paired atomic 9274 has completed 9275 before invalidating 9276 the 9277 cache. Therefore 9278 any following 9279 locations read must 9280 be no older than 9281 the value read by 9282 the 9283 fence-paired-atomic. 9284 9285 3. buffer_gl0_inv 9286 9287 - If CU wavefront execution 9288 mode, omit. 9289 - Ensures that 9290 following 9291 loads will not see 9292 stale data. 9293 9294 fence acquire - agent *none* 1. s_waitcnt lgkmcnt(0) & 9295 - system vmcnt(0) & vscnt(0) 9296 9297 - If OpenCL and 9298 address space is 9299 not generic, omit 9300 lgkmcnt(0). 9301 - If OpenCL and 9302 address space is 9303 local, omit 9304 vmcnt(0) and vscnt(0). 9305 - However, since LLVM 9306 currently has no 9307 address space on 9308 the fence need to 9309 conservatively 9310 always generate 9311 (see comment for 9312 previous fence). 9313 - Could be split into 9314 separate s_waitcnt 9315 vmcnt(0), s_waitcnt 9316 vscnt(0) and s_waitcnt 9317 lgkmcnt(0) to allow 9318 them to be 9319 independently moved 9320 according to the 9321 following rules. 9322 - s_waitcnt vmcnt(0) 9323 must happen after 9324 any preceding 9325 global/generic load 9326 atomic/ 9327 atomicrmw-with-return-value 9328 with an equal or 9329 wider sync scope 9330 and memory ordering 9331 stronger than 9332 unordered (this is 9333 termed the 9334 fence-paired-atomic). 9335 - s_waitcnt vscnt(0) 9336 must happen after 9337 any preceding 9338 global/generic 9339 atomicrmw-no-return-value 9340 with an equal or 9341 wider sync scope 9342 and memory ordering 9343 stronger than 9344 unordered (this is 9345 termed the 9346 fence-paired-atomic). 9347 - s_waitcnt lgkmcnt(0) 9348 must happen after 9349 any preceding 9350 local/generic load 9351 atomic/atomicrmw 9352 with an equal or 9353 wider sync scope 9354 and memory ordering 9355 stronger than 9356 unordered (this is 9357 termed the 9358 fence-paired-atomic). 9359 - Must happen before 9360 the following 9361 buffer_gl*_inv. 9362 - Ensures that the 9363 fence-paired atomic 9364 has completed 9365 before invalidating 9366 the 9367 caches. Therefore 9368 any following 9369 locations read must 9370 be no older than 9371 the value read by 9372 the 9373 fence-paired-atomic. 9374 9375 2. buffer_gl0_inv; 9376 buffer_gl1_inv 9377 9378 - Must happen before any 9379 following global/generic 9380 load/load 9381 atomic/store/store 9382 atomic/atomicrmw. 9383 - Ensures that 9384 following loads 9385 will not see stale 9386 global data. 9387 9388 **Release Atomic** 9389 ------------------------------------------------------------------------------------ 9390 store atomic release - singlethread - global 1. buffer/global/ds/flat_store 9391 - wavefront - local 9392 - generic 9393 store atomic release - workgroup - global 1. s_waitcnt lgkmcnt(0) & 9394 - generic vmcnt(0) & vscnt(0) 9395 9396 - If CU wavefront execution 9397 mode, omit vmcnt(0) and 9398 vscnt(0). 9399 - If OpenCL, omit 9400 lgkmcnt(0). 9401 - Could be split into 9402 separate s_waitcnt 9403 vmcnt(0), s_waitcnt 9404 vscnt(0) and s_waitcnt 9405 lgkmcnt(0) to allow 9406 them to be 9407 independently moved 9408 according to the 9409 following rules. 9410 - s_waitcnt vmcnt(0) 9411 must happen after 9412 any preceding 9413 global/generic load/load 9414 atomic/ 9415 atomicrmw-with-return-value. 9416 - s_waitcnt vscnt(0) 9417 must happen after 9418 any preceding 9419 global/generic 9420 store/store 9421 atomic/ 9422 atomicrmw-no-return-value. 9423 - s_waitcnt lgkmcnt(0) 9424 must happen after 9425 any preceding 9426 local/generic 9427 load/store/load 9428 atomic/store 9429 atomic/atomicrmw. 9430 - Must happen before 9431 the following 9432 store. 9433 - Ensures that all 9434 memory operations 9435 have 9436 completed before 9437 performing the 9438 store that is being 9439 released. 9440 9441 2. buffer/global/flat_store 9442 store atomic release - workgroup - local 1. s_waitcnt vmcnt(0) & vscnt(0) 9443 9444 - If CU wavefront execution 9445 mode, omit. 9446 - If OpenCL, omit. 9447 - Could be split into 9448 separate s_waitcnt 9449 vmcnt(0) and s_waitcnt 9450 vscnt(0) to allow 9451 them to be 9452 independently moved 9453 according to the 9454 following rules. 9455 - s_waitcnt vmcnt(0) 9456 must happen after 9457 any preceding 9458 global/generic load/load 9459 atomic/ 9460 atomicrmw-with-return-value. 9461 - s_waitcnt vscnt(0) 9462 must happen after 9463 any preceding 9464 global/generic 9465 store/store atomic/ 9466 atomicrmw-no-return-value. 9467 - Must happen before 9468 the following 9469 store. 9470 - Ensures that all 9471 global memory 9472 operations have 9473 completed before 9474 performing the 9475 store that is being 9476 released. 9477 9478 2. ds_store 9479 store atomic release - agent - global 1. s_waitcnt lgkmcnt(0) & 9480 - system - generic vmcnt(0) & vscnt(0) 9481 9482 - If OpenCL and 9483 address space is 9484 not generic, omit 9485 lgkmcnt(0). 9486 - Could be split into 9487 separate s_waitcnt 9488 vmcnt(0), s_waitcnt vscnt(0) 9489 and s_waitcnt 9490 lgkmcnt(0) to allow 9491 them to be 9492 independently moved 9493 according to the 9494 following rules. 9495 - s_waitcnt vmcnt(0) 9496 must happen after 9497 any preceding 9498 global/generic 9499 load/load 9500 atomic/ 9501 atomicrmw-with-return-value. 9502 - s_waitcnt vscnt(0) 9503 must happen after 9504 any preceding 9505 global/generic 9506 store/store atomic/ 9507 atomicrmw-no-return-value. 9508 - s_waitcnt lgkmcnt(0) 9509 must happen after 9510 any preceding 9511 local/generic 9512 load/store/load 9513 atomic/store 9514 atomic/atomicrmw. 9515 - Must happen before 9516 the following 9517 store. 9518 - Ensures that all 9519 memory operations 9520 have 9521 completed before 9522 performing the 9523 store that is being 9524 released. 9525 9526 2. buffer/global/flat_store 9527 atomicrmw release - singlethread - global 1. buffer/global/ds/flat_atomic 9528 - wavefront - local 9529 - generic 9530 atomicrmw release - workgroup - global 1. s_waitcnt lgkmcnt(0) & 9531 - generic vmcnt(0) & vscnt(0) 9532 9533 - If CU wavefront execution 9534 mode, omit vmcnt(0) and 9535 vscnt(0). 9536 - If OpenCL, omit lgkmcnt(0). 9537 - Could be split into 9538 separate s_waitcnt 9539 vmcnt(0), s_waitcnt 9540 vscnt(0) and s_waitcnt 9541 lgkmcnt(0) to allow 9542 them to be 9543 independently moved 9544 according to the 9545 following rules. 9546 - s_waitcnt vmcnt(0) 9547 must happen after 9548 any preceding 9549 global/generic load/load 9550 atomic/ 9551 atomicrmw-with-return-value. 9552 - s_waitcnt vscnt(0) 9553 must happen after 9554 any preceding 9555 global/generic 9556 store/store 9557 atomic/ 9558 atomicrmw-no-return-value. 9559 - s_waitcnt lgkmcnt(0) 9560 must happen after 9561 any preceding 9562 local/generic 9563 load/store/load 9564 atomic/store 9565 atomic/atomicrmw. 9566 - Must happen before 9567 the following 9568 atomicrmw. 9569 - Ensures that all 9570 memory operations 9571 have 9572 completed before 9573 performing the 9574 atomicrmw that is 9575 being released. 9576 9577 2. buffer/global/flat_atomic 9578 atomicrmw release - workgroup - local 1. s_waitcnt vmcnt(0) & vscnt(0) 9579 9580 - If CU wavefront execution 9581 mode, omit. 9582 - If OpenCL, omit. 9583 - Could be split into 9584 separate s_waitcnt 9585 vmcnt(0) and s_waitcnt 9586 vscnt(0) to allow 9587 them to be 9588 independently moved 9589 according to the 9590 following rules. 9591 - s_waitcnt vmcnt(0) 9592 must happen after 9593 any preceding 9594 global/generic load/load 9595 atomic/ 9596 atomicrmw-with-return-value. 9597 - s_waitcnt vscnt(0) 9598 must happen after 9599 any preceding 9600 global/generic 9601 store/store atomic/ 9602 atomicrmw-no-return-value. 9603 - Must happen before 9604 the following 9605 store. 9606 - Ensures that all 9607 global memory 9608 operations have 9609 completed before 9610 performing the 9611 store that is being 9612 released. 9613 9614 2. ds_atomic 9615 atomicrmw release - agent - global 1. s_waitcnt lgkmcnt(0) & 9616 - system - generic vmcnt(0) & vscnt(0) 9617 9618 - If OpenCL, omit 9619 lgkmcnt(0). 9620 - Could be split into 9621 separate s_waitcnt 9622 vmcnt(0), s_waitcnt 9623 vscnt(0) and s_waitcnt 9624 lgkmcnt(0) to allow 9625 them to be 9626 independently moved 9627 according to the 9628 following rules. 9629 - s_waitcnt vmcnt(0) 9630 must happen after 9631 any preceding 9632 global/generic 9633 load/load atomic/ 9634 atomicrmw-with-return-value. 9635 - s_waitcnt vscnt(0) 9636 must happen after 9637 any preceding 9638 global/generic 9639 store/store atomic/ 9640 atomicrmw-no-return-value. 9641 - s_waitcnt lgkmcnt(0) 9642 must happen after 9643 any preceding 9644 local/generic 9645 load/store/load 9646 atomic/store 9647 atomic/atomicrmw. 9648 - Must happen before 9649 the following 9650 atomicrmw. 9651 - Ensures that all 9652 memory operations 9653 to global and local 9654 have completed 9655 before performing 9656 the atomicrmw that 9657 is being released. 9658 9659 2. buffer/global/flat_atomic 9660 fence release - singlethread *none* *none* 9661 - wavefront 9662 fence release - workgroup *none* 1. s_waitcnt lgkmcnt(0) & 9663 vmcnt(0) & vscnt(0) 9664 9665 - If CU wavefront execution 9666 mode, omit vmcnt(0) and 9667 vscnt(0). 9668 - If OpenCL and 9669 address space is 9670 not generic, omit 9671 lgkmcnt(0). 9672 - If OpenCL and 9673 address space is 9674 local, omit 9675 vmcnt(0) and vscnt(0). 9676 - However, since LLVM 9677 currently has no 9678 address space on 9679 the fence need to 9680 conservatively 9681 always generate. If 9682 fence had an 9683 address space then 9684 set to address 9685 space of OpenCL 9686 fence flag, or to 9687 generic if both 9688 local and global 9689 flags are 9690 specified. 9691 - Could be split into 9692 separate s_waitcnt 9693 vmcnt(0), s_waitcnt 9694 vscnt(0) and s_waitcnt 9695 lgkmcnt(0) to allow 9696 them to be 9697 independently moved 9698 according to the 9699 following rules. 9700 - s_waitcnt vmcnt(0) 9701 must happen after 9702 any preceding 9703 global/generic 9704 load/load 9705 atomic/ 9706 atomicrmw-with-return-value. 9707 - s_waitcnt vscnt(0) 9708 must happen after 9709 any preceding 9710 global/generic 9711 store/store atomic/ 9712 atomicrmw-no-return-value. 9713 - s_waitcnt lgkmcnt(0) 9714 must happen after 9715 any preceding 9716 local/generic 9717 load/store/load 9718 atomic/store atomic/ 9719 atomicrmw. 9720 - Must happen before 9721 any following store 9722 atomic/atomicrmw 9723 with an equal or 9724 wider sync scope 9725 and memory ordering 9726 stronger than 9727 unordered (this is 9728 termed the 9729 fence-paired-atomic). 9730 - Ensures that all 9731 memory operations 9732 have 9733 completed before 9734 performing the 9735 following 9736 fence-paired-atomic. 9737 9738 fence release - agent *none* 1. s_waitcnt lgkmcnt(0) & 9739 - system vmcnt(0) & vscnt(0) 9740 9741 - If OpenCL and 9742 address space is 9743 not generic, omit 9744 lgkmcnt(0). 9745 - If OpenCL and 9746 address space is 9747 local, omit 9748 vmcnt(0) and vscnt(0). 9749 - However, since LLVM 9750 currently has no 9751 address space on 9752 the fence need to 9753 conservatively 9754 always generate. If 9755 fence had an 9756 address space then 9757 set to address 9758 space of OpenCL 9759 fence flag, or to 9760 generic if both 9761 local and global 9762 flags are 9763 specified. 9764 - Could be split into 9765 separate s_waitcnt 9766 vmcnt(0), s_waitcnt 9767 vscnt(0) and s_waitcnt 9768 lgkmcnt(0) to allow 9769 them to be 9770 independently moved 9771 according to the 9772 following rules. 9773 - s_waitcnt vmcnt(0) 9774 must happen after 9775 any preceding 9776 global/generic 9777 load/load atomic/ 9778 atomicrmw-with-return-value. 9779 - s_waitcnt vscnt(0) 9780 must happen after 9781 any preceding 9782 global/generic 9783 store/store atomic/ 9784 atomicrmw-no-return-value. 9785 - s_waitcnt lgkmcnt(0) 9786 must happen after 9787 any preceding 9788 local/generic 9789 load/store/load 9790 atomic/store 9791 atomic/atomicrmw. 9792 - Must happen before 9793 any following store 9794 atomic/atomicrmw 9795 with an equal or 9796 wider sync scope 9797 and memory ordering 9798 stronger than 9799 unordered (this is 9800 termed the 9801 fence-paired-atomic). 9802 - Ensures that all 9803 memory operations 9804 have 9805 completed before 9806 performing the 9807 following 9808 fence-paired-atomic. 9809 9810 **Acquire-Release Atomic** 9811 ------------------------------------------------------------------------------------ 9812 atomicrmw acq_rel - singlethread - global 1. buffer/global/ds/flat_atomic 9813 - wavefront - local 9814 - generic 9815 atomicrmw acq_rel - workgroup - global 1. s_waitcnt lgkmcnt(0) & 9816 vmcnt(0) & vscnt(0) 9817 9818 - If CU wavefront execution 9819 mode, omit vmcnt(0) and 9820 vscnt(0). 9821 - If OpenCL, omit 9822 lgkmcnt(0). 9823 - Must happen after 9824 any preceding 9825 local/generic 9826 load/store/load 9827 atomic/store 9828 atomic/atomicrmw. 9829 - Could be split into 9830 separate s_waitcnt 9831 vmcnt(0), s_waitcnt 9832 vscnt(0), and s_waitcnt 9833 lgkmcnt(0) to allow 9834 them to be 9835 independently moved 9836 according to the 9837 following rules. 9838 - s_waitcnt vmcnt(0) 9839 must happen after 9840 any preceding 9841 global/generic load/load 9842 atomic/ 9843 atomicrmw-with-return-value. 9844 - s_waitcnt vscnt(0) 9845 must happen after 9846 any preceding 9847 global/generic 9848 store/store 9849 atomic/ 9850 atomicrmw-no-return-value. 9851 - s_waitcnt lgkmcnt(0) 9852 must happen after 9853 any preceding 9854 local/generic 9855 load/store/load 9856 atomic/store 9857 atomic/atomicrmw. 9858 - Must happen before 9859 the following 9860 atomicrmw. 9861 - Ensures that all 9862 memory operations 9863 have 9864 completed before 9865 performing the 9866 atomicrmw that is 9867 being released. 9868 9869 2. buffer/global_atomic 9870 3. s_waitcnt vm/vscnt(0) 9871 9872 - If CU wavefront execution 9873 mode, omit. 9874 - Use vmcnt(0) if atomic with 9875 return and vscnt(0) if 9876 atomic with no-return. 9877 - Must happen before 9878 the following 9879 buffer_gl0_inv. 9880 - Ensures any 9881 following global 9882 data read is no 9883 older than the 9884 atomicrmw value 9885 being acquired. 9886 9887 4. buffer_gl0_inv 9888 9889 - If CU wavefront execution 9890 mode, omit. 9891 - Ensures that 9892 following 9893 loads will not see 9894 stale data. 9895 9896 atomicrmw acq_rel - workgroup - local 1. s_waitcnt vmcnt(0) & vscnt(0) 9897 9898 - If CU wavefront execution 9899 mode, omit. 9900 - If OpenCL, omit. 9901 - Could be split into 9902 separate s_waitcnt 9903 vmcnt(0) and s_waitcnt 9904 vscnt(0) to allow 9905 them to be 9906 independently moved 9907 according to the 9908 following rules. 9909 - s_waitcnt vmcnt(0) 9910 must happen after 9911 any preceding 9912 global/generic load/load 9913 atomic/ 9914 atomicrmw-with-return-value. 9915 - s_waitcnt vscnt(0) 9916 must happen after 9917 any preceding 9918 global/generic 9919 store/store atomic/ 9920 atomicrmw-no-return-value. 9921 - Must happen before 9922 the following 9923 store. 9924 - Ensures that all 9925 global memory 9926 operations have 9927 completed before 9928 performing the 9929 store that is being 9930 released. 9931 9932 2. ds_atomic 9933 3. s_waitcnt lgkmcnt(0) 9934 9935 - If OpenCL, omit. 9936 - Must happen before 9937 the following 9938 buffer_gl0_inv. 9939 - Ensures any 9940 following global 9941 data read is no 9942 older than the local load 9943 atomic value being 9944 acquired. 9945 9946 4. buffer_gl0_inv 9947 9948 - If CU wavefront execution 9949 mode, omit. 9950 - If OpenCL omit. 9951 - Ensures that 9952 following 9953 loads will not see 9954 stale data. 9955 9956 atomicrmw acq_rel - workgroup - generic 1. s_waitcnt lgkmcnt(0) & 9957 vmcnt(0) & vscnt(0) 9958 9959 - If CU wavefront execution 9960 mode, omit vmcnt(0) and 9961 vscnt(0). 9962 - If OpenCL, omit lgkmcnt(0). 9963 - Could be split into 9964 separate s_waitcnt 9965 vmcnt(0), s_waitcnt 9966 vscnt(0) and s_waitcnt 9967 lgkmcnt(0) to allow 9968 them to be 9969 independently moved 9970 according to the 9971 following rules. 9972 - s_waitcnt vmcnt(0) 9973 must happen after 9974 any preceding 9975 global/generic load/load 9976 atomic/ 9977 atomicrmw-with-return-value. 9978 - s_waitcnt vscnt(0) 9979 must happen after 9980 any preceding 9981 global/generic 9982 store/store 9983 atomic/ 9984 atomicrmw-no-return-value. 9985 - s_waitcnt lgkmcnt(0) 9986 must happen after 9987 any preceding 9988 local/generic 9989 load/store/load 9990 atomic/store 9991 atomic/atomicrmw. 9992 - Must happen before 9993 the following 9994 atomicrmw. 9995 - Ensures that all 9996 memory operations 9997 have 9998 completed before 9999 performing the 10000 atomicrmw that is 10001 being released. 10002 10003 2. flat_atomic 10004 3. s_waitcnt lgkmcnt(0) & 10005 vmcnt(0) & vscnt(0) 10006 10007 - If CU wavefront execution 10008 mode, omit vmcnt(0) and 10009 vscnt(0). 10010 - If OpenCL, omit lgkmcnt(0). 10011 - Must happen before 10012 the following 10013 buffer_gl0_inv. 10014 - Ensures any 10015 following global 10016 data read is no 10017 older than the load 10018 atomic value being 10019 acquired. 10020 10021 3. buffer_gl0_inv 10022 10023 - If CU wavefront execution 10024 mode, omit. 10025 - Ensures that 10026 following 10027 loads will not see 10028 stale data. 10029 10030 atomicrmw acq_rel - agent - global 1. s_waitcnt lgkmcnt(0) & 10031 - system vmcnt(0) & vscnt(0) 10032 10033 - If OpenCL, omit 10034 lgkmcnt(0). 10035 - Could be split into 10036 separate s_waitcnt 10037 vmcnt(0), s_waitcnt 10038 vscnt(0) and s_waitcnt 10039 lgkmcnt(0) to allow 10040 them to be 10041 independently moved 10042 according to the 10043 following rules. 10044 - s_waitcnt vmcnt(0) 10045 must happen after 10046 any preceding 10047 global/generic 10048 load/load atomic/ 10049 atomicrmw-with-return-value. 10050 - s_waitcnt vscnt(0) 10051 must happen after 10052 any preceding 10053 global/generic 10054 store/store atomic/ 10055 atomicrmw-no-return-value. 10056 - s_waitcnt lgkmcnt(0) 10057 must happen after 10058 any preceding 10059 local/generic 10060 load/store/load 10061 atomic/store 10062 atomic/atomicrmw. 10063 - Must happen before 10064 the following 10065 atomicrmw. 10066 - Ensures that all 10067 memory operations 10068 to global have 10069 completed before 10070 performing the 10071 atomicrmw that is 10072 being released. 10073 10074 2. buffer/global_atomic 10075 3. s_waitcnt vm/vscnt(0) 10076 10077 - Use vmcnt(0) if atomic with 10078 return and vscnt(0) if 10079 atomic with no-return. 10080 - Must happen before 10081 following 10082 buffer_gl*_inv. 10083 - Ensures the 10084 atomicrmw has 10085 completed before 10086 invalidating the 10087 caches. 10088 10089 4. buffer_gl0_inv; 10090 buffer_gl1_inv 10091 10092 - Must happen before 10093 any following 10094 global/generic 10095 load/load 10096 atomic/atomicrmw. 10097 - Ensures that 10098 following loads 10099 will not see stale 10100 global data. 10101 10102 atomicrmw acq_rel - agent - generic 1. s_waitcnt lgkmcnt(0) & 10103 - system vmcnt(0) & vscnt(0) 10104 10105 - If OpenCL, omit 10106 lgkmcnt(0). 10107 - Could be split into 10108 separate s_waitcnt 10109 vmcnt(0), s_waitcnt 10110 vscnt(0), and s_waitcnt 10111 lgkmcnt(0) to allow 10112 them to be 10113 independently moved 10114 according to the 10115 following rules. 10116 - s_waitcnt vmcnt(0) 10117 must happen after 10118 any preceding 10119 global/generic 10120 load/load atomic 10121 atomicrmw-with-return-value. 10122 - s_waitcnt vscnt(0) 10123 must happen after 10124 any preceding 10125 global/generic 10126 store/store atomic/ 10127 atomicrmw-no-return-value. 10128 - s_waitcnt lgkmcnt(0) 10129 must happen after 10130 any preceding 10131 local/generic 10132 load/store/load 10133 atomic/store 10134 atomic/atomicrmw. 10135 - Must happen before 10136 the following 10137 atomicrmw. 10138 - Ensures that all 10139 memory operations 10140 have 10141 completed before 10142 performing the 10143 atomicrmw that is 10144 being released. 10145 10146 2. flat_atomic 10147 3. s_waitcnt vm/vscnt(0) & 10148 lgkmcnt(0) 10149 10150 - If OpenCL, omit 10151 lgkmcnt(0). 10152 - Use vmcnt(0) if atomic with 10153 return and vscnt(0) if 10154 atomic with no-return. 10155 - Must happen before 10156 following 10157 buffer_gl*_inv. 10158 - Ensures the 10159 atomicrmw has 10160 completed before 10161 invalidating the 10162 caches. 10163 10164 4. buffer_gl0_inv; 10165 buffer_gl1_inv 10166 10167 - Must happen before 10168 any following 10169 global/generic 10170 load/load 10171 atomic/atomicrmw. 10172 - Ensures that 10173 following loads 10174 will not see stale 10175 global data. 10176 10177 fence acq_rel - singlethread *none* *none* 10178 - wavefront 10179 fence acq_rel - workgroup *none* 1. s_waitcnt lgkmcnt(0) & 10180 vmcnt(0) & vscnt(0) 10181 10182 - If CU wavefront execution 10183 mode, omit vmcnt(0) and 10184 vscnt(0). 10185 - If OpenCL and 10186 address space is 10187 not generic, omit 10188 lgkmcnt(0). 10189 - If OpenCL and 10190 address space is 10191 local, omit 10192 vmcnt(0) and vscnt(0). 10193 - However, 10194 since LLVM 10195 currently has no 10196 address space on 10197 the fence need to 10198 conservatively 10199 always generate 10200 (see comment for 10201 previous fence). 10202 - Could be split into 10203 separate s_waitcnt 10204 vmcnt(0), s_waitcnt 10205 vscnt(0) and s_waitcnt 10206 lgkmcnt(0) to allow 10207 them to be 10208 independently moved 10209 according to the 10210 following rules. 10211 - s_waitcnt vmcnt(0) 10212 must happen after 10213 any preceding 10214 global/generic 10215 load/load 10216 atomic/ 10217 atomicrmw-with-return-value. 10218 - s_waitcnt vscnt(0) 10219 must happen after 10220 any preceding 10221 global/generic 10222 store/store atomic/ 10223 atomicrmw-no-return-value. 10224 - s_waitcnt lgkmcnt(0) 10225 must happen after 10226 any preceding 10227 local/generic 10228 load/store/load 10229 atomic/store atomic/ 10230 atomicrmw. 10231 - Must happen before 10232 any following 10233 global/generic 10234 load/load 10235 atomic/store/store 10236 atomic/atomicrmw. 10237 - Ensures that all 10238 memory operations 10239 have 10240 completed before 10241 performing any 10242 following global 10243 memory operations. 10244 - Ensures that the 10245 preceding 10246 local/generic load 10247 atomic/atomicrmw 10248 with an equal or 10249 wider sync scope 10250 and memory ordering 10251 stronger than 10252 unordered (this is 10253 termed the 10254 acquire-fence-paired-atomic) 10255 has completed 10256 before following 10257 global memory 10258 operations. This 10259 satisfies the 10260 requirements of 10261 acquire. 10262 - Ensures that all 10263 previous memory 10264 operations have 10265 completed before a 10266 following 10267 local/generic store 10268 atomic/atomicrmw 10269 with an equal or 10270 wider sync scope 10271 and memory ordering 10272 stronger than 10273 unordered (this is 10274 termed the 10275 release-fence-paired-atomic). 10276 This satisfies the 10277 requirements of 10278 release. 10279 - Must happen before 10280 the following 10281 buffer_gl0_inv. 10282 - Ensures that the 10283 acquire-fence-paired 10284 atomic has completed 10285 before invalidating 10286 the 10287 cache. Therefore 10288 any following 10289 locations read must 10290 be no older than 10291 the value read by 10292 the 10293 acquire-fence-paired-atomic. 10294 10295 3. buffer_gl0_inv 10296 10297 - If CU wavefront execution 10298 mode, omit. 10299 - Ensures that 10300 following 10301 loads will not see 10302 stale data. 10303 10304 fence acq_rel - agent *none* 1. s_waitcnt lgkmcnt(0) & 10305 - system vmcnt(0) & vscnt(0) 10306 10307 - If OpenCL and 10308 address space is 10309 not generic, omit 10310 lgkmcnt(0). 10311 - If OpenCL and 10312 address space is 10313 local, omit 10314 vmcnt(0) and vscnt(0). 10315 - However, since LLVM 10316 currently has no 10317 address space on 10318 the fence need to 10319 conservatively 10320 always generate 10321 (see comment for 10322 previous fence). 10323 - Could be split into 10324 separate s_waitcnt 10325 vmcnt(0), s_waitcnt 10326 vscnt(0) and s_waitcnt 10327 lgkmcnt(0) to allow 10328 them to be 10329 independently moved 10330 according to the 10331 following rules. 10332 - s_waitcnt vmcnt(0) 10333 must happen after 10334 any preceding 10335 global/generic 10336 load/load 10337 atomic/ 10338 atomicrmw-with-return-value. 10339 - s_waitcnt vscnt(0) 10340 must happen after 10341 any preceding 10342 global/generic 10343 store/store atomic/ 10344 atomicrmw-no-return-value. 10345 - s_waitcnt lgkmcnt(0) 10346 must happen after 10347 any preceding 10348 local/generic 10349 load/store/load 10350 atomic/store 10351 atomic/atomicrmw. 10352 - Must happen before 10353 the following 10354 buffer_gl*_inv. 10355 - Ensures that the 10356 preceding 10357 global/local/generic 10358 load 10359 atomic/atomicrmw 10360 with an equal or 10361 wider sync scope 10362 and memory ordering 10363 stronger than 10364 unordered (this is 10365 termed the 10366 acquire-fence-paired-atomic) 10367 has completed 10368 before invalidating 10369 the caches. This 10370 satisfies the 10371 requirements of 10372 acquire. 10373 - Ensures that all 10374 previous memory 10375 operations have 10376 completed before a 10377 following 10378 global/local/generic 10379 store 10380 atomic/atomicrmw 10381 with an equal or 10382 wider sync scope 10383 and memory ordering 10384 stronger than 10385 unordered (this is 10386 termed the 10387 release-fence-paired-atomic). 10388 This satisfies the 10389 requirements of 10390 release. 10391 10392 2. buffer_gl0_inv; 10393 buffer_gl1_inv 10394 10395 - Must happen before 10396 any following 10397 global/generic 10398 load/load 10399 atomic/store/store 10400 atomic/atomicrmw. 10401 - Ensures that 10402 following loads 10403 will not see stale 10404 global data. This 10405 satisfies the 10406 requirements of 10407 acquire. 10408 10409 **Sequential Consistent Atomic** 10410 ------------------------------------------------------------------------------------ 10411 load atomic seq_cst - singlethread - global *Same as corresponding 10412 - wavefront - local load atomic acquire, 10413 - generic except must generate 10414 all instructions even 10415 for OpenCL.* 10416 load atomic seq_cst - workgroup - global 1. s_waitcnt lgkmcnt(0) & 10417 - generic vmcnt(0) & vscnt(0) 10418 10419 - If CU wavefront execution 10420 mode, omit vmcnt(0) and 10421 vscnt(0). 10422 - Could be split into 10423 separate s_waitcnt 10424 vmcnt(0), s_waitcnt 10425 vscnt(0), and s_waitcnt 10426 lgkmcnt(0) to allow 10427 them to be 10428 independently moved 10429 according to the 10430 following rules. 10431 - s_waitcnt lgkmcnt(0) must 10432 happen after 10433 preceding 10434 local/generic load 10435 atomic/store 10436 atomic/atomicrmw 10437 with memory 10438 ordering of seq_cst 10439 and with equal or 10440 wider sync scope. 10441 (Note that seq_cst 10442 fences have their 10443 own s_waitcnt 10444 lgkmcnt(0) and so do 10445 not need to be 10446 considered.) 10447 - s_waitcnt vmcnt(0) 10448 must happen after 10449 preceding 10450 global/generic load 10451 atomic/ 10452 atomicrmw-with-return-value 10453 with memory 10454 ordering of seq_cst 10455 and with equal or 10456 wider sync scope. 10457 (Note that seq_cst 10458 fences have their 10459 own s_waitcnt 10460 vmcnt(0) and so do 10461 not need to be 10462 considered.) 10463 - s_waitcnt vscnt(0) 10464 Must happen after 10465 preceding 10466 global/generic store 10467 atomic/ 10468 atomicrmw-no-return-value 10469 with memory 10470 ordering of seq_cst 10471 and with equal or 10472 wider sync scope. 10473 (Note that seq_cst 10474 fences have their 10475 own s_waitcnt 10476 vscnt(0) and so do 10477 not need to be 10478 considered.) 10479 - Ensures any 10480 preceding 10481 sequential 10482 consistent global/local 10483 memory instructions 10484 have completed 10485 before executing 10486 this sequentially 10487 consistent 10488 instruction. This 10489 prevents reordering 10490 a seq_cst store 10491 followed by a 10492 seq_cst load. (Note 10493 that seq_cst is 10494 stronger than 10495 acquire/release as 10496 the reordering of 10497 load acquire 10498 followed by a store 10499 release is 10500 prevented by the 10501 s_waitcnt of 10502 the release, but 10503 there is nothing 10504 preventing a store 10505 release followed by 10506 load acquire from 10507 completing out of 10508 order. The s_waitcnt 10509 could be placed after 10510 seq_store or before 10511 the seq_load. We 10512 choose the load to 10513 make the s_waitcnt be 10514 as late as possible 10515 so that the store 10516 may have already 10517 completed.) 10518 10519 2. *Following 10520 instructions same as 10521 corresponding load 10522 atomic acquire, 10523 except must generate 10524 all instructions even 10525 for OpenCL.* 10526 load atomic seq_cst - workgroup - local 10527 10528 1. s_waitcnt vmcnt(0) & vscnt(0) 10529 10530 - If CU wavefront execution 10531 mode, omit. 10532 - Could be split into 10533 separate s_waitcnt 10534 vmcnt(0) and s_waitcnt 10535 vscnt(0) to allow 10536 them to be 10537 independently moved 10538 according to the 10539 following rules. 10540 - s_waitcnt vmcnt(0) 10541 Must happen after 10542 preceding 10543 global/generic load 10544 atomic/ 10545 atomicrmw-with-return-value 10546 with memory 10547 ordering of seq_cst 10548 and with equal or 10549 wider sync scope. 10550 (Note that seq_cst 10551 fences have their 10552 own s_waitcnt 10553 vmcnt(0) and so do 10554 not need to be 10555 considered.) 10556 - s_waitcnt vscnt(0) 10557 Must happen after 10558 preceding 10559 global/generic store 10560 atomic/ 10561 atomicrmw-no-return-value 10562 with memory 10563 ordering of seq_cst 10564 and with equal or 10565 wider sync scope. 10566 (Note that seq_cst 10567 fences have their 10568 own s_waitcnt 10569 vscnt(0) and so do 10570 not need to be 10571 considered.) 10572 - Ensures any 10573 preceding 10574 sequential 10575 consistent global 10576 memory instructions 10577 have completed 10578 before executing 10579 this sequentially 10580 consistent 10581 instruction. This 10582 prevents reordering 10583 a seq_cst store 10584 followed by a 10585 seq_cst load. (Note 10586 that seq_cst is 10587 stronger than 10588 acquire/release as 10589 the reordering of 10590 load acquire 10591 followed by a store 10592 release is 10593 prevented by the 10594 s_waitcnt of 10595 the release, but 10596 there is nothing 10597 preventing a store 10598 release followed by 10599 load acquire from 10600 completing out of 10601 order. The s_waitcnt 10602 could be placed after 10603 seq_store or before 10604 the seq_load. We 10605 choose the load to 10606 make the s_waitcnt be 10607 as late as possible 10608 so that the store 10609 may have already 10610 completed.) 10611 10612 2. *Following 10613 instructions same as 10614 corresponding load 10615 atomic acquire, 10616 except must generate 10617 all instructions even 10618 for OpenCL.* 10619 10620 load atomic seq_cst - agent - global 1. s_waitcnt lgkmcnt(0) & 10621 - system - generic vmcnt(0) & vscnt(0) 10622 10623 - Could be split into 10624 separate s_waitcnt 10625 vmcnt(0), s_waitcnt 10626 vscnt(0) and s_waitcnt 10627 lgkmcnt(0) to allow 10628 them to be 10629 independently moved 10630 according to the 10631 following rules. 10632 - s_waitcnt lgkmcnt(0) 10633 must happen after 10634 preceding 10635 local load 10636 atomic/store 10637 atomic/atomicrmw 10638 with memory 10639 ordering of seq_cst 10640 and with equal or 10641 wider sync scope. 10642 (Note that seq_cst 10643 fences have their 10644 own s_waitcnt 10645 lgkmcnt(0) and so do 10646 not need to be 10647 considered.) 10648 - s_waitcnt vmcnt(0) 10649 must happen after 10650 preceding 10651 global/generic load 10652 atomic/ 10653 atomicrmw-with-return-value 10654 with memory 10655 ordering of seq_cst 10656 and with equal or 10657 wider sync scope. 10658 (Note that seq_cst 10659 fences have their 10660 own s_waitcnt 10661 vmcnt(0) and so do 10662 not need to be 10663 considered.) 10664 - s_waitcnt vscnt(0) 10665 Must happen after 10666 preceding 10667 global/generic store 10668 atomic/ 10669 atomicrmw-no-return-value 10670 with memory 10671 ordering of seq_cst 10672 and with equal or 10673 wider sync scope. 10674 (Note that seq_cst 10675 fences have their 10676 own s_waitcnt 10677 vscnt(0) and so do 10678 not need to be 10679 considered.) 10680 - Ensures any 10681 preceding 10682 sequential 10683 consistent global 10684 memory instructions 10685 have completed 10686 before executing 10687 this sequentially 10688 consistent 10689 instruction. This 10690 prevents reordering 10691 a seq_cst store 10692 followed by a 10693 seq_cst load. (Note 10694 that seq_cst is 10695 stronger than 10696 acquire/release as 10697 the reordering of 10698 load acquire 10699 followed by a store 10700 release is 10701 prevented by the 10702 s_waitcnt of 10703 the release, but 10704 there is nothing 10705 preventing a store 10706 release followed by 10707 load acquire from 10708 completing out of 10709 order. The s_waitcnt 10710 could be placed after 10711 seq_store or before 10712 the seq_load. We 10713 choose the load to 10714 make the s_waitcnt be 10715 as late as possible 10716 so that the store 10717 may have already 10718 completed.) 10719 10720 2. *Following 10721 instructions same as 10722 corresponding load 10723 atomic acquire, 10724 except must generate 10725 all instructions even 10726 for OpenCL.* 10727 store atomic seq_cst - singlethread - global *Same as corresponding 10728 - wavefront - local store atomic release, 10729 - workgroup - generic except must generate 10730 - agent all instructions even 10731 - system for OpenCL.* 10732 atomicrmw seq_cst - singlethread - global *Same as corresponding 10733 - wavefront - local atomicrmw acq_rel, 10734 - workgroup - generic except must generate 10735 - agent all instructions even 10736 - system for OpenCL.* 10737 fence seq_cst - singlethread *none* *Same as corresponding 10738 - wavefront fence acq_rel, 10739 - workgroup except must generate 10740 - agent all instructions even 10741 - system for OpenCL.* 10742 ============ ============ ============== ========== ================================ 10743 10744.. _amdgpu-amdhsa-trap-handler-abi: 10745 10746Trap Handler ABI 10747~~~~~~~~~~~~~~~~ 10748 10749For code objects generated by the AMDGPU backend for HSA [HSA]_ compatible 10750runtimes (see :ref:`amdgpu-os`), the runtime installs a trap handler that 10751supports the ``s_trap`` instruction. For usage see: 10752 10753- :ref:`amdgpu-trap-handler-for-amdhsa-os-v2-table` 10754- :ref:`amdgpu-trap-handler-for-amdhsa-os-v3-table` 10755- :ref:`amdgpu-trap-handler-for-amdhsa-os-v4-onwards-table` 10756 10757 .. table:: AMDGPU Trap Handler for AMDHSA OS Code Object V2 10758 :name: amdgpu-trap-handler-for-amdhsa-os-v2-table 10759 10760 =================== =============== =============== ======================================= 10761 Usage Code Sequence Trap Handler Description 10762 Inputs 10763 =================== =============== =============== ======================================= 10764 reserved ``s_trap 0x00`` Reserved by hardware. 10765 ``debugtrap(arg)`` ``s_trap 0x01`` ``SGPR0-1``: Reserved for Finalizer HSA ``debugtrap`` 10766 ``queue_ptr`` intrinsic (not implemented). 10767 ``VGPR0``: 10768 ``arg`` 10769 ``llvm.trap`` ``s_trap 0x02`` ``SGPR0-1``: Causes wave to be halted with the PC at 10770 ``queue_ptr`` the trap instruction. The associated 10771 queue is signalled to put it into the 10772 error state. When the queue is put in 10773 the error state, the waves executing 10774 dispatches on the queue will be 10775 terminated. 10776 ``llvm.debugtrap`` ``s_trap 0x03`` *none* - If debugger not enabled then behaves 10777 as a no-operation. The trap handler 10778 is entered and immediately returns to 10779 continue execution of the wavefront. 10780 - If the debugger is enabled, causes 10781 the debug trap to be reported by the 10782 debugger and the wavefront is put in 10783 the halt state with the PC at the 10784 instruction. The debugger must 10785 increment the PC and resume the wave. 10786 reserved ``s_trap 0x04`` Reserved. 10787 reserved ``s_trap 0x05`` Reserved. 10788 reserved ``s_trap 0x06`` Reserved. 10789 reserved ``s_trap 0x07`` Reserved. 10790 reserved ``s_trap 0x08`` Reserved. 10791 reserved ``s_trap 0xfe`` Reserved. 10792 reserved ``s_trap 0xff`` Reserved. 10793 =================== =============== =============== ======================================= 10794 10795.. 10796 10797 .. table:: AMDGPU Trap Handler for AMDHSA OS Code Object V3 10798 :name: amdgpu-trap-handler-for-amdhsa-os-v3-table 10799 10800 =================== =============== =============== ======================================= 10801 Usage Code Sequence Trap Handler Description 10802 Inputs 10803 =================== =============== =============== ======================================= 10804 reserved ``s_trap 0x00`` Reserved by hardware. 10805 debugger breakpoint ``s_trap 0x01`` *none* Reserved for debugger to use for 10806 breakpoints. Causes wave to be halted 10807 with the PC at the trap instruction. 10808 The debugger is responsible to resume 10809 the wave, including the instruction 10810 that the breakpoint overwrote. 10811 ``llvm.trap`` ``s_trap 0x02`` ``SGPR0-1``: Causes wave to be halted with the PC at 10812 ``queue_ptr`` the trap instruction. The associated 10813 queue is signalled to put it into the 10814 error state. When the queue is put in 10815 the error state, the waves executing 10816 dispatches on the queue will be 10817 terminated. 10818 ``llvm.debugtrap`` ``s_trap 0x03`` *none* - If debugger not enabled then behaves 10819 as a no-operation. The trap handler 10820 is entered and immediately returns to 10821 continue execution of the wavefront. 10822 - If the debugger is enabled, causes 10823 the debug trap to be reported by the 10824 debugger and the wavefront is put in 10825 the halt state with the PC at the 10826 instruction. The debugger must 10827 increment the PC and resume the wave. 10828 reserved ``s_trap 0x04`` Reserved. 10829 reserved ``s_trap 0x05`` Reserved. 10830 reserved ``s_trap 0x06`` Reserved. 10831 reserved ``s_trap 0x07`` Reserved. 10832 reserved ``s_trap 0x08`` Reserved. 10833 reserved ``s_trap 0xfe`` Reserved. 10834 reserved ``s_trap 0xff`` Reserved. 10835 =================== =============== =============== ======================================= 10836 10837.. 10838 10839 .. table:: AMDGPU Trap Handler for AMDHSA OS Code Object V4 and Above 10840 :name: amdgpu-trap-handler-for-amdhsa-os-v4-onwards-table 10841 10842 =================== =============== ================ ================= ======================================= 10843 Usage Code Sequence GFX6-GFX8 Inputs GFX9-GFX10 Inputs Description 10844 =================== =============== ================ ================= ======================================= 10845 reserved ``s_trap 0x00`` Reserved by hardware. 10846 debugger breakpoint ``s_trap 0x01`` *none* *none* Reserved for debugger to use for 10847 breakpoints. Causes wave to be halted 10848 with the PC at the trap instruction. 10849 The debugger is responsible to resume 10850 the wave, including the instruction 10851 that the breakpoint overwrote. 10852 ``llvm.trap`` ``s_trap 0x02`` ``SGPR0-1``: *none* Causes wave to be halted with the PC at 10853 ``queue_ptr`` the trap instruction. The associated 10854 queue is signalled to put it into the 10855 error state. When the queue is put in 10856 the error state, the waves executing 10857 dispatches on the queue will be 10858 terminated. 10859 ``llvm.debugtrap`` ``s_trap 0x03`` *none* *none* - If debugger not enabled then behaves 10860 as a no-operation. The trap handler 10861 is entered and immediately returns to 10862 continue execution of the wavefront. 10863 - If the debugger is enabled, causes 10864 the debug trap to be reported by the 10865 debugger and the wavefront is put in 10866 the halt state with the PC at the 10867 instruction. The debugger must 10868 increment the PC and resume the wave. 10869 reserved ``s_trap 0x04`` Reserved. 10870 reserved ``s_trap 0x05`` Reserved. 10871 reserved ``s_trap 0x06`` Reserved. 10872 reserved ``s_trap 0x07`` Reserved. 10873 reserved ``s_trap 0x08`` Reserved. 10874 reserved ``s_trap 0xfe`` Reserved. 10875 reserved ``s_trap 0xff`` Reserved. 10876 =================== =============== ================ ================= ======================================= 10877 10878.. _amdgpu-amdhsa-function-call-convention: 10879 10880Call Convention 10881~~~~~~~~~~~~~~~ 10882 10883.. note:: 10884 10885 This section is currently incomplete and has inaccuracies. It is WIP that will 10886 be updated as information is determined. 10887 10888See :ref:`amdgpu-dwarf-address-space-identifier` for information on swizzled 10889addresses. Unswizzled addresses are normal linear addresses. 10890 10891.. _amdgpu-amdhsa-function-call-convention-kernel-functions: 10892 10893Kernel Functions 10894++++++++++++++++ 10895 10896This section describes the call convention ABI for the outer kernel function. 10897 10898See :ref:`amdgpu-amdhsa-initial-kernel-execution-state` for the kernel call 10899convention. 10900 10901The following is not part of the AMDGPU kernel calling convention but describes 10902how the AMDGPU implements function calls: 10903 109041. Clang decides the kernarg layout to match the *HSA Programmer's Language 10905 Reference* [HSA]_. 10906 10907 - All structs are passed directly. 10908 - Lambda values are passed *TBA*. 10909 10910 .. TODO:: 10911 10912 - Does this really follow HSA rules? Or are structs >16 bytes passed 10913 by-value struct? 10914 - What is ABI for lambda values? 10915 109164. The kernel performs certain setup in its prolog, as described in 10917 :ref:`amdgpu-amdhsa-kernel-prolog`. 10918 10919.. _amdgpu-amdhsa-function-call-convention-non-kernel-functions: 10920 10921Non-Kernel Functions 10922++++++++++++++++++++ 10923 10924This section describes the call convention ABI for functions other than the 10925outer kernel function. 10926 10927If a kernel has function calls then scratch is always allocated and used for 10928the call stack which grows from low address to high address using the swizzled 10929scratch address space. 10930 10931On entry to a function: 10932 109331. SGPR0-3 contain a V# with the following properties (see 10934 :ref:`amdgpu-amdhsa-kernel-prolog-private-segment-buffer`): 10935 10936 * Base address pointing to the beginning of the wavefront scratch backing 10937 memory. 10938 * Swizzled with dword element size and stride of wavefront size elements. 10939 109402. The FLAT_SCRATCH register pair is setup. See 10941 :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`. 109423. GFX6-GFX8: M0 register set to the size of LDS in bytes. See 10943 :ref:`amdgpu-amdhsa-kernel-prolog-m0`. 109444. The EXEC register is set to the lanes active on entry to the function. 109455. MODE register: *TBD* 109466. VGPR0-31 and SGPR4-29 are used to pass function input arguments as described 10947 below. 109487. SGPR30-31 return address (RA). The code address that the function must 10949 return to when it completes. The value is undefined if the function is *no 10950 return*. 109518. SGPR32 is used for the stack pointer (SP). It is an unswizzled scratch 10952 offset relative to the beginning of the wavefront scratch backing memory. 10953 10954 The unswizzled SP can be used with buffer instructions as an unswizzled SGPR 10955 offset with the scratch V# in SGPR0-3 to access the stack in a swizzled 10956 manner. 10957 10958 The unswizzled SP value can be converted into the swizzled SP value by: 10959 10960 | swizzled SP = unswizzled SP / wavefront size 10961 10962 This may be used to obtain the private address space address of stack 10963 objects and to convert this address to a flat address by adding the flat 10964 scratch aperture base address. 10965 10966 The swizzled SP value is always 4 bytes aligned for the ``r600`` 10967 architecture and 16 byte aligned for the ``amdgcn`` architecture. 10968 10969 .. note:: 10970 10971 The ``amdgcn`` value is selected to avoid dynamic stack alignment for the 10972 OpenCL language which has the largest base type defined as 16 bytes. 10973 10974 On entry, the swizzled SP value is the address of the first function 10975 argument passed on the stack. Other stack passed arguments are positive 10976 offsets from the entry swizzled SP value. 10977 10978 The function may use positive offsets beyond the last stack passed argument 10979 for stack allocated local variables and register spill slots. If necessary, 10980 the function may align these to greater alignment than 16 bytes. After these 10981 the function may dynamically allocate space for such things as runtime sized 10982 ``alloca`` local allocations. 10983 10984 If the function calls another function, it will place any stack allocated 10985 arguments after the last local allocation and adjust SGPR32 to the address 10986 after the last local allocation. 10987 109889. All other registers are unspecified. 1098910. Any necessary ``s_waitcnt`` has been performed to ensure memory is available 10990 to the function. 10991 10992On exit from a function: 10993 109941. VGPR0-31 and SGPR4-29 are used to pass function result arguments as 10995 described below. Any registers used are considered clobbered registers. 109962. The following registers are preserved and have the same value as on entry: 10997 10998 * FLAT_SCRATCH 10999 * EXEC 11000 * GFX6-GFX8: M0 11001 * All SGPR registers except the clobbered registers of SGPR4-31. 11002 * VGPR40-47 11003 * VGPR56-63 11004 * VGPR72-79 11005 * VGPR88-95 11006 * VGPR104-111 11007 * VGPR120-127 11008 * VGPR136-143 11009 * VGPR152-159 11010 * VGPR168-175 11011 * VGPR184-191 11012 * VGPR200-207 11013 * VGPR216-223 11014 * VGPR232-239 11015 * VGPR248-255 11016 11017 .. note:: 11018 11019 Except the argument registers, the VGPRs clobbered and the preserved 11020 registers are intermixed at regular intervals in order to keep a 11021 similar ratio independent of the number of allocated VGPRs. 11022 11023 * GFX90A: All AGPR registers except the clobbered registers AGPR0-31. 11024 * Lanes of all VGPRs that are inactive at the call site. 11025 11026 For the AMDGPU backend, an inter-procedural register allocation (IPRA) 11027 optimization may mark some of clobbered SGPR and VGPR registers as 11028 preserved if it can be determined that the called function does not change 11029 their value. 11030 110312. The PC is set to the RA provided on entry. 110323. MODE register: *TBD*. 110334. All other registers are clobbered. 110345. Any necessary ``s_waitcnt`` has been performed to ensure memory accessed by 11035 function is available to the caller. 11036 11037.. TODO:: 11038 11039 - How are function results returned? The address of structured types is passed 11040 by reference, but what about other types? 11041 11042The function input arguments are made up of the formal arguments explicitly 11043declared by the source language function plus the implicit input arguments used 11044by the implementation. 11045 11046The source language input arguments are: 11047 110481. Any source language implicit ``this`` or ``self`` argument comes first as a 11049 pointer type. 110502. Followed by the function formal arguments in left to right source order. 11051 11052The source language result arguments are: 11053 110541. The function result argument. 11055 11056The source language input or result struct type arguments that are less than or 11057equal to 16 bytes, are decomposed recursively into their base type fields, and 11058each field is passed as if a separate argument. For input arguments, if the 11059called function requires the struct to be in memory, for example because its 11060address is taken, then the function body is responsible for allocating a stack 11061location and copying the field arguments into it. Clang terms this *direct 11062struct*. 11063 11064The source language input struct type arguments that are greater than 16 bytes, 11065are passed by reference. The caller is responsible for allocating a stack 11066location to make a copy of the struct value and pass the address as the input 11067argument. The called function is responsible to perform the dereference when 11068accessing the input argument. Clang terms this *by-value struct*. 11069 11070A source language result struct type argument that is greater than 16 bytes, is 11071returned by reference. The caller is responsible for allocating a stack location 11072to hold the result value and passes the address as the last input argument 11073(before the implicit input arguments). In this case there are no result 11074arguments. The called function is responsible to perform the dereference when 11075storing the result value. Clang terms this *structured return (sret)*. 11076 11077*TODO: correct the ``sret`` definition.* 11078 11079.. TODO:: 11080 11081 Is this definition correct? Or is ``sret`` only used if passing in registers, and 11082 pass as non-decomposed struct as stack argument? Or something else? Is the 11083 memory location in the caller stack frame, or a stack memory argument and so 11084 no address is passed as the caller can directly write to the argument stack 11085 location? But then the stack location is still live after return. If an 11086 argument stack location is it the first stack argument or the last one? 11087 11088Lambda argument types are treated as struct types with an implementation defined 11089set of fields. 11090 11091.. TODO:: 11092 11093 Need to specify the ABI for lambda types for AMDGPU. 11094 11095For AMDGPU backend all source language arguments (including the decomposed 11096struct type arguments) are passed in VGPRs unless marked ``inreg`` in which case 11097they are passed in SGPRs. 11098 11099The AMDGPU backend walks the function call graph from the leaves to determine 11100which implicit input arguments are used, propagating to each caller of the 11101function. The used implicit arguments are appended to the function arguments 11102after the source language arguments in the following order: 11103 11104.. TODO:: 11105 11106 Is recursion or external functions supported? 11107 111081. Work-Item ID (1 VGPR) 11109 11110 The X, Y and Z work-item ID are packed into a single VGRP with the following 11111 layout. Only fields actually used by the function are set. The other bits 11112 are undefined. 11113 11114 The values come from the initial kernel execution state. See 11115 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`. 11116 11117 .. table:: Work-item implicit argument layout 11118 :name: amdgpu-amdhsa-workitem-implicit-argument-layout-table 11119 11120 ======= ======= ============== 11121 Bits Size Field Name 11122 ======= ======= ============== 11123 9:0 10 bits X Work-Item ID 11124 19:10 10 bits Y Work-Item ID 11125 29:20 10 bits Z Work-Item ID 11126 31:30 2 bits Unused 11127 ======= ======= ============== 11128 111292. Dispatch Ptr (2 SGPRs) 11130 11131 The value comes from the initial kernel execution state. See 11132 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 11133 111343. Queue Ptr (2 SGPRs) 11135 11136 The value comes from the initial kernel execution state. See 11137 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 11138 111394. Kernarg Segment Ptr (2 SGPRs) 11140 11141 The value comes from the initial kernel execution state. See 11142 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 11143 111445. Dispatch id (2 SGPRs) 11145 11146 The value comes from the initial kernel execution state. See 11147 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 11148 111496. Work-Group ID X (1 SGPR) 11150 11151 The value comes from the initial kernel execution state. See 11152 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 11153 111547. Work-Group ID Y (1 SGPR) 11155 11156 The value comes from the initial kernel execution state. See 11157 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 11158 111598. Work-Group ID Z (1 SGPR) 11160 11161 The value comes from the initial kernel execution state. See 11162 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 11163 111649. Implicit Argument Ptr (2 SGPRs) 11165 11166 The value is computed by adding an offset to Kernarg Segment Ptr to get the 11167 global address space pointer to the first kernarg implicit argument. 11168 11169The input and result arguments are assigned in order in the following manner: 11170 11171.. note:: 11172 11173 There are likely some errors and omissions in the following description that 11174 need correction. 11175 11176 .. TODO:: 11177 11178 Check the Clang source code to decipher how function arguments and return 11179 results are handled. Also see the AMDGPU specific values used. 11180 11181* VGPR arguments are assigned to consecutive VGPRs starting at VGPR0 up to 11182 VGPR31. 11183 11184 If there are more arguments than will fit in these registers, the remaining 11185 arguments are allocated on the stack in order on naturally aligned 11186 addresses. 11187 11188 .. TODO:: 11189 11190 How are overly aligned structures allocated on the stack? 11191 11192* SGPR arguments are assigned to consecutive SGPRs starting at SGPR0 up to 11193 SGPR29. 11194 11195 If there are more arguments than will fit in these registers, the remaining 11196 arguments are allocated on the stack in order on naturally aligned 11197 addresses. 11198 11199Note that decomposed struct type arguments may have some fields passed in 11200registers and some in memory. 11201 11202.. TODO:: 11203 11204 So, a struct which can pass some fields as decomposed register arguments, will 11205 pass the rest as decomposed stack elements? But an argument that will not start 11206 in registers will not be decomposed and will be passed as a non-decomposed 11207 stack value? 11208 11209The following is not part of the AMDGPU function calling convention but 11210describes how the AMDGPU implements function calls: 11211 112121. SGPR33 is used as a frame pointer (FP) if necessary. Like the SP it is an 11213 unswizzled scratch address. It is only needed if runtime sized ``alloca`` 11214 are used, or for the reasons defined in ``SIFrameLowering``. 112152. Runtime stack alignment is supported. SGPR34 is used as a base pointer (BP) 11216 to access the incoming stack arguments in the function. The BP is needed 11217 only when the function requires the runtime stack alignment. 11218 112193. Allocating SGPR arguments on the stack are not supported. 11220 112214. No CFI is currently generated. See 11222 :ref:`amdgpu-dwarf-call-frame-information`. 11223 11224 .. note:: 11225 11226 CFI will be generated that defines the CFA as the unswizzled address 11227 relative to the wave scratch base in the unswizzled private address space 11228 of the lowest address stack allocated local variable. 11229 11230 ``DW_AT_frame_base`` will be defined as the swizzled address in the 11231 swizzled private address space by dividing the CFA by the wavefront size 11232 (since CFA is always at least dword aligned which matches the scratch 11233 swizzle element size). 11234 11235 If no dynamic stack alignment was performed, the stack allocated arguments 11236 are accessed as negative offsets relative to ``DW_AT_frame_base``, and the 11237 local variables and register spill slots are accessed as positive offsets 11238 relative to ``DW_AT_frame_base``. 11239 112405. Function argument passing is implemented by copying the input physical 11241 registers to virtual registers on entry. The register allocator can spill if 11242 necessary. These are copied back to physical registers at call sites. The 11243 net effect is that each function call can have these values in entirely 11244 distinct locations. The IPRA can help avoid shuffling argument registers. 112456. Call sites are implemented by setting up the arguments at positive offsets 11246 from SP. Then SP is incremented to account for the known frame size before 11247 the call and decremented after the call. 11248 11249 .. note:: 11250 11251 The CFI will reflect the changed calculation needed to compute the CFA 11252 from SP. 11253 112547. 4 byte spill slots are used in the stack frame. One slot is allocated for an 11255 emergency spill slot. Buffer instructions are used for stack accesses and 11256 not the ``flat_scratch`` instruction. 11257 11258 .. TODO:: 11259 11260 Explain when the emergency spill slot is used. 11261 11262.. TODO:: 11263 11264 Possible broken issues: 11265 11266 - Stack arguments must be aligned to required alignment. 11267 - Stack is aligned to max(16, max formal argument alignment) 11268 - Direct argument < 64 bits should check register budget. 11269 - Register budget calculation should respect ``inreg`` for SGPR. 11270 - SGPR overflow is not handled. 11271 - struct with 1 member unpeeling is not checking size of member. 11272 - ``sret`` is after ``this`` pointer. 11273 - Caller is not implementing stack realignment: need an extra pointer. 11274 - Should say AMDGPU passes FP rather than SP. 11275 - Should CFI define CFA as address of locals or arguments. Difference is 11276 apparent when have implemented dynamic alignment. 11277 - If ``SCRATCH`` instruction could allow negative offsets, then can make FP be 11278 highest address of stack frame and use negative offset for locals. Would 11279 allow SP to be the same as FP and could support signal-handler-like as now 11280 have a real SP for the top of the stack. 11281 - How is ``sret`` passed on the stack? In argument stack area? Can it overlay 11282 arguments? 11283 11284AMDPAL 11285------ 11286 11287This section provides code conventions used when the target triple OS is 11288``amdpal`` (see :ref:`amdgpu-target-triples`). 11289 11290.. _amdgpu-amdpal-code-object-metadata-section: 11291 11292Code Object Metadata 11293~~~~~~~~~~~~~~~~~~~~ 11294 11295.. note:: 11296 11297 The metadata is currently in development and is subject to major 11298 changes. Only the current version is supported. *When this document 11299 was generated the version was 2.6.* 11300 11301Code object metadata is specified by the ``NT_AMDGPU_METADATA`` note 11302record (see :ref:`amdgpu-note-records-v3-onwards`). 11303 11304The metadata is represented as Message Pack formatted binary data (see 11305[MsgPack]_). The top level is a Message Pack map that includes the keys 11306defined in table :ref:`amdgpu-amdpal-code-object-metadata-map-table` 11307and referenced tables. 11308 11309Additional information can be added to the maps. To avoid conflicts, any 11310key names should be prefixed by "*vendor-name*." where ``vendor-name`` 11311can be the name of the vendor and specific vendor tool that generates the 11312information. The prefix is abbreviated to simply "." when it appears 11313within a map that has been added by the same *vendor-name*. 11314 11315 .. table:: AMDPAL Code Object Metadata Map 11316 :name: amdgpu-amdpal-code-object-metadata-map-table 11317 11318 =================== ============== ========= ====================================================================== 11319 String Key Value Type Required? Description 11320 =================== ============== ========= ====================================================================== 11321 "amdpal.version" sequence of Required PAL code object metadata (major, minor) version. The current values 11322 2 integers are defined by *Util::Abi::PipelineMetadata(Major|Minor)Version*. 11323 "amdpal.pipelines" sequence of Required Per-pipeline metadata. See 11324 map :ref:`amdgpu-amdpal-code-object-pipeline-metadata-map-table` for the 11325 definition of the keys included in that map. 11326 =================== ============== ========= ====================================================================== 11327 11328.. 11329 11330 .. table:: AMDPAL Code Object Pipeline Metadata Map 11331 :name: amdgpu-amdpal-code-object-pipeline-metadata-map-table 11332 11333 ====================================== ============== ========= =================================================== 11334 String Key Value Type Required? Description 11335 ====================================== ============== ========= =================================================== 11336 ".name" string Source name of the pipeline. 11337 ".type" string Pipeline type, e.g. VsPs. Values include: 11338 11339 - "VsPs" 11340 - "Gs" 11341 - "Cs" 11342 - "Ngg" 11343 - "Tess" 11344 - "GsTess" 11345 - "NggTess" 11346 11347 ".internal_pipeline_hash" sequence of Required Internal compiler hash for this pipeline. Lower 11348 2 integers 64 bits is the "stable" portion of the hash, used 11349 for e.g. shader replacement lookup. Upper 64 bits 11350 is the "unique" portion of the hash, used for 11351 e.g. pipeline cache lookup. The value is 11352 implementation defined, and can not be relied on 11353 between different builds of the compiler. 11354 ".shaders" map Per-API shader metadata. See 11355 :ref:`amdgpu-amdpal-code-object-shader-map-table` 11356 for the definition of the keys included in that 11357 map. 11358 ".hardware_stages" map Per-hardware stage metadata. See 11359 :ref:`amdgpu-amdpal-code-object-hardware-stage-map-table` 11360 for the definition of the keys included in that 11361 map. 11362 ".shader_functions" map Per-shader function metadata. See 11363 :ref:`amdgpu-amdpal-code-object-shader-function-map-table` 11364 for the definition of the keys included in that 11365 map. 11366 ".registers" map Required Hardware register configuration. See 11367 :ref:`amdgpu-amdpal-code-object-register-map-table` 11368 for the definition of the keys included in that 11369 map. 11370 ".user_data_limit" integer Number of user data entries accessed by this 11371 pipeline. 11372 ".spill_threshold" integer The user data spill threshold. 0xFFFF for 11373 NoUserDataSpilling. 11374 ".uses_viewport_array_index" boolean Indicates whether or not the pipeline uses the 11375 viewport array index feature. Pipelines which use 11376 this feature can render into all 16 viewports, 11377 whereas pipelines which do not use it are 11378 restricted to viewport #0. 11379 ".es_gs_lds_size" integer Size in bytes of LDS space used internally for 11380 handling data-passing between the ES and GS 11381 shader stages. This can be zero if the data is 11382 passed using off-chip buffers. This value should 11383 be used to program all user-SGPRs which have been 11384 marked with "UserDataMapping::EsGsLdsSize" 11385 (typically only the GS and VS HW stages will ever 11386 have a user-SGPR so marked). 11387 ".nggSubgroupSize" integer Explicit maximum subgroup size for NGG shaders 11388 (maximum number of threads in a subgroup). 11389 ".num_interpolants" integer Graphics only. Number of PS interpolants. 11390 ".mesh_scratch_memory_size" integer Max mesh shader scratch memory used. 11391 ".api" string Name of the client graphics API. 11392 ".api_create_info" binary Graphics API shader create info binary blob. Can 11393 be defined by the driver using the compiler if 11394 they want to be able to correlate API-specific 11395 information used during creation at a later time. 11396 ====================================== ============== ========= =================================================== 11397 11398.. 11399 11400 .. table:: AMDPAL Code Object Shader Map 11401 :name: amdgpu-amdpal-code-object-shader-map-table 11402 11403 11404 +-------------+--------------+-------------------------------------------------------------------+ 11405 |String Key |Value Type |Description | 11406 +=============+==============+===================================================================+ 11407 |- ".compute" |map |See :ref:`amdgpu-amdpal-code-object-api-shader-metadata-map-table` | 11408 |- ".vertex" | |for the definition of the keys included in that map. | 11409 |- ".hull" | | | 11410 |- ".domain" | | | 11411 |- ".geometry"| | | 11412 |- ".pixel" | | | 11413 +-------------+--------------+-------------------------------------------------------------------+ 11414 11415.. 11416 11417 .. table:: AMDPAL Code Object API Shader Metadata Map 11418 :name: amdgpu-amdpal-code-object-api-shader-metadata-map-table 11419 11420 ==================== ============== ========= ===================================================================== 11421 String Key Value Type Required? Description 11422 ==================== ============== ========= ===================================================================== 11423 ".api_shader_hash" sequence of Required Input shader hash, typically passed in from the client. The value 11424 2 integers is implementation defined, and can not be relied on between 11425 different builds of the compiler. 11426 ".hardware_mapping" sequence of Required Flags indicating the HW stages this API shader maps to. Values 11427 string include: 11428 11429 - ".ls" 11430 - ".hs" 11431 - ".es" 11432 - ".gs" 11433 - ".vs" 11434 - ".ps" 11435 - ".cs" 11436 11437 ==================== ============== ========= ===================================================================== 11438 11439.. 11440 11441 .. table:: AMDPAL Code Object Hardware Stage Map 11442 :name: amdgpu-amdpal-code-object-hardware-stage-map-table 11443 11444 +-------------+--------------+-----------------------------------------------------------------------+ 11445 |String Key |Value Type |Description | 11446 +=============+==============+=======================================================================+ 11447 |- ".ls" |map |See :ref:`amdgpu-amdpal-code-object-hardware-stage-metadata-map-table` | 11448 |- ".hs" | |for the definition of the keys included in that map. | 11449 |- ".es" | | | 11450 |- ".gs" | | | 11451 |- ".vs" | | | 11452 |- ".ps" | | | 11453 |- ".cs" | | | 11454 +-------------+--------------+-----------------------------------------------------------------------+ 11455 11456.. 11457 11458 .. table:: AMDPAL Code Object Hardware Stage Metadata Map 11459 :name: amdgpu-amdpal-code-object-hardware-stage-metadata-map-table 11460 11461 ========================== ============== ========= =============================================================== 11462 String Key Value Type Required? Description 11463 ========================== ============== ========= =============================================================== 11464 ".entry_point" string The ELF symbol pointing to this pipeline's stage entry point. 11465 ".scratch_memory_size" integer Scratch memory size in bytes. 11466 ".lds_size" integer Local Data Share size in bytes. 11467 ".perf_data_buffer_size" integer Performance data buffer size in bytes. 11468 ".vgpr_count" integer Number of VGPRs used. 11469 ".agpr_count" integer Number of AGPRs used. 11470 ".sgpr_count" integer Number of SGPRs used. 11471 ".vgpr_limit" integer If non-zero, indicates the shader was compiled with a 11472 directive to instruct the compiler to limit the VGPR usage to 11473 be less than or equal to the specified value (only set if 11474 different from HW default). 11475 ".sgpr_limit" integer SGPR count upper limit (only set if different from HW 11476 default). 11477 ".threadgroup_dimensions" sequence of Thread-group X/Y/Z dimensions (Compute only). 11478 3 integers 11479 ".wavefront_size" integer Wavefront size (only set if different from HW default). 11480 ".uses_uavs" boolean The shader reads or writes UAVs. 11481 ".uses_rovs" boolean The shader reads or writes ROVs. 11482 ".writes_uavs" boolean The shader writes to one or more UAVs. 11483 ".writes_depth" boolean The shader writes out a depth value. 11484 ".uses_append_consume" boolean The shader uses append and/or consume operations, either 11485 memory or GDS. 11486 ".uses_prim_id" boolean The shader uses PrimID. 11487 ========================== ============== ========= =============================================================== 11488 11489.. 11490 11491 .. table:: AMDPAL Code Object Shader Function Map 11492 :name: amdgpu-amdpal-code-object-shader-function-map-table 11493 11494 =============== ============== ==================================================================== 11495 String Key Value Type Description 11496 =============== ============== ==================================================================== 11497 *symbol name* map *symbol name* is the ELF symbol name of the shader function code 11498 entry address. The value is the function's metadata. See 11499 :ref:`amdgpu-amdpal-code-object-shader-function-metadata-map-table`. 11500 =============== ============== ==================================================================== 11501 11502.. 11503 11504 .. table:: AMDPAL Code Object Shader Function Metadata Map 11505 :name: amdgpu-amdpal-code-object-shader-function-metadata-map-table 11506 11507 ============================= ============== ================================================================= 11508 String Key Value Type Description 11509 ============================= ============== ================================================================= 11510 ".api_shader_hash" sequence of Input shader hash, typically passed in from the client. The value 11511 2 integers is implementation defined, and can not be relied on between 11512 different builds of the compiler. 11513 ".scratch_memory_size" integer Size in bytes of scratch memory used by the shader. 11514 ".lds_size" integer Size in bytes of LDS memory. 11515 ".vgpr_count" integer Number of VGPRs used by the shader. 11516 ".sgpr_count" integer Number of SGPRs used by the shader. 11517 ".stack_frame_size_in_bytes" integer Amount of stack size used by the shader. 11518 ".shader_subtype" string Shader subtype/kind. Values include: 11519 11520 - "Unknown" 11521 11522 ============================= ============== ================================================================= 11523 11524.. 11525 11526 .. table:: AMDPAL Code Object Register Map 11527 :name: amdgpu-amdpal-code-object-register-map-table 11528 11529 ========================== ============== ==================================================================== 11530 32-bit Integer Key Value Type Description 11531 ========================== ============== ==================================================================== 11532 ``reg offset`` 32-bit integer ``reg offset`` is the dword offset into the GFXIP register space of 11533 a GRBM register (i.e., driver accessible GPU register number, not 11534 shader GPR register number). The driver is required to program each 11535 specified register to the corresponding specified value when 11536 executing this pipeline. Typically, the ``reg offsets`` are the 11537 ``uint16_t`` offsets to each register as defined by the hardware 11538 chip headers. The register is set to the provided value. However, a 11539 ``reg offset`` that specifies a user data register (e.g., 11540 COMPUTE_USER_DATA_0) needs special treatment. See 11541 :ref:`amdgpu-amdpal-code-object-user-data-section` section for more 11542 information. 11543 ========================== ============== ==================================================================== 11544 11545.. _amdgpu-amdpal-code-object-user-data-section: 11546 11547User Data 11548+++++++++ 11549 11550Each hardware stage has a set of 32-bit physical SPI *user data registers* 11551(either 16 or 32 based on graphics IP and the stage) which can be 11552written from a command buffer and then loaded into SGPRs when waves are 11553launched via a subsequent dispatch or draw operation. This is the way 11554most arguments are passed from the application/runtime to a hardware 11555shader. 11556 11557PAL abstracts this functionality by exposing a set of 128 *user data 11558entries* per pipeline a client can use to pass arguments from a command 11559buffer to one or more shaders in that pipeline. The ELF code object must 11560specify a mapping from virtualized *user data entries* to physical *user 11561data registers*, and PAL is responsible for implementing that mapping, 11562including spilling overflow *user data entries* to memory if needed. 11563 11564Since the *user data registers* are GRBM-accessible SPI registers, this 11565mapping is actually embedded in the ``.registers`` metadata entry. For 11566most registers, the value in that map is a literal 32-bit value that 11567should be written to the register by the driver. However, when the 11568register is a *user data register* (any USER_DATA register e.g., 11569SPI_SHADER_USER_DATA_PS_5), the value is instead an encoding that tells 11570the driver to write either a *user data entry* value or one of several 11571driver-internal values to the register. This encoding is described in 11572the following table: 11573 11574.. note:: 11575 11576 Currently, *user data registers* 0 and 1 (e.g., SPI_SHADER_USER_DATA_PS_0, 11577 and SPI_SHADER_USER_DATA_PS_1) are reserved. *User data register* 0 must 11578 always be programmed to the address of the GlobalTable, and *user data 11579 register* 1 must always be programmed to the address of the PerShaderTable. 11580 11581.. 11582 11583 .. table:: AMDPAL User Data Mapping 11584 :name: amdgpu-amdpal-code-object-metadata-user-data-mapping-table 11585 11586 ========== ================= =============================================================================== 11587 Value Name Description 11588 ========== ================= =============================================================================== 11589 0..127 *User Data Entry* 32-bit value of user_data_entry[N] as specified via *CmdSetUserData()* 11590 0x10000000 GlobalTable 32-bit pointer to GPU memory containing the global internal table (should 11591 always point to *user data register* 0). 11592 0x10000001 PerShaderTable 32-bit pointer to GPU memory containing the per-shader internal table. See 11593 :ref:`amdgpu-amdpal-code-object-metadata-user-data-per-shader-table-section` 11594 for more detail (should always point to *user data register* 1). 11595 0x10000002 SpillTable 32-bit pointer to GPU memory containing the user data spill table. See 11596 :ref:`amdgpu-amdpal-code-object-metadata-user-data-spill-table-section` for 11597 more detail. 11598 0x10000003 BaseVertex Vertex offset (32-bit unsigned integer). Not needed if the pipeline doesn't 11599 reference the draw index in the vertex shader. Only supported by the first 11600 stage in a graphics pipeline. 11601 0x10000004 BaseInstance Instance offset (32-bit unsigned integer). Only supported by the first stage in 11602 a graphics pipeline. 11603 0x10000005 DrawIndex Draw index (32-bit unsigned integer). Only supported by the first stage in a 11604 graphics pipeline. 11605 0x10000006 Workgroup Thread group count (32-bit unsigned integer). Low half of a 64-bit address of 11606 a buffer containing the grid dimensions for a Compute dispatch operation. The 11607 high half of the address is stored in the next sequential user-SGPR. Only 11608 supported by compute pipelines. 11609 0x1000000A EsGsLdsSize Indicates that PAL will program this user-SGPR to contain the amount of LDS 11610 space used for the ES/GS pseudo-ring-buffer for passing data between shader 11611 stages. 11612 0x1000000B ViewId View id (32-bit unsigned integer) identifies a view of graphic 11613 pipeline instancing. 11614 0x1000000C StreamOutTable 32-bit pointer to GPU memory containing the stream out target SRD table. This 11615 can only appear for one shader stage per pipeline. 11616 0x1000000D PerShaderPerfData 32-bit pointer to GPU memory containing the per-shader performance data buffer. 11617 0x1000000F VertexBufferTable 32-bit pointer to GPU memory containing the vertex buffer SRD table. This can 11618 only appear for one shader stage per pipeline. 11619 0x10000010 UavExportTable 32-bit pointer to GPU memory containing the UAV export SRD table. This can 11620 only appear for one shader stage per pipeline (PS). These replace color targets 11621 and are completely separate from any UAVs used by the shader. This is optional, 11622 and only used by the PS when UAV exports are used to replace color-target 11623 exports to optimize specific shaders. 11624 0x10000011 NggCullingData 64-bit pointer to GPU memory containing the hardware register data needed by 11625 some NGG pipelines to perform culling. This value contains the address of the 11626 first of two consecutive registers which provide the full GPU address. 11627 0x10000015 FetchShaderPtr 64-bit pointer to GPU memory containing the fetch shader subroutine. 11628 ========== ================= =============================================================================== 11629 11630.. _amdgpu-amdpal-code-object-metadata-user-data-per-shader-table-section: 11631 11632Per-Shader Table 11633################ 11634 11635Low 32 bits of the GPU address for an optional buffer in the ``.data`` 11636section of the ELF. The high 32 bits of the address match the high 32 bits 11637of the shader's program counter. 11638 11639The buffer can be anything the shader compiler needs it for, and 11640allows each shader to have its own region of the ``.data`` section. 11641Typically, this could be a table of buffer SRD's and the data pointed to 11642by the buffer SRD's, but it could be a flat-address region of memory as 11643well. Its layout and usage are defined by the shader compiler. 11644 11645Each shader's table in the ``.data`` section is referenced by the symbol 11646``_amdgpu_``\ *xs*\ ``_shdr_intrl_data`` where *xs* corresponds with the 11647hardware shader stage the data is for. E.g., 11648``_amdgpu_cs_shdr_intrl_data`` for the compute shader hardware stage. 11649 11650.. _amdgpu-amdpal-code-object-metadata-user-data-spill-table-section: 11651 11652Spill Table 11653########### 11654 11655It is possible for a hardware shader to need access to more *user data 11656entries* than there are slots available in user data registers for one 11657or more hardware shader stages. In that case, the PAL runtime expects 11658the necessary *user data entries* to be spilled to GPU memory and use 11659one user data register to point to the spilled user data memory. The 11660value of the *user data entry* must then represent the location where 11661a shader expects to read the low 32-bits of the table's GPU virtual 11662address. The *spill table* itself represents a set of 32-bit values 11663managed by the PAL runtime in GPU-accessible memory that can be made 11664indirectly accessible to a hardware shader. 11665 11666Unspecified OS 11667-------------- 11668 11669This section provides code conventions used when the target triple OS is 11670empty (see :ref:`amdgpu-target-triples`). 11671 11672Trap Handler ABI 11673~~~~~~~~~~~~~~~~ 11674 11675For code objects generated by AMDGPU backend for non-amdhsa OS, the runtime does 11676not install a trap handler. The ``llvm.trap`` and ``llvm.debugtrap`` 11677instructions are handled as follows: 11678 11679 .. table:: AMDGPU Trap Handler for Non-AMDHSA OS 11680 :name: amdgpu-trap-handler-for-non-amdhsa-os-table 11681 11682 =============== =============== =========================================== 11683 Usage Code Sequence Description 11684 =============== =============== =========================================== 11685 llvm.trap s_endpgm Causes wavefront to be terminated. 11686 llvm.debugtrap *none* Compiler warning given that there is no 11687 trap handler installed. 11688 =============== =============== =========================================== 11689 11690Source Languages 11691================ 11692 11693.. _amdgpu-opencl: 11694 11695OpenCL 11696------ 11697 11698When the language is OpenCL the following differences occur: 11699 117001. The OpenCL memory model is used (see :ref:`amdgpu-amdhsa-memory-model`). 117012. The AMDGPU backend appends additional arguments to the kernel's explicit 11702 arguments for the AMDHSA OS (see 11703 :ref:`opencl-kernel-implicit-arguments-appended-for-amdhsa-os-table`). 117043. Additional metadata is generated 11705 (see :ref:`amdgpu-amdhsa-code-object-metadata`). 11706 11707 .. table:: OpenCL kernel implicit arguments appended for AMDHSA OS 11708 :name: opencl-kernel-implicit-arguments-appended-for-amdhsa-os-table 11709 11710 ======== ==== ========= =========================================== 11711 Position Byte Byte Description 11712 Size Alignment 11713 ======== ==== ========= =========================================== 11714 1 8 8 OpenCL Global Offset X 11715 2 8 8 OpenCL Global Offset Y 11716 3 8 8 OpenCL Global Offset Z 11717 4 8 8 OpenCL address of printf buffer 11718 5 8 8 OpenCL address of virtual queue used by 11719 enqueue_kernel. 11720 6 8 8 OpenCL address of AqlWrap struct used by 11721 enqueue_kernel. 11722 7 8 8 Pointer argument used for Multi-gird 11723 synchronization. 11724 ======== ==== ========= =========================================== 11725 11726.. _amdgpu-hcc: 11727 11728HCC 11729--- 11730 11731When the language is HCC the following differences occur: 11732 117331. The HSA memory model is used (see :ref:`amdgpu-amdhsa-memory-model`). 11734 11735.. _amdgpu-assembler: 11736 11737Assembler 11738--------- 11739 11740AMDGPU backend has LLVM-MC based assembler which is currently in development. 11741It supports AMDGCN GFX6-GFX10. 11742 11743This section describes general syntax for instructions and operands. 11744 11745Instructions 11746~~~~~~~~~~~~ 11747 11748An instruction has the following :doc:`syntax<AMDGPUInstructionSyntax>`: 11749 11750 | ``<``\ *opcode*\ ``> <``\ *operand0*\ ``>, <``\ *operand1*\ ``>,... 11751 <``\ *modifier0*\ ``> <``\ *modifier1*\ ``>...`` 11752 11753:doc:`Operands<AMDGPUOperandSyntax>` are comma-separated while 11754:doc:`modifiers<AMDGPUModifierSyntax>` are space-separated. 11755 11756The order of operands and modifiers is fixed. 11757Most modifiers are optional and may be omitted. 11758 11759Links to detailed instruction syntax description may be found in the following 11760table. Note that features under development are not included 11761in this description. 11762 11763 =================================== ======================================= 11764 Core ISA ISA Extensions 11765 =================================== ======================================= 11766 :doc:`GFX7<AMDGPU/AMDGPUAsmGFX7>` \- 11767 :doc:`GFX8<AMDGPU/AMDGPUAsmGFX8>` \- 11768 :doc:`GFX9<AMDGPU/AMDGPUAsmGFX9>` :doc:`gfx900<AMDGPU/AMDGPUAsmGFX900>` 11769 11770 :doc:`gfx902<AMDGPU/AMDGPUAsmGFX900>` 11771 11772 :doc:`gfx904<AMDGPU/AMDGPUAsmGFX904>` 11773 11774 :doc:`gfx906<AMDGPU/AMDGPUAsmGFX906>` 11775 11776 :doc:`gfx908<AMDGPU/AMDGPUAsmGFX908>` 11777 11778 :doc:`gfx909<AMDGPU/AMDGPUAsmGFX900>` 11779 11780 :doc:`gfx90a<AMDGPU/AMDGPUAsmGFX90a>` 11781 11782 :doc:`GFX10<AMDGPU/AMDGPUAsmGFX10>` :doc:`gfx1011<AMDGPU/AMDGPUAsmGFX1011>` 11783 11784 :doc:`gfx1012<AMDGPU/AMDGPUAsmGFX1011>` 11785 =================================== ======================================= 11786 11787For more information about instructions, their semantics and supported 11788combinations of operands, refer to one of instruction set architecture manuals 11789[AMD-GCN-GFX6]_, [AMD-GCN-GFX7]_, [AMD-GCN-GFX8]_, 11790[AMD-GCN-GFX900-GFX904-VEGA]_, [AMD-GCN-GFX906-VEGA7NM]_ 11791[AMD-GCN-GFX908-CDNA1]_, [AMD-GCN-GFX10-RDNA1]_ and [AMD-GCN-GFX10-RDNA2]_. 11792 11793Operands 11794~~~~~~~~ 11795 11796Detailed description of operands may be found :doc:`here<AMDGPUOperandSyntax>`. 11797 11798Modifiers 11799~~~~~~~~~ 11800 11801Detailed description of modifiers may be found 11802:doc:`here<AMDGPUModifierSyntax>`. 11803 11804Instruction Examples 11805~~~~~~~~~~~~~~~~~~~~ 11806 11807DS 11808++ 11809 11810.. code-block:: nasm 11811 11812 ds_add_u32 v2, v4 offset:16 11813 ds_write_src2_b64 v2 offset0:4 offset1:8 11814 ds_cmpst_f32 v2, v4, v6 11815 ds_min_rtn_f64 v[8:9], v2, v[4:5] 11816 11817For full list of supported instructions, refer to "LDS/GDS instructions" in ISA 11818Manual. 11819 11820FLAT 11821++++ 11822 11823.. code-block:: nasm 11824 11825 flat_load_dword v1, v[3:4] 11826 flat_store_dwordx3 v[3:4], v[5:7] 11827 flat_atomic_swap v1, v[3:4], v5 glc 11828 flat_atomic_cmpswap v1, v[3:4], v[5:6] glc slc 11829 flat_atomic_fmax_x2 v[1:2], v[3:4], v[5:6] glc 11830 11831For full list of supported instructions, refer to "FLAT instructions" in ISA 11832Manual. 11833 11834MUBUF 11835+++++ 11836 11837.. code-block:: nasm 11838 11839 buffer_load_dword v1, off, s[4:7], s1 11840 buffer_store_dwordx4 v[1:4], v2, ttmp[4:7], s1 offen offset:4 glc tfe 11841 buffer_store_format_xy v[1:2], off, s[4:7], s1 11842 buffer_wbinvl1 11843 buffer_atomic_inc v1, v2, s[8:11], s4 idxen offset:4 slc 11844 11845For full list of supported instructions, refer to "MUBUF Instructions" in ISA 11846Manual. 11847 11848SMRD/SMEM 11849+++++++++ 11850 11851.. code-block:: nasm 11852 11853 s_load_dword s1, s[2:3], 0xfc 11854 s_load_dwordx8 s[8:15], s[2:3], s4 11855 s_load_dwordx16 s[88:103], s[2:3], s4 11856 s_dcache_inv_vol 11857 s_memtime s[4:5] 11858 11859For full list of supported instructions, refer to "Scalar Memory Operations" in 11860ISA Manual. 11861 11862SOP1 11863++++ 11864 11865.. code-block:: nasm 11866 11867 s_mov_b32 s1, s2 11868 s_mov_b64 s[0:1], 0x80000000 11869 s_cmov_b32 s1, 200 11870 s_wqm_b64 s[2:3], s[4:5] 11871 s_bcnt0_i32_b64 s1, s[2:3] 11872 s_swappc_b64 s[2:3], s[4:5] 11873 s_cbranch_join s[4:5] 11874 11875For full list of supported instructions, refer to "SOP1 Instructions" in ISA 11876Manual. 11877 11878SOP2 11879++++ 11880 11881.. code-block:: nasm 11882 11883 s_add_u32 s1, s2, s3 11884 s_and_b64 s[2:3], s[4:5], s[6:7] 11885 s_cselect_b32 s1, s2, s3 11886 s_andn2_b32 s2, s4, s6 11887 s_lshr_b64 s[2:3], s[4:5], s6 11888 s_ashr_i32 s2, s4, s6 11889 s_bfm_b64 s[2:3], s4, s6 11890 s_bfe_i64 s[2:3], s[4:5], s6 11891 s_cbranch_g_fork s[4:5], s[6:7] 11892 11893For full list of supported instructions, refer to "SOP2 Instructions" in ISA 11894Manual. 11895 11896SOPC 11897++++ 11898 11899.. code-block:: nasm 11900 11901 s_cmp_eq_i32 s1, s2 11902 s_bitcmp1_b32 s1, s2 11903 s_bitcmp0_b64 s[2:3], s4 11904 s_setvskip s3, s5 11905 11906For full list of supported instructions, refer to "SOPC Instructions" in ISA 11907Manual. 11908 11909SOPP 11910++++ 11911 11912.. code-block:: nasm 11913 11914 s_barrier 11915 s_nop 2 11916 s_endpgm 11917 s_waitcnt 0 ; Wait for all counters to be 0 11918 s_waitcnt vmcnt(0) & expcnt(0) & lgkmcnt(0) ; Equivalent to above 11919 s_waitcnt vmcnt(1) ; Wait for vmcnt counter to be 1. 11920 s_sethalt 9 11921 s_sleep 10 11922 s_sendmsg 0x1 11923 s_sendmsg sendmsg(MSG_INTERRUPT) 11924 s_trap 1 11925 11926For full list of supported instructions, refer to "SOPP Instructions" in ISA 11927Manual. 11928 11929Unless otherwise mentioned, little verification is performed on the operands 11930of SOPP Instructions, so it is up to the programmer to be familiar with the 11931range or acceptable values. 11932 11933VALU 11934++++ 11935 11936For vector ALU instruction opcodes (VOP1, VOP2, VOP3, VOPC, VOP_DPP, VOP_SDWA), 11937the assembler will automatically use optimal encoding based on its operands. To 11938force specific encoding, one can add a suffix to the opcode of the instruction: 11939 11940* _e32 for 32-bit VOP1/VOP2/VOPC 11941* _e64 for 64-bit VOP3 11942* _dpp for VOP_DPP 11943* _sdwa for VOP_SDWA 11944 11945VOP1/VOP2/VOP3/VOPC examples: 11946 11947.. code-block:: nasm 11948 11949 v_mov_b32 v1, v2 11950 v_mov_b32_e32 v1, v2 11951 v_nop 11952 v_cvt_f64_i32_e32 v[1:2], v2 11953 v_floor_f32_e32 v1, v2 11954 v_bfrev_b32_e32 v1, v2 11955 v_add_f32_e32 v1, v2, v3 11956 v_mul_i32_i24_e64 v1, v2, 3 11957 v_mul_i32_i24_e32 v1, -3, v3 11958 v_mul_i32_i24_e32 v1, -100, v3 11959 v_addc_u32 v1, s[0:1], v2, v3, s[2:3] 11960 v_max_f16_e32 v1, v2, v3 11961 11962VOP_DPP examples: 11963 11964.. code-block:: nasm 11965 11966 v_mov_b32 v0, v0 quad_perm:[0,2,1,1] 11967 v_sin_f32 v0, v0 row_shl:1 row_mask:0xa bank_mask:0x1 bound_ctrl:0 11968 v_mov_b32 v0, v0 wave_shl:1 11969 v_mov_b32 v0, v0 row_mirror 11970 v_mov_b32 v0, v0 row_bcast:31 11971 v_mov_b32 v0, v0 quad_perm:[1,3,0,1] row_mask:0xa bank_mask:0x1 bound_ctrl:0 11972 v_add_f32 v0, v0, |v0| row_shl:1 row_mask:0xa bank_mask:0x1 bound_ctrl:0 11973 v_max_f16 v1, v2, v3 row_shl:1 row_mask:0xa bank_mask:0x1 bound_ctrl:0 11974 11975VOP_SDWA examples: 11976 11977.. code-block:: nasm 11978 11979 v_mov_b32 v1, v2 dst_sel:BYTE_0 dst_unused:UNUSED_PRESERVE src0_sel:DWORD 11980 v_min_u32 v200, v200, v1 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:BYTE_1 src1_sel:DWORD 11981 v_sin_f32 v0, v0 dst_unused:UNUSED_PAD src0_sel:WORD_1 11982 v_fract_f32 v0, |v0| dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 11983 v_cmpx_le_u32 vcc, v1, v2 src0_sel:BYTE_2 src1_sel:WORD_0 11984 11985For full list of supported instructions, refer to "Vector ALU instructions". 11986 11987.. _amdgpu-amdhsa-assembler-predefined-symbols-v2: 11988 11989Code Object V2 Predefined Symbols 11990~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 11991 11992.. warning:: 11993 Code object V2 is not the default code object version emitted by 11994 this version of LLVM. 11995 11996The AMDGPU assembler defines and updates some symbols automatically. These 11997symbols do not affect code generation. 11998 11999.option.machine_version_major 12000+++++++++++++++++++++++++++++ 12001 12002Set to the GFX major generation number of the target being assembled for. For 12003example, when assembling for a "GFX9" target this will be set to the integer 12004value "9". The possible GFX major generation numbers are presented in 12005:ref:`amdgpu-processors`. 12006 12007.option.machine_version_minor 12008+++++++++++++++++++++++++++++ 12009 12010Set to the GFX minor generation number of the target being assembled for. For 12011example, when assembling for a "GFX810" target this will be set to the integer 12012value "1". The possible GFX minor generation numbers are presented in 12013:ref:`amdgpu-processors`. 12014 12015.option.machine_version_stepping 12016++++++++++++++++++++++++++++++++ 12017 12018Set to the GFX stepping generation number of the target being assembled for. 12019For example, when assembling for a "GFX704" target this will be set to the 12020integer value "4". The possible GFX stepping generation numbers are presented 12021in :ref:`amdgpu-processors`. 12022 12023.kernel.vgpr_count 12024++++++++++++++++++ 12025 12026Set to zero each time a 12027:ref:`amdgpu-amdhsa-assembler-directive-amdgpu_hsa_kernel` directive is 12028encountered. At each instruction, if the current value of this symbol is less 12029than or equal to the maximum VGPR number explicitly referenced within that 12030instruction then the symbol value is updated to equal that VGPR number plus 12031one. 12032 12033.kernel.sgpr_count 12034++++++++++++++++++ 12035 12036Set to zero each time a 12037:ref:`amdgpu-amdhsa-assembler-directive-amdgpu_hsa_kernel` directive is 12038encountered. At each instruction, if the current value of this symbol is less 12039than or equal to the maximum VGPR number explicitly referenced within that 12040instruction then the symbol value is updated to equal that SGPR number plus 12041one. 12042 12043.. _amdgpu-amdhsa-assembler-directives-v2: 12044 12045Code Object V2 Directives 12046~~~~~~~~~~~~~~~~~~~~~~~~~ 12047 12048.. warning:: 12049 Code object V2 is not the default code object version emitted by 12050 this version of LLVM. 12051 12052AMDGPU ABI defines auxiliary data in output code object. In assembly source, 12053one can specify them with assembler directives. 12054 12055.hsa_code_object_version major, minor 12056+++++++++++++++++++++++++++++++++++++ 12057 12058*major* and *minor* are integers that specify the version of the HSA code 12059object that will be generated by the assembler. 12060 12061.hsa_code_object_isa [major, minor, stepping, vendor, arch] 12062+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 12063 12064 12065*major*, *minor*, and *stepping* are all integers that describe the instruction 12066set architecture (ISA) version of the assembly program. 12067 12068*vendor* and *arch* are quoted strings. *vendor* should always be equal to 12069"AMD" and *arch* should always be equal to "AMDGPU". 12070 12071By default, the assembler will derive the ISA version, *vendor*, and *arch* 12072from the value of the -mcpu option that is passed to the assembler. 12073 12074.. _amdgpu-amdhsa-assembler-directive-amdgpu_hsa_kernel: 12075 12076.amdgpu_hsa_kernel (name) 12077+++++++++++++++++++++++++ 12078 12079This directives specifies that the symbol with given name is a kernel entry 12080point (label) and the object should contain corresponding symbol of type 12081STT_AMDGPU_HSA_KERNEL. 12082 12083.amd_kernel_code_t 12084++++++++++++++++++ 12085 12086This directive marks the beginning of a list of key / value pairs that are used 12087to specify the amd_kernel_code_t object that will be emitted by the assembler. 12088The list must be terminated by the *.end_amd_kernel_code_t* directive. For any 12089amd_kernel_code_t values that are unspecified a default value will be used. The 12090default value for all keys is 0, with the following exceptions: 12091 12092- *amd_code_version_major* defaults to 1. 12093- *amd_kernel_code_version_minor* defaults to 2. 12094- *amd_machine_kind* defaults to 1. 12095- *amd_machine_version_major*, *machine_version_minor*, and 12096 *amd_machine_version_stepping* are derived from the value of the -mcpu option 12097 that is passed to the assembler. 12098- *kernel_code_entry_byte_offset* defaults to 256. 12099- *wavefront_size* defaults 6 for all targets before GFX10. For GFX10 onwards 12100 defaults to 6 if target feature ``wavefrontsize64`` is enabled, otherwise 5. 12101 Note that wavefront size is specified as a power of two, so a value of **n** 12102 means a size of 2^ **n**. 12103- *call_convention* defaults to -1. 12104- *kernarg_segment_alignment*, *group_segment_alignment*, and 12105 *private_segment_alignment* default to 4. Note that alignments are specified 12106 as a power of 2, so a value of **n** means an alignment of 2^ **n**. 12107- *enable_tg_split* defaults to 1 if target feature ``tgsplit`` is enabled for 12108 GFX90A onwards. 12109- *enable_wgp_mode* defaults to 1 if target feature ``cumode`` is disabled for 12110 GFX10 onwards. 12111- *enable_mem_ordered* defaults to 1 for GFX10 onwards. 12112 12113The *.amd_kernel_code_t* directive must be placed immediately after the 12114function label and before any instructions. 12115 12116For a full list of amd_kernel_code_t keys, refer to AMDGPU ABI document, 12117comments in lib/Target/AMDGPU/AmdKernelCodeT.h and test/CodeGen/AMDGPU/hsa.s. 12118 12119.. _amdgpu-amdhsa-assembler-example-v2: 12120 12121Code Object V2 Example Source Code 12122~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 12123 12124.. warning:: 12125 Code Object V2 is not the default code object version emitted by 12126 this version of LLVM. 12127 12128Here is an example of a minimal assembly source file, defining one HSA kernel: 12129 12130.. code:: 12131 :number-lines: 12132 12133 .hsa_code_object_version 1,0 12134 .hsa_code_object_isa 12135 12136 .hsatext 12137 .globl hello_world 12138 .p2align 8 12139 .amdgpu_hsa_kernel hello_world 12140 12141 hello_world: 12142 12143 .amd_kernel_code_t 12144 enable_sgpr_kernarg_segment_ptr = 1 12145 is_ptr64 = 1 12146 compute_pgm_rsrc1_vgprs = 0 12147 compute_pgm_rsrc1_sgprs = 0 12148 compute_pgm_rsrc2_user_sgpr = 2 12149 compute_pgm_rsrc1_wgp_mode = 0 12150 compute_pgm_rsrc1_mem_ordered = 0 12151 compute_pgm_rsrc1_fwd_progress = 1 12152 .end_amd_kernel_code_t 12153 12154 s_load_dwordx2 s[0:1], s[0:1] 0x0 12155 v_mov_b32 v0, 3.14159 12156 s_waitcnt lgkmcnt(0) 12157 v_mov_b32 v1, s0 12158 v_mov_b32 v2, s1 12159 flat_store_dword v[1:2], v0 12160 s_endpgm 12161 .Lfunc_end0: 12162 .size hello_world, .Lfunc_end0-hello_world 12163 12164.. _amdgpu-amdhsa-assembler-predefined-symbols-v3-onwards: 12165 12166Code Object V3 and Above Predefined Symbols 12167~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 12168 12169The AMDGPU assembler defines and updates some symbols automatically. These 12170symbols do not affect code generation. 12171 12172.amdgcn.gfx_generation_number 12173+++++++++++++++++++++++++++++ 12174 12175Set to the GFX major generation number of the target being assembled for. For 12176example, when assembling for a "GFX9" target this will be set to the integer 12177value "9". The possible GFX major generation numbers are presented in 12178:ref:`amdgpu-processors`. 12179 12180.amdgcn.gfx_generation_minor 12181++++++++++++++++++++++++++++ 12182 12183Set to the GFX minor generation number of the target being assembled for. For 12184example, when assembling for a "GFX810" target this will be set to the integer 12185value "1". The possible GFX minor generation numbers are presented in 12186:ref:`amdgpu-processors`. 12187 12188.amdgcn.gfx_generation_stepping 12189+++++++++++++++++++++++++++++++ 12190 12191Set to the GFX stepping generation number of the target being assembled for. 12192For example, when assembling for a "GFX704" target this will be set to the 12193integer value "4". The possible GFX stepping generation numbers are presented 12194in :ref:`amdgpu-processors`. 12195 12196.. _amdgpu-amdhsa-assembler-symbol-next_free_vgpr: 12197 12198.amdgcn.next_free_vgpr 12199++++++++++++++++++++++ 12200 12201Set to zero before assembly begins. At each instruction, if the current value 12202of this symbol is less than or equal to the maximum VGPR number explicitly 12203referenced within that instruction then the symbol value is updated to equal 12204that VGPR number plus one. 12205 12206May be used to set the `.amdhsa_next_free_vgpr` directive in 12207:ref:`amdhsa-kernel-directives-table`. 12208 12209May be set at any time, e.g. manually set to zero at the start of each kernel. 12210 12211.. _amdgpu-amdhsa-assembler-symbol-next_free_sgpr: 12212 12213.amdgcn.next_free_sgpr 12214++++++++++++++++++++++ 12215 12216Set to zero before assembly begins. At each instruction, if the current value 12217of this symbol is less than or equal the maximum SGPR number explicitly 12218referenced within that instruction then the symbol value is updated to equal 12219that SGPR number plus one. 12220 12221May be used to set the `.amdhsa_next_free_spgr` directive in 12222:ref:`amdhsa-kernel-directives-table`. 12223 12224May be set at any time, e.g. manually set to zero at the start of each kernel. 12225 12226.. _amdgpu-amdhsa-assembler-directives-v3-onwards: 12227 12228Code Object V3 and Above Directives 12229~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 12230 12231Directives which begin with ``.amdgcn`` are valid for all ``amdgcn`` 12232architecture processors, and are not OS-specific. Directives which begin with 12233``.amdhsa`` are specific to ``amdgcn`` architecture processors when the 12234``amdhsa`` OS is specified. See :ref:`amdgpu-target-triples` and 12235:ref:`amdgpu-processors`. 12236 12237.. _amdgpu-assembler-directive-amdgcn-target: 12238 12239.amdgcn_target <target-triple> "-" <target-id> 12240++++++++++++++++++++++++++++++++++++++++++++++ 12241 12242Optional directive which declares the ``<target-triple>-<target-id>`` supported 12243by the containing assembler source file. Used by the assembler to validate 12244command-line options such as ``-triple``, ``-mcpu``, and 12245``--offload-arch=<target-id>``. A non-canonical target ID is allowed. See 12246:ref:`amdgpu-target-triples` and :ref:`amdgpu-target-id`. 12247 12248.. note:: 12249 12250 The target ID syntax used for code object V2 to V3 for this directive differs 12251 from that used elsewhere. See :ref:`amdgpu-target-id-v2-v3`. 12252 12253.amdhsa_kernel <name> 12254+++++++++++++++++++++ 12255 12256Creates a correctly aligned AMDHSA kernel descriptor and a symbol, 12257``<name>.kd``, in the current location of the current section. Only valid when 12258the OS is ``amdhsa``. ``<name>`` must be a symbol that labels the first 12259instruction to execute, and does not need to be previously defined. 12260 12261Marks the beginning of a list of directives used to generate the bytes of a 12262kernel descriptor, as described in :ref:`amdgpu-amdhsa-kernel-descriptor`. 12263Directives which may appear in this list are described in 12264:ref:`amdhsa-kernel-directives-table`. Directives may appear in any order, must 12265be valid for the target being assembled for, and cannot be repeated. Directives 12266support the range of values specified by the field they reference in 12267:ref:`amdgpu-amdhsa-kernel-descriptor`. If a directive is not specified, it is 12268assumed to have its default value, unless it is marked as "Required", in which 12269case it is an error to omit the directive. This list of directives is 12270terminated by an ``.end_amdhsa_kernel`` directive. 12271 12272 .. table:: AMDHSA Kernel Assembler Directives 12273 :name: amdhsa-kernel-directives-table 12274 12275 ======================================================== =================== ============ =================== 12276 Directive Default Supported On Description 12277 ======================================================== =================== ============ =================== 12278 ``.amdhsa_group_segment_fixed_size`` 0 GFX6-GFX10 Controls GROUP_SEGMENT_FIXED_SIZE in 12279 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 12280 ``.amdhsa_private_segment_fixed_size`` 0 GFX6-GFX10 Controls PRIVATE_SEGMENT_FIXED_SIZE in 12281 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 12282 ``.amdhsa_kernarg_size`` 0 GFX6-GFX10 Controls KERNARG_SIZE in 12283 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 12284 ``.amdhsa_user_sgpr_count`` 0 GFX6-GFX10 Controls USER_SGPR_COUNT in COMPUTE_PGM_RSRC2 12285 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table` 12286 ``.amdhsa_user_sgpr_private_segment_buffer`` 0 GFX6-GFX10 Controls ENABLE_SGPR_PRIVATE_SEGMENT_BUFFER in 12287 (except :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 12288 GFX940) 12289 ``.amdhsa_user_sgpr_dispatch_ptr`` 0 GFX6-GFX10 Controls ENABLE_SGPR_DISPATCH_PTR in 12290 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 12291 ``.amdhsa_user_sgpr_queue_ptr`` 0 GFX6-GFX10 Controls ENABLE_SGPR_QUEUE_PTR in 12292 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 12293 ``.amdhsa_user_sgpr_kernarg_segment_ptr`` 0 GFX6-GFX10 Controls ENABLE_SGPR_KERNARG_SEGMENT_PTR in 12294 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 12295 ``.amdhsa_user_sgpr_dispatch_id`` 0 GFX6-GFX10 Controls ENABLE_SGPR_DISPATCH_ID in 12296 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 12297 ``.amdhsa_user_sgpr_flat_scratch_init`` 0 GFX6-GFX10 Controls ENABLE_SGPR_FLAT_SCRATCH_INIT in 12298 (except :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 12299 GFX940) 12300 ``.amdhsa_user_sgpr_private_segment_size`` 0 GFX6-GFX10 Controls ENABLE_SGPR_PRIVATE_SEGMENT_SIZE in 12301 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 12302 ``.amdhsa_wavefront_size32`` Target GFX10 Controls ENABLE_WAVEFRONT_SIZE32 in 12303 Feature :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 12304 Specific 12305 (wavefrontsize64) 12306 ``.amdhsa_system_sgpr_private_segment_wavefront_offset`` 0 GFX6-GFX10 Controls ENABLE_PRIVATE_SEGMENT in 12307 (except :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 12308 GFX940) 12309 ``.amdhsa_enable_private_segment`` 0 GFX940 Controls ENABLE_PRIVATE_SEGMENT in 12310 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 12311 ``.amdhsa_system_sgpr_workgroup_id_x`` 1 GFX6-GFX10 Controls ENABLE_SGPR_WORKGROUP_ID_X in 12312 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 12313 ``.amdhsa_system_sgpr_workgroup_id_y`` 0 GFX6-GFX10 Controls ENABLE_SGPR_WORKGROUP_ID_Y in 12314 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 12315 ``.amdhsa_system_sgpr_workgroup_id_z`` 0 GFX6-GFX10 Controls ENABLE_SGPR_WORKGROUP_ID_Z in 12316 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 12317 ``.amdhsa_system_sgpr_workgroup_info`` 0 GFX6-GFX10 Controls ENABLE_SGPR_WORKGROUP_INFO in 12318 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 12319 ``.amdhsa_system_vgpr_workitem_id`` 0 GFX6-GFX10 Controls ENABLE_VGPR_WORKITEM_ID in 12320 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 12321 Possible values are defined in 12322 :ref:`amdgpu-amdhsa-system-vgpr-work-item-id-enumeration-values-table`. 12323 ``.amdhsa_next_free_vgpr`` Required GFX6-GFX10 Maximum VGPR number explicitly referenced, plus one. 12324 Used to calculate GRANULATED_WORKITEM_VGPR_COUNT in 12325 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 12326 ``.amdhsa_next_free_sgpr`` Required GFX6-GFX10 Maximum SGPR number explicitly referenced, plus one. 12327 Used to calculate GRANULATED_WAVEFRONT_SGPR_COUNT in 12328 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 12329 ``.amdhsa_accum_offset`` Required GFX90A, Offset of a first AccVGPR in the unified register file. 12330 GFX940 Used to calculate ACCUM_OFFSET in 12331 :ref:`amdgpu-amdhsa-compute_pgm_rsrc3-gfx90a-table`. 12332 ``.amdhsa_reserve_vcc`` 1 GFX6-GFX10 Whether the kernel may use the special VCC SGPR. 12333 Used to calculate GRANULATED_WAVEFRONT_SGPR_COUNT in 12334 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 12335 ``.amdhsa_reserve_flat_scratch`` 1 GFX7-GFX10 Whether the kernel may use flat instructions to access 12336 (except scratch memory. Used to calculate 12337 GFX940) GRANULATED_WAVEFRONT_SGPR_COUNT in 12338 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 12339 ``.amdhsa_reserve_xnack_mask`` Target GFX8-GFX10 Whether the kernel may trigger XNACK replay. 12340 Feature Used to calculate GRANULATED_WAVEFRONT_SGPR_COUNT in 12341 Specific :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 12342 (xnack) 12343 ``.amdhsa_float_round_mode_32`` 0 GFX6-GFX10 Controls FLOAT_ROUND_MODE_32 in 12344 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 12345 Possible values are defined in 12346 :ref:`amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table`. 12347 ``.amdhsa_float_round_mode_16_64`` 0 GFX6-GFX10 Controls FLOAT_ROUND_MODE_16_64 in 12348 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 12349 Possible values are defined in 12350 :ref:`amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table`. 12351 ``.amdhsa_float_denorm_mode_32`` 0 GFX6-GFX10 Controls FLOAT_DENORM_MODE_32 in 12352 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 12353 Possible values are defined in 12354 :ref:`amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table`. 12355 ``.amdhsa_float_denorm_mode_16_64`` 3 GFX6-GFX10 Controls FLOAT_DENORM_MODE_16_64 in 12356 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 12357 Possible values are defined in 12358 :ref:`amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table`. 12359 ``.amdhsa_dx10_clamp`` 1 GFX6-GFX10 Controls ENABLE_DX10_CLAMP in 12360 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 12361 ``.amdhsa_ieee_mode`` 1 GFX6-GFX10 Controls ENABLE_IEEE_MODE in 12362 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 12363 ``.amdhsa_fp16_overflow`` 0 GFX9-GFX10 Controls FP16_OVFL in 12364 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 12365 ``.amdhsa_tg_split`` Target GFX90A, Controls TG_SPLIT in 12366 Feature GFX940 :ref:`amdgpu-amdhsa-compute_pgm_rsrc3-gfx90a-table`. 12367 Specific 12368 (tgsplit) 12369 ``.amdhsa_workgroup_processor_mode`` Target GFX10 Controls ENABLE_WGP_MODE in 12370 Feature :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 12371 Specific 12372 (cumode) 12373 ``.amdhsa_memory_ordered`` 1 GFX10 Controls MEM_ORDERED in 12374 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 12375 ``.amdhsa_forward_progress`` 0 GFX10 Controls FWD_PROGRESS in 12376 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 12377 ``.amdhsa_shared_vgpr_count`` 0 GFX10 Controls SHARED_VGPR_COUNT in 12378 :ref:`amdgpu-amdhsa-compute_pgm_rsrc3-gfx10-table`. 12379 ``.amdhsa_exception_fp_ieee_invalid_op`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_INVALID_OPERATION in 12380 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 12381 ``.amdhsa_exception_fp_denorm_src`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_FP_DENORMAL_SOURCE in 12382 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 12383 ``.amdhsa_exception_fp_ieee_div_zero`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_DIVISION_BY_ZERO in 12384 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 12385 ``.amdhsa_exception_fp_ieee_overflow`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_OVERFLOW in 12386 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 12387 ``.amdhsa_exception_fp_ieee_underflow`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_UNDERFLOW in 12388 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 12389 ``.amdhsa_exception_fp_ieee_inexact`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_INEXACT in 12390 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 12391 ``.amdhsa_exception_int_div_zero`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_INT_DIVIDE_BY_ZERO in 12392 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 12393 ======================================================== =================== ============ =================== 12394 12395.amdgpu_metadata 12396++++++++++++++++ 12397 12398Optional directive which declares the contents of the ``NT_AMDGPU_METADATA`` 12399note record (see :ref:`amdgpu-elf-note-records-table-v3-onwards`). 12400 12401The contents must be in the [YAML]_ markup format, with the same structure and 12402semantics described in :ref:`amdgpu-amdhsa-code-object-metadata-v3`, 12403:ref:`amdgpu-amdhsa-code-object-metadata-v4` or 12404:ref:`amdgpu-amdhsa-code-object-metadata-v5`. 12405 12406This directive is terminated by an ``.end_amdgpu_metadata`` directive. 12407 12408.. _amdgpu-amdhsa-assembler-example-v3-onwards: 12409 12410Code Object V3 and Above Example Source Code 12411~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 12412 12413Here is an example of a minimal assembly source file, defining one HSA kernel: 12414 12415.. code:: 12416 :number-lines: 12417 12418 .amdgcn_target "amdgcn-amd-amdhsa--gfx900+xnack" // optional 12419 12420 .text 12421 .globl hello_world 12422 .p2align 8 12423 .type hello_world,@function 12424 hello_world: 12425 s_load_dwordx2 s[0:1], s[0:1] 0x0 12426 v_mov_b32 v0, 3.14159 12427 s_waitcnt lgkmcnt(0) 12428 v_mov_b32 v1, s0 12429 v_mov_b32 v2, s1 12430 flat_store_dword v[1:2], v0 12431 s_endpgm 12432 .Lfunc_end0: 12433 .size hello_world, .Lfunc_end0-hello_world 12434 12435 .rodata 12436 .p2align 6 12437 .amdhsa_kernel hello_world 12438 .amdhsa_user_sgpr_kernarg_segment_ptr 1 12439 .amdhsa_next_free_vgpr .amdgcn.next_free_vgpr 12440 .amdhsa_next_free_sgpr .amdgcn.next_free_sgpr 12441 .end_amdhsa_kernel 12442 12443 .amdgpu_metadata 12444 --- 12445 amdhsa.version: 12446 - 1 12447 - 0 12448 amdhsa.kernels: 12449 - .name: hello_world 12450 .symbol: hello_world.kd 12451 .kernarg_segment_size: 48 12452 .group_segment_fixed_size: 0 12453 .private_segment_fixed_size: 0 12454 .kernarg_segment_align: 4 12455 .wavefront_size: 64 12456 .sgpr_count: 2 12457 .vgpr_count: 3 12458 .max_flat_workgroup_size: 256 12459 .args: 12460 - .size: 8 12461 .offset: 0 12462 .value_kind: global_buffer 12463 .address_space: global 12464 .actual_access: write_only 12465 //... 12466 .end_amdgpu_metadata 12467 12468This kernel is equivalent to the following HIP program: 12469 12470.. code:: 12471 :number-lines: 12472 12473 __global__ void hello_world(float *p) { 12474 *p = 3.14159f; 12475 } 12476 12477If an assembly source file contains multiple kernels and/or functions, the 12478:ref:`amdgpu-amdhsa-assembler-symbol-next_free_vgpr` and 12479:ref:`amdgpu-amdhsa-assembler-symbol-next_free_sgpr` symbols may be reset using 12480the ``.set <symbol>, <expression>`` directive. For example, in the case of two 12481kernels, where ``function1`` is only called from ``kernel1`` it is sufficient 12482to group the function with the kernel that calls it and reset the symbols 12483between the two connected components: 12484 12485.. code:: 12486 :number-lines: 12487 12488 .amdgcn_target "amdgcn-amd-amdhsa--gfx900+xnack" // optional 12489 12490 // gpr tracking symbols are implicitly set to zero 12491 12492 .text 12493 .globl kern0 12494 .p2align 8 12495 .type kern0,@function 12496 kern0: 12497 // ... 12498 s_endpgm 12499 .Lkern0_end: 12500 .size kern0, .Lkern0_end-kern0 12501 12502 .rodata 12503 .p2align 6 12504 .amdhsa_kernel kern0 12505 // ... 12506 .amdhsa_next_free_vgpr .amdgcn.next_free_vgpr 12507 .amdhsa_next_free_sgpr .amdgcn.next_free_sgpr 12508 .end_amdhsa_kernel 12509 12510 // reset symbols to begin tracking usage in func1 and kern1 12511 .set .amdgcn.next_free_vgpr, 0 12512 .set .amdgcn.next_free_sgpr, 0 12513 12514 .text 12515 .hidden func1 12516 .global func1 12517 .p2align 2 12518 .type func1,@function 12519 func1: 12520 // ... 12521 s_setpc_b64 s[30:31] 12522 .Lfunc1_end: 12523 .size func1, .Lfunc1_end-func1 12524 12525 .globl kern1 12526 .p2align 8 12527 .type kern1,@function 12528 kern1: 12529 // ... 12530 s_getpc_b64 s[4:5] 12531 s_add_u32 s4, s4, func1@rel32@lo+4 12532 s_addc_u32 s5, s5, func1@rel32@lo+4 12533 s_swappc_b64 s[30:31], s[4:5] 12534 // ... 12535 s_endpgm 12536 .Lkern1_end: 12537 .size kern1, .Lkern1_end-kern1 12538 12539 .rodata 12540 .p2align 6 12541 .amdhsa_kernel kern1 12542 // ... 12543 .amdhsa_next_free_vgpr .amdgcn.next_free_vgpr 12544 .amdhsa_next_free_sgpr .amdgcn.next_free_sgpr 12545 .end_amdhsa_kernel 12546 12547These symbols cannot identify connected components in order to automatically 12548track the usage for each kernel. However, in some cases careful organization of 12549the kernels and functions in the source file means there is minimal additional 12550effort required to accurately calculate GPR usage. 12551 12552Additional Documentation 12553======================== 12554 12555.. [AMD-GCN-GFX6] `AMD Southern Islands Series ISA <http://developer.amd.com/wordpress/media/2012/12/AMD_Southern_Islands_Instruction_Set_Architecture.pdf>`__ 12556.. [AMD-GCN-GFX7] `AMD Sea Islands Series ISA <http://developer.amd.com/wordpress/media/2013/07/AMD_Sea_Islands_Instruction_Set_Architecture.pdf>`_ 12557.. [AMD-GCN-GFX8] `AMD GCN3 Instruction Set Architecture <http://amd-dev.wpengine.netdna-cdn.com/wordpress/media/2013/12/AMD_GCN3_Instruction_Set_Architecture_rev1.1.pdf>`__ 12558.. [AMD-GCN-GFX900-GFX904-VEGA] `AMD Vega Instruction Set Architecture <http://developer.amd.com/wordpress/media/2013/12/Vega_Shader_ISA_28July2017.pdf>`__ 12559.. [AMD-GCN-GFX906-VEGA7NM] `AMD Vega 7nm Instruction Set Architecture <https://gpuopen.com/wp-content/uploads/2019/11/Vega_7nm_Shader_ISA_26November2019.pdf>`__ 12560.. [AMD-GCN-GFX908-CDNA1] `AMD Instinct MI100 Instruction Set Architecture <https://developer.amd.com/wp-content/resources/CDNA1_Shader_ISA_14December2020.pdf>`__ 12561.. [AMD-GCN-GFX10-RDNA1] `AMD RDNA 1.0 Instruction Set Architecture <https://gpuopen.com/wp-content/uploads/2019/08/RDNA_Shader_ISA_5August2019.pdf>`__ 12562.. [AMD-GCN-GFX10-RDNA2] `AMD RDNA 2 Instruction Set Architecture <https://developer.amd.com/wp-content/resources/RDNA2_Shader_ISA_November2020.pdf>`__ 12563.. [AMD-RADEON-HD-2000-3000] `AMD R6xx shader ISA <http://developer.amd.com/wordpress/media/2012/10/R600_Instruction_Set_Architecture.pdf>`__ 12564.. [AMD-RADEON-HD-4000] `AMD R7xx shader ISA <http://developer.amd.com/wordpress/media/2012/10/R700-Family_Instruction_Set_Architecture.pdf>`__ 12565.. [AMD-RADEON-HD-5000] `AMD Evergreen shader ISA <http://developer.amd.com/wordpress/media/2012/10/AMD_Evergreen-Family_Instruction_Set_Architecture.pdf>`__ 12566.. [AMD-RADEON-HD-6000] `AMD Cayman/Trinity shader ISA <http://developer.amd.com/wordpress/media/2012/10/AMD_HD_6900_Series_Instruction_Set_Architecture.pdf>`__ 12567.. [AMD-ROCm] `AMD ROCm™ Platform <https://rocmdocs.amd.com/>`__ 12568.. [AMD-ROCm-github] `AMD ROCm™ github <http://github.com/RadeonOpenCompute>`__ 12569.. [AMD-ROCm-Release-Notes] `AMD ROCm Release Notes <https://github.com/RadeonOpenCompute/ROCm>`__ 12570.. [CLANG-ATTR] `Attributes in Clang <https://clang.llvm.org/docs/AttributeReference.html>`__ 12571.. [DWARF] `DWARF Debugging Information Format <http://dwarfstd.org/>`__ 12572.. [ELF] `Executable and Linkable Format (ELF) <http://www.sco.com/developers/gabi/>`__ 12573.. [HRF] `Heterogeneous-race-free Memory Models <http://benedictgaster.org/wp-content/uploads/2014/01/asplos269-FINAL.pdf>`__ 12574.. [HSA] `Heterogeneous System Architecture (HSA) Foundation <http://www.hsafoundation.com/>`__ 12575.. [MsgPack] `Message Pack <http://www.msgpack.org/>`__ 12576.. [OpenCL] `The OpenCL Specification Version 2.0 <http://www.khronos.org/registry/cl/specs/opencl-2.0.pdf>`__ 12577.. [SEMVER] `Semantic Versioning <https://semver.org/>`__ 12578.. [YAML] `YAML Ain't Markup Language (YAML™) Version 1.2 <http://www.yaml.org/spec/1.2/spec.html>`__ 12579