1============================= 2User Guide for AMDGPU Backend 3============================= 4 5.. contents:: 6 :local: 7 8.. toctree:: 9 :hidden: 10 11 AMDGPU/AMDGPUAsmGFX7 12 AMDGPU/AMDGPUAsmGFX8 13 AMDGPU/AMDGPUAsmGFX9 14 AMDGPU/AMDGPUAsmGFX900 15 AMDGPU/AMDGPUAsmGFX904 16 AMDGPU/AMDGPUAsmGFX906 17 AMDGPU/AMDGPUAsmGFX908 18 AMDGPU/AMDGPUAsmGFX90a 19 AMDGPU/AMDGPUAsmGFX10 20 AMDGPU/AMDGPUAsmGFX1011 21 AMDGPUModifierSyntax 22 AMDGPUOperandSyntax 23 AMDGPUInstructionSyntax 24 AMDGPUInstructionNotation 25 AMDGPUDwarfExtensionsForHeterogeneousDebugging 26 27Introduction 28============ 29 30The AMDGPU backend provides ISA code generation for AMD GPUs, starting with the 31R600 family up until the current GCN families. It lives in the 32``llvm/lib/Target/AMDGPU`` directory. 33 34LLVM 35==== 36 37.. _amdgpu-target-triples: 38 39Target Triples 40-------------- 41 42Use the Clang option ``-target <Architecture>-<Vendor>-<OS>-<Environment>`` 43to specify the target triple: 44 45 .. table:: AMDGPU Architectures 46 :name: amdgpu-architecture-table 47 48 ============ ============================================================== 49 Architecture Description 50 ============ ============================================================== 51 ``r600`` AMD GPUs HD2XXX-HD6XXX for graphics and compute shaders. 52 ``amdgcn`` AMD GPUs GCN GFX6 onwards for graphics and compute shaders. 53 ============ ============================================================== 54 55 .. table:: AMDGPU Vendors 56 :name: amdgpu-vendor-table 57 58 ============ ============================================================== 59 Vendor Description 60 ============ ============================================================== 61 ``amd`` Can be used for all AMD GPU usage. 62 ``mesa3d`` Can be used if the OS is ``mesa3d``. 63 ============ ============================================================== 64 65 .. table:: AMDGPU Operating Systems 66 :name: amdgpu-os 67 68 ============== ============================================================ 69 OS Description 70 ============== ============================================================ 71 *<empty>* Defaults to the *unknown* OS. 72 ``amdhsa`` Compute kernels executed on HSA [HSA]_ compatible runtimes 73 such as: 74 75 - AMD's ROCm™ runtime [AMD-ROCm]_ using the *rocm-amdhsa* 76 loader on Linux. See *AMD ROCm Platform Release Notes* 77 [AMD-ROCm-Release-Notes]_ for supported hardware and 78 software. 79 - AMD's PAL runtime using the *pal-amdhsa* loader on 80 Windows. 81 82 ``amdpal`` Graphic shaders and compute kernels executed on AMD's PAL 83 runtime using the *pal-amdpal* loader on Windows and Linux 84 Pro. 85 ``mesa3d`` Graphic shaders and compute kernels executed on AMD's Mesa 86 3D runtime using the *mesa-mesa3d* loader on Linux. 87 ============== ============================================================ 88 89 .. table:: AMDGPU Environments 90 :name: amdgpu-environment-table 91 92 ============ ============================================================== 93 Environment Description 94 ============ ============================================================== 95 *<empty>* Default. 96 ============ ============================================================== 97 98.. _amdgpu-processors: 99 100Processors 101---------- 102 103Use the Clang options ``-mcpu=<target-id>`` or ``--offload-arch=<target-id>`` to 104specify the AMDGPU processor together with optional target features. See 105:ref:`amdgpu-target-id` and :ref:`amdgpu-target-features` for AMD GPU target 106specific information. 107 108Every processor supports every OS ABI (see :ref:`amdgpu-os`) with the following exceptions: 109 110* ``amdhsa`` is not supported in ``r600`` architecture (see :ref:`amdgpu-architecture-table`). 111 112 113 .. table:: AMDGPU Processors 114 :name: amdgpu-processor-table 115 116 =========== =============== ============ ===== ================= =============== =============== ====================== 117 Processor Alternative Target dGPU/ Target Target OS Support Example 118 Processor Triple APU Features Properties *(see* Products 119 Architecture Supported `amdgpu-os`_ 120 *and 121 corresponding 122 runtime release 123 notes for 124 current 125 information and 126 level of 127 support)* 128 =========== =============== ============ ===== ================= =============== =============== ====================== 129 **Radeon HD 2000/3000 Series (R600)** [AMD-RADEON-HD-2000-3000]_ 130 ----------------------------------------------------------------------------------------------------------------------- 131 ``r600`` ``r600`` dGPU - Does not 132 support 133 generic 134 address 135 space 136 ``r630`` ``r600`` dGPU - Does not 137 support 138 generic 139 address 140 space 141 ``rs880`` ``r600`` dGPU - Does not 142 support 143 generic 144 address 145 space 146 ``rv670`` ``r600`` dGPU - Does not 147 support 148 generic 149 address 150 space 151 **Radeon HD 4000 Series (R700)** [AMD-RADEON-HD-4000]_ 152 ----------------------------------------------------------------------------------------------------------------------- 153 ``rv710`` ``r600`` dGPU - Does not 154 support 155 generic 156 address 157 space 158 ``rv730`` ``r600`` dGPU - Does not 159 support 160 generic 161 address 162 space 163 ``rv770`` ``r600`` dGPU - Does not 164 support 165 generic 166 address 167 space 168 **Radeon HD 5000 Series (Evergreen)** [AMD-RADEON-HD-5000]_ 169 ----------------------------------------------------------------------------------------------------------------------- 170 ``cedar`` ``r600`` dGPU - Does not 171 support 172 generic 173 address 174 space 175 ``cypress`` ``r600`` dGPU - Does not 176 support 177 generic 178 address 179 space 180 ``juniper`` ``r600`` dGPU - Does not 181 support 182 generic 183 address 184 space 185 ``redwood`` ``r600`` dGPU - Does not 186 support 187 generic 188 address 189 space 190 ``sumo`` ``r600`` dGPU - Does not 191 support 192 generic 193 address 194 space 195 **Radeon HD 6000 Series (Northern Islands)** [AMD-RADEON-HD-6000]_ 196 ----------------------------------------------------------------------------------------------------------------------- 197 ``barts`` ``r600`` dGPU - Does not 198 support 199 generic 200 address 201 space 202 ``caicos`` ``r600`` dGPU - Does not 203 support 204 generic 205 address 206 space 207 ``cayman`` ``r600`` dGPU - Does not 208 support 209 generic 210 address 211 space 212 ``turks`` ``r600`` dGPU - Does not 213 support 214 generic 215 address 216 space 217 **GCN GFX6 (Southern Islands (SI))** [AMD-GCN-GFX6]_ 218 ----------------------------------------------------------------------------------------------------------------------- 219 ``gfx600`` - ``tahiti`` ``amdgcn`` dGPU - Does not - *pal-amdpal* 220 support 221 generic 222 address 223 space 224 ``gfx601`` - ``pitcairn`` ``amdgcn`` dGPU - Does not - *pal-amdpal* 225 - ``verde`` support 226 generic 227 address 228 space 229 ``gfx602`` - ``hainan`` ``amdgcn`` dGPU - Does not - *pal-amdpal* 230 - ``oland`` support 231 generic 232 address 233 space 234 **GCN GFX7 (Sea Islands (CI))** [AMD-GCN-GFX7]_ 235 ----------------------------------------------------------------------------------------------------------------------- 236 ``gfx700`` - ``kaveri`` ``amdgcn`` APU - Offset - *rocm-amdhsa* - A6-7000 237 flat - *pal-amdhsa* - A6 Pro-7050B 238 scratch - *pal-amdpal* - A8-7100 239 - A8 Pro-7150B 240 - A10-7300 241 - A10 Pro-7350B 242 - FX-7500 243 - A8-7200P 244 - A10-7400P 245 - FX-7600P 246 ``gfx701`` - ``hawaii`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - FirePro W8100 247 flat - *pal-amdhsa* - FirePro W9100 248 scratch - *pal-amdpal* - FirePro S9150 249 - FirePro S9170 250 ``gfx702`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - Radeon R9 290 251 flat - *pal-amdhsa* - Radeon R9 290x 252 scratch - *pal-amdpal* - Radeon R390 253 - Radeon R390x 254 ``gfx703`` - ``kabini`` ``amdgcn`` APU - Offset - *pal-amdhsa* - E1-2100 255 - ``mullins`` flat - *pal-amdpal* - E1-2200 256 scratch - E1-2500 257 - E2-3000 258 - E2-3800 259 - A4-5000 260 - A4-5100 261 - A6-5200 262 - A4 Pro-3340B 263 ``gfx704`` - ``bonaire`` ``amdgcn`` dGPU - Offset - *pal-amdhsa* - Radeon HD 7790 264 flat - *pal-amdpal* - Radeon HD 8770 265 scratch - R7 260 266 - R7 260X 267 ``gfx705`` ``amdgcn`` APU - Offset - *pal-amdhsa* *TBA* 268 flat - *pal-amdpal* 269 scratch .. TODO:: 270 271 Add product 272 names. 273 274 **GCN GFX8 (Volcanic Islands (VI))** [AMD-GCN-GFX8]_ 275 ----------------------------------------------------------------------------------------------------------------------- 276 ``gfx801`` - ``carrizo`` ``amdgcn`` APU - xnack - Offset - *rocm-amdhsa* - A6-8500P 277 flat - *pal-amdhsa* - Pro A6-8500B 278 scratch - *pal-amdpal* - A8-8600P 279 - Pro A8-8600B 280 - FX-8800P 281 - Pro A12-8800B 282 - A10-8700P 283 - Pro A10-8700B 284 - A10-8780P 285 - A10-9600P 286 - A10-9630P 287 - A12-9700P 288 - A12-9730P 289 - FX-9800P 290 - FX-9830P 291 - E2-9010 292 - A6-9210 293 - A9-9410 294 ``gfx802`` - ``iceland`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - Radeon R9 285 295 - ``tonga`` flat - *pal-amdhsa* - Radeon R9 380 296 scratch - *pal-amdpal* - Radeon R9 385 297 ``gfx803`` - ``fiji`` ``amdgcn`` dGPU - *rocm-amdhsa* - Radeon R9 Nano 298 - *pal-amdhsa* - Radeon R9 Fury 299 - *pal-amdpal* - Radeon R9 FuryX 300 - Radeon Pro Duo 301 - FirePro S9300x2 302 - Radeon Instinct MI8 303 \ - ``polaris10`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - Radeon RX 470 304 flat - *pal-amdhsa* - Radeon RX 480 305 scratch - *pal-amdpal* - Radeon Instinct MI6 306 \ - ``polaris11`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - Radeon RX 460 307 flat - *pal-amdhsa* 308 scratch - *pal-amdpal* 309 ``gfx805`` - ``tongapro`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - FirePro S7150 310 flat - *pal-amdhsa* - FirePro S7100 311 scratch - *pal-amdpal* - FirePro W7100 312 - Mobile FirePro 313 M7170 314 ``gfx810`` - ``stoney`` ``amdgcn`` APU - xnack - Offset - *rocm-amdhsa* *TBA* 315 flat - *pal-amdhsa* 316 scratch - *pal-amdpal* .. TODO:: 317 318 Add product 319 names. 320 321 **GCN GFX9 (Vega)** [AMD-GCN-GFX900-GFX904-VEGA]_ [AMD-GCN-GFX906-VEGA7NM]_ [AMD-GCN-GFX908-CDNA1]_ 322 ----------------------------------------------------------------------------------------------------------------------- 323 ``gfx900`` ``amdgcn`` dGPU - xnack - Absolute - *rocm-amdhsa* - Radeon Vega 324 flat - *pal-amdhsa* Frontier Edition 325 scratch - *pal-amdpal* - Radeon RX Vega 56 326 - Radeon RX Vega 64 327 - Radeon RX Vega 64 328 Liquid 329 - Radeon Instinct MI25 330 ``gfx902`` ``amdgcn`` APU - xnack - Absolute - *rocm-amdhsa* - Ryzen 3 2200G 331 flat - *pal-amdhsa* - Ryzen 5 2400G 332 scratch - *pal-amdpal* 333 ``gfx904`` ``amdgcn`` dGPU - xnack - *rocm-amdhsa* *TBA* 334 - *pal-amdhsa* 335 - *pal-amdpal* .. TODO:: 336 337 Add product 338 names. 339 340 ``gfx906`` ``amdgcn`` dGPU - sramecc - Absolute - *rocm-amdhsa* - Radeon Instinct MI50 341 - xnack flat - *pal-amdhsa* - Radeon Instinct MI60 342 scratch - *pal-amdpal* - Radeon VII 343 - Radeon Pro VII 344 ``gfx908`` ``amdgcn`` dGPU - sramecc - *rocm-amdhsa* - AMD Instinct MI100 Accelerator 345 - xnack - Absolute 346 flat 347 scratch 348 ``gfx909`` ``amdgcn`` APU - xnack - Absolute - *pal-amdpal* *TBA* 349 flat 350 scratch .. TODO:: 351 352 Add product 353 names. 354 355 ``gfx90a`` ``amdgcn`` dGPU - sramecc - Absolute - *rocm-amdhsa* *TBA* 356 - tgsplit flat 357 - xnack scratch .. TODO:: 358 - Packed 359 work-item Add product 360 IDs names. 361 362 ``gfx90c`` ``amdgcn`` APU - xnack - Absolute - *pal-amdpal* - Ryzen 7 4700G 363 flat - Ryzen 7 4700GE 364 scratch - Ryzen 5 4600G 365 - Ryzen 5 4600GE 366 - Ryzen 3 4300G 367 - Ryzen 3 4300GE 368 - Ryzen Pro 4000G 369 - Ryzen 7 Pro 4700G 370 - Ryzen 7 Pro 4750GE 371 - Ryzen 5 Pro 4650G 372 - Ryzen 5 Pro 4650GE 373 - Ryzen 3 Pro 4350G 374 - Ryzen 3 Pro 4350GE 375 376 **GCN GFX10 (RDNA 1)** [AMD-GCN-GFX10-RDNA1]_ 377 ----------------------------------------------------------------------------------------------------------------------- 378 ``gfx1010`` ``amdgcn`` dGPU - cumode - Absolute - *rocm-amdhsa* - Radeon RX 5700 379 - wavefrontsize64 flat - *pal-amdhsa* - Radeon RX 5700 XT 380 - xnack scratch - *pal-amdpal* - Radeon Pro 5600 XT 381 - Radeon Pro 5600M 382 ``gfx1011`` ``amdgcn`` dGPU - cumode - *rocm-amdhsa* - Radeon Pro V520 383 - wavefrontsize64 - Absolute - *pal-amdhsa* 384 - xnack flat - *pal-amdpal* 385 scratch 386 ``gfx1012`` ``amdgcn`` dGPU - cumode - Absolute - *rocm-amdhsa* - Radeon RX 5500 387 - wavefrontsize64 flat - *pal-amdhsa* - Radeon RX 5500 XT 388 - xnack scratch - *pal-amdpal* 389 ``gfx1013`` ``amdgcn`` APU - cumode - Absolute - *rocm-amdhsa* *TBA* 390 - wavefrontsize64 flat - *pal-amdhsa* 391 - xnack scratch - *pal-amdpal* .. TODO:: 392 393 Add product 394 names. 395 396 **GCN GFX10 (RDNA 2)** [AMD-GCN-GFX10-RDNA2]_ 397 ----------------------------------------------------------------------------------------------------------------------- 398 ``gfx1030`` ``amdgcn`` dGPU - cumode - Absolute - *rocm-amdhsa* - Radeon RX 6800 399 - wavefrontsize64 flat - *pal-amdhsa* - Radeon RX 6800 XT 400 scratch - *pal-amdpal* - Radeon RX 6900 XT 401 ``gfx1031`` ``amdgcn`` dGPU - cumode - Absolute - *rocm-amdhsa* - Radeon RX 6700 XT 402 - wavefrontsize64 flat - *pal-amdhsa* 403 scratch - *pal-amdpal* 404 ``gfx1032`` ``amdgcn`` dGPU - cumode - Absolute - *rocm-amdhsa* *TBA* 405 - wavefrontsize64 flat - *pal-amdhsa* 406 scratch - *pal-amdpal* .. TODO:: 407 408 Add product 409 names. 410 411 ``gfx1033`` ``amdgcn`` APU - cumode - Absolute - *pal-amdpal* *TBA* 412 - wavefrontsize64 flat 413 scratch .. TODO:: 414 415 Add product 416 names. 417 ``gfx1034`` ``amdgcn`` dGPU - cumode - Absolute - *pal-amdpal* *TBA* 418 - wavefrontsize64 flat 419 scratch .. TODO:: 420 421 Add product 422 names. 423 424 ``gfx1035`` ``amdgcn`` APU - cumode - Absolute - *pal-amdpal* *TBA* 425 - wavefrontsize64 flat 426 scratch .. TODO:: 427 Add product 428 names. 429 430 =========== =============== ============ ===== ================= =============== =============== ====================== 431 432.. _amdgpu-target-features: 433 434Target Features 435--------------- 436 437Target features control how code is generated to support certain 438processor specific features. Not all target features are supported by 439all processors. The runtime must ensure that the features supported by 440the device used to execute the code match the features enabled when 441generating the code. A mismatch of features may result in incorrect 442execution, or a reduction in performance. 443 444The target features supported by each processor is listed in 445:ref:`amdgpu-processor-table`. 446 447Target features are controlled by exactly one of the following Clang 448options: 449 450``-mcpu=<target-id>`` or ``--offload-arch=<target-id>`` 451 452 The ``-mcpu`` and ``--offload-arch`` can specify the target feature as 453 optional components of the target ID. If omitted, the target feature has the 454 ``any`` value. See :ref:`amdgpu-target-id`. 455 456``-m[no-]<target-feature>`` 457 458 Target features not specified by the target ID are specified using a 459 separate option. These target features can have an ``on`` or ``off`` 460 value. ``on`` is specified by omitting the ``no-`` prefix, and 461 ``off`` is specified by including the ``no-`` prefix. The default 462 if not specified is ``off``. 463 464For example: 465 466``-mcpu=gfx908:xnack+`` 467 Enable the ``xnack`` feature. 468``-mcpu=gfx908:xnack-`` 469 Disable the ``xnack`` feature. 470``-mcumode`` 471 Enable the ``cumode`` feature. 472``-mno-cumode`` 473 Disable the ``cumode`` feature. 474 475 .. table:: AMDGPU Target Features 476 :name: amdgpu-target-features-table 477 478 =============== ============================ ================================================== 479 Target Feature Clang Option to Control Description 480 Name 481 =============== ============================ ================================================== 482 cumode - ``-m[no-]cumode`` Control the wavefront execution mode used 483 when generating code for kernels. When disabled 484 native WGP wavefront execution mode is used, 485 when enabled CU wavefront execution mode is used 486 (see :ref:`amdgpu-amdhsa-memory-model`). 487 488 sramecc - ``-mcpu`` If specified, generate code that can only be 489 - ``--offload-arch`` loaded and executed in a process that has a 490 matching setting for SRAMECC. 491 492 If not specified for code object V2 to V3, generate 493 code that can be loaded and executed in a process 494 with SRAMECC enabled. 495 496 If not specified for code object V4, generate 497 code that can be loaded and executed in a process 498 with either setting of SRAMECC. 499 500 tgsplit ``-m[no-]tgsplit`` Enable/disable generating code that assumes 501 work-groups are launched in threadgroup split mode. 502 When enabled the waves of a work-group may be 503 launched in different CUs. 504 505 wavefrontsize64 - ``-m[no-]wavefrontsize64`` Control the wavefront size used when 506 generating code for kernels. When disabled 507 native wavefront size 32 is used, when enabled 508 wavefront size 64 is used. 509 510 xnack - ``-mcpu`` If specified, generate code that can only be 511 - ``--offload-arch`` loaded and executed in a process that has a 512 matching setting for XNACK replay. 513 514 If not specified for code object V2 to V3, generate 515 code that can be loaded and executed in a process 516 with XNACK replay enabled. 517 518 If not specified for code object V4, generate 519 code that can be loaded and executed in a process 520 with either setting of XNACK replay. 521 522 XNACK replay can be used for demand paging and 523 page migration. If enabled in the device, then if 524 a page fault occurs the code may execute 525 incorrectly unless generated with XNACK replay 526 enabled, or generated for code object V4 without 527 specifying XNACK replay. Executing code that was 528 generated with XNACK replay enabled, or generated 529 for code object V4 without specifying XNACK replay, 530 on a device that does not have XNACK replay 531 enabled will execute correctly but may be less 532 performant than code generated for XNACK replay 533 disabled. 534 =============== ============================ ================================================== 535 536.. _amdgpu-target-id: 537 538Target ID 539--------- 540 541AMDGPU supports target IDs. See `Clang Offload Bundler 542<https://clang.llvm.org/docs/ClangOffloadBundler.html>`_ for a general 543description. The AMDGPU target specific information is: 544 545**processor** 546 Is an AMDGPU processor or alternative processor name specified in 547 :ref:`amdgpu-processor-table`. The non-canonical form target ID allows both 548 the primary processor and alternative processor names. The canonical form 549 target ID only allow the primary processor name. 550 551**target-feature** 552 Is a target feature name specified in :ref:`amdgpu-target-features-table` that 553 is supported by the processor. The target features supported by each processor 554 is specified in :ref:`amdgpu-processor-table`. Those that can be specified in 555 a target ID are marked as being controlled by ``-mcpu`` and 556 ``--offload-arch``. Each target feature must appear at most once in a target 557 ID. The non-canonical form target ID allows the target features to be 558 specified in any order. The canonical form target ID requires the target 559 features to be specified in alphabetic order. 560 561.. _amdgpu-target-id-v2-v3: 562 563Code Object V2 to V3 Target ID 564~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 565 566The target ID syntax for code object V2 to V3 is the same as defined in `Clang 567Offload Bundler <https://clang.llvm.org/docs/ClangOffloadBundler.html>`_ except 568when used in the :ref:`amdgpu-assembler-directive-amdgcn-target` assembler 569directive and the bundle entry ID. In those cases it has the following BNF 570syntax: 571 572.. code:: 573 574 <target-id> ::== <processor> ( "+" <target-feature> )* 575 576Where a target feature is omitted if *Off* and present if *On* or *Any*. 577 578.. note:: 579 580 The code object V2 to V3 cannot represent *Any* and treats it the same as 581 *On*. 582 583.. _amdgpu-embedding-bundled-objects: 584 585Embedding Bundled Code Objects 586------------------------------ 587 588AMDGPU supports the HIP and OpenMP languages that perform code object embedding 589as described in `Clang Offload Bundler 590<https://clang.llvm.org/docs/ClangOffloadBundler.html>`_. 591 592.. note:: 593 594 The target ID syntax used for code object V2 to V3 for a bundle entry ID 595 differs from that used elsewhere. See :ref:`amdgpu-target-id-v2-v3`. 596 597.. _amdgpu-address-spaces: 598 599Address Spaces 600-------------- 601 602The AMDGPU architecture supports a number of memory address spaces. The address 603space names use the OpenCL standard names, with some additions. 604 605The AMDGPU address spaces correspond to target architecture specific LLVM 606address space numbers used in LLVM IR. 607 608The AMDGPU address spaces are described in 609:ref:`amdgpu-address-spaces-table`. Only 64-bit process address spaces are 610supported for the ``amdgcn`` target. 611 612 .. table:: AMDGPU Address Spaces 613 :name: amdgpu-address-spaces-table 614 615 ================================= =============== =========== ================ ======= ============================ 616 .. 64-Bit Process Address Space 617 --------------------------------- --------------- ----------- ---------------- ------------------------------------ 618 Address Space Name LLVM IR Address HSA Segment Hardware Address NULL Value 619 Space Number Name Name Size 620 ================================= =============== =========== ================ ======= ============================ 621 Generic 0 flat flat 64 0x0000000000000000 622 Global 1 global global 64 0x0000000000000000 623 Region 2 N/A GDS 32 *not implemented for AMDHSA* 624 Local 3 group LDS 32 0xFFFFFFFF 625 Constant 4 constant *same as global* 64 0x0000000000000000 626 Private 5 private scratch 32 0xFFFFFFFF 627 Constant 32-bit 6 *TODO* 0x00000000 628 Buffer Fat Pointer (experimental) 7 *TODO* 629 ================================= =============== =========== ================ ======= ============================ 630 631**Generic** 632 The generic address space is supported unless the *Target Properties* column 633 of :ref:`amdgpu-processor-table` specifies *Does not support generic address 634 space*. 635 636 The generic address space uses the hardware flat address support for two fixed 637 ranges of virtual addresses (the private and local apertures), that are 638 outside the range of addressable global memory, to map from a flat address to 639 a private or local address. This uses FLAT instructions that can take a flat 640 address and access global, private (scratch), and group (LDS) memory depending 641 on if the address is within one of the aperture ranges. 642 643 Flat access to scratch requires hardware aperture setup and setup in the 644 kernel prologue (see :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`). Flat 645 access to LDS requires hardware aperture setup and M0 (GFX7-GFX8) register 646 setup (see :ref:`amdgpu-amdhsa-kernel-prolog-m0`). 647 648 To convert between a private or group address space address (termed a segment 649 address) and a flat address the base address of the corresponding aperture 650 can be used. For GFX7-GFX8 these are available in the 651 :ref:`amdgpu-amdhsa-hsa-aql-queue` the address of which can be obtained with 652 Queue Ptr SGPR (see :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). For 653 GFX9-GFX10 the aperture base addresses are directly available as inline 654 constant registers ``SRC_SHARED_BASE/LIMIT`` and ``SRC_PRIVATE_BASE/LIMIT``. 655 In 64-bit address mode the aperture sizes are 2^32 bytes and the base is 656 aligned to 2^32 which makes it easier to convert from flat to segment or 657 segment to flat. 658 659 A global address space address has the same value when used as a flat address 660 so no conversion is needed. 661 662**Global and Constant** 663 The global and constant address spaces both use global virtual addresses, 664 which are the same virtual address space used by the CPU. However, some 665 virtual addresses may only be accessible to the CPU, some only accessible 666 by the GPU, and some by both. 667 668 Using the constant address space indicates that the data will not change 669 during the execution of the kernel. This allows scalar read instructions to 670 be used. As the constant address space could only be modified on the host 671 side, a generic pointer loaded from the constant address space is safe to be 672 assumed as a global pointer since only the device global memory is visible 673 and managed on the host side. The vector and scalar L1 caches are invalidated 674 of volatile data before each kernel dispatch execution to allow constant 675 memory to change values between kernel dispatches. 676 677**Region** 678 The region address space uses the hardware Global Data Store (GDS). All 679 wavefronts executing on the same device will access the same memory for any 680 given region address. However, the same region address accessed by wavefronts 681 executing on different devices will access different memory. It is higher 682 performance than global memory. It is allocated by the runtime. The data 683 store (DS) instructions can be used to access it. 684 685**Local** 686 The local address space uses the hardware Local Data Store (LDS) which is 687 automatically allocated when the hardware creates the wavefronts of a 688 work-group, and freed when all the wavefronts of a work-group have 689 terminated. All wavefronts belonging to the same work-group will access the 690 same memory for any given local address. However, the same local address 691 accessed by wavefronts belonging to different work-groups will access 692 different memory. It is higher performance than global memory. The data store 693 (DS) instructions can be used to access it. 694 695**Private** 696 The private address space uses the hardware scratch memory support which 697 automatically allocates memory when it creates a wavefront and frees it when 698 a wavefronts terminates. The memory accessed by a lane of a wavefront for any 699 given private address will be different to the memory accessed by another lane 700 of the same or different wavefront for the same private address. 701 702 If a kernel dispatch uses scratch, then the hardware allocates memory from a 703 pool of backing memory allocated by the runtime for each wavefront. The lanes 704 of the wavefront access this using dword (4 byte) interleaving. The mapping 705 used from private address to backing memory address is: 706 707 ``wavefront-scratch-base + 708 ((private-address / 4) * wavefront-size * 4) + 709 (wavefront-lane-id * 4) + (private-address % 4)`` 710 711 If each lane of a wavefront accesses the same private address, the 712 interleaving results in adjacent dwords being accessed and hence requires 713 fewer cache lines to be fetched. 714 715 There are different ways that the wavefront scratch base address is 716 determined by a wavefront (see 717 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 718 719 Scratch memory can be accessed in an interleaved manner using buffer 720 instructions with the scratch buffer descriptor and per wavefront scratch 721 offset, by the scratch instructions, or by flat instructions. Multi-dword 722 access is not supported except by flat and scratch instructions in 723 GFX9-GFX10. 724 725**Constant 32-bit** 726 *TODO* 727 728**Buffer Fat Pointer** 729 The buffer fat pointer is an experimental address space that is currently 730 unsupported in the backend. It exposes a non-integral pointer that is in 731 the future intended to support the modelling of 128-bit buffer descriptors 732 plus a 32-bit offset into the buffer (in total encapsulating a 160-bit 733 *pointer*), allowing normal LLVM load/store/atomic operations to be used to 734 model the buffer descriptors used heavily in graphics workloads targeting 735 the backend. 736 737.. _amdgpu-memory-scopes: 738 739Memory Scopes 740------------- 741 742This section provides LLVM memory synchronization scopes supported by the AMDGPU 743backend memory model when the target triple OS is ``amdhsa`` (see 744:ref:`amdgpu-amdhsa-memory-model` and :ref:`amdgpu-target-triples`). 745 746The memory model supported is based on the HSA memory model [HSA]_ which is 747based in turn on HRF-indirect with scope inclusion [HRF]_. The happens-before 748relation is transitive over the synchronizes-with relation independent of scope 749and synchronizes-with allows the memory scope instances to be inclusive (see 750table :ref:`amdgpu-amdhsa-llvm-sync-scopes-table`). 751 752This is different to the OpenCL [OpenCL]_ memory model which does not have scope 753inclusion and requires the memory scopes to exactly match. However, this 754is conservatively correct for OpenCL. 755 756 .. table:: AMDHSA LLVM Sync Scopes 757 :name: amdgpu-amdhsa-llvm-sync-scopes-table 758 759 ======================= =================================================== 760 LLVM Sync Scope Description 761 ======================= =================================================== 762 *none* The default: ``system``. 763 764 Synchronizes with, and participates in modification 765 and seq_cst total orderings with, other operations 766 (except image operations) for all address spaces 767 (except private, or generic that accesses private) 768 provided the other operation's sync scope is: 769 770 - ``system``. 771 - ``agent`` and executed by a thread on the same 772 agent. 773 - ``workgroup`` and executed by a thread in the 774 same work-group. 775 - ``wavefront`` and executed by a thread in the 776 same wavefront. 777 778 ``agent`` Synchronizes with, and participates in modification 779 and seq_cst total orderings with, other operations 780 (except image operations) for all address spaces 781 (except private, or generic that accesses private) 782 provided the other operation's sync scope is: 783 784 - ``system`` or ``agent`` and executed by a thread 785 on the same agent. 786 - ``workgroup`` and executed by a thread in the 787 same work-group. 788 - ``wavefront`` and executed by a thread in the 789 same wavefront. 790 791 ``workgroup`` Synchronizes with, and participates in modification 792 and seq_cst total orderings with, other operations 793 (except image operations) for all address spaces 794 (except private, or generic that accesses private) 795 provided the other operation's sync scope is: 796 797 - ``system``, ``agent`` or ``workgroup`` and 798 executed by a thread in the same work-group. 799 - ``wavefront`` and executed by a thread in the 800 same wavefront. 801 802 ``wavefront`` Synchronizes with, and participates in modification 803 and seq_cst total orderings with, other operations 804 (except image operations) for all address spaces 805 (except private, or generic that accesses private) 806 provided the other operation's sync scope is: 807 808 - ``system``, ``agent``, ``workgroup`` or 809 ``wavefront`` and executed by a thread in the 810 same wavefront. 811 812 ``singlethread`` Only synchronizes with and participates in 813 modification and seq_cst total orderings with, 814 other operations (except image operations) running 815 in the same thread for all address spaces (for 816 example, in signal handlers). 817 818 ``one-as`` Same as ``system`` but only synchronizes with other 819 operations within the same address space. 820 821 ``agent-one-as`` Same as ``agent`` but only synchronizes with other 822 operations within the same address space. 823 824 ``workgroup-one-as`` Same as ``workgroup`` but only synchronizes with 825 other operations within the same address space. 826 827 ``wavefront-one-as`` Same as ``wavefront`` but only synchronizes with 828 other operations within the same address space. 829 830 ``singlethread-one-as`` Same as ``singlethread`` but only synchronizes with 831 other operations within the same address space. 832 ======================= =================================================== 833 834LLVM IR Intrinsics 835------------------ 836 837The AMDGPU backend implements the following LLVM IR intrinsics. 838 839*This section is WIP.* 840 841.. TODO:: 842 843 List AMDGPU intrinsics. 844 845LLVM IR Attributes 846------------------ 847 848The AMDGPU backend supports the following LLVM IR attributes. 849 850 .. table:: AMDGPU LLVM IR Attributes 851 :name: amdgpu-llvm-ir-attributes-table 852 853 ======================================= ========================================================== 854 LLVM Attribute Description 855 ======================================= ========================================================== 856 "amdgpu-flat-work-group-size"="min,max" Specify the minimum and maximum flat work group sizes that 857 will be specified when the kernel is dispatched. Generated 858 by the ``amdgpu_flat_work_group_size`` CLANG attribute [CLANG-ATTR]_. 859 The implied default value is 1,1024. 860 861 "amdgpu-implicitarg-num-bytes"="n" Number of kernel argument bytes to add to the kernel 862 argument block size for the implicit arguments. This 863 varies by OS and language (for OpenCL see 864 :ref:`opencl-kernel-implicit-arguments-appended-for-amdhsa-os-table`). 865 "amdgpu-num-sgpr"="n" Specifies the number of SGPRs to use. Generated by 866 the ``amdgpu_num_sgpr`` CLANG attribute [CLANG-ATTR]_. 867 "amdgpu-num-vgpr"="n" Specifies the number of VGPRs to use. Generated by the 868 ``amdgpu_num_vgpr`` CLANG attribute [CLANG-ATTR]_. 869 "amdgpu-waves-per-eu"="m,n" Specify the minimum and maximum number of waves per 870 execution unit. Generated by the ``amdgpu_waves_per_eu`` 871 CLANG attribute [CLANG-ATTR]_. This is an optimization hint, 872 and the backend may not be able to satisfy the request. If 873 the specified range is incompatible with the function's 874 "amdgpu-flat-work-group-size" value, the implied occupancy 875 bounds by the workgroup size takes precedence. 876 877 "amdgpu-ieee" true/false. Specify whether the function expects the IEEE field of the 878 mode register to be set on entry. Overrides the default for 879 the calling convention. 880 "amdgpu-dx10-clamp" true/false. Specify whether the function expects the DX10_CLAMP field of 881 the mode register to be set on entry. Overrides the default 882 for the calling convention. 883 884 "amdgpu-no-workitem-id-x" Indicates the function does not depend on the value of the 885 llvm.amdgcn.workitem.id.x intrinsic. If a function is marked with this 886 attribute, or reached through a call site marked with this attribute, 887 the value returned by the intrinsic is undefined. The backend can 888 generally infer this during code generation, so typically there is no 889 benefit to frontends marking functions with this. 890 891 "amdgpu-no-workitem-id-y" The same as amdgpu-no-workitem-id-x, except for the 892 llvm.amdgcn.workitem.id.y intrinsic. 893 894 "amdgpu-no-workitem-id-z" The same as amdgpu-no-workitem-id-x, except for the 895 llvm.amdgcn.workitem.id.z intrinsic. 896 897 "amdgpu-no-workgroup-id-x" The same as amdgpu-no-workitem-id-x, except for the 898 llvm.amdgcn.workgroup.id.x intrinsic. 899 900 "amdgpu-no-workgroup-id-y" The same as amdgpu-no-workitem-id-x, except for the 901 llvm.amdgcn.workgroup.id.y intrinsic. 902 903 "amdgpu-no-workgroup-id-z" The same as amdgpu-no-workitem-id-x, except for the 904 llvm.amdgcn.workgroup.id.z intrinsic. 905 906 "amdgpu-no-dispatch-ptr" The same as amdgpu-no-workitem-id-x, except for the 907 llvm.amdgcn.dispatch.ptr intrinsic. 908 909 "amdgpu-no-implicitarg-ptr" The same as amdgpu-no-workitem-id-x, except for the 910 llvm.amdgcn.implicitarg.ptr intrinsic. 911 912 "amdgpu-no-dispatch-id" The same as amdgpu-no-workitem-id-x, except for the 913 llvm.amdgcn.dispatch.id intrinsic. 914 915 "amdgpu-no-queue-ptr" Similar to amdgpu-no-workitem-id-x, except for the 916 llvm.amdgcn.queue.ptr intrinsic. Note that unlike the other ABI hint 917 attributes, the queue pointer may be required in situations where the 918 intrinsic call does not directly appear in the program. Some subtargets 919 require the queue pointer for to handle some addrspacecasts, as well 920 as the llvm.amdgcn.is.shared, llvm.amdgcn.is.private, llvm.trap, and 921 llvm.debug intrinsics. 922 923 ======================================= ========================================================== 924 925.. _amdgpu-elf-code-object: 926 927ELF Code Object 928=============== 929 930The AMDGPU backend generates a standard ELF [ELF]_ relocatable code object that 931can be linked by ``lld`` to produce a standard ELF shared code object which can 932be loaded and executed on an AMDGPU target. 933 934.. _amdgpu-elf-header: 935 936Header 937------ 938 939The AMDGPU backend uses the following ELF header: 940 941 .. table:: AMDGPU ELF Header 942 :name: amdgpu-elf-header-table 943 944 ========================== =============================== 945 Field Value 946 ========================== =============================== 947 ``e_ident[EI_CLASS]`` ``ELFCLASS64`` 948 ``e_ident[EI_DATA]`` ``ELFDATA2LSB`` 949 ``e_ident[EI_OSABI]`` - ``ELFOSABI_NONE`` 950 - ``ELFOSABI_AMDGPU_HSA`` 951 - ``ELFOSABI_AMDGPU_PAL`` 952 - ``ELFOSABI_AMDGPU_MESA3D`` 953 ``e_ident[EI_ABIVERSION]`` - ``ELFABIVERSION_AMDGPU_HSA_V2`` 954 - ``ELFABIVERSION_AMDGPU_HSA_V3`` 955 - ``ELFABIVERSION_AMDGPU_HSA_V4`` 956 - ``ELFABIVERSION_AMDGPU_PAL`` 957 - ``ELFABIVERSION_AMDGPU_MESA3D`` 958 ``e_type`` - ``ET_REL`` 959 - ``ET_DYN`` 960 ``e_machine`` ``EM_AMDGPU`` 961 ``e_entry`` 0 962 ``e_flags`` See :ref:`amdgpu-elf-header-e_flags-v2-table`, 963 :ref:`amdgpu-elf-header-e_flags-table-v3`, 964 and :ref:`amdgpu-elf-header-e_flags-table-v4` 965 ========================== =============================== 966 967.. 968 969 .. table:: AMDGPU ELF Header Enumeration Values 970 :name: amdgpu-elf-header-enumeration-values-table 971 972 =============================== ===== 973 Name Value 974 =============================== ===== 975 ``EM_AMDGPU`` 224 976 ``ELFOSABI_NONE`` 0 977 ``ELFOSABI_AMDGPU_HSA`` 64 978 ``ELFOSABI_AMDGPU_PAL`` 65 979 ``ELFOSABI_AMDGPU_MESA3D`` 66 980 ``ELFABIVERSION_AMDGPU_HSA_V2`` 0 981 ``ELFABIVERSION_AMDGPU_HSA_V3`` 1 982 ``ELFABIVERSION_AMDGPU_HSA_V4`` 2 983 ``ELFABIVERSION_AMDGPU_PAL`` 0 984 ``ELFABIVERSION_AMDGPU_MESA3D`` 0 985 =============================== ===== 986 987``e_ident[EI_CLASS]`` 988 The ELF class is: 989 990 * ``ELFCLASS32`` for ``r600`` architecture. 991 992 * ``ELFCLASS64`` for ``amdgcn`` architecture which only supports 64-bit 993 process address space applications. 994 995``e_ident[EI_DATA]`` 996 All AMDGPU targets use ``ELFDATA2LSB`` for little-endian byte ordering. 997 998``e_ident[EI_OSABI]`` 999 One of the following AMDGPU target architecture specific OS ABIs 1000 (see :ref:`amdgpu-os`): 1001 1002 * ``ELFOSABI_NONE`` for *unknown* OS. 1003 1004 * ``ELFOSABI_AMDGPU_HSA`` for ``amdhsa`` OS. 1005 1006 * ``ELFOSABI_AMDGPU_PAL`` for ``amdpal`` OS. 1007 1008 * ``ELFOSABI_AMDGPU_MESA3D`` for ``mesa3D`` OS. 1009 1010``e_ident[EI_ABIVERSION]`` 1011 The ABI version of the AMDGPU target architecture specific OS ABI to which the code 1012 object conforms: 1013 1014 * ``ELFABIVERSION_AMDGPU_HSA_V2`` is used to specify the version of AMD HSA 1015 runtime ABI for code object V2. Specify using the Clang option 1016 ``-mcode-object-version=2``. 1017 1018 * ``ELFABIVERSION_AMDGPU_HSA_V3`` is used to specify the version of AMD HSA 1019 runtime ABI for code object V3. Specify using the Clang option 1020 ``-mcode-object-version=3``. This is the default code object 1021 version if not specified. 1022 1023 * ``ELFABIVERSION_AMDGPU_HSA_V4`` is used to specify the version of AMD HSA 1024 runtime ABI for code object V4. Specify using the Clang option 1025 ``-mcode-object-version=4``. 1026 1027 * ``ELFABIVERSION_AMDGPU_PAL`` is used to specify the version of AMD PAL 1028 runtime ABI. 1029 1030 * ``ELFABIVERSION_AMDGPU_MESA3D`` is used to specify the version of AMD MESA 1031 3D runtime ABI. 1032 1033``e_type`` 1034 Can be one of the following values: 1035 1036 1037 ``ET_REL`` 1038 The type produced by the AMDGPU backend compiler as it is relocatable code 1039 object. 1040 1041 ``ET_DYN`` 1042 The type produced by the linker as it is a shared code object. 1043 1044 The AMD HSA runtime loader requires a ``ET_DYN`` code object. 1045 1046``e_machine`` 1047 The value ``EM_AMDGPU`` is used for the machine for all processors supported 1048 by the ``r600`` and ``amdgcn`` architectures (see 1049 :ref:`amdgpu-processor-table`). The specific processor is specified in the 1050 ``NT_AMD_HSA_ISA_VERSION`` note record for code object V2 (see 1051 :ref:`amdgpu-note-records-v2`) and in the ``EF_AMDGPU_MACH`` bit field of the 1052 ``e_flags`` for code object V3 to V4 (see 1053 :ref:`amdgpu-elf-header-e_flags-table-v3` and 1054 :ref:`amdgpu-elf-header-e_flags-table-v4`). 1055 1056``e_entry`` 1057 The entry point is 0 as the entry points for individual kernels must be 1058 selected in order to invoke them through AQL packets. 1059 1060``e_flags`` 1061 The AMDGPU backend uses the following ELF header flags: 1062 1063 .. table:: AMDGPU ELF Header ``e_flags`` for Code Object V2 1064 :name: amdgpu-elf-header-e_flags-v2-table 1065 1066 ===================================== ===== ============================= 1067 Name Value Description 1068 ===================================== ===== ============================= 1069 ``EF_AMDGPU_FEATURE_XNACK_V2`` 0x01 Indicates if the ``xnack`` 1070 target feature is 1071 enabled for all code 1072 contained in the code object. 1073 If the processor 1074 does not support the 1075 ``xnack`` target 1076 feature then must 1077 be 0. 1078 See 1079 :ref:`amdgpu-target-features`. 1080 ``EF_AMDGPU_FEATURE_TRAP_HANDLER_V2`` 0x02 Indicates if the trap 1081 handler is enabled for all 1082 code contained in the code 1083 object. If the processor 1084 does not support a trap 1085 handler then must be 0. 1086 See 1087 :ref:`amdgpu-target-features`. 1088 ===================================== ===== ============================= 1089 1090 .. table:: AMDGPU ELF Header ``e_flags`` for Code Object V3 1091 :name: amdgpu-elf-header-e_flags-table-v3 1092 1093 ================================= ===== ============================= 1094 Name Value Description 1095 ================================= ===== ============================= 1096 ``EF_AMDGPU_MACH`` 0x0ff AMDGPU processor selection 1097 mask for 1098 ``EF_AMDGPU_MACH_xxx`` values 1099 defined in 1100 :ref:`amdgpu-ef-amdgpu-mach-table`. 1101 ``EF_AMDGPU_FEATURE_XNACK_V3`` 0x100 Indicates if the ``xnack`` 1102 target feature is 1103 enabled for all code 1104 contained in the code object. 1105 If the processor 1106 does not support the 1107 ``xnack`` target 1108 feature then must 1109 be 0. 1110 See 1111 :ref:`amdgpu-target-features`. 1112 ``EF_AMDGPU_FEATURE_SRAMECC_V3`` 0x200 Indicates if the ``sramecc`` 1113 target feature is 1114 enabled for all code 1115 contained in the code object. 1116 If the processor 1117 does not support the 1118 ``sramecc`` target 1119 feature then must 1120 be 0. 1121 See 1122 :ref:`amdgpu-target-features`. 1123 ================================= ===== ============================= 1124 1125 .. table:: AMDGPU ELF Header ``e_flags`` for Code Object V4 1126 :name: amdgpu-elf-header-e_flags-table-v4 1127 1128 ============================================ ===== =================================== 1129 Name Value Description 1130 ============================================ ===== =================================== 1131 ``EF_AMDGPU_MACH`` 0x0ff AMDGPU processor selection 1132 mask for 1133 ``EF_AMDGPU_MACH_xxx`` values 1134 defined in 1135 :ref:`amdgpu-ef-amdgpu-mach-table`. 1136 ``EF_AMDGPU_FEATURE_XNACK_V4`` 0x300 XNACK selection mask for 1137 ``EF_AMDGPU_FEATURE_XNACK_*_V4`` 1138 values. 1139 ``EF_AMDGPU_FEATURE_XNACK_UNSUPPORTED_V4`` 0x000 XNACK unsuppored. 1140 ``EF_AMDGPU_FEATURE_XNACK_ANY_V4`` 0x100 XNACK can have any value. 1141 ``EF_AMDGPU_FEATURE_XNACK_OFF_V4`` 0x200 XNACK disabled. 1142 ``EF_AMDGPU_FEATURE_XNACK_ON_V4`` 0x300 XNACK enabled. 1143 ``EF_AMDGPU_FEATURE_SRAMECC_V4`` 0xc00 SRAMECC selection mask for 1144 ``EF_AMDGPU_FEATURE_SRAMECC_*_V4`` 1145 values. 1146 ``EF_AMDGPU_FEATURE_SRAMECC_UNSUPPORTED_V4`` 0x000 SRAMECC unsuppored. 1147 ``EF_AMDGPU_FEATURE_SRAMECC_ANY_V4`` 0x400 SRAMECC can have any value. 1148 ``EF_AMDGPU_FEATURE_SRAMECC_OFF_V4`` 0x800 SRAMECC disabled, 1149 ``EF_AMDGPU_FEATURE_SRAMECC_ON_V4`` 0xc00 SRAMECC enabled. 1150 ============================================ ===== =================================== 1151 1152 .. table:: AMDGPU ``EF_AMDGPU_MACH`` Values 1153 :name: amdgpu-ef-amdgpu-mach-table 1154 1155 ==================================== ========== ============================= 1156 Name Value Description (see 1157 :ref:`amdgpu-processor-table`) 1158 ==================================== ========== ============================= 1159 ``EF_AMDGPU_MACH_NONE`` 0x000 *not specified* 1160 ``EF_AMDGPU_MACH_R600_R600`` 0x001 ``r600`` 1161 ``EF_AMDGPU_MACH_R600_R630`` 0x002 ``r630`` 1162 ``EF_AMDGPU_MACH_R600_RS880`` 0x003 ``rs880`` 1163 ``EF_AMDGPU_MACH_R600_RV670`` 0x004 ``rv670`` 1164 ``EF_AMDGPU_MACH_R600_RV710`` 0x005 ``rv710`` 1165 ``EF_AMDGPU_MACH_R600_RV730`` 0x006 ``rv730`` 1166 ``EF_AMDGPU_MACH_R600_RV770`` 0x007 ``rv770`` 1167 ``EF_AMDGPU_MACH_R600_CEDAR`` 0x008 ``cedar`` 1168 ``EF_AMDGPU_MACH_R600_CYPRESS`` 0x009 ``cypress`` 1169 ``EF_AMDGPU_MACH_R600_JUNIPER`` 0x00a ``juniper`` 1170 ``EF_AMDGPU_MACH_R600_REDWOOD`` 0x00b ``redwood`` 1171 ``EF_AMDGPU_MACH_R600_SUMO`` 0x00c ``sumo`` 1172 ``EF_AMDGPU_MACH_R600_BARTS`` 0x00d ``barts`` 1173 ``EF_AMDGPU_MACH_R600_CAICOS`` 0x00e ``caicos`` 1174 ``EF_AMDGPU_MACH_R600_CAYMAN`` 0x00f ``cayman`` 1175 ``EF_AMDGPU_MACH_R600_TURKS`` 0x010 ``turks`` 1176 *reserved* 0x011 - Reserved for ``r600`` 1177 0x01f architecture processors. 1178 ``EF_AMDGPU_MACH_AMDGCN_GFX600`` 0x020 ``gfx600`` 1179 ``EF_AMDGPU_MACH_AMDGCN_GFX601`` 0x021 ``gfx601`` 1180 ``EF_AMDGPU_MACH_AMDGCN_GFX700`` 0x022 ``gfx700`` 1181 ``EF_AMDGPU_MACH_AMDGCN_GFX701`` 0x023 ``gfx701`` 1182 ``EF_AMDGPU_MACH_AMDGCN_GFX702`` 0x024 ``gfx702`` 1183 ``EF_AMDGPU_MACH_AMDGCN_GFX703`` 0x025 ``gfx703`` 1184 ``EF_AMDGPU_MACH_AMDGCN_GFX704`` 0x026 ``gfx704`` 1185 *reserved* 0x027 Reserved. 1186 ``EF_AMDGPU_MACH_AMDGCN_GFX801`` 0x028 ``gfx801`` 1187 ``EF_AMDGPU_MACH_AMDGCN_GFX802`` 0x029 ``gfx802`` 1188 ``EF_AMDGPU_MACH_AMDGCN_GFX803`` 0x02a ``gfx803`` 1189 ``EF_AMDGPU_MACH_AMDGCN_GFX810`` 0x02b ``gfx810`` 1190 ``EF_AMDGPU_MACH_AMDGCN_GFX900`` 0x02c ``gfx900`` 1191 ``EF_AMDGPU_MACH_AMDGCN_GFX902`` 0x02d ``gfx902`` 1192 ``EF_AMDGPU_MACH_AMDGCN_GFX904`` 0x02e ``gfx904`` 1193 ``EF_AMDGPU_MACH_AMDGCN_GFX906`` 0x02f ``gfx906`` 1194 ``EF_AMDGPU_MACH_AMDGCN_GFX908`` 0x030 ``gfx908`` 1195 ``EF_AMDGPU_MACH_AMDGCN_GFX909`` 0x031 ``gfx909`` 1196 ``EF_AMDGPU_MACH_AMDGCN_GFX90C`` 0x032 ``gfx90c`` 1197 ``EF_AMDGPU_MACH_AMDGCN_GFX1010`` 0x033 ``gfx1010`` 1198 ``EF_AMDGPU_MACH_AMDGCN_GFX1011`` 0x034 ``gfx1011`` 1199 ``EF_AMDGPU_MACH_AMDGCN_GFX1012`` 0x035 ``gfx1012`` 1200 ``EF_AMDGPU_MACH_AMDGCN_GFX1030`` 0x036 ``gfx1030`` 1201 ``EF_AMDGPU_MACH_AMDGCN_GFX1031`` 0x037 ``gfx1031`` 1202 ``EF_AMDGPU_MACH_AMDGCN_GFX1032`` 0x038 ``gfx1032`` 1203 ``EF_AMDGPU_MACH_AMDGCN_GFX1033`` 0x039 ``gfx1033`` 1204 ``EF_AMDGPU_MACH_AMDGCN_GFX602`` 0x03a ``gfx602`` 1205 ``EF_AMDGPU_MACH_AMDGCN_GFX705`` 0x03b ``gfx705`` 1206 ``EF_AMDGPU_MACH_AMDGCN_GFX805`` 0x03c ``gfx805`` 1207 ``EF_AMDGPU_MACH_AMDGCN_GFX1035`` 0x03d ``gfx1035`` 1208 ``EF_AMDGPU_MACH_AMDGCN_GFX1034`` 0x03e ``gfx1034`` 1209 ``EF_AMDGPU_MACH_AMDGCN_GFX90A`` 0x03f ``gfx90a`` 1210 *reserved* 0x040 Reserved. 1211 *reserved* 0x041 Reserved. 1212 ``EF_AMDGPU_MACH_AMDGCN_GFX1013`` 0x042 ``gfx1013`` 1213 *reserved* 0x043 Reserved. 1214 *reserved* 0x044 Reserved. 1215 *reserved* 0x045 Reserved. 1216 ==================================== ========== ============================= 1217 1218Sections 1219-------- 1220 1221An AMDGPU target ELF code object has the standard ELF sections which include: 1222 1223 .. table:: AMDGPU ELF Sections 1224 :name: amdgpu-elf-sections-table 1225 1226 ================== ================ ================================= 1227 Name Type Attributes 1228 ================== ================ ================================= 1229 ``.bss`` ``SHT_NOBITS`` ``SHF_ALLOC`` + ``SHF_WRITE`` 1230 ``.data`` ``SHT_PROGBITS`` ``SHF_ALLOC`` + ``SHF_WRITE`` 1231 ``.debug_``\ *\** ``SHT_PROGBITS`` *none* 1232 ``.dynamic`` ``SHT_DYNAMIC`` ``SHF_ALLOC`` 1233 ``.dynstr`` ``SHT_PROGBITS`` ``SHF_ALLOC`` 1234 ``.dynsym`` ``SHT_PROGBITS`` ``SHF_ALLOC`` 1235 ``.got`` ``SHT_PROGBITS`` ``SHF_ALLOC`` + ``SHF_WRITE`` 1236 ``.hash`` ``SHT_HASH`` ``SHF_ALLOC`` 1237 ``.note`` ``SHT_NOTE`` *none* 1238 ``.rela``\ *name* ``SHT_RELA`` *none* 1239 ``.rela.dyn`` ``SHT_RELA`` *none* 1240 ``.rodata`` ``SHT_PROGBITS`` ``SHF_ALLOC`` 1241 ``.shstrtab`` ``SHT_STRTAB`` *none* 1242 ``.strtab`` ``SHT_STRTAB`` *none* 1243 ``.symtab`` ``SHT_SYMTAB`` *none* 1244 ``.text`` ``SHT_PROGBITS`` ``SHF_ALLOC`` + ``SHF_EXECINSTR`` 1245 ================== ================ ================================= 1246 1247These sections have their standard meanings (see [ELF]_) and are only generated 1248if needed. 1249 1250``.debug``\ *\** 1251 The standard DWARF sections. See :ref:`amdgpu-dwarf-debug-information` for 1252 information on the DWARF produced by the AMDGPU backend. 1253 1254``.dynamic``, ``.dynstr``, ``.dynsym``, ``.hash`` 1255 The standard sections used by a dynamic loader. 1256 1257``.note`` 1258 See :ref:`amdgpu-note-records` for the note records supported by the AMDGPU 1259 backend. 1260 1261``.rela``\ *name*, ``.rela.dyn`` 1262 For relocatable code objects, *name* is the name of the section that the 1263 relocation records apply. For example, ``.rela.text`` is the section name for 1264 relocation records associated with the ``.text`` section. 1265 1266 For linked shared code objects, ``.rela.dyn`` contains all the relocation 1267 records from each of the relocatable code object's ``.rela``\ *name* sections. 1268 1269 See :ref:`amdgpu-relocation-records` for the relocation records supported by 1270 the AMDGPU backend. 1271 1272``.text`` 1273 The executable machine code for the kernels and functions they call. Generated 1274 as position independent code. See :ref:`amdgpu-code-conventions` for 1275 information on conventions used in the isa generation. 1276 1277.. _amdgpu-note-records: 1278 1279Note Records 1280------------ 1281 1282The AMDGPU backend code object contains ELF note records in the ``.note`` 1283section. The set of generated notes and their semantics depend on the code 1284object version; see :ref:`amdgpu-note-records-v2` and 1285:ref:`amdgpu-note-records-v3-v4`. 1286 1287As required by ``ELFCLASS32`` and ``ELFCLASS64``, minimal zero-byte padding 1288must be generated after the ``name`` field to ensure the ``desc`` field is 4 1289byte aligned. In addition, minimal zero-byte padding must be generated to 1290ensure the ``desc`` field size is a multiple of 4 bytes. The ``sh_addralign`` 1291field of the ``.note`` section must be at least 4 to indicate at least 8 byte 1292alignment. 1293 1294.. _amdgpu-note-records-v2: 1295 1296Code Object V2 Note Records 1297~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1298 1299.. warning:: 1300 Code object V2 is not the default code object version emitted by 1301 this version of LLVM. 1302 1303The AMDGPU backend code object uses the following ELF note record in the 1304``.note`` section when compiling for code object V2. 1305 1306The note record vendor field is "AMD". 1307 1308Additional note records may be present, but any which are not documented here 1309are deprecated and should not be used. 1310 1311 .. table:: AMDGPU Code Object V2 ELF Note Records 1312 :name: amdgpu-elf-note-records-v2-table 1313 1314 ===== ===================================== ====================================== 1315 Name Type Description 1316 ===== ===================================== ====================================== 1317 "AMD" ``NT_AMD_HSA_CODE_OBJECT_VERSION`` Code object version. 1318 "AMD" ``NT_AMD_HSA_HSAIL`` HSAIL properties generated by the HSAIL 1319 Finalizer and not the LLVM compiler. 1320 "AMD" ``NT_AMD_HSA_ISA_VERSION`` Target ISA version. 1321 "AMD" ``NT_AMD_HSA_METADATA`` Metadata null terminated string in 1322 YAML [YAML]_ textual format. 1323 "AMD" ``NT_AMD_HSA_ISA_NAME`` Target ISA name. 1324 ===== ===================================== ====================================== 1325 1326.. 1327 1328 .. table:: AMDGPU Code Object V2 ELF Note Record Enumeration Values 1329 :name: amdgpu-elf-note-record-enumeration-values-v2-table 1330 1331 ===================================== ===== 1332 Name Value 1333 ===================================== ===== 1334 ``NT_AMD_HSA_CODE_OBJECT_VERSION`` 1 1335 ``NT_AMD_HSA_HSAIL`` 2 1336 ``NT_AMD_HSA_ISA_VERSION`` 3 1337 *reserved* 4-9 1338 ``NT_AMD_HSA_METADATA`` 10 1339 ``NT_AMD_HSA_ISA_NAME`` 11 1340 ===================================== ===== 1341 1342``NT_AMD_HSA_CODE_OBJECT_VERSION`` 1343 Specifies the code object version number. The description field has the 1344 following layout: 1345 1346 .. code:: c 1347 1348 struct amdgpu_hsa_note_code_object_version_s { 1349 uint32_t major_version; 1350 uint32_t minor_version; 1351 }; 1352 1353 The ``major_version`` has a value less than or equal to 2. 1354 1355``NT_AMD_HSA_HSAIL`` 1356 Specifies the HSAIL properties used by the HSAIL Finalizer. The description 1357 field has the following layout: 1358 1359 .. code:: c 1360 1361 struct amdgpu_hsa_note_hsail_s { 1362 uint32_t hsail_major_version; 1363 uint32_t hsail_minor_version; 1364 uint8_t profile; 1365 uint8_t machine_model; 1366 uint8_t default_float_round; 1367 }; 1368 1369``NT_AMD_HSA_ISA_VERSION`` 1370 Specifies the target ISA version. The description field has the following layout: 1371 1372 .. code:: c 1373 1374 struct amdgpu_hsa_note_isa_s { 1375 uint16_t vendor_name_size; 1376 uint16_t architecture_name_size; 1377 uint32_t major; 1378 uint32_t minor; 1379 uint32_t stepping; 1380 char vendor_and_architecture_name[1]; 1381 }; 1382 1383 ``vendor_name_size`` and ``architecture_name_size`` are the length of the 1384 vendor and architecture names respectively, including the NUL character. 1385 1386 ``vendor_and_architecture_name`` contains the NUL terminates string for the 1387 vendor, immediately followed by the NUL terminated string for the 1388 architecture. 1389 1390 This note record is used by the HSA runtime loader. 1391 1392 Code object V2 only supports a limited number of processors and has fixed 1393 settings for target features. See 1394 :ref:`amdgpu-elf-note-record-supported_processors-v2-table` for a list of 1395 processors and the corresponding target ID. In the table the note record ISA 1396 name is a concatenation of the vendor name, architecture name, major, minor, 1397 and stepping separated by a ":". 1398 1399 The target ID column shows the processor name and fixed target features used 1400 by the LLVM compiler. The LLVM compiler does not generate a 1401 ``NT_AMD_HSA_HSAIL`` note record. 1402 1403 A code object generated by the Finalizer also uses code object V2 and always 1404 generates a ``NT_AMD_HSA_HSAIL`` note record. The processor name and 1405 ``sramecc`` target feature is as shown in 1406 :ref:`amdgpu-elf-note-record-supported_processors-v2-table` but the ``xnack`` 1407 target feature is specified by the ``EF_AMDGPU_FEATURE_XNACK_V2`` ``e_flags`` 1408 bit. 1409 1410``NT_AMD_HSA_ISA_NAME`` 1411 Specifies the target ISA name as a non-NUL terminated string. 1412 1413 This note record is not used by the HSA runtime loader. 1414 1415 See the ``NT_AMD_HSA_ISA_VERSION`` note record description of the code object 1416 V2's limited support of processors and fixed settings for target features. 1417 1418 See :ref:`amdgpu-elf-note-record-supported_processors-v2-table` for a mapping 1419 from the string to the corresponding target ID. If the ``xnack`` target 1420 feature is supported and enabled, the string produced by the LLVM compiler 1421 will may have a ``+xnack`` appended. The Finlizer did not do the appending and 1422 instead used the ``EF_AMDGPU_FEATURE_XNACK_V2`` ``e_flags`` bit. 1423 1424``NT_AMD_HSA_METADATA`` 1425 Specifies extensible metadata associated with the code objects executed on HSA 1426 [HSA]_ compatible runtimes (see :ref:`amdgpu-os`). It is required when the 1427 target triple OS is ``amdhsa`` (see :ref:`amdgpu-target-triples`). See 1428 :ref:`amdgpu-amdhsa-code-object-metadata-v2` for the syntax of the code object 1429 metadata string. 1430 1431 .. table:: AMDGPU Code Object V2 Supported Processors and Fixed Target Feature Settings 1432 :name: amdgpu-elf-note-record-supported_processors-v2-table 1433 1434 ===================== ========================== 1435 Note Record ISA Name Target ID 1436 ===================== ========================== 1437 ``AMD:AMDGPU:6:0:0`` ``gfx600`` 1438 ``AMD:AMDGPU:6:0:1`` ``gfx601`` 1439 ``AMD:AMDGPU:6:0:2`` ``gfx602`` 1440 ``AMD:AMDGPU:7:0:0`` ``gfx700`` 1441 ``AMD:AMDGPU:7:0:1`` ``gfx701`` 1442 ``AMD:AMDGPU:7:0:2`` ``gfx702`` 1443 ``AMD:AMDGPU:7:0:3`` ``gfx703`` 1444 ``AMD:AMDGPU:7:0:4`` ``gfx704`` 1445 ``AMD:AMDGPU:7:0:5`` ``gfx705`` 1446 ``AMD:AMDGPU:8:0:0`` ``gfx802`` 1447 ``AMD:AMDGPU:8:0:1`` ``gfx801:xnack+`` 1448 ``AMD:AMDGPU:8:0:2`` ``gfx802`` 1449 ``AMD:AMDGPU:8:0:3`` ``gfx803`` 1450 ``AMD:AMDGPU:8:0:4`` ``gfx803`` 1451 ``AMD:AMDGPU:8:0:5`` ``gfx805`` 1452 ``AMD:AMDGPU:8:1:0`` ``gfx810:xnack+`` 1453 ``AMD:AMDGPU:9:0:0`` ``gfx900:xnack-`` 1454 ``AMD:AMDGPU:9:0:1`` ``gfx900:xnack+`` 1455 ``AMD:AMDGPU:9:0:2`` ``gfx902:xnack-`` 1456 ``AMD:AMDGPU:9:0:3`` ``gfx902:xnack+`` 1457 ``AMD:AMDGPU:9:0:4`` ``gfx904:xnack-`` 1458 ``AMD:AMDGPU:9:0:5`` ``gfx904:xnack+`` 1459 ``AMD:AMDGPU:9:0:6`` ``gfx906:sramecc-:xnack-`` 1460 ``AMD:AMDGPU:9:0:7`` ``gfx906:sramecc-:xnack+`` 1461 ``AMD:AMDGPU:9:0:12`` ``gfx90c:xnack-`` 1462 ===================== ========================== 1463 1464.. _amdgpu-note-records-v3-v4: 1465 1466Code Object V3 to V4 Note Records 1467~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1468 1469The AMDGPU backend code object uses the following ELF note record in the 1470``.note`` section when compiling for code object V3 to V4. 1471 1472The note record vendor field is "AMDGPU". 1473 1474Additional note records may be present, but any which are not documented here 1475are deprecated and should not be used. 1476 1477 .. table:: AMDGPU Code Object V3 to V4 ELF Note Records 1478 :name: amdgpu-elf-note-records-table-v3-v4 1479 1480 ======== ============================== ====================================== 1481 Name Type Description 1482 ======== ============================== ====================================== 1483 "AMDGPU" ``NT_AMDGPU_METADATA`` Metadata in Message Pack [MsgPack]_ 1484 binary format. 1485 ======== ============================== ====================================== 1486 1487.. 1488 1489 .. table:: AMDGPU Code Object V3 to V4 ELF Note Record Enumeration Values 1490 :name: amdgpu-elf-note-record-enumeration-values-table-v3-v4 1491 1492 ============================== ===== 1493 Name Value 1494 ============================== ===== 1495 *reserved* 0-31 1496 ``NT_AMDGPU_METADATA`` 32 1497 ============================== ===== 1498 1499``NT_AMDGPU_METADATA`` 1500 Specifies extensible metadata associated with an AMDGPU code object. It is 1501 encoded as a map in the Message Pack [MsgPack]_ binary data format. See 1502 :ref:`amdgpu-amdhsa-code-object-metadata-v3` and 1503 :ref:`amdgpu-amdhsa-code-object-metadata-v4` for the map keys defined for the 1504 ``amdhsa`` OS. 1505 1506.. _amdgpu-symbols: 1507 1508Symbols 1509------- 1510 1511Symbols include the following: 1512 1513 .. table:: AMDGPU ELF Symbols 1514 :name: amdgpu-elf-symbols-table 1515 1516 ===================== ================== ================ ================== 1517 Name Type Section Description 1518 ===================== ================== ================ ================== 1519 *link-name* ``STT_OBJECT`` - ``.data`` Global variable 1520 - ``.rodata`` 1521 - ``.bss`` 1522 *link-name*\ ``.kd`` ``STT_OBJECT`` - ``.rodata`` Kernel descriptor 1523 *link-name* ``STT_FUNC`` - ``.text`` Kernel entry point 1524 *link-name* ``STT_OBJECT`` - SHN_AMDGPU_LDS Global variable in LDS 1525 ===================== ================== ================ ================== 1526 1527Global variable 1528 Global variables both used and defined by the compilation unit. 1529 1530 If the symbol is defined in the compilation unit then it is allocated in the 1531 appropriate section according to if it has initialized data or is readonly. 1532 1533 If the symbol is external then its section is ``STN_UNDEF`` and the loader 1534 will resolve relocations using the definition provided by another code object 1535 or explicitly defined by the runtime. 1536 1537 If the symbol resides in local/group memory (LDS) then its section is the 1538 special processor specific section name ``SHN_AMDGPU_LDS``, and the 1539 ``st_value`` field describes alignment requirements as it does for common 1540 symbols. 1541 1542 .. TODO:: 1543 1544 Add description of linked shared object symbols. Seems undefined symbols 1545 are marked as STT_NOTYPE. 1546 1547Kernel descriptor 1548 Every HSA kernel has an associated kernel descriptor. It is the address of the 1549 kernel descriptor that is used in the AQL dispatch packet used to invoke the 1550 kernel, not the kernel entry point. The layout of the HSA kernel descriptor is 1551 defined in :ref:`amdgpu-amdhsa-kernel-descriptor`. 1552 1553Kernel entry point 1554 Every HSA kernel also has a symbol for its machine code entry point. 1555 1556.. _amdgpu-relocation-records: 1557 1558Relocation Records 1559------------------ 1560 1561AMDGPU backend generates ``Elf64_Rela`` relocation records. Supported 1562relocatable fields are: 1563 1564``word32`` 1565 This specifies a 32-bit field occupying 4 bytes with arbitrary byte 1566 alignment. These values use the same byte order as other word values in the 1567 AMDGPU architecture. 1568 1569``word64`` 1570 This specifies a 64-bit field occupying 8 bytes with arbitrary byte 1571 alignment. These values use the same byte order as other word values in the 1572 AMDGPU architecture. 1573 1574Following notations are used for specifying relocation calculations: 1575 1576**A** 1577 Represents the addend used to compute the value of the relocatable field. 1578 1579**G** 1580 Represents the offset into the global offset table at which the relocation 1581 entry's symbol will reside during execution. 1582 1583**GOT** 1584 Represents the address of the global offset table. 1585 1586**P** 1587 Represents the place (section offset for ``et_rel`` or address for ``et_dyn``) 1588 of the storage unit being relocated (computed using ``r_offset``). 1589 1590**S** 1591 Represents the value of the symbol whose index resides in the relocation 1592 entry. Relocations not using this must specify a symbol index of 1593 ``STN_UNDEF``. 1594 1595**B** 1596 Represents the base address of a loaded executable or shared object which is 1597 the difference between the ELF address and the actual load address. 1598 Relocations using this are only valid in executable or shared objects. 1599 1600The following relocation types are supported: 1601 1602 .. table:: AMDGPU ELF Relocation Records 1603 :name: amdgpu-elf-relocation-records-table 1604 1605 ========================== ======= ===== ========== ============================== 1606 Relocation Type Kind Value Field Calculation 1607 ========================== ======= ===== ========== ============================== 1608 ``R_AMDGPU_NONE`` 0 *none* *none* 1609 ``R_AMDGPU_ABS32_LO`` Static, 1 ``word32`` (S + A) & 0xFFFFFFFF 1610 Dynamic 1611 ``R_AMDGPU_ABS32_HI`` Static, 2 ``word32`` (S + A) >> 32 1612 Dynamic 1613 ``R_AMDGPU_ABS64`` Static, 3 ``word64`` S + A 1614 Dynamic 1615 ``R_AMDGPU_REL32`` Static 4 ``word32`` S + A - P 1616 ``R_AMDGPU_REL64`` Static 5 ``word64`` S + A - P 1617 ``R_AMDGPU_ABS32`` Static, 6 ``word32`` S + A 1618 Dynamic 1619 ``R_AMDGPU_GOTPCREL`` Static 7 ``word32`` G + GOT + A - P 1620 ``R_AMDGPU_GOTPCREL32_LO`` Static 8 ``word32`` (G + GOT + A - P) & 0xFFFFFFFF 1621 ``R_AMDGPU_GOTPCREL32_HI`` Static 9 ``word32`` (G + GOT + A - P) >> 32 1622 ``R_AMDGPU_REL32_LO`` Static 10 ``word32`` (S + A - P) & 0xFFFFFFFF 1623 ``R_AMDGPU_REL32_HI`` Static 11 ``word32`` (S + A - P) >> 32 1624 *reserved* 12 1625 ``R_AMDGPU_RELATIVE64`` Dynamic 13 ``word64`` B + A 1626 ``R_AMDGPU_REL16`` Static 14 ``word16`` ((S + A - P) - 4) / 4 1627 ========================== ======= ===== ========== ============================== 1628 1629``R_AMDGPU_ABS32_LO`` and ``R_AMDGPU_ABS32_HI`` are only supported by 1630the ``mesa3d`` OS, which does not support ``R_AMDGPU_ABS64``. 1631 1632There is no current OS loader support for 32-bit programs and so 1633``R_AMDGPU_ABS32`` is not used. 1634 1635.. _amdgpu-loaded-code-object-path-uniform-resource-identifier: 1636 1637Loaded Code Object Path Uniform Resource Identifier (URI) 1638--------------------------------------------------------- 1639 1640The AMD GPU code object loader represents the path of the ELF shared object from 1641which the code object was loaded as a textual Uniform Resource Identifier (URI). 1642Note that the code object is the in memory loaded relocated form of the ELF 1643shared object. Multiple code objects may be loaded at different memory 1644addresses in the same process from the same ELF shared object. 1645 1646The loaded code object path URI syntax is defined by the following BNF syntax: 1647 1648.. code:: 1649 1650 code_object_uri ::== file_uri | memory_uri 1651 file_uri ::== "file://" file_path [ range_specifier ] 1652 memory_uri ::== "memory://" process_id range_specifier 1653 range_specifier ::== [ "#" | "?" ] "offset=" number "&" "size=" number 1654 file_path ::== URI_ENCODED_OS_FILE_PATH 1655 process_id ::== DECIMAL_NUMBER 1656 number ::== HEX_NUMBER | DECIMAL_NUMBER | OCTAL_NUMBER 1657 1658**number** 1659 Is a C integral literal where hexadecimal values are prefixed by "0x" or "0X", 1660 and octal values by "0". 1661 1662**file_path** 1663 Is the file's path specified as a URI encoded UTF-8 string. In URI encoding, 1664 every character that is not in the regular expression ``[a-zA-Z0-9/_.~-]`` is 1665 encoded as two uppercase hexadecimal digits proceeded by "%". Directories in 1666 the path are separated by "/". 1667 1668**offset** 1669 Is a 0-based byte offset to the start of the code object. For a file URI, it 1670 is from the start of the file specified by the ``file_path``, and if omitted 1671 defaults to 0. For a memory URI, it is the memory address and is required. 1672 1673**size** 1674 Is the number of bytes in the code object. For a file URI, if omitted it 1675 defaults to the size of the file. It is required for a memory URI. 1676 1677**process_id** 1678 Is the identity of the process owning the memory. For Linux it is the C 1679 unsigned integral decimal literal for the process ID (PID). 1680 1681For example: 1682 1683.. code:: 1684 1685 file:///dir1/dir2/file1 1686 file:///dir3/dir4/file2#offset=0x2000&size=3000 1687 memory://1234#offset=0x20000&size=3000 1688 1689.. _amdgpu-dwarf-debug-information: 1690 1691DWARF Debug Information 1692======================= 1693 1694.. warning:: 1695 1696 This section describes **provisional support** for AMDGPU DWARF [DWARF]_ that 1697 is not currently fully implemented and is subject to change. 1698 1699AMDGPU generates DWARF [DWARF]_ debugging information ELF sections (see 1700:ref:`amdgpu-elf-code-object`) which contain information that maps the code 1701object executable code and data to the source language constructs. It can be 1702used by tools such as debuggers and profilers. It uses features defined in 1703:doc:`AMDGPUDwarfExtensionsForHeterogeneousDebugging` that are made available in 1704DWARF Version 4 and DWARF Version 5 as an LLVM vendor extension. 1705 1706This section defines the AMDGPU target architecture specific DWARF mappings. 1707 1708.. _amdgpu-dwarf-register-identifier: 1709 1710Register Identifier 1711------------------- 1712 1713This section defines the AMDGPU target architecture register numbers used in 1714DWARF operation expressions (see DWARF Version 5 section 2.5 and 1715:ref:`amdgpu-dwarf-operation-expressions`) and Call Frame Information 1716instructions (see DWARF Version 5 section 6.4 and 1717:ref:`amdgpu-dwarf-call-frame-information`). 1718 1719A single code object can contain code for kernels that have different wavefront 1720sizes. The vector registers and some scalar registers are based on the wavefront 1721size. AMDGPU defines distinct DWARF registers for each wavefront size. This 1722simplifies the consumer of the DWARF so that each register has a fixed size, 1723rather than being dynamic according to the wavefront size mode. Similarly, 1724distinct DWARF registers are defined for those registers that vary in size 1725according to the process address size. This allows a consumer to treat a 1726specific AMDGPU processor as a single architecture regardless of how it is 1727configured at run time. The compiler explicitly specifies the DWARF registers 1728that match the mode in which the code it is generating will be executed. 1729 1730DWARF registers are encoded as numbers, which are mapped to architecture 1731registers. The mapping for AMDGPU is defined in 1732:ref:`amdgpu-dwarf-register-mapping-table`. All AMDGPU targets use the same 1733mapping. 1734 1735.. table:: AMDGPU DWARF Register Mapping 1736 :name: amdgpu-dwarf-register-mapping-table 1737 1738 ============== ================= ======== ================================== 1739 DWARF Register AMDGPU Register Bit Size Description 1740 ============== ================= ======== ================================== 1741 0 PC_32 32 Program Counter (PC) when 1742 executing in a 32-bit process 1743 address space. Used in the CFI to 1744 describe the PC of the calling 1745 frame. 1746 1 EXEC_MASK_32 32 Execution Mask Register when 1747 executing in wavefront 32 mode. 1748 2-15 *Reserved* *Reserved for highly accessed 1749 registers using DWARF shortcut.* 1750 16 PC_64 64 Program Counter (PC) when 1751 executing in a 64-bit process 1752 address space. Used in the CFI to 1753 describe the PC of the calling 1754 frame. 1755 17 EXEC_MASK_64 64 Execution Mask Register when 1756 executing in wavefront 64 mode. 1757 18-31 *Reserved* *Reserved for highly accessed 1758 registers using DWARF shortcut.* 1759 32-95 SGPR0-SGPR63 32 Scalar General Purpose 1760 Registers. 1761 96-127 *Reserved* *Reserved for frequently accessed 1762 registers using DWARF 1-byte ULEB.* 1763 128 STATUS 32 Status Register. 1764 129-511 *Reserved* *Reserved for future Scalar 1765 Architectural Registers.* 1766 512 VCC_32 32 Vector Condition Code Register 1767 when executing in wavefront 32 1768 mode. 1769 513-767 *Reserved* *Reserved for future Vector 1770 Architectural Registers when 1771 executing in wavefront 32 mode.* 1772 768 VCC_64 64 Vector Condition Code Register 1773 when executing in wavefront 64 1774 mode. 1775 769-1023 *Reserved* *Reserved for future Vector 1776 Architectural Registers when 1777 executing in wavefront 64 mode.* 1778 1024-1087 *Reserved* *Reserved for padding.* 1779 1088-1129 SGPR64-SGPR105 32 Scalar General Purpose Registers. 1780 1130-1535 *Reserved* *Reserved for future Scalar 1781 General Purpose Registers.* 1782 1536-1791 VGPR0-VGPR255 32*32 Vector General Purpose Registers 1783 when executing in wavefront 32 1784 mode. 1785 1792-2047 *Reserved* *Reserved for future Vector 1786 General Purpose Registers when 1787 executing in wavefront 32 mode.* 1788 2048-2303 AGPR0-AGPR255 32*32 Vector Accumulation Registers 1789 when executing in wavefront 32 1790 mode. 1791 2304-2559 *Reserved* *Reserved for future Vector 1792 Accumulation Registers when 1793 executing in wavefront 32 mode.* 1794 2560-2815 VGPR0-VGPR255 64*32 Vector General Purpose Registers 1795 when executing in wavefront 64 1796 mode. 1797 2816-3071 *Reserved* *Reserved for future Vector 1798 General Purpose Registers when 1799 executing in wavefront 64 mode.* 1800 3072-3327 AGPR0-AGPR255 64*32 Vector Accumulation Registers 1801 when executing in wavefront 64 1802 mode. 1803 3328-3583 *Reserved* *Reserved for future Vector 1804 Accumulation Registers when 1805 executing in wavefront 64 mode.* 1806 ============== ================= ======== ================================== 1807 1808The vector registers are represented as the full size for the wavefront. They 1809are organized as consecutive dwords (32-bits), one per lane, with the dword at 1810the least significant bit position corresponding to lane 0 and so forth. DWARF 1811location expressions involving the ``DW_OP_LLVM_offset`` and 1812``DW_OP_LLVM_push_lane`` operations are used to select the part of the vector 1813register corresponding to the lane that is executing the current thread of 1814execution in languages that are implemented using a SIMD or SIMT execution 1815model. 1816 1817If the wavefront size is 32 lanes then the wavefront 32 mode register 1818definitions are used. If the wavefront size is 64 lanes then the wavefront 64 1819mode register definitions are used. Some AMDGPU targets support executing in 1820both wavefront 32 and wavefront 64 mode. The register definitions corresponding 1821to the wavefront mode of the generated code will be used. 1822 1823If code is generated to execute in a 32-bit process address space, then the 182432-bit process address space register definitions are used. If code is generated 1825to execute in a 64-bit process address space, then the 64-bit process address 1826space register definitions are used. The ``amdgcn`` target only supports the 182764-bit process address space. 1828 1829.. _amdgpu-dwarf-address-class-identifier: 1830 1831Address Class Identifier 1832------------------------ 1833 1834The DWARF address class represents the source language memory space. See DWARF 1835Version 5 section 2.12 which is updated by the *DWARF Extensions For 1836Heterogeneous Debugging* section :ref:`amdgpu-dwarf-segment_addresses`. 1837 1838The DWARF address class mapping used for AMDGPU is defined in 1839:ref:`amdgpu-dwarf-address-class-mapping-table`. 1840 1841.. table:: AMDGPU DWARF Address Class Mapping 1842 :name: amdgpu-dwarf-address-class-mapping-table 1843 1844 ========================= ====== ================= 1845 DWARF AMDGPU 1846 -------------------------------- ----------------- 1847 Address Class Name Value Address Space 1848 ========================= ====== ================= 1849 ``DW_ADDR_none`` 0x0000 Generic (Flat) 1850 ``DW_ADDR_LLVM_global`` 0x0001 Global 1851 ``DW_ADDR_LLVM_constant`` 0x0002 Global 1852 ``DW_ADDR_LLVM_group`` 0x0003 Local (group/LDS) 1853 ``DW_ADDR_LLVM_private`` 0x0004 Private (Scratch) 1854 ``DW_ADDR_AMDGPU_region`` 0x8000 Region (GDS) 1855 ========================= ====== ================= 1856 1857The DWARF address class values defined in the *DWARF Extensions For 1858Heterogeneous Debugging* section :ref:`amdgpu-dwarf-segment_addresses` are used. 1859 1860In addition, ``DW_ADDR_AMDGPU_region`` is encoded as a vendor extension. This is 1861available for use for the AMD extension for access to the hardware GDS memory 1862which is scratchpad memory allocated per device. 1863 1864For AMDGPU if no ``DW_AT_address_class`` attribute is present, then the default 1865address class of ``DW_ADDR_none`` is used. 1866 1867See :ref:`amdgpu-dwarf-address-space-identifier` for information on the AMDGPU 1868mapping of DWARF address classes to DWARF address spaces, including address size 1869and NULL value. 1870 1871.. _amdgpu-dwarf-address-space-identifier: 1872 1873Address Space Identifier 1874------------------------ 1875 1876DWARF address spaces correspond to target architecture specific linear 1877addressable memory areas. See DWARF Version 5 section 2.12 and *DWARF Extensions 1878For Heterogeneous Debugging* section :ref:`amdgpu-dwarf-segment_addresses`. 1879 1880The DWARF address space mapping used for AMDGPU is defined in 1881:ref:`amdgpu-dwarf-address-space-mapping-table`. 1882 1883.. table:: AMDGPU DWARF Address Space Mapping 1884 :name: amdgpu-dwarf-address-space-mapping-table 1885 1886 ======================================= ===== ======= ======== ================= ======================= 1887 DWARF AMDGPU Notes 1888 --------------------------------------- ----- ---------------- ----------------- ----------------------- 1889 Address Space Name Value Address Bit Size Address Space 1890 --------------------------------------- ----- ------- -------- ----------------- ----------------------- 1891 .. 64-bit 32-bit 1892 process process 1893 address address 1894 space space 1895 ======================================= ===== ======= ======== ================= ======================= 1896 ``DW_ASPACE_none`` 0x00 64 32 Global *default address space* 1897 ``DW_ASPACE_AMDGPU_generic`` 0x01 64 32 Generic (Flat) 1898 ``DW_ASPACE_AMDGPU_region`` 0x02 32 32 Region (GDS) 1899 ``DW_ASPACE_AMDGPU_local`` 0x03 32 32 Local (group/LDS) 1900 *Reserved* 0x04 1901 ``DW_ASPACE_AMDGPU_private_lane`` 0x05 32 32 Private (Scratch) *focused lane* 1902 ``DW_ASPACE_AMDGPU_private_wave`` 0x06 32 32 Private (Scratch) *unswizzled wavefront* 1903 ======================================= ===== ======= ======== ================= ======================= 1904 1905See :ref:`amdgpu-address-spaces` for information on the AMDGPU address spaces 1906including address size and NULL value. 1907 1908The ``DW_ASPACE_none`` address space is the default target architecture address 1909space used in DWARF operations that do not specify an address space. It 1910therefore has to map to the global address space so that the ``DW_OP_addr*`` and 1911related operations can refer to addresses in the program code. 1912 1913The ``DW_ASPACE_AMDGPU_generic`` address space allows location expressions to 1914specify the flat address space. If the address corresponds to an address in the 1915local address space, then it corresponds to the wavefront that is executing the 1916focused thread of execution. If the address corresponds to an address in the 1917private address space, then it corresponds to the lane that is executing the 1918focused thread of execution for languages that are implemented using a SIMD or 1919SIMT execution model. 1920 1921.. note:: 1922 1923 CUDA-like languages such as HIP that do not have address spaces in the 1924 language type system, but do allow variables to be allocated in different 1925 address spaces, need to explicitly specify the ``DW_ASPACE_AMDGPU_generic`` 1926 address space in the DWARF expression operations as the default address space 1927 is the global address space. 1928 1929The ``DW_ASPACE_AMDGPU_local`` address space allows location expressions to 1930specify the local address space corresponding to the wavefront that is executing 1931the focused thread of execution. 1932 1933The ``DW_ASPACE_AMDGPU_private_lane`` address space allows location expressions 1934to specify the private address space corresponding to the lane that is executing 1935the focused thread of execution for languages that are implemented using a SIMD 1936or SIMT execution model. 1937 1938The ``DW_ASPACE_AMDGPU_private_wave`` address space allows location expressions 1939to specify the unswizzled private address space corresponding to the wavefront 1940that is executing the focused thread of execution. The wavefront view of private 1941memory is the per wavefront unswizzled backing memory layout defined in 1942:ref:`amdgpu-address-spaces`, such that address 0 corresponds to the first 1943location for the backing memory of the wavefront (namely the address is not 1944offset by ``wavefront-scratch-base``). The following formula can be used to 1945convert from a ``DW_ASPACE_AMDGPU_private_lane`` address to a 1946``DW_ASPACE_AMDGPU_private_wave`` address: 1947 1948:: 1949 1950 private-address-wavefront = 1951 ((private-address-lane / 4) * wavefront-size * 4) + 1952 (wavefront-lane-id * 4) + (private-address-lane % 4) 1953 1954If the ``DW_ASPACE_AMDGPU_private_lane`` address is dword aligned, and the start 1955of the dwords for each lane starting with lane 0 is required, then this 1956simplifies to: 1957 1958:: 1959 1960 private-address-wavefront = 1961 private-address-lane * wavefront-size 1962 1963A compiler can use the ``DW_ASPACE_AMDGPU_private_wave`` address space to read a 1964complete spilled vector register back into a complete vector register in the 1965CFI. The frame pointer can be a private lane address which is dword aligned, 1966which can be shifted to multiply by the wavefront size, and then used to form a 1967private wavefront address that gives a location for a contiguous set of dwords, 1968one per lane, where the vector register dwords are spilled. The compiler knows 1969the wavefront size since it generates the code. Note that the type of the 1970address may have to be converted as the size of a 1971``DW_ASPACE_AMDGPU_private_lane`` address may be smaller than the size of a 1972``DW_ASPACE_AMDGPU_private_wave`` address. 1973 1974.. _amdgpu-dwarf-lane-identifier: 1975 1976Lane identifier 1977--------------- 1978 1979DWARF lane identifies specify a target architecture lane position for hardware 1980that executes in a SIMD or SIMT manner, and on which a source language maps its 1981threads of execution onto those lanes. The DWARF lane identifier is pushed by 1982the ``DW_OP_LLVM_push_lane`` DWARF expression operation. See DWARF Version 5 1983section 2.5 which is updated by *DWARF Extensions For Heterogeneous Debugging* 1984section :ref:`amdgpu-dwarf-operation-expressions`. 1985 1986For AMDGPU, the lane identifier corresponds to the hardware lane ID of a 1987wavefront. It is numbered from 0 to the wavefront size minus 1. 1988 1989Operation Expressions 1990--------------------- 1991 1992DWARF expressions are used to compute program values and the locations of 1993program objects. See DWARF Version 5 section 2.5 and 1994:ref:`amdgpu-dwarf-operation-expressions`. 1995 1996DWARF location descriptions describe how to access storage which includes memory 1997and registers. When accessing storage on AMDGPU, bytes are ordered with least 1998significant bytes first, and bits are ordered within bytes with least 1999significant bits first. 2000 2001For AMDGPU CFI expressions, ``DW_OP_LLVM_select_bit_piece`` is used to describe 2002unwinding vector registers that are spilled under the execution mask to memory: 2003the zero-single location description is the vector register, and the one-single 2004location description is the spilled memory location description. The 2005``DW_OP_LLVM_form_aspace_address`` is used to specify the address space of the 2006memory location description. 2007 2008In AMDGPU expressions, ``DW_OP_LLVM_select_bit_piece`` is used by the 2009``DW_AT_LLVM_lane_pc`` attribute expression where divergent control flow is 2010controlled by the execution mask. An undefined location description together 2011with ``DW_OP_LLVM_extend`` is used to indicate the lane was not active on entry 2012to the subprogram. See :ref:`amdgpu-dwarf-dw-at-llvm-lane-pc` for an example. 2013 2014Debugger Information Entry Attributes 2015------------------------------------- 2016 2017This section describes how certain debugger information entry attributes are 2018used by AMDGPU. See the sections in DWARF Version 5 section 2 which are updated 2019by *DWARF Extensions For Heterogeneous Debugging* section 2020:ref:`amdgpu-dwarf-debugging-information-entry-attributes`. 2021 2022.. _amdgpu-dwarf-dw-at-llvm-lane-pc: 2023 2024``DW_AT_LLVM_lane_pc`` 2025~~~~~~~~~~~~~~~~~~~~~~ 2026 2027For AMDGPU, the ``DW_AT_LLVM_lane_pc`` attribute is used to specify the program 2028location of the separate lanes of a SIMT thread. 2029 2030If the lane is an active lane then this will be the same as the current program 2031location. 2032 2033If the lane is inactive, but was active on entry to the subprogram, then this is 2034the program location in the subprogram at which execution of the lane is 2035conceptual positioned. 2036 2037If the lane was not active on entry to the subprogram, then this will be the 2038undefined location. A client debugger can check if the lane is part of a valid 2039work-group by checking that the lane is in the range of the associated 2040work-group within the grid, accounting for partial work-groups. If it is not, 2041then the debugger can omit any information for the lane. Otherwise, the debugger 2042may repeatedly unwind the stack and inspect the ``DW_AT_LLVM_lane_pc`` of the 2043calling subprogram until it finds a non-undefined location. Conceptually the 2044lane only has the call frames that it has a non-undefined 2045``DW_AT_LLVM_lane_pc``. 2046 2047The following example illustrates how the AMDGPU backend can generate a DWARF 2048location list expression for the nested ``IF/THEN/ELSE`` structures of the 2049following subprogram pseudo code for a target with 64 lanes per wavefront. 2050 2051.. code:: 2052 :number-lines: 2053 2054 SUBPROGRAM X 2055 BEGIN 2056 a; 2057 IF (c1) THEN 2058 b; 2059 IF (c2) THEN 2060 c; 2061 ELSE 2062 d; 2063 ENDIF 2064 e; 2065 ELSE 2066 f; 2067 ENDIF 2068 g; 2069 END 2070 2071The AMDGPU backend may generate the following pseudo LLVM MIR to manipulate the 2072execution mask (``EXEC``) to linearize the control flow. The condition is 2073evaluated to make a mask of the lanes for which the condition evaluates to true. 2074First the ``THEN`` region is executed by setting the ``EXEC`` mask to the 2075logical ``AND`` of the current ``EXEC`` mask with the condition mask. Then the 2076``ELSE`` region is executed by negating the ``EXEC`` mask and logical ``AND`` of 2077the saved ``EXEC`` mask at the start of the region. After the ``IF/THEN/ELSE`` 2078region the ``EXEC`` mask is restored to the value it had at the beginning of the 2079region. This is shown below. Other approaches are possible, but the basic 2080concept is the same. 2081 2082.. code:: 2083 :number-lines: 2084 2085 $lex_start: 2086 a; 2087 %1 = EXEC 2088 %2 = c1 2089 $lex_1_start: 2090 EXEC = %1 & %2 2091 $if_1_then: 2092 b; 2093 %3 = EXEC 2094 %4 = c2 2095 $lex_1_1_start: 2096 EXEC = %3 & %4 2097 $lex_1_1_then: 2098 c; 2099 EXEC = ~EXEC & %3 2100 $lex_1_1_else: 2101 d; 2102 EXEC = %3 2103 $lex_1_1_end: 2104 e; 2105 EXEC = ~EXEC & %1 2106 $lex_1_else: 2107 f; 2108 EXEC = %1 2109 $lex_1_end: 2110 g; 2111 $lex_end: 2112 2113To create the DWARF location list expression that defines the location 2114description of a vector of lane program locations, the LLVM MIR ``DBG_VALUE`` 2115pseudo instruction can be used to annotate the linearized control flow. This can 2116be done by defining an artificial variable for the lane PC. The DWARF location 2117list expression created for it is used as the value of the 2118``DW_AT_LLVM_lane_pc`` attribute on the subprogram's debugger information entry. 2119 2120A DWARF procedure is defined for each well nested structured control flow region 2121which provides the conceptual lane program location for a lane if it is not 2122active (namely it is divergent). The DWARF operation expression for each region 2123conceptually inherits the value of the immediately enclosing region and modifies 2124it according to the semantics of the region. 2125 2126For an ``IF/THEN/ELSE`` region the divergent program location is at the start of 2127the region for the ``THEN`` region since it is executed first. For the ``ELSE`` 2128region the divergent program location is at the end of the ``IF/THEN/ELSE`` 2129region since the ``THEN`` region has completed. 2130 2131The lane PC artificial variable is assigned at each region transition. It uses 2132the immediately enclosing region's DWARF procedure to compute the program 2133location for each lane assuming they are divergent, and then modifies the result 2134by inserting the current program location for each lane that the ``EXEC`` mask 2135indicates is active. 2136 2137By having separate DWARF procedures for each region, they can be reused to 2138define the value for any nested region. This reduces the total size of the DWARF 2139operation expressions. 2140 2141The following provides an example using pseudo LLVM MIR. 2142 2143.. code:: 2144 :number-lines: 2145 2146 $lex_start: 2147 DEFINE_DWARF %__uint_64 = DW_TAG_base_type[ 2148 DW_AT_name = "__uint64"; 2149 DW_AT_byte_size = 8; 2150 DW_AT_encoding = DW_ATE_unsigned; 2151 ]; 2152 DEFINE_DWARF %__active_lane_pc = DW_TAG_dwarf_procedure[ 2153 DW_AT_name = "__active_lane_pc"; 2154 DW_AT_location = [ 2155 DW_OP_regx PC; 2156 DW_OP_LLVM_extend 64, 64; 2157 DW_OP_regval_type EXEC, %uint_64; 2158 DW_OP_LLVM_select_bit_piece 64, 64; 2159 ]; 2160 ]; 2161 DEFINE_DWARF %__divergent_lane_pc = DW_TAG_dwarf_procedure[ 2162 DW_AT_name = "__divergent_lane_pc"; 2163 DW_AT_location = [ 2164 DW_OP_LLVM_undefined; 2165 DW_OP_LLVM_extend 64, 64; 2166 ]; 2167 ]; 2168 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 2169 DW_OP_call_ref %__divergent_lane_pc; 2170 DW_OP_call_ref %__active_lane_pc; 2171 ]; 2172 a; 2173 %1 = EXEC; 2174 DBG_VALUE %1, $noreg, %__lex_1_save_exec; 2175 %2 = c1; 2176 $lex_1_start: 2177 EXEC = %1 & %2; 2178 $lex_1_then: 2179 DEFINE_DWARF %__divergent_lane_pc_1_then = DW_TAG_dwarf_procedure[ 2180 DW_AT_name = "__divergent_lane_pc_1_then"; 2181 DW_AT_location = DIExpression[ 2182 DW_OP_call_ref %__divergent_lane_pc; 2183 DW_OP_addrx &lex_1_start; 2184 DW_OP_stack_value; 2185 DW_OP_LLVM_extend 64, 64; 2186 DW_OP_call_ref %__lex_1_save_exec; 2187 DW_OP_deref_type 64, %__uint_64; 2188 DW_OP_LLVM_select_bit_piece 64, 64; 2189 ]; 2190 ]; 2191 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 2192 DW_OP_call_ref %__divergent_lane_pc_1_then; 2193 DW_OP_call_ref %__active_lane_pc; 2194 ]; 2195 b; 2196 %3 = EXEC; 2197 DBG_VALUE %3, %__lex_1_1_save_exec; 2198 %4 = c2; 2199 $lex_1_1_start: 2200 EXEC = %3 & %4; 2201 $lex_1_1_then: 2202 DEFINE_DWARF %__divergent_lane_pc_1_1_then = DW_TAG_dwarf_procedure[ 2203 DW_AT_name = "__divergent_lane_pc_1_1_then"; 2204 DW_AT_location = DIExpression[ 2205 DW_OP_call_ref %__divergent_lane_pc_1_then; 2206 DW_OP_addrx &lex_1_1_start; 2207 DW_OP_stack_value; 2208 DW_OP_LLVM_extend 64, 64; 2209 DW_OP_call_ref %__lex_1_1_save_exec; 2210 DW_OP_deref_type 64, %__uint_64; 2211 DW_OP_LLVM_select_bit_piece 64, 64; 2212 ]; 2213 ]; 2214 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 2215 DW_OP_call_ref %__divergent_lane_pc_1_1_then; 2216 DW_OP_call_ref %__active_lane_pc; 2217 ]; 2218 c; 2219 EXEC = ~EXEC & %3; 2220 $lex_1_1_else: 2221 DEFINE_DWARF %__divergent_lane_pc_1_1_else = DW_TAG_dwarf_procedure[ 2222 DW_AT_name = "__divergent_lane_pc_1_1_else"; 2223 DW_AT_location = DIExpression[ 2224 DW_OP_call_ref %__divergent_lane_pc_1_then; 2225 DW_OP_addrx &lex_1_1_end; 2226 DW_OP_stack_value; 2227 DW_OP_LLVM_extend 64, 64; 2228 DW_OP_call_ref %__lex_1_1_save_exec; 2229 DW_OP_deref_type 64, %__uint_64; 2230 DW_OP_LLVM_select_bit_piece 64, 64; 2231 ]; 2232 ]; 2233 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 2234 DW_OP_call_ref %__divergent_lane_pc_1_1_else; 2235 DW_OP_call_ref %__active_lane_pc; 2236 ]; 2237 d; 2238 EXEC = %3; 2239 $lex_1_1_end: 2240 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 2241 DW_OP_call_ref %__divergent_lane_pc; 2242 DW_OP_call_ref %__active_lane_pc; 2243 ]; 2244 e; 2245 EXEC = ~EXEC & %1; 2246 $lex_1_else: 2247 DEFINE_DWARF %__divergent_lane_pc_1_else = DW_TAG_dwarf_procedure[ 2248 DW_AT_name = "__divergent_lane_pc_1_else"; 2249 DW_AT_location = DIExpression[ 2250 DW_OP_call_ref %__divergent_lane_pc; 2251 DW_OP_addrx &lex_1_end; 2252 DW_OP_stack_value; 2253 DW_OP_LLVM_extend 64, 64; 2254 DW_OP_call_ref %__lex_1_save_exec; 2255 DW_OP_deref_type 64, %__uint_64; 2256 DW_OP_LLVM_select_bit_piece 64, 64; 2257 ]; 2258 ]; 2259 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 2260 DW_OP_call_ref %__divergent_lane_pc_1_else; 2261 DW_OP_call_ref %__active_lane_pc; 2262 ]; 2263 f; 2264 EXEC = %1; 2265 $lex_1_end: 2266 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc DIExpression[ 2267 DW_OP_call_ref %__divergent_lane_pc; 2268 DW_OP_call_ref %__active_lane_pc; 2269 ]; 2270 g; 2271 $lex_end: 2272 2273The DWARF procedure ``%__active_lane_pc`` is used to update the lane pc elements 2274that are active, with the current program location. 2275 2276Artificial variables %__lex_1_save_exec and %__lex_1_1_save_exec are created for 2277the execution masks saved on entry to a region. Using the ``DBG_VALUE`` pseudo 2278instruction, location list entries will be created that describe where the 2279artificial variables are allocated at any given program location. The compiler 2280may allocate them to registers or spill them to memory. 2281 2282The DWARF procedures for each region use the values of the saved execution mask 2283artificial variables to only update the lanes that are active on entry to the 2284region. All other lanes retain the value of the enclosing region where they were 2285last active. If they were not active on entry to the subprogram, then will have 2286the undefined location description. 2287 2288Other structured control flow regions can be handled similarly. For example, 2289loops would set the divergent program location for the region at the end of the 2290loop. Any lanes active will be in the loop, and any lanes not active must have 2291exited the loop. 2292 2293An ``IF/THEN/ELSEIF/ELSEIF/...`` region can be treated as a nest of 2294``IF/THEN/ELSE`` regions. 2295 2296The DWARF procedures can use the active lane artificial variable described in 2297:ref:`amdgpu-dwarf-amdgpu-dw-at-llvm-active-lane` rather than the actual 2298``EXEC`` mask in order to support whole or quad wavefront mode. 2299 2300.. _amdgpu-dwarf-amdgpu-dw-at-llvm-active-lane: 2301 2302``DW_AT_LLVM_active_lane`` 2303~~~~~~~~~~~~~~~~~~~~~~~~~~ 2304 2305The ``DW_AT_LLVM_active_lane`` attribute on a subprogram debugger information 2306entry is used to specify the lanes that are conceptually active for a SIMT 2307thread. 2308 2309The execution mask may be modified to implement whole or quad wavefront mode 2310operations. For example, all lanes may need to temporarily be made active to 2311execute a whole wavefront operation. Such regions would save the ``EXEC`` mask, 2312update it to enable the necessary lanes, perform the operations, and then 2313restore the ``EXEC`` mask from the saved value. While executing the whole 2314wavefront region, the conceptual execution mask is the saved value, not the 2315``EXEC`` value. 2316 2317This is handled by defining an artificial variable for the active lane mask. The 2318active lane mask artificial variable would be the actual ``EXEC`` mask for 2319normal regions, and the saved execution mask for regions where the mask is 2320temporarily updated. The location list expression created for this artificial 2321variable is used to define the value of the ``DW_AT_LLVM_active_lane`` 2322attribute. 2323 2324``DW_AT_LLVM_augmentation`` 2325~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2326 2327For AMDGPU, the ``DW_AT_LLVM_augmentation`` attribute of a compilation unit 2328debugger information entry has the following value for the augmentation string: 2329 2330:: 2331 2332 [amdgpu:v0.0] 2333 2334The "vX.Y" specifies the major X and minor Y version number of the AMDGPU 2335extensions used in the DWARF of the compilation unit. The version number 2336conforms to [SEMVER]_. 2337 2338Call Frame Information 2339---------------------- 2340 2341DWARF Call Frame Information (CFI) describes how a consumer can virtually 2342*unwind* call frames in a running process or core dump. See DWARF Version 5 2343section 6.4 and :ref:`amdgpu-dwarf-call-frame-information`. 2344 2345For AMDGPU, the Common Information Entry (CIE) fields have the following values: 2346 23471. ``augmentation`` string contains the following null-terminated UTF-8 string: 2348 2349 :: 2350 2351 [amd:v0.0] 2352 2353 The ``vX.Y`` specifies the major X and minor Y version number of the AMDGPU 2354 extensions used in this CIE or to the FDEs that use it. The version number 2355 conforms to [SEMVER]_. 2356 23572. ``address_size`` for the ``Global`` address space is defined in 2358 :ref:`amdgpu-dwarf-address-space-identifier`. 2359 23603. ``segment_selector_size`` is 0 as AMDGPU does not use a segment selector. 2361 23624. ``code_alignment_factor`` is 4 bytes. 2363 2364 .. TODO:: 2365 2366 Add to :ref:`amdgpu-processor-table` table. 2367 23685. ``data_alignment_factor`` is 4 bytes. 2369 2370 .. TODO:: 2371 2372 Add to :ref:`amdgpu-processor-table` table. 2373 23746. ``return_address_register`` is ``PC_32`` for 32-bit processes and ``PC_64`` 2375 for 64-bit processes defined in :ref:`amdgpu-dwarf-register-identifier`. 2376 23777. ``initial_instructions`` Since a subprogram X with fewer registers can be 2378 called from subprogram Y that has more allocated, X will not change any of 2379 the extra registers as it cannot access them. Therefore, the default rule 2380 for all columns is ``same value``. 2381 2382For AMDGPU the register number follows the numbering defined in 2383:ref:`amdgpu-dwarf-register-identifier`. 2384 2385For AMDGPU the instructions are variable size. A consumer can subtract 1 from 2386the return address to get the address of a byte within the call site 2387instructions. See DWARF Version 5 section 6.4.4. 2388 2389Accelerated Access 2390------------------ 2391 2392See DWARF Version 5 section 6.1. 2393 2394Lookup By Name Section Header 2395~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2396 2397See DWARF Version 5 section 6.1.1.4.1 and :ref:`amdgpu-dwarf-lookup-by-name`. 2398 2399For AMDGPU the lookup by name section header table: 2400 2401``augmentation_string_size`` (uword) 2402 2403 Set to the length of the ``augmentation_string`` value which is always a 2404 multiple of 4. 2405 2406``augmentation_string`` (sequence of UTF-8 characters) 2407 2408 Contains the following UTF-8 string null padded to a multiple of 4 bytes: 2409 2410 :: 2411 2412 [amdgpu:v0.0] 2413 2414 The "vX.Y" specifies the major X and minor Y version number of the AMDGPU 2415 extensions used in the DWARF of this index. The version number conforms to 2416 [SEMVER]_. 2417 2418 .. note:: 2419 2420 This is different to the DWARF Version 5 definition that requires the first 2421 4 characters to be the vendor ID. But this is consistent with the other 2422 augmentation strings and does allow multiple vendor contributions. However, 2423 backwards compatibility may be more desirable. 2424 2425Lookup By Address Section Header 2426~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2427 2428See DWARF Version 5 section 6.1.2. 2429 2430For AMDGPU the lookup by address section header table: 2431 2432``address_size`` (ubyte) 2433 2434 Match the address size for the ``Global`` address space defined in 2435 :ref:`amdgpu-dwarf-address-space-identifier`. 2436 2437``segment_selector_size`` (ubyte) 2438 2439 AMDGPU does not use a segment selector so this is 0. The entries in the 2440 ``.debug_aranges`` do not have a segment selector. 2441 2442Line Number Information 2443----------------------- 2444 2445See DWARF Version 5 section 6.2 and :ref:`amdgpu-dwarf-line-number-information`. 2446 2447AMDGPU does not use the ``isa`` state machine registers and always sets it to 0. 2448The instruction set must be obtained from the ELF file header ``e_flags`` field 2449in the ``EF_AMDGPU_MACH`` bit position (see :ref:`ELF Header 2450<amdgpu-elf-header>`). See DWARF Version 5 section 6.2.2. 2451 2452.. TODO:: 2453 2454 Should the ``isa`` state machine register be used to indicate if the code is 2455 in wavefront32 or wavefront64 mode? Or used to specify the architecture ISA? 2456 2457For AMDGPU the line number program header fields have the following values (see 2458DWARF Version 5 section 6.2.4): 2459 2460``address_size`` (ubyte) 2461 Matches the address size for the ``Global`` address space defined in 2462 :ref:`amdgpu-dwarf-address-space-identifier`. 2463 2464``segment_selector_size`` (ubyte) 2465 AMDGPU does not use a segment selector so this is 0. 2466 2467``minimum_instruction_length`` (ubyte) 2468 For GFX9-GFX10 this is 4. 2469 2470``maximum_operations_per_instruction`` (ubyte) 2471 For GFX9-GFX10 this is 1. 2472 2473Source text for online-compiled programs (for example, those compiled by the 2474OpenCL language runtime) may be embedded into the DWARF Version 5 line table. 2475See DWARF Version 5 section 6.2.4.1 which is updated by *DWARF Extensions For 2476Heterogeneous Debugging* section :ref:`DW_LNCT_LLVM_source 2477<amdgpu-dwarf-line-number-information-dw-lnct-llvm-source>`. 2478 2479The Clang option used to control source embedding in AMDGPU is defined in 2480:ref:`amdgpu-clang-debug-options-table`. 2481 2482 .. table:: AMDGPU Clang Debug Options 2483 :name: amdgpu-clang-debug-options-table 2484 2485 ==================== ================================================== 2486 Debug Flag Description 2487 ==================== ================================================== 2488 -g[no-]embed-source Enable/disable embedding source text in DWARF 2489 debug sections. Useful for environments where 2490 source cannot be written to disk, such as 2491 when performing online compilation. 2492 ==================== ================================================== 2493 2494For example: 2495 2496``-gembed-source`` 2497 Enable the embedded source. 2498 2499``-gno-embed-source`` 2500 Disable the embedded source. 2501 250232-Bit and 64-Bit DWARF Formats 2503------------------------------- 2504 2505See DWARF Version 5 section 7.4 and 2506:ref:`amdgpu-dwarf-32-bit-and-64-bit-dwarf-formats`. 2507 2508For AMDGPU: 2509 2510* For the ``amdgcn`` target architecture only the 64-bit process address space 2511 is supported. 2512 2513* The producer can generate either 32-bit or 64-bit DWARF format. LLVM generates 2514 the 32-bit DWARF format. 2515 2516Unit Headers 2517------------ 2518 2519For AMDGPU the following values apply for each of the unit headers described in 2520DWARF Version 5 sections 7.5.1.1, 7.5.1.2, and 7.5.1.3: 2521 2522``address_size`` (ubyte) 2523 Matches the address size for the ``Global`` address space defined in 2524 :ref:`amdgpu-dwarf-address-space-identifier`. 2525 2526.. _amdgpu-code-conventions: 2527 2528Code Conventions 2529================ 2530 2531This section provides code conventions used for each supported target triple OS 2532(see :ref:`amdgpu-target-triples`). 2533 2534AMDHSA 2535------ 2536 2537This section provides code conventions used when the target triple OS is 2538``amdhsa`` (see :ref:`amdgpu-target-triples`). 2539 2540.. _amdgpu-amdhsa-code-object-metadata: 2541 2542Code Object Metadata 2543~~~~~~~~~~~~~~~~~~~~ 2544 2545The code object metadata specifies extensible metadata associated with the code 2546objects executed on HSA [HSA]_ compatible runtimes (see :ref:`amdgpu-os`). The 2547encoding and semantics of this metadata depends on the code object version; see 2548:ref:`amdgpu-amdhsa-code-object-metadata-v2`, 2549:ref:`amdgpu-amdhsa-code-object-metadata-v3`, and 2550:ref:`amdgpu-amdhsa-code-object-metadata-v4`. 2551 2552Code object metadata is specified in a note record (see 2553:ref:`amdgpu-note-records`) and is required when the target triple OS is 2554``amdhsa`` (see :ref:`amdgpu-target-triples`). It must contain the minimum 2555information necessary to support the HSA compatible runtime kernel queries. For 2556example, the segment sizes needed in a dispatch packet. In addition, a 2557high-level language runtime may require other information to be included. For 2558example, the AMD OpenCL runtime records kernel argument information. 2559 2560.. _amdgpu-amdhsa-code-object-metadata-v2: 2561 2562Code Object V2 Metadata 2563+++++++++++++++++++++++ 2564 2565.. warning:: 2566 Code object V2 is not the default code object version emitted by this version 2567 of LLVM. 2568 2569Code object V2 metadata is specified by the ``NT_AMD_HSA_METADATA`` note record 2570(see :ref:`amdgpu-note-records-v2`). 2571 2572The metadata is specified as a YAML formatted string (see [YAML]_ and 2573:doc:`YamlIO`). 2574 2575.. TODO:: 2576 2577 Is the string null terminated? It probably should not if YAML allows it to 2578 contain null characters, otherwise it should be. 2579 2580The metadata is represented as a single YAML document comprised of the mapping 2581defined in table :ref:`amdgpu-amdhsa-code-object-metadata-map-v2-table` and 2582referenced tables. 2583 2584For boolean values, the string values of ``false`` and ``true`` are used for 2585false and true respectively. 2586 2587Additional information can be added to the mappings. To avoid conflicts, any 2588non-AMD key names should be prefixed by "*vendor-name*.". 2589 2590 .. table:: AMDHSA Code Object V2 Metadata Map 2591 :name: amdgpu-amdhsa-code-object-metadata-map-v2-table 2592 2593 ========== ============== ========= ======================================= 2594 String Key Value Type Required? Description 2595 ========== ============== ========= ======================================= 2596 "Version" sequence of Required - The first integer is the major 2597 2 integers version. Currently 1. 2598 - The second integer is the minor 2599 version. Currently 0. 2600 "Printf" sequence of Each string is encoded information 2601 strings about a printf function call. The 2602 encoded information is organized as 2603 fields separated by colon (':'): 2604 2605 ``ID:N:S[0]:S[1]:...:S[N-1]:FormatString`` 2606 2607 where: 2608 2609 ``ID`` 2610 A 32-bit integer as a unique id for 2611 each printf function call 2612 2613 ``N`` 2614 A 32-bit integer equal to the number 2615 of arguments of printf function call 2616 minus 1 2617 2618 ``S[i]`` (where i = 0, 1, ... , N-1) 2619 32-bit integers for the size in bytes 2620 of the i-th FormatString argument of 2621 the printf function call 2622 2623 FormatString 2624 The format string passed to the 2625 printf function call. 2626 "Kernels" sequence of Required Sequence of the mappings for each 2627 mapping kernel in the code object. See 2628 :ref:`amdgpu-amdhsa-code-object-kernel-metadata-map-v2-table` 2629 for the definition of the mapping. 2630 ========== ============== ========= ======================================= 2631 2632.. 2633 2634 .. table:: AMDHSA Code Object V2 Kernel Metadata Map 2635 :name: amdgpu-amdhsa-code-object-kernel-metadata-map-v2-table 2636 2637 ================= ============== ========= ================================ 2638 String Key Value Type Required? Description 2639 ================= ============== ========= ================================ 2640 "Name" string Required Source name of the kernel. 2641 "SymbolName" string Required Name of the kernel 2642 descriptor ELF symbol. 2643 "Language" string Source language of the kernel. 2644 Values include: 2645 2646 - "OpenCL C" 2647 - "OpenCL C++" 2648 - "HCC" 2649 - "OpenMP" 2650 2651 "LanguageVersion" sequence of - The first integer is the major 2652 2 integers version. 2653 - The second integer is the 2654 minor version. 2655 "Attrs" mapping Mapping of kernel attributes. 2656 See 2657 :ref:`amdgpu-amdhsa-code-object-kernel-attribute-metadata-map-v2-table` 2658 for the mapping definition. 2659 "Args" sequence of Sequence of mappings of the 2660 mapping kernel arguments. See 2661 :ref:`amdgpu-amdhsa-code-object-kernel-argument-metadata-map-v2-table` 2662 for the definition of the mapping. 2663 "CodeProps" mapping Mapping of properties related to 2664 the kernel code. See 2665 :ref:`amdgpu-amdhsa-code-object-kernel-code-properties-metadata-map-v2-table` 2666 for the mapping definition. 2667 ================= ============== ========= ================================ 2668 2669.. 2670 2671 .. table:: AMDHSA Code Object V2 Kernel Attribute Metadata Map 2672 :name: amdgpu-amdhsa-code-object-kernel-attribute-metadata-map-v2-table 2673 2674 =================== ============== ========= ============================== 2675 String Key Value Type Required? Description 2676 =================== ============== ========= ============================== 2677 "ReqdWorkGroupSize" sequence of If not 0, 0, 0 then all values 2678 3 integers must be >=1 and the dispatch 2679 work-group size X, Y, Z must 2680 correspond to the specified 2681 values. Defaults to 0, 0, 0. 2682 2683 Corresponds to the OpenCL 2684 ``reqd_work_group_size`` 2685 attribute. 2686 "WorkGroupSizeHint" sequence of The dispatch work-group size 2687 3 integers X, Y, Z is likely to be the 2688 specified values. 2689 2690 Corresponds to the OpenCL 2691 ``work_group_size_hint`` 2692 attribute. 2693 "VecTypeHint" string The name of a scalar or vector 2694 type. 2695 2696 Corresponds to the OpenCL 2697 ``vec_type_hint`` attribute. 2698 2699 "RuntimeHandle" string The external symbol name 2700 associated with a kernel. 2701 OpenCL runtime allocates a 2702 global buffer for the symbol 2703 and saves the kernel's address 2704 to it, which is used for 2705 device side enqueueing. Only 2706 available for device side 2707 enqueued kernels. 2708 =================== ============== ========= ============================== 2709 2710.. 2711 2712 .. table:: AMDHSA Code Object V2 Kernel Argument Metadata Map 2713 :name: amdgpu-amdhsa-code-object-kernel-argument-metadata-map-v2-table 2714 2715 ================= ============== ========= ================================ 2716 String Key Value Type Required? Description 2717 ================= ============== ========= ================================ 2718 "Name" string Kernel argument name. 2719 "TypeName" string Kernel argument type name. 2720 "Size" integer Required Kernel argument size in bytes. 2721 "Align" integer Required Kernel argument alignment in 2722 bytes. Must be a power of two. 2723 "ValueKind" string Required Kernel argument kind that 2724 specifies how to set up the 2725 corresponding argument. 2726 Values include: 2727 2728 "ByValue" 2729 The argument is copied 2730 directly into the kernarg. 2731 2732 "GlobalBuffer" 2733 A global address space pointer 2734 to the buffer data is passed 2735 in the kernarg. 2736 2737 "DynamicSharedPointer" 2738 A group address space pointer 2739 to dynamically allocated LDS 2740 is passed in the kernarg. 2741 2742 "Sampler" 2743 A global address space 2744 pointer to a S# is passed in 2745 the kernarg. 2746 2747 "Image" 2748 A global address space 2749 pointer to a T# is passed in 2750 the kernarg. 2751 2752 "Pipe" 2753 A global address space pointer 2754 to an OpenCL pipe is passed in 2755 the kernarg. 2756 2757 "Queue" 2758 A global address space pointer 2759 to an OpenCL device enqueue 2760 queue is passed in the 2761 kernarg. 2762 2763 "HiddenGlobalOffsetX" 2764 The OpenCL grid dispatch 2765 global offset for the X 2766 dimension is passed in the 2767 kernarg. 2768 2769 "HiddenGlobalOffsetY" 2770 The OpenCL grid dispatch 2771 global offset for the Y 2772 dimension is passed in the 2773 kernarg. 2774 2775 "HiddenGlobalOffsetZ" 2776 The OpenCL grid dispatch 2777 global offset for the Z 2778 dimension is passed in the 2779 kernarg. 2780 2781 "HiddenNone" 2782 An argument that is not used 2783 by the kernel. Space needs to 2784 be left for it, but it does 2785 not need to be set up. 2786 2787 "HiddenPrintfBuffer" 2788 A global address space pointer 2789 to the runtime printf buffer 2790 is passed in kernarg. 2791 2792 "HiddenHostcallBuffer" 2793 A global address space pointer 2794 to the runtime hostcall buffer 2795 is passed in kernarg. 2796 2797 "HiddenDefaultQueue" 2798 A global address space pointer 2799 to the OpenCL device enqueue 2800 queue that should be used by 2801 the kernel by default is 2802 passed in the kernarg. 2803 2804 "HiddenCompletionAction" 2805 A global address space pointer 2806 to help link enqueued kernels into 2807 the ancestor tree for determining 2808 when the parent kernel has finished. 2809 2810 "HiddenMultiGridSyncArg" 2811 A global address space pointer for 2812 multi-grid synchronization is 2813 passed in the kernarg. 2814 2815 "ValueType" string Unused and deprecated. This should no longer 2816 be emitted, but is accepted for compatibility. 2817 2818 2819 "PointeeAlign" integer Alignment in bytes of pointee 2820 type for pointer type kernel 2821 argument. Must be a power 2822 of 2. Only present if 2823 "ValueKind" is 2824 "DynamicSharedPointer". 2825 "AddrSpaceQual" string Kernel argument address space 2826 qualifier. Only present if 2827 "ValueKind" is "GlobalBuffer" or 2828 "DynamicSharedPointer". Values 2829 are: 2830 2831 - "Private" 2832 - "Global" 2833 - "Constant" 2834 - "Local" 2835 - "Generic" 2836 - "Region" 2837 2838 .. TODO:: 2839 2840 Is GlobalBuffer only Global 2841 or Constant? Is 2842 DynamicSharedPointer always 2843 Local? Can HCC allow Generic? 2844 How can Private or Region 2845 ever happen? 2846 2847 "AccQual" string Kernel argument access 2848 qualifier. Only present if 2849 "ValueKind" is "Image" or 2850 "Pipe". Values 2851 are: 2852 2853 - "ReadOnly" 2854 - "WriteOnly" 2855 - "ReadWrite" 2856 2857 .. TODO:: 2858 2859 Does this apply to 2860 GlobalBuffer? 2861 2862 "ActualAccQual" string The actual memory accesses 2863 performed by the kernel on the 2864 kernel argument. Only present if 2865 "ValueKind" is "GlobalBuffer", 2866 "Image", or "Pipe". This may be 2867 more restrictive than indicated 2868 by "AccQual" to reflect what the 2869 kernel actual does. If not 2870 present then the runtime must 2871 assume what is implied by 2872 "AccQual" and "IsConst". Values 2873 are: 2874 2875 - "ReadOnly" 2876 - "WriteOnly" 2877 - "ReadWrite" 2878 2879 "IsConst" boolean Indicates if the kernel argument 2880 is const qualified. Only present 2881 if "ValueKind" is 2882 "GlobalBuffer". 2883 2884 "IsRestrict" boolean Indicates if the kernel argument 2885 is restrict qualified. Only 2886 present if "ValueKind" is 2887 "GlobalBuffer". 2888 2889 "IsVolatile" boolean Indicates if the kernel argument 2890 is volatile qualified. Only 2891 present if "ValueKind" is 2892 "GlobalBuffer". 2893 2894 "IsPipe" boolean Indicates if the kernel argument 2895 is pipe qualified. Only present 2896 if "ValueKind" is "Pipe". 2897 2898 .. TODO:: 2899 2900 Can GlobalBuffer be pipe 2901 qualified? 2902 2903 ================= ============== ========= ================================ 2904 2905.. 2906 2907 .. table:: AMDHSA Code Object V2 Kernel Code Properties Metadata Map 2908 :name: amdgpu-amdhsa-code-object-kernel-code-properties-metadata-map-v2-table 2909 2910 ============================ ============== ========= ===================== 2911 String Key Value Type Required? Description 2912 ============================ ============== ========= ===================== 2913 "KernargSegmentSize" integer Required The size in bytes of 2914 the kernarg segment 2915 that holds the values 2916 of the arguments to 2917 the kernel. 2918 "GroupSegmentFixedSize" integer Required The amount of group 2919 segment memory 2920 required by a 2921 work-group in 2922 bytes. This does not 2923 include any 2924 dynamically allocated 2925 group segment memory 2926 that may be added 2927 when the kernel is 2928 dispatched. 2929 "PrivateSegmentFixedSize" integer Required The amount of fixed 2930 private address space 2931 memory required for a 2932 work-item in 2933 bytes. If the kernel 2934 uses a dynamic call 2935 stack then additional 2936 space must be added 2937 to this value for the 2938 call stack. 2939 "KernargSegmentAlign" integer Required The maximum byte 2940 alignment of 2941 arguments in the 2942 kernarg segment. Must 2943 be a power of 2. 2944 "WavefrontSize" integer Required Wavefront size. Must 2945 be a power of 2. 2946 "NumSGPRs" integer Required Number of scalar 2947 registers used by a 2948 wavefront for 2949 GFX6-GFX10. This 2950 includes the special 2951 SGPRs for VCC, Flat 2952 Scratch (GFX7-GFX10) 2953 and XNACK (for 2954 GFX8-GFX10). It does 2955 not include the 16 2956 SGPR added if a trap 2957 handler is 2958 enabled. It is not 2959 rounded up to the 2960 allocation 2961 granularity. 2962 "NumVGPRs" integer Required Number of vector 2963 registers used by 2964 each work-item for 2965 GFX6-GFX10 2966 "MaxFlatWorkGroupSize" integer Required Maximum flat 2967 work-group size 2968 supported by the 2969 kernel in work-items. 2970 Must be >=1 and 2971 consistent with 2972 ReqdWorkGroupSize if 2973 not 0, 0, 0. 2974 "NumSpilledSGPRs" integer Number of stores from 2975 a scalar register to 2976 a register allocator 2977 created spill 2978 location. 2979 "NumSpilledVGPRs" integer Number of stores from 2980 a vector register to 2981 a register allocator 2982 created spill 2983 location. 2984 ============================ ============== ========= ===================== 2985 2986.. _amdgpu-amdhsa-code-object-metadata-v3: 2987 2988Code Object V3 Metadata 2989+++++++++++++++++++++++ 2990 2991Code object V3 to V4 metadata is specified by the ``NT_AMDGPU_METADATA`` note 2992record (see :ref:`amdgpu-note-records-v3-v4`). 2993 2994The metadata is represented as Message Pack formatted binary data (see 2995[MsgPack]_). The top level is a Message Pack map that includes the 2996keys defined in table 2997:ref:`amdgpu-amdhsa-code-object-metadata-map-table-v3` and referenced 2998tables. 2999 3000Additional information can be added to the maps. To avoid conflicts, 3001any key names should be prefixed by "*vendor-name*." where 3002``vendor-name`` can be the name of the vendor and specific vendor 3003tool that generates the information. The prefix is abbreviated to 3004simply "." when it appears within a map that has been added by the 3005same *vendor-name*. 3006 3007 .. table:: AMDHSA Code Object V3 Metadata Map 3008 :name: amdgpu-amdhsa-code-object-metadata-map-table-v3 3009 3010 ================= ============== ========= ======================================= 3011 String Key Value Type Required? Description 3012 ================= ============== ========= ======================================= 3013 "amdhsa.version" sequence of Required - The first integer is the major 3014 2 integers version. Currently 1. 3015 - The second integer is the minor 3016 version. Currently 0. 3017 "amdhsa.printf" sequence of Each string is encoded information 3018 strings about a printf function call. The 3019 encoded information is organized as 3020 fields separated by colon (':'): 3021 3022 ``ID:N:S[0]:S[1]:...:S[N-1]:FormatString`` 3023 3024 where: 3025 3026 ``ID`` 3027 A 32-bit integer as a unique id for 3028 each printf function call 3029 3030 ``N`` 3031 A 32-bit integer equal to the number 3032 of arguments of printf function call 3033 minus 1 3034 3035 ``S[i]`` (where i = 0, 1, ... , N-1) 3036 32-bit integers for the size in bytes 3037 of the i-th FormatString argument of 3038 the printf function call 3039 3040 FormatString 3041 The format string passed to the 3042 printf function call. 3043 "amdhsa.kernels" sequence of Required Sequence of the maps for each 3044 map kernel in the code object. See 3045 :ref:`amdgpu-amdhsa-code-object-kernel-metadata-map-table-v3` 3046 for the definition of the keys included 3047 in that map. 3048 ================= ============== ========= ======================================= 3049 3050.. 3051 3052 .. table:: AMDHSA Code Object V3 Kernel Metadata Map 3053 :name: amdgpu-amdhsa-code-object-kernel-metadata-map-table-v3 3054 3055 =================================== ============== ========= ================================ 3056 String Key Value Type Required? Description 3057 =================================== ============== ========= ================================ 3058 ".name" string Required Source name of the kernel. 3059 ".symbol" string Required Name of the kernel 3060 descriptor ELF symbol. 3061 ".language" string Source language of the kernel. 3062 Values include: 3063 3064 - "OpenCL C" 3065 - "OpenCL C++" 3066 - "HCC" 3067 - "HIP" 3068 - "OpenMP" 3069 - "Assembler" 3070 3071 ".language_version" sequence of - The first integer is the major 3072 2 integers version. 3073 - The second integer is the 3074 minor version. 3075 ".args" sequence of Sequence of maps of the 3076 map kernel arguments. See 3077 :ref:`amdgpu-amdhsa-code-object-kernel-argument-metadata-map-table-v3` 3078 for the definition of the keys 3079 included in that map. 3080 ".reqd_workgroup_size" sequence of If not 0, 0, 0 then all values 3081 3 integers must be >=1 and the dispatch 3082 work-group size X, Y, Z must 3083 correspond to the specified 3084 values. Defaults to 0, 0, 0. 3085 3086 Corresponds to the OpenCL 3087 ``reqd_work_group_size`` 3088 attribute. 3089 ".workgroup_size_hint" sequence of The dispatch work-group size 3090 3 integers X, Y, Z is likely to be the 3091 specified values. 3092 3093 Corresponds to the OpenCL 3094 ``work_group_size_hint`` 3095 attribute. 3096 ".vec_type_hint" string The name of a scalar or vector 3097 type. 3098 3099 Corresponds to the OpenCL 3100 ``vec_type_hint`` attribute. 3101 3102 ".device_enqueue_symbol" string The external symbol name 3103 associated with a kernel. 3104 OpenCL runtime allocates a 3105 global buffer for the symbol 3106 and saves the kernel's address 3107 to it, which is used for 3108 device side enqueueing. Only 3109 available for device side 3110 enqueued kernels. 3111 ".kernarg_segment_size" integer Required The size in bytes of 3112 the kernarg segment 3113 that holds the values 3114 of the arguments to 3115 the kernel. 3116 ".group_segment_fixed_size" integer Required The amount of group 3117 segment memory 3118 required by a 3119 work-group in 3120 bytes. This does not 3121 include any 3122 dynamically allocated 3123 group segment memory 3124 that may be added 3125 when the kernel is 3126 dispatched. 3127 ".private_segment_fixed_size" integer Required The amount of fixed 3128 private address space 3129 memory required for a 3130 work-item in 3131 bytes. If the kernel 3132 uses a dynamic call 3133 stack then additional 3134 space must be added 3135 to this value for the 3136 call stack. 3137 ".kernarg_segment_align" integer Required The maximum byte 3138 alignment of 3139 arguments in the 3140 kernarg segment. Must 3141 be a power of 2. 3142 ".wavefront_size" integer Required Wavefront size. Must 3143 be a power of 2. 3144 ".sgpr_count" integer Required Number of scalar 3145 registers required by a 3146 wavefront for 3147 GFX6-GFX9. A register 3148 is required if it is 3149 used explicitly, or 3150 if a higher numbered 3151 register is used 3152 explicitly. This 3153 includes the special 3154 SGPRs for VCC, Flat 3155 Scratch (GFX7-GFX9) 3156 and XNACK (for 3157 GFX8-GFX9). It does 3158 not include the 16 3159 SGPR added if a trap 3160 handler is 3161 enabled. It is not 3162 rounded up to the 3163 allocation 3164 granularity. 3165 ".vgpr_count" integer Required Number of vector 3166 registers required by 3167 each work-item for 3168 GFX6-GFX9. A register 3169 is required if it is 3170 used explicitly, or 3171 if a higher numbered 3172 register is used 3173 explicitly. 3174 ".max_flat_workgroup_size" integer Required Maximum flat 3175 work-group size 3176 supported by the 3177 kernel in work-items. 3178 Must be >=1 and 3179 consistent with 3180 ReqdWorkGroupSize if 3181 not 0, 0, 0. 3182 ".sgpr_spill_count" integer Number of stores from 3183 a scalar register to 3184 a register allocator 3185 created spill 3186 location. 3187 ".vgpr_spill_count" integer Number of stores from 3188 a vector register to 3189 a register allocator 3190 created spill 3191 location. 3192 ".kind" string The kind of the kernel 3193 with the following 3194 values: 3195 3196 "normal" 3197 Regular kernels. 3198 3199 "init" 3200 These kernels must be 3201 invoked after loading 3202 the containing code 3203 object and must 3204 complete before any 3205 normal and fini 3206 kernels in the same 3207 code object are 3208 invoked. 3209 3210 "fini" 3211 These kernels must be 3212 invoked before 3213 unloading the 3214 containing code object 3215 and after all init and 3216 normal kernels in the 3217 same code object have 3218 been invoked and 3219 completed. 3220 3221 If omitted, "normal" is 3222 assumed. 3223 =================================== ============== ========= ================================ 3224 3225.. 3226 3227 .. table:: AMDHSA Code Object V3 Kernel Argument Metadata Map 3228 :name: amdgpu-amdhsa-code-object-kernel-argument-metadata-map-table-v3 3229 3230 ====================== ============== ========= ================================ 3231 String Key Value Type Required? Description 3232 ====================== ============== ========= ================================ 3233 ".name" string Kernel argument name. 3234 ".type_name" string Kernel argument type name. 3235 ".size" integer Required Kernel argument size in bytes. 3236 ".offset" integer Required Kernel argument offset in 3237 bytes. The offset must be a 3238 multiple of the alignment 3239 required by the argument. 3240 ".value_kind" string Required Kernel argument kind that 3241 specifies how to set up the 3242 corresponding argument. 3243 Values include: 3244 3245 "by_value" 3246 The argument is copied 3247 directly into the kernarg. 3248 3249 "global_buffer" 3250 A global address space pointer 3251 to the buffer data is passed 3252 in the kernarg. 3253 3254 "dynamic_shared_pointer" 3255 A group address space pointer 3256 to dynamically allocated LDS 3257 is passed in the kernarg. 3258 3259 "sampler" 3260 A global address space 3261 pointer to a S# is passed in 3262 the kernarg. 3263 3264 "image" 3265 A global address space 3266 pointer to a T# is passed in 3267 the kernarg. 3268 3269 "pipe" 3270 A global address space pointer 3271 to an OpenCL pipe is passed in 3272 the kernarg. 3273 3274 "queue" 3275 A global address space pointer 3276 to an OpenCL device enqueue 3277 queue is passed in the 3278 kernarg. 3279 3280 "hidden_global_offset_x" 3281 The OpenCL grid dispatch 3282 global offset for the X 3283 dimension is passed in the 3284 kernarg. 3285 3286 "hidden_global_offset_y" 3287 The OpenCL grid dispatch 3288 global offset for the Y 3289 dimension is passed in the 3290 kernarg. 3291 3292 "hidden_global_offset_z" 3293 The OpenCL grid dispatch 3294 global offset for the Z 3295 dimension is passed in the 3296 kernarg. 3297 3298 "hidden_none" 3299 An argument that is not used 3300 by the kernel. Space needs to 3301 be left for it, but it does 3302 not need to be set up. 3303 3304 "hidden_printf_buffer" 3305 A global address space pointer 3306 to the runtime printf buffer 3307 is passed in kernarg. 3308 3309 "hidden_hostcall_buffer" 3310 A global address space pointer 3311 to the runtime hostcall buffer 3312 is passed in kernarg. 3313 3314 "hidden_default_queue" 3315 A global address space pointer 3316 to the OpenCL device enqueue 3317 queue that should be used by 3318 the kernel by default is 3319 passed in the kernarg. 3320 3321 "hidden_completion_action" 3322 A global address space pointer 3323 to help link enqueued kernels into 3324 the ancestor tree for determining 3325 when the parent kernel has finished. 3326 3327 "hidden_multigrid_sync_arg" 3328 A global address space pointer for 3329 multi-grid synchronization is 3330 passed in the kernarg. 3331 3332 ".value_type" string Unused and deprecated. This should no longer 3333 be emitted, but is accepted for compatibility. 3334 3335 ".pointee_align" integer Alignment in bytes of pointee 3336 type for pointer type kernel 3337 argument. Must be a power 3338 of 2. Only present if 3339 ".value_kind" is 3340 "dynamic_shared_pointer". 3341 ".address_space" string Kernel argument address space 3342 qualifier. Only present if 3343 ".value_kind" is "global_buffer" or 3344 "dynamic_shared_pointer". Values 3345 are: 3346 3347 - "private" 3348 - "global" 3349 - "constant" 3350 - "local" 3351 - "generic" 3352 - "region" 3353 3354 .. TODO:: 3355 3356 Is "global_buffer" only "global" 3357 or "constant"? Is 3358 "dynamic_shared_pointer" always 3359 "local"? Can HCC allow "generic"? 3360 How can "private" or "region" 3361 ever happen? 3362 3363 ".access" string Kernel argument access 3364 qualifier. Only present if 3365 ".value_kind" is "image" or 3366 "pipe". Values 3367 are: 3368 3369 - "read_only" 3370 - "write_only" 3371 - "read_write" 3372 3373 .. TODO:: 3374 3375 Does this apply to 3376 "global_buffer"? 3377 3378 ".actual_access" string The actual memory accesses 3379 performed by the kernel on the 3380 kernel argument. Only present if 3381 ".value_kind" is "global_buffer", 3382 "image", or "pipe". This may be 3383 more restrictive than indicated 3384 by ".access" to reflect what the 3385 kernel actual does. If not 3386 present then the runtime must 3387 assume what is implied by 3388 ".access" and ".is_const" . Values 3389 are: 3390 3391 - "read_only" 3392 - "write_only" 3393 - "read_write" 3394 3395 ".is_const" boolean Indicates if the kernel argument 3396 is const qualified. Only present 3397 if ".value_kind" is 3398 "global_buffer". 3399 3400 ".is_restrict" boolean Indicates if the kernel argument 3401 is restrict qualified. Only 3402 present if ".value_kind" is 3403 "global_buffer". 3404 3405 ".is_volatile" boolean Indicates if the kernel argument 3406 is volatile qualified. Only 3407 present if ".value_kind" is 3408 "global_buffer". 3409 3410 ".is_pipe" boolean Indicates if the kernel argument 3411 is pipe qualified. Only present 3412 if ".value_kind" is "pipe". 3413 3414 .. TODO:: 3415 3416 Can "global_buffer" be pipe 3417 qualified? 3418 3419 ====================== ============== ========= ================================ 3420 3421.. _amdgpu-amdhsa-code-object-metadata-v4: 3422 3423Code Object V4 Metadata 3424+++++++++++++++++++++++ 3425 3426.. warning:: 3427 Code object V4 is not the default code object version emitted by this version 3428 of LLVM. 3429 3430Code object V4 metadata is the same as 3431:ref:`amdgpu-amdhsa-code-object-metadata-v3` with the changes and additions 3432defined in table :ref:`amdgpu-amdhsa-code-object-metadata-map-table-v3`. 3433 3434 .. table:: AMDHSA Code Object V4 Metadata Map Changes from :ref:`amdgpu-amdhsa-code-object-metadata-v3` 3435 :name: amdgpu-amdhsa-code-object-metadata-map-table-v4 3436 3437 ================= ============== ========= ======================================= 3438 String Key Value Type Required? Description 3439 ================= ============== ========= ======================================= 3440 "amdhsa.version" sequence of Required - The first integer is the major 3441 2 integers version. Currently 1. 3442 - The second integer is the minor 3443 version. Currently 1. 3444 "amdhsa.target" string Required The target name of the code using the syntax: 3445 3446 .. code:: 3447 3448 <target-triple> [ "-" <target-id> ] 3449 3450 A canonical target ID must be 3451 used. See :ref:`amdgpu-target-triples` 3452 and :ref:`amdgpu-target-id`. 3453 ================= ============== ========= ======================================= 3454 3455.. 3456 3457Kernel Dispatch 3458~~~~~~~~~~~~~~~ 3459 3460The HSA architected queuing language (AQL) defines a user space memory interface 3461that can be used to control the dispatch of kernels, in an agent independent 3462way. An agent can have zero or more AQL queues created for it using an HSA 3463compatible runtime (see :ref:`amdgpu-os`), in which AQL packets (all of which 3464are 64 bytes) can be placed. See the *HSA Platform System Architecture 3465Specification* [HSA]_ for the AQL queue mechanics and packet layouts. 3466 3467The packet processor of a kernel agent is responsible for detecting and 3468dispatching HSA kernels from the AQL queues associated with it. For AMD GPUs the 3469packet processor is implemented by the hardware command processor (CP), 3470asynchronous dispatch controller (ADC) and shader processor input controller 3471(SPI). 3472 3473An HSA compatible runtime can be used to allocate an AQL queue object. It uses 3474the kernel mode driver to initialize and register the AQL queue with CP. 3475 3476To dispatch a kernel the following actions are performed. This can occur in the 3477CPU host program, or from an HSA kernel executing on a GPU. 3478 34791. A pointer to an AQL queue for the kernel agent on which the kernel is to be 3480 executed is obtained. 34812. A pointer to the kernel descriptor (see 3482 :ref:`amdgpu-amdhsa-kernel-descriptor`) of the kernel to execute is obtained. 3483 It must be for a kernel that is contained in a code object that that was 3484 loaded by an HSA compatible runtime on the kernel agent with which the AQL 3485 queue is associated. 34863. Space is allocated for the kernel arguments using the HSA compatible runtime 3487 allocator for a memory region with the kernarg property for the kernel agent 3488 that will execute the kernel. It must be at least 16-byte aligned. 34894. Kernel argument values are assigned to the kernel argument memory 3490 allocation. The layout is defined in the *HSA Programmer's Language 3491 Reference* [HSA]_. For AMDGPU the kernel execution directly accesses the 3492 kernel argument memory in the same way constant memory is accessed. (Note 3493 that the HSA specification allows an implementation to copy the kernel 3494 argument contents to another location that is accessed by the kernel.) 34955. An AQL kernel dispatch packet is created on the AQL queue. The HSA compatible 3496 runtime api uses 64-bit atomic operations to reserve space in the AQL queue 3497 for the packet. The packet must be set up, and the final write must use an 3498 atomic store release to set the packet kind to ensure the packet contents are 3499 visible to the kernel agent. AQL defines a doorbell signal mechanism to 3500 notify the kernel agent that the AQL queue has been updated. These rules, and 3501 the layout of the AQL queue and kernel dispatch packet is defined in the *HSA 3502 System Architecture Specification* [HSA]_. 35036. A kernel dispatch packet includes information about the actual dispatch, 3504 such as grid and work-group size, together with information from the code 3505 object about the kernel, such as segment sizes. The HSA compatible runtime 3506 queries on the kernel symbol can be used to obtain the code object values 3507 which are recorded in the :ref:`amdgpu-amdhsa-code-object-metadata`. 35087. CP executes micro-code and is responsible for detecting and setting up the 3509 GPU to execute the wavefronts of a kernel dispatch. 35108. CP ensures that when the a wavefront starts executing the kernel machine 3511 code, the scalar general purpose registers (SGPR) and vector general purpose 3512 registers (VGPR) are set up as required by the machine code. The required 3513 setup is defined in the :ref:`amdgpu-amdhsa-kernel-descriptor`. The initial 3514 register state is defined in 3515 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`. 35169. The prolog of the kernel machine code (see 3517 :ref:`amdgpu-amdhsa-kernel-prolog`) sets up the machine state as necessary 3518 before continuing executing the machine code that corresponds to the kernel. 351910. When the kernel dispatch has completed execution, CP signals the completion 3520 signal specified in the kernel dispatch packet if not 0. 3521 3522.. _amdgpu-amdhsa-memory-spaces: 3523 3524Memory Spaces 3525~~~~~~~~~~~~~ 3526 3527The memory space properties are: 3528 3529 .. table:: AMDHSA Memory Spaces 3530 :name: amdgpu-amdhsa-memory-spaces-table 3531 3532 ================= =========== ======== ======= ================== 3533 Memory Space Name HSA Segment Hardware Address NULL Value 3534 Name Name Size 3535 ================= =========== ======== ======= ================== 3536 Private private scratch 32 0x00000000 3537 Local group LDS 32 0xFFFFFFFF 3538 Global global global 64 0x0000000000000000 3539 Constant constant *same as 64 0x0000000000000000 3540 global* 3541 Generic flat flat 64 0x0000000000000000 3542 Region N/A GDS 32 *not implemented 3543 for AMDHSA* 3544 ================= =========== ======== ======= ================== 3545 3546The global and constant memory spaces both use global virtual addresses, which 3547are the same virtual address space used by the CPU. However, some virtual 3548addresses may only be accessible to the CPU, some only accessible by the GPU, 3549and some by both. 3550 3551Using the constant memory space indicates that the data will not change during 3552the execution of the kernel. This allows scalar read instructions to be 3553used. The vector and scalar L1 caches are invalidated of volatile data before 3554each kernel dispatch execution to allow constant memory to change values between 3555kernel dispatches. 3556 3557The local memory space uses the hardware Local Data Store (LDS) which is 3558automatically allocated when the hardware creates work-groups of wavefronts, and 3559freed when all the wavefronts of a work-group have terminated. The data store 3560(DS) instructions can be used to access it. 3561 3562The private memory space uses the hardware scratch memory support. If the kernel 3563uses scratch, then the hardware allocates memory that is accessed using 3564wavefront lane dword (4 byte) interleaving. The mapping used from private 3565address to physical address is: 3566 3567 ``wavefront-scratch-base + 3568 (private-address * wavefront-size * 4) + 3569 (wavefront-lane-id * 4)`` 3570 3571There are different ways that the wavefront scratch base address is determined 3572by a wavefront (see :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). This 3573memory can be accessed in an interleaved manner using buffer instruction with 3574the scratch buffer descriptor and per wavefront scratch offset, by the scratch 3575instructions, or by flat instructions. If each lane of a wavefront accesses the 3576same private address, the interleaving results in adjacent dwords being accessed 3577and hence requires fewer cache lines to be fetched. Multi-dword access is not 3578supported except by flat and scratch instructions in GFX9-GFX10. 3579 3580The generic address space uses the hardware flat address support available in 3581GFX7-GFX10. This uses two fixed ranges of virtual addresses (the private and 3582local apertures), that are outside the range of addressible global memory, to 3583map from a flat address to a private or local address. 3584 3585FLAT instructions can take a flat address and access global, private (scratch) 3586and group (LDS) memory depending in if the address is within one of the 3587aperture ranges. Flat access to scratch requires hardware aperture setup and 3588setup in the kernel prologue (see 3589:ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`). Flat access to LDS requires 3590hardware aperture setup and M0 (GFX7-GFX8) register setup (see 3591:ref:`amdgpu-amdhsa-kernel-prolog-m0`). 3592 3593To convert between a segment address and a flat address the base address of the 3594apertures address can be used. For GFX7-GFX8 these are available in the 3595:ref:`amdgpu-amdhsa-hsa-aql-queue` the address of which can be obtained with 3596Queue Ptr SGPR (see :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). For 3597GFX9-GFX10 the aperture base addresses are directly available as inline constant 3598registers ``SRC_SHARED_BASE/LIMIT`` and ``SRC_PRIVATE_BASE/LIMIT``. In 64 bit 3599address mode the aperture sizes are 2^32 bytes and the base is aligned to 2^32 3600which makes it easier to convert from flat to segment or segment to flat. 3601 3602Image and Samplers 3603~~~~~~~~~~~~~~~~~~ 3604 3605Image and sample handles created by an HSA compatible runtime (see 3606:ref:`amdgpu-os`) are 64-bit addresses of a hardware 32-byte V# and 48 byte S# 3607object respectively. In order to support the HSA ``query_sampler`` operations 3608two extra dwords are used to store the HSA BRIG enumeration values for the 3609queries that are not trivially deducible from the S# representation. 3610 3611HSA Signals 3612~~~~~~~~~~~ 3613 3614HSA signal handles created by an HSA compatible runtime (see :ref:`amdgpu-os`) 3615are 64-bit addresses of a structure allocated in memory accessible from both the 3616CPU and GPU. The structure is defined by the runtime and subject to change 3617between releases. For example, see [AMD-ROCm-github]_. 3618 3619.. _amdgpu-amdhsa-hsa-aql-queue: 3620 3621HSA AQL Queue 3622~~~~~~~~~~~~~ 3623 3624The HSA AQL queue structure is defined by an HSA compatible runtime (see 3625:ref:`amdgpu-os`) and subject to change between releases. For example, see 3626[AMD-ROCm-github]_. For some processors it contains fields needed to implement 3627certain language features such as the flat address aperture bases. It also 3628contains fields used by CP such as managing the allocation of scratch memory. 3629 3630.. _amdgpu-amdhsa-kernel-descriptor: 3631 3632Kernel Descriptor 3633~~~~~~~~~~~~~~~~~ 3634 3635A kernel descriptor consists of the information needed by CP to initiate the 3636execution of a kernel, including the entry point address of the machine code 3637that implements the kernel. 3638 3639Code Object V3 Kernel Descriptor 3640++++++++++++++++++++++++++++++++ 3641 3642CP microcode requires the Kernel descriptor to be allocated on 64-byte 3643alignment. 3644 3645The fields used by CP for code objects before V3 also match those specified in 3646:ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 3647 3648 .. table:: Code Object V3 Kernel Descriptor 3649 :name: amdgpu-amdhsa-kernel-descriptor-v3-table 3650 3651 ======= ======= =============================== ============================ 3652 Bits Size Field Name Description 3653 ======= ======= =============================== ============================ 3654 31:0 4 bytes GROUP_SEGMENT_FIXED_SIZE The amount of fixed local 3655 address space memory 3656 required for a work-group 3657 in bytes. This does not 3658 include any dynamically 3659 allocated local address 3660 space memory that may be 3661 added when the kernel is 3662 dispatched. 3663 63:32 4 bytes PRIVATE_SEGMENT_FIXED_SIZE The amount of fixed 3664 private address space 3665 memory required for a 3666 work-item in bytes. 3667 Additional space may need to 3668 be added to this value if 3669 the call stack has 3670 non-inlined function calls. 3671 95:64 4 bytes KERNARG_SIZE The size of the kernarg 3672 memory pointed to by the 3673 AQL dispatch packet. The 3674 kernarg memory is used to 3675 pass arguments to the 3676 kernel. 3677 3678 * If the kernarg pointer in 3679 the dispatch packet is NULL 3680 then there are no kernel 3681 arguments. 3682 * If the kernarg pointer in 3683 the dispatch packet is 3684 not NULL and this value 3685 is 0 then the kernarg 3686 memory size is 3687 unspecified. 3688 * If the kernarg pointer in 3689 the dispatch packet is 3690 not NULL and this value 3691 is not 0 then the value 3692 specifies the kernarg 3693 memory size in bytes. It 3694 is recommended to provide 3695 a value as it may be used 3696 by CP to optimize making 3697 the kernarg memory 3698 visible to the kernel 3699 code. 3700 3701 127:96 4 bytes Reserved, must be 0. 3702 191:128 8 bytes KERNEL_CODE_ENTRY_BYTE_OFFSET Byte offset (possibly 3703 negative) from base 3704 address of kernel 3705 descriptor to kernel's 3706 entry point instruction 3707 which must be 256 byte 3708 aligned. 3709 351:272 20 Reserved, must be 0. 3710 bytes 3711 383:352 4 bytes COMPUTE_PGM_RSRC3 GFX6-GFX9 3712 Reserved, must be 0. 3713 GFX90A 3714 Compute Shader (CS) 3715 program settings used by 3716 CP to set up 3717 ``COMPUTE_PGM_RSRC3`` 3718 configuration 3719 register. See 3720 :ref:`amdgpu-amdhsa-compute_pgm_rsrc3-gfx90a-table`. 3721 GFX10 3722 Compute Shader (CS) 3723 program settings used by 3724 CP to set up 3725 ``COMPUTE_PGM_RSRC3`` 3726 configuration 3727 register. See 3728 :ref:`amdgpu-amdhsa-compute_pgm_rsrc3-gfx10-table`. 3729 415:384 4 bytes COMPUTE_PGM_RSRC1 Compute Shader (CS) 3730 program settings used by 3731 CP to set up 3732 ``COMPUTE_PGM_RSRC1`` 3733 configuration 3734 register. See 3735 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 3736 447:416 4 bytes COMPUTE_PGM_RSRC2 Compute Shader (CS) 3737 program settings used by 3738 CP to set up 3739 ``COMPUTE_PGM_RSRC2`` 3740 configuration 3741 register. See 3742 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 3743 458:448 7 bits *See separate bits below.* Enable the setup of the 3744 SGPR user data registers 3745 (see 3746 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 3747 3748 The total number of SGPR 3749 user data registers 3750 requested must not exceed 3751 16 and match value in 3752 ``compute_pgm_rsrc2.user_sgpr.user_sgpr_count``. 3753 Any requests beyond 16 3754 will be ignored. 3755 >448 1 bit ENABLE_SGPR_PRIVATE_SEGMENT If the *Target Properties* 3756 _BUFFER column of 3757 :ref:`amdgpu-processor-table` 3758 specifies *Architected flat 3759 scratch* then not supported 3760 and must be 0, 3761 >449 1 bit ENABLE_SGPR_DISPATCH_PTR 3762 >450 1 bit ENABLE_SGPR_QUEUE_PTR 3763 >451 1 bit ENABLE_SGPR_KERNARG_SEGMENT_PTR 3764 >452 1 bit ENABLE_SGPR_DISPATCH_ID 3765 >453 1 bit ENABLE_SGPR_FLAT_SCRATCH_INIT If the *Target Properties* 3766 column of 3767 :ref:`amdgpu-processor-table` 3768 specifies *Architected flat 3769 scratch* then not supported 3770 and must be 0, 3771 >454 1 bit ENABLE_SGPR_PRIVATE_SEGMENT 3772 _SIZE 3773 457:455 3 bits Reserved, must be 0. 3774 458 1 bit ENABLE_WAVEFRONT_SIZE32 GFX6-GFX9 3775 Reserved, must be 0. 3776 GFX10 3777 - If 0 execute in 3778 wavefront size 64 mode. 3779 - If 1 execute in 3780 native wavefront size 3781 32 mode. 3782 463:459 1 bit Reserved, must be 0. 3783 464 1 bit RESERVED_464 Deprecated, must be 0. 3784 467:465 3 bits Reserved, must be 0. 3785 468 1 bit RESERVED_468 Deprecated, must be 0. 3786 469:471 3 bits Reserved, must be 0. 3787 511:472 5 bytes Reserved, must be 0. 3788 512 **Total size 64 bytes.** 3789 ======= ==================================================================== 3790 3791.. 3792 3793 .. table:: compute_pgm_rsrc1 for GFX6-GFX10 3794 :name: amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table 3795 3796 ======= ======= =============================== =========================================================================== 3797 Bits Size Field Name Description 3798 ======= ======= =============================== =========================================================================== 3799 5:0 6 bits GRANULATED_WORKITEM_VGPR_COUNT Number of vector register 3800 blocks used by each work-item; 3801 granularity is device 3802 specific: 3803 3804 GFX6-GFX9 3805 - vgprs_used 0..256 3806 - max(0, ceil(vgprs_used / 4) - 1) 3807 GFX90A 3808 - vgprs_used 0..512 3809 - vgprs_used = align(arch_vgprs, 4) 3810 + acc_vgprs 3811 - max(0, ceil(vgprs_used / 8) - 1) 3812 GFX10 (wavefront size 64) 3813 - max_vgpr 1..256 3814 - max(0, ceil(vgprs_used / 4) - 1) 3815 GFX10 (wavefront size 32) 3816 - max_vgpr 1..256 3817 - max(0, ceil(vgprs_used / 8) - 1) 3818 3819 Where vgprs_used is defined 3820 as the highest VGPR number 3821 explicitly referenced plus 3822 one. 3823 3824 Used by CP to set up 3825 ``COMPUTE_PGM_RSRC1.VGPRS``. 3826 3827 The 3828 :ref:`amdgpu-assembler` 3829 calculates this 3830 automatically for the 3831 selected processor from 3832 values provided to the 3833 `.amdhsa_kernel` directive 3834 by the 3835 `.amdhsa_next_free_vgpr` 3836 nested directive (see 3837 :ref:`amdhsa-kernel-directives-table`). 3838 9:6 4 bits GRANULATED_WAVEFRONT_SGPR_COUNT Number of scalar register 3839 blocks used by a wavefront; 3840 granularity is device 3841 specific: 3842 3843 GFX6-GFX8 3844 - sgprs_used 0..112 3845 - max(0, ceil(sgprs_used / 8) - 1) 3846 GFX9 3847 - sgprs_used 0..112 3848 - 2 * max(0, ceil(sgprs_used / 16) - 1) 3849 GFX10 3850 Reserved, must be 0. 3851 (128 SGPRs always 3852 allocated.) 3853 3854 Where sgprs_used is 3855 defined as the highest 3856 SGPR number explicitly 3857 referenced plus one, plus 3858 a target specific number 3859 of additional special 3860 SGPRs for VCC, 3861 FLAT_SCRATCH (GFX7+) and 3862 XNACK_MASK (GFX8+), and 3863 any additional 3864 target specific 3865 limitations. It does not 3866 include the 16 SGPRs added 3867 if a trap handler is 3868 enabled. 3869 3870 The target specific 3871 limitations and special 3872 SGPR layout are defined in 3873 the hardware 3874 documentation, which can 3875 be found in the 3876 :ref:`amdgpu-processors` 3877 table. 3878 3879 Used by CP to set up 3880 ``COMPUTE_PGM_RSRC1.SGPRS``. 3881 3882 The 3883 :ref:`amdgpu-assembler` 3884 calculates this 3885 automatically for the 3886 selected processor from 3887 values provided to the 3888 `.amdhsa_kernel` directive 3889 by the 3890 `.amdhsa_next_free_sgpr` 3891 and `.amdhsa_reserve_*` 3892 nested directives (see 3893 :ref:`amdhsa-kernel-directives-table`). 3894 11:10 2 bits PRIORITY Must be 0. 3895 3896 Start executing wavefront 3897 at the specified priority. 3898 3899 CP is responsible for 3900 filling in 3901 ``COMPUTE_PGM_RSRC1.PRIORITY``. 3902 13:12 2 bits FLOAT_ROUND_MODE_32 Wavefront starts execution 3903 with specified rounding 3904 mode for single (32 3905 bit) floating point 3906 precision floating point 3907 operations. 3908 3909 Floating point rounding 3910 mode values are defined in 3911 :ref:`amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table`. 3912 3913 Used by CP to set up 3914 ``COMPUTE_PGM_RSRC1.FLOAT_MODE``. 3915 15:14 2 bits FLOAT_ROUND_MODE_16_64 Wavefront starts execution 3916 with specified rounding 3917 denorm mode for half/double (16 3918 and 64-bit) floating point 3919 precision floating point 3920 operations. 3921 3922 Floating point rounding 3923 mode values are defined in 3924 :ref:`amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table`. 3925 3926 Used by CP to set up 3927 ``COMPUTE_PGM_RSRC1.FLOAT_MODE``. 3928 17:16 2 bits FLOAT_DENORM_MODE_32 Wavefront starts execution 3929 with specified denorm mode 3930 for single (32 3931 bit) floating point 3932 precision floating point 3933 operations. 3934 3935 Floating point denorm mode 3936 values are defined in 3937 :ref:`amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table`. 3938 3939 Used by CP to set up 3940 ``COMPUTE_PGM_RSRC1.FLOAT_MODE``. 3941 19:18 2 bits FLOAT_DENORM_MODE_16_64 Wavefront starts execution 3942 with specified denorm mode 3943 for half/double (16 3944 and 64-bit) floating point 3945 precision floating point 3946 operations. 3947 3948 Floating point denorm mode 3949 values are defined in 3950 :ref:`amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table`. 3951 3952 Used by CP to set up 3953 ``COMPUTE_PGM_RSRC1.FLOAT_MODE``. 3954 20 1 bit PRIV Must be 0. 3955 3956 Start executing wavefront 3957 in privilege trap handler 3958 mode. 3959 3960 CP is responsible for 3961 filling in 3962 ``COMPUTE_PGM_RSRC1.PRIV``. 3963 21 1 bit ENABLE_DX10_CLAMP Wavefront starts execution 3964 with DX10 clamp mode 3965 enabled. Used by the vector 3966 ALU to force DX10 style 3967 treatment of NaN's (when 3968 set, clamp NaN to zero, 3969 otherwise pass NaN 3970 through). 3971 3972 Used by CP to set up 3973 ``COMPUTE_PGM_RSRC1.DX10_CLAMP``. 3974 22 1 bit DEBUG_MODE Must be 0. 3975 3976 Start executing wavefront 3977 in single step mode. 3978 3979 CP is responsible for 3980 filling in 3981 ``COMPUTE_PGM_RSRC1.DEBUG_MODE``. 3982 23 1 bit ENABLE_IEEE_MODE Wavefront starts execution 3983 with IEEE mode 3984 enabled. Floating point 3985 opcodes that support 3986 exception flag gathering 3987 will quiet and propagate 3988 signaling-NaN inputs per 3989 IEEE 754-2008. Min_dx10 and 3990 max_dx10 become IEEE 3991 754-2008 compliant due to 3992 signaling-NaN propagation 3993 and quieting. 3994 3995 Used by CP to set up 3996 ``COMPUTE_PGM_RSRC1.IEEE_MODE``. 3997 24 1 bit BULKY Must be 0. 3998 3999 Only one work-group allowed 4000 to execute on a compute 4001 unit. 4002 4003 CP is responsible for 4004 filling in 4005 ``COMPUTE_PGM_RSRC1.BULKY``. 4006 25 1 bit CDBG_USER Must be 0. 4007 4008 Flag that can be used to 4009 control debugging code. 4010 4011 CP is responsible for 4012 filling in 4013 ``COMPUTE_PGM_RSRC1.CDBG_USER``. 4014 26 1 bit FP16_OVFL GFX6-GFX8 4015 Reserved, must be 0. 4016 GFX9-GFX10 4017 Wavefront starts execution 4018 with specified fp16 overflow 4019 mode. 4020 4021 - If 0, fp16 overflow generates 4022 +/-INF values. 4023 - If 1, fp16 overflow that is the 4024 result of an +/-INF input value 4025 or divide by 0 produces a +/-INF, 4026 otherwise clamps computed 4027 overflow to +/-MAX_FP16 as 4028 appropriate. 4029 4030 Used by CP to set up 4031 ``COMPUTE_PGM_RSRC1.FP16_OVFL``. 4032 28:27 2 bits Reserved, must be 0. 4033 29 1 bit WGP_MODE GFX6-GFX9 4034 Reserved, must be 0. 4035 GFX10 4036 - If 0 execute work-groups in 4037 CU wavefront execution mode. 4038 - If 1 execute work-groups on 4039 in WGP wavefront execution mode. 4040 4041 See :ref:`amdgpu-amdhsa-memory-model`. 4042 4043 Used by CP to set up 4044 ``COMPUTE_PGM_RSRC1.WGP_MODE``. 4045 30 1 bit MEM_ORDERED GFX6-GFX9 4046 Reserved, must be 0. 4047 GFX10 4048 Controls the behavior of the 4049 s_waitcnt's vmcnt and vscnt 4050 counters. 4051 4052 - If 0 vmcnt reports completion 4053 of load and atomic with return 4054 out of order with sample 4055 instructions, and the vscnt 4056 reports the completion of 4057 store and atomic without 4058 return in order. 4059 - If 1 vmcnt reports completion 4060 of load, atomic with return 4061 and sample instructions in 4062 order, and the vscnt reports 4063 the completion of store and 4064 atomic without return in order. 4065 4066 Used by CP to set up 4067 ``COMPUTE_PGM_RSRC1.MEM_ORDERED``. 4068 31 1 bit FWD_PROGRESS GFX6-GFX9 4069 Reserved, must be 0. 4070 GFX10 4071 - If 0 execute SIMD wavefronts 4072 using oldest first policy. 4073 - If 1 execute SIMD wavefronts to 4074 ensure wavefronts will make some 4075 forward progress. 4076 4077 Used by CP to set up 4078 ``COMPUTE_PGM_RSRC1.FWD_PROGRESS``. 4079 32 **Total size 4 bytes** 4080 ======= =================================================================================================================== 4081 4082.. 4083 4084 .. table:: compute_pgm_rsrc2 for GFX6-GFX10 4085 :name: amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table 4086 4087 ======= ======= =============================== =========================================================================== 4088 Bits Size Field Name Description 4089 ======= ======= =============================== =========================================================================== 4090 0 1 bit ENABLE_PRIVATE_SEGMENT * Enable the setup of the 4091 private segment. 4092 * If the *Target Properties* 4093 column of 4094 :ref:`amdgpu-processor-table` 4095 does not specify 4096 *Architected flat 4097 scratch* then enable the 4098 setup of the SGPR 4099 wavefront scratch offset 4100 system register (see 4101 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4102 * If the *Target Properties* 4103 column of 4104 :ref:`amdgpu-processor-table` 4105 specifies *Architected 4106 flat scratch* then enable 4107 the setup of the 4108 FLAT_SCRATCH register 4109 pair (see 4110 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4111 4112 Used by CP to set up 4113 ``COMPUTE_PGM_RSRC2.SCRATCH_EN``. 4114 5:1 5 bits USER_SGPR_COUNT The total number of SGPR 4115 user data registers 4116 requested. This number must 4117 match the number of user 4118 data registers enabled. 4119 4120 Used by CP to set up 4121 ``COMPUTE_PGM_RSRC2.USER_SGPR``. 4122 6 1 bit ENABLE_TRAP_HANDLER Must be 0. 4123 4124 This bit represents 4125 ``COMPUTE_PGM_RSRC2.TRAP_PRESENT``, 4126 which is set by the CP if 4127 the runtime has installed a 4128 trap handler. 4129 7 1 bit ENABLE_SGPR_WORKGROUP_ID_X Enable the setup of the 4130 system SGPR register for 4131 the work-group id in the X 4132 dimension (see 4133 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4134 4135 Used by CP to set up 4136 ``COMPUTE_PGM_RSRC2.TGID_X_EN``. 4137 8 1 bit ENABLE_SGPR_WORKGROUP_ID_Y Enable the setup of the 4138 system SGPR register for 4139 the work-group id in the Y 4140 dimension (see 4141 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4142 4143 Used by CP to set up 4144 ``COMPUTE_PGM_RSRC2.TGID_Y_EN``. 4145 9 1 bit ENABLE_SGPR_WORKGROUP_ID_Z Enable the setup of the 4146 system SGPR register for 4147 the work-group id in the Z 4148 dimension (see 4149 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4150 4151 Used by CP to set up 4152 ``COMPUTE_PGM_RSRC2.TGID_Z_EN``. 4153 10 1 bit ENABLE_SGPR_WORKGROUP_INFO Enable the setup of the 4154 system SGPR register for 4155 work-group information (see 4156 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4157 4158 Used by CP to set up 4159 ``COMPUTE_PGM_RSRC2.TGID_SIZE_EN``. 4160 12:11 2 bits ENABLE_VGPR_WORKITEM_ID Enable the setup of the 4161 VGPR system registers used 4162 for the work-item ID. 4163 :ref:`amdgpu-amdhsa-system-vgpr-work-item-id-enumeration-values-table` 4164 defines the values. 4165 4166 Used by CP to set up 4167 ``COMPUTE_PGM_RSRC2.TIDIG_CMP_CNT``. 4168 13 1 bit ENABLE_EXCEPTION_ADDRESS_WATCH Must be 0. 4169 4170 Wavefront starts execution 4171 with address watch 4172 exceptions enabled which 4173 are generated when L1 has 4174 witnessed a thread access 4175 an *address of 4176 interest*. 4177 4178 CP is responsible for 4179 filling in the address 4180 watch bit in 4181 ``COMPUTE_PGM_RSRC2.EXCP_EN_MSB`` 4182 according to what the 4183 runtime requests. 4184 14 1 bit ENABLE_EXCEPTION_MEMORY Must be 0. 4185 4186 Wavefront starts execution 4187 with memory violation 4188 exceptions exceptions 4189 enabled which are generated 4190 when a memory violation has 4191 occurred for this wavefront from 4192 L1 or LDS 4193 (write-to-read-only-memory, 4194 mis-aligned atomic, LDS 4195 address out of range, 4196 illegal address, etc.). 4197 4198 CP sets the memory 4199 violation bit in 4200 ``COMPUTE_PGM_RSRC2.EXCP_EN_MSB`` 4201 according to what the 4202 runtime requests. 4203 23:15 9 bits GRANULATED_LDS_SIZE Must be 0. 4204 4205 CP uses the rounded value 4206 from the dispatch packet, 4207 not this value, as the 4208 dispatch may contain 4209 dynamically allocated group 4210 segment memory. CP writes 4211 directly to 4212 ``COMPUTE_PGM_RSRC2.LDS_SIZE``. 4213 4214 Amount of group segment 4215 (LDS) to allocate for each 4216 work-group. Granularity is 4217 device specific: 4218 4219 GFX6 4220 roundup(lds-size / (64 * 4)) 4221 GFX7-GFX10 4222 roundup(lds-size / (128 * 4)) 4223 4224 24 1 bit ENABLE_EXCEPTION_IEEE_754_FP Wavefront starts execution 4225 _INVALID_OPERATION with specified exceptions 4226 enabled. 4227 4228 Used by CP to set up 4229 ``COMPUTE_PGM_RSRC2.EXCP_EN`` 4230 (set from bits 0..6). 4231 4232 IEEE 754 FP Invalid 4233 Operation 4234 25 1 bit ENABLE_EXCEPTION_FP_DENORMAL FP Denormal one or more 4235 _SOURCE input operands is a 4236 denormal number 4237 26 1 bit ENABLE_EXCEPTION_IEEE_754_FP IEEE 754 FP Division by 4238 _DIVISION_BY_ZERO Zero 4239 27 1 bit ENABLE_EXCEPTION_IEEE_754_FP IEEE 754 FP FP Overflow 4240 _OVERFLOW 4241 28 1 bit ENABLE_EXCEPTION_IEEE_754_FP IEEE 754 FP Underflow 4242 _UNDERFLOW 4243 29 1 bit ENABLE_EXCEPTION_IEEE_754_FP IEEE 754 FP Inexact 4244 _INEXACT 4245 30 1 bit ENABLE_EXCEPTION_INT_DIVIDE_BY Integer Division by Zero 4246 _ZERO (rcp_iflag_f32 instruction 4247 only) 4248 31 1 bit Reserved, must be 0. 4249 32 **Total size 4 bytes.** 4250 ======= =================================================================================================================== 4251 4252.. 4253 4254 .. table:: compute_pgm_rsrc3 for GFX90A 4255 :name: amdgpu-amdhsa-compute_pgm_rsrc3-gfx90a-table 4256 4257 ======= ======= =============================== =========================================================================== 4258 Bits Size Field Name Description 4259 ======= ======= =============================== =========================================================================== 4260 5:0 6 bits ACCUM_OFFSET Offset of a first AccVGPR in the unified register file. Granularity 4. 4261 Value 0-63. 0 - accum-offset = 4, 1 - accum-offset = 8, ..., 4262 63 - accum-offset = 256. 4263 6:15 10 Reserved, must be 0. 4264 bits 4265 16 1 bit TG_SPLIT - If 0 the waves of a work-group are 4266 launched in the same CU. 4267 - If 1 the waves of a work-group can be 4268 launched in different CUs. The waves 4269 cannot use S_BARRIER or LDS. 4270 17:31 15 Reserved, must be 0. 4271 bits 4272 32 **Total size 4 bytes.** 4273 ======= =================================================================================================================== 4274 4275.. 4276 4277 .. table:: compute_pgm_rsrc3 for GFX10 4278 :name: amdgpu-amdhsa-compute_pgm_rsrc3-gfx10-table 4279 4280 ======= ======= =============================== =========================================================================== 4281 Bits Size Field Name Description 4282 ======= ======= =============================== =========================================================================== 4283 3:0 4 bits SHARED_VGPR_COUNT Number of shared VGPRs for wavefront size 64. Granularity 8. Value 0-120. 4284 compute_pgm_rsrc1.vgprs + shared_vgpr_cnt cannot exceed 64. 4285 31:4 28 Reserved, must be 0. 4286 bits 4287 32 **Total size 4 bytes.** 4288 ======= =================================================================================================================== 4289 4290.. 4291 4292 .. table:: Floating Point Rounding Mode Enumeration Values 4293 :name: amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table 4294 4295 ====================================== ===== ============================== 4296 Enumeration Name Value Description 4297 ====================================== ===== ============================== 4298 FLOAT_ROUND_MODE_NEAR_EVEN 0 Round Ties To Even 4299 FLOAT_ROUND_MODE_PLUS_INFINITY 1 Round Toward +infinity 4300 FLOAT_ROUND_MODE_MINUS_INFINITY 2 Round Toward -infinity 4301 FLOAT_ROUND_MODE_ZERO 3 Round Toward 0 4302 ====================================== ===== ============================== 4303 4304.. 4305 4306 .. table:: Floating Point Denorm Mode Enumeration Values 4307 :name: amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table 4308 4309 ====================================== ===== ============================== 4310 Enumeration Name Value Description 4311 ====================================== ===== ============================== 4312 FLOAT_DENORM_MODE_FLUSH_SRC_DST 0 Flush Source and Destination 4313 Denorms 4314 FLOAT_DENORM_MODE_FLUSH_DST 1 Flush Output Denorms 4315 FLOAT_DENORM_MODE_FLUSH_SRC 2 Flush Source Denorms 4316 FLOAT_DENORM_MODE_FLUSH_NONE 3 No Flush 4317 ====================================== ===== ============================== 4318 4319.. 4320 4321 .. table:: System VGPR Work-Item ID Enumeration Values 4322 :name: amdgpu-amdhsa-system-vgpr-work-item-id-enumeration-values-table 4323 4324 ======================================== ===== ============================ 4325 Enumeration Name Value Description 4326 ======================================== ===== ============================ 4327 SYSTEM_VGPR_WORKITEM_ID_X 0 Set work-item X dimension 4328 ID. 4329 SYSTEM_VGPR_WORKITEM_ID_X_Y 1 Set work-item X and Y 4330 dimensions ID. 4331 SYSTEM_VGPR_WORKITEM_ID_X_Y_Z 2 Set work-item X, Y and Z 4332 dimensions ID. 4333 SYSTEM_VGPR_WORKITEM_ID_UNDEFINED 3 Undefined. 4334 ======================================== ===== ============================ 4335 4336.. _amdgpu-amdhsa-initial-kernel-execution-state: 4337 4338Initial Kernel Execution State 4339~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 4340 4341This section defines the register state that will be set up by the packet 4342processor prior to the start of execution of every wavefront. This is limited by 4343the constraints of the hardware controllers of CP/ADC/SPI. 4344 4345The order of the SGPR registers is defined, but the compiler can specify which 4346ones are actually setup in the kernel descriptor using the ``enable_sgpr_*`` bit 4347fields (see :ref:`amdgpu-amdhsa-kernel-descriptor`). The register numbers used 4348for enabled registers are dense starting at SGPR0: the first enabled register is 4349SGPR0, the next enabled register is SGPR1 etc.; disabled registers do not have 4350an SGPR number. 4351 4352The initial SGPRs comprise up to 16 User SRGPs that are set by CP and apply to 4353all wavefronts of the grid. It is possible to specify more than 16 User SGPRs 4354using the ``enable_sgpr_*`` bit fields, in which case only the first 16 are 4355actually initialized. These are then immediately followed by the System SGPRs 4356that are set up by ADC/SPI and can have different values for each wavefront of 4357the grid dispatch. 4358 4359SGPR register initial state is defined in 4360:ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 4361 4362 .. table:: SGPR Register Set Up Order 4363 :name: amdgpu-amdhsa-sgpr-register-set-up-order-table 4364 4365 ========== ========================== ====== ============================== 4366 SGPR Order Name Number Description 4367 (kernel descriptor enable of 4368 field) SGPRs 4369 ========== ========================== ====== ============================== 4370 First Private Segment Buffer 4 See 4371 (enable_sgpr_private :ref:`amdgpu-amdhsa-kernel-prolog-private-segment-buffer`. 4372 _segment_buffer) 4373 then Dispatch Ptr 2 64-bit address of AQL dispatch 4374 (enable_sgpr_dispatch_ptr) packet for kernel dispatch 4375 actually executing. 4376 then Queue Ptr 2 64-bit address of amd_queue_t 4377 (enable_sgpr_queue_ptr) object for AQL queue on which 4378 the dispatch packet was 4379 queued. 4380 then Kernarg Segment Ptr 2 64-bit address of Kernarg 4381 (enable_sgpr_kernarg segment. This is directly 4382 _segment_ptr) copied from the 4383 kernarg_address in the kernel 4384 dispatch packet. 4385 4386 Having CP load it once avoids 4387 loading it at the beginning of 4388 every wavefront. 4389 then Dispatch Id 2 64-bit Dispatch ID of the 4390 (enable_sgpr_dispatch_id) dispatch packet being 4391 executed. 4392 then Flat Scratch Init 2 See 4393 (enable_sgpr_flat_scratch :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`. 4394 _init) 4395 then Private Segment Size 1 The 32-bit byte size of a 4396 (enable_sgpr_private single work-item's memory 4397 _segment_size) allocation. This is the 4398 value from the kernel 4399 dispatch packet Private 4400 Segment Byte Size rounded up 4401 by CP to a multiple of 4402 DWORD. 4403 4404 Having CP load it once avoids 4405 loading it at the beginning of 4406 every wavefront. 4407 4408 This is not used for 4409 GFX7-GFX8 since it is the same 4410 value as the second SGPR of 4411 Flat Scratch Init. However, it 4412 may be needed for GFX9-GFX10 which 4413 changes the meaning of the 4414 Flat Scratch Init value. 4415 then Work-Group Id X 1 32-bit work-group id in X 4416 (enable_sgpr_workgroup_id dimension of grid for 4417 _X) wavefront. 4418 then Work-Group Id Y 1 32-bit work-group id in Y 4419 (enable_sgpr_workgroup_id dimension of grid for 4420 _Y) wavefront. 4421 then Work-Group Id Z 1 32-bit work-group id in Z 4422 (enable_sgpr_workgroup_id dimension of grid for 4423 _Z) wavefront. 4424 then Work-Group Info 1 {first_wavefront, 14'b0000, 4425 (enable_sgpr_workgroup ordered_append_term[10:0], 4426 _info) threadgroup_size_in_wavefronts[5:0]} 4427 then Scratch Wavefront Offset 1 See 4428 (enable_sgpr_private :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`. 4429 _segment_wavefront_offset) and 4430 :ref:`amdgpu-amdhsa-kernel-prolog-private-segment-buffer`. 4431 ========== ========================== ====== ============================== 4432 4433The order of the VGPR registers is defined, but the compiler can specify which 4434ones are actually setup in the kernel descriptor using the ``enable_vgpr*`` bit 4435fields (see :ref:`amdgpu-amdhsa-kernel-descriptor`). The register numbers used 4436for enabled registers are dense starting at VGPR0: the first enabled register is 4437VGPR0, the next enabled register is VGPR1 etc.; disabled registers do not have a 4438VGPR number. 4439 4440There are different methods used for the VGPR initial state: 4441 4442* Unless the *Target Properties* column of :ref:`amdgpu-processor-table` 4443 specifies otherwise, a separate VGPR register is used per work-item ID. The 4444 VGPR register initial state for this method is defined in 4445 :ref:`amdgpu-amdhsa-vgpr-register-set-up-order-for-unpacked-work-item-id-method-table`. 4446* If *Target Properties* column of :ref:`amdgpu-processor-table` 4447 specifies *Packed work-item IDs*, the initial value of VGPR0 register is used 4448 for all work-item IDs. The register layout for this method is defined in 4449 :ref:`amdgpu-amdhsa-register-layout-for-packed-work-item-id-method-table`. 4450 4451 .. table:: VGPR Register Set Up Order for Unpacked Work-Item ID Method 4452 :name: amdgpu-amdhsa-vgpr-register-set-up-order-for-unpacked-work-item-id-method-table 4453 4454 ========== ========================== ====== ============================== 4455 VGPR Order Name Number Description 4456 (kernel descriptor enable of 4457 field) VGPRs 4458 ========== ========================== ====== ============================== 4459 First Work-Item Id X 1 32-bit work-item id in X 4460 (Always initialized) dimension of work-group for 4461 wavefront lane. 4462 then Work-Item Id Y 1 32-bit work-item id in Y 4463 (enable_vgpr_workitem_id dimension of work-group for 4464 > 0) wavefront lane. 4465 then Work-Item Id Z 1 32-bit work-item id in Z 4466 (enable_vgpr_workitem_id dimension of work-group for 4467 > 1) wavefront lane. 4468 ========== ========================== ====== ============================== 4469 4470.. 4471 4472 .. table:: Register Layout for Packed Work-Item ID Method 4473 :name: amdgpu-amdhsa-register-layout-for-packed-work-item-id-method-table 4474 4475 ======= ======= ================ ========================================= 4476 Bits Size Field Name Description 4477 ======= ======= ================ ========================================= 4478 0:9 10 bits Work-Item Id X Work-item id in X 4479 dimension of work-group for 4480 wavefront lane. 4481 4482 Always initialized. 4483 4484 10:19 10 bits Work-Item Id Y Work-item id in Y 4485 dimension of work-group for 4486 wavefront lane. 4487 4488 Initialized if enable_vgpr_workitem_id > 4489 0, otherwise set to 0. 4490 20:29 10 bits Work-Item Id Z Work-item id in Z 4491 dimension of work-group for 4492 wavefront lane. 4493 4494 Initialized if enable_vgpr_workitem_id > 4495 1, otherwise set to 0. 4496 30:31 2 bits Reserved, set to 0. 4497 ======= ======= ================ ========================================= 4498 4499The setting of registers is done by GPU CP/ADC/SPI hardware as follows: 4500 45011. SGPRs before the Work-Group Ids are set by CP using the 16 User Data 4502 registers. 45032. Work-group Id registers X, Y, Z are set by ADC which supports any 4504 combination including none. 45053. Scratch Wavefront Offset is set by SPI in a per wavefront basis which is why 4506 its value cannot be included with the flat scratch init value which is per 4507 queue (see :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`). 45084. The VGPRs are set by SPI which only supports specifying either (X), (X, Y) 4509 or (X, Y, Z). 45105. Flat Scratch register pair initialization is described in 4511 :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`. 4512 4513The global segment can be accessed either using buffer instructions (GFX6 which 4514has V# 64-bit address support), flat instructions (GFX7-GFX10), or global 4515instructions (GFX9-GFX10). 4516 4517If buffer operations are used, then the compiler can generate a V# with the 4518following properties: 4519 4520* base address of 0 4521* no swizzle 4522* ATC: 1 if IOMMU present (such as APU) 4523* ptr64: 1 4524* MTYPE set to support memory coherence that matches the runtime (such as CC for 4525 APU and NC for dGPU). 4526 4527.. _amdgpu-amdhsa-kernel-prolog: 4528 4529Kernel Prolog 4530~~~~~~~~~~~~~ 4531 4532The compiler performs initialization in the kernel prologue depending on the 4533target and information about things like stack usage in the kernel and called 4534functions. Some of this initialization requires the compiler to request certain 4535User and System SGPRs be present in the 4536:ref:`amdgpu-amdhsa-initial-kernel-execution-state` via the 4537:ref:`amdgpu-amdhsa-kernel-descriptor`. 4538 4539.. _amdgpu-amdhsa-kernel-prolog-cfi: 4540 4541CFI 4542+++ 4543 45441. The CFI return address is undefined. 4545 45462. The CFI CFA is defined using an expression which evaluates to a location 4547 description that comprises one memory location description for the 4548 ``DW_ASPACE_AMDGPU_private_lane`` address space address ``0``. 4549 4550.. _amdgpu-amdhsa-kernel-prolog-m0: 4551 4552M0 4553++ 4554 4555GFX6-GFX8 4556 The M0 register must be initialized with a value at least the total LDS size 4557 if the kernel may access LDS via DS or flat operations. Total LDS size is 4558 available in dispatch packet. For M0, it is also possible to use maximum 4559 possible value of LDS for given target (0x7FFF for GFX6 and 0xFFFF for 4560 GFX7-GFX8). 4561GFX9-GFX10 4562 The M0 register is not used for range checking LDS accesses and so does not 4563 need to be initialized in the prolog. 4564 4565.. _amdgpu-amdhsa-kernel-prolog-stack-pointer: 4566 4567Stack Pointer 4568+++++++++++++ 4569 4570If the kernel has function calls it must set up the ABI stack pointer described 4571in :ref:`amdgpu-amdhsa-function-call-convention-non-kernel-functions` by setting 4572SGPR32 to the unswizzled scratch offset of the address past the last local 4573allocation. 4574 4575.. _amdgpu-amdhsa-kernel-prolog-frame-pointer: 4576 4577Frame Pointer 4578+++++++++++++ 4579 4580If the kernel needs a frame pointer for the reasons defined in 4581``SIFrameLowering`` then SGPR33 is used and is always set to ``0`` in the 4582kernel prolog. If a frame pointer is not required then all uses of the frame 4583pointer are replaced with immediate ``0`` offsets. 4584 4585.. _amdgpu-amdhsa-kernel-prolog-flat-scratch: 4586 4587Flat Scratch 4588++++++++++++ 4589 4590There are different methods used for initializing flat scratch: 4591 4592* If the *Target Properties* column of :ref:`amdgpu-processor-table` 4593 specifies *Does not support generic address space*: 4594 4595 Flat scratch is not supported and there is no flat scratch register pair. 4596 4597* If the *Target Properties* column of :ref:`amdgpu-processor-table` 4598 specifies *Offset flat scratch*: 4599 4600 If the kernel or any function it calls may use flat operations to access 4601 scratch memory, the prolog code must set up the FLAT_SCRATCH register pair 4602 (FLAT_SCRATCH_LO/FLAT_SCRATCH_HI). Initialization uses Flat Scratch Init and 4603 Scratch Wavefront Offset SGPR registers (see 4604 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`): 4605 4606 1. The low word of Flat Scratch Init is the 32-bit byte offset from 4607 ``SH_HIDDEN_PRIVATE_BASE_VIMID`` to the base of scratch backing memory 4608 being managed by SPI for the queue executing the kernel dispatch. This is 4609 the same value used in the Scratch Segment Buffer V# base address. 4610 4611 CP obtains this from the runtime. (The Scratch Segment Buffer base address 4612 is ``SH_HIDDEN_PRIVATE_BASE_VIMID`` plus this offset.) 4613 4614 The prolog must add the value of Scratch Wavefront Offset to get the 4615 wavefront's byte scratch backing memory offset from 4616 ``SH_HIDDEN_PRIVATE_BASE_VIMID``. 4617 4618 The Scratch Wavefront Offset must also be used as an offset with Private 4619 segment address when using the Scratch Segment Buffer. 4620 4621 Since FLAT_SCRATCH_LO is in units of 256 bytes, the offset must be right 4622 shifted by 8 before moving into FLAT_SCRATCH_HI. 4623 4624 FLAT_SCRATCH_HI corresponds to SGPRn-4 on GFX7, and SGPRn-6 on GFX8 (where 4625 SGPRn is the highest numbered SGPR allocated to the wavefront). 4626 FLAT_SCRATCH_HI is multiplied by 256 (as it is in units of 256 bytes) and 4627 added to ``SH_HIDDEN_PRIVATE_BASE_VIMID`` to calculate the per wavefront 4628 FLAT SCRATCH BASE in flat memory instructions that access the scratch 4629 aperture. 4630 2. The second word of Flat Scratch Init is 32-bit byte size of a single 4631 work-items scratch memory usage. 4632 4633 CP obtains this from the runtime, and it is always a multiple of DWORD. CP 4634 checks that the value in the kernel dispatch packet Private Segment Byte 4635 Size is not larger and requests the runtime to increase the queue's scratch 4636 size if necessary. 4637 4638 CP directly loads from the kernel dispatch packet Private Segment Byte Size 4639 field and rounds up to a multiple of DWORD. Having CP load it once avoids 4640 loading it at the beginning of every wavefront. 4641 4642 The kernel prolog code must move it to FLAT_SCRATCH_LO which is SGPRn-3 on 4643 GFX7 and SGPRn-5 on GFX8. FLAT_SCRATCH_LO is used as the FLAT SCRATCH SIZE 4644 in flat memory instructions. 4645 4646* If the *Target Properties* column of :ref:`amdgpu-processor-table` 4647 specifies *Absolute flat scratch*: 4648 4649 If the kernel or any function it calls may use flat operations to access 4650 scratch memory, the prolog code must set up the FLAT_SCRATCH register pair 4651 (FLAT_SCRATCH_LO/FLAT_SCRATCH_HI which are in SGPRn-4/SGPRn-3). Initialization 4652 uses Flat Scratch Init and Scratch Wavefront Offset SGPR registers (see 4653 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`): 4654 4655 The Flat Scratch Init is the 64-bit address of the base of scratch backing 4656 memory being managed by SPI for the queue executing the kernel dispatch. 4657 4658 CP obtains this from the runtime. 4659 4660 The kernel prolog must add the value of the wave's Scratch Wavefront Offset 4661 and move the result as a 64-bit value to the FLAT_SCRATCH SGPR register pair 4662 which is SGPRn-6 and SGPRn-5. It is used as the FLAT SCRATCH BASE in flat 4663 memory instructions. 4664 4665 The Scratch Wavefront Offset must also be used as an offset with Private 4666 segment address when using the Scratch Segment Buffer (see 4667 :ref:`amdgpu-amdhsa-kernel-prolog-private-segment-buffer`). 4668 4669* If the *Target Properties* column of :ref:`amdgpu-processor-table` 4670 specifies *Architected flat scratch*: 4671 4672 If ENABLE_PRIVATE_SEGMENT is enabled in 4673 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table` then the FLAT_SCRATCH 4674 register pair will be initialized to the 64-bit address of the base of scratch 4675 backing memory being managed by SPI for the queue executing the kernel 4676 dispatch plus the value of the wave's Scratch Wavefront Offset for use as the 4677 flat scratch base in flat memory instructions. 4678 4679.. _amdgpu-amdhsa-kernel-prolog-private-segment-buffer: 4680 4681Private Segment Buffer 4682++++++++++++++++++++++ 4683 4684If the *Target Properties* column of :ref:`amdgpu-processor-table` specifies 4685*Architected flat scratch* then a Private Segment Buffer is not supported. 4686Instead the flat SCRATCH instructions are used. 4687 4688Otherwise, Private Segment Buffer SGPR register is used to initialize 4 SGPRs 4689that are used as a V# to access scratch. CP uses the value provided by the 4690runtime. It is used, together with Scratch Wavefront Offset as an offset, to 4691access the private memory space using a segment address. See 4692:ref:`amdgpu-amdhsa-initial-kernel-execution-state`. 4693 4694The scratch V# is a four-aligned SGPR and always selected for the kernel as 4695follows: 4696 4697 - If it is known during instruction selection that there is stack usage, 4698 SGPR0-3 is reserved for use as the scratch V#. Stack usage is assumed if 4699 optimizations are disabled (``-O0``), if stack objects already exist (for 4700 locals, etc.), or if there are any function calls. 4701 4702 - Otherwise, four high numbered SGPRs beginning at a four-aligned SGPR index 4703 are reserved for the tentative scratch V#. These will be used if it is 4704 determined that spilling is needed. 4705 4706 - If no use is made of the tentative scratch V#, then it is unreserved, 4707 and the register count is determined ignoring it. 4708 - If use is made of the tentative scratch V#, then its register numbers 4709 are shifted to the first four-aligned SGPR index after the highest one 4710 allocated by the register allocator, and all uses are updated. The 4711 register count includes them in the shifted location. 4712 - In either case, if the processor has the SGPR allocation bug, the 4713 tentative allocation is not shifted or unreserved in order to ensure 4714 the register count is higher to workaround the bug. 4715 4716 .. note:: 4717 4718 This approach of using a tentative scratch V# and shifting the register 4719 numbers if used avoids having to perform register allocation a second 4720 time if the tentative V# is eliminated. This is more efficient and 4721 avoids the problem that the second register allocation may perform 4722 spilling which will fail as there is no longer a scratch V#. 4723 4724When the kernel prolog code is being emitted it is known whether the scratch V# 4725described above is actually used. If it is, the prolog code must set it up by 4726copying the Private Segment Buffer to the scratch V# registers and then adding 4727the Private Segment Wavefront Offset to the queue base address in the V#. The 4728result is a V# with a base address pointing to the beginning of the wavefront 4729scratch backing memory. 4730 4731The Private Segment Buffer is always requested, but the Private Segment 4732Wavefront Offset is only requested if it is used (see 4733:ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4734 4735.. _amdgpu-amdhsa-memory-model: 4736 4737Memory Model 4738~~~~~~~~~~~~ 4739 4740This section describes the mapping of the LLVM memory model onto AMDGPU machine 4741code (see :ref:`memmodel`). 4742 4743The AMDGPU backend supports the memory synchronization scopes specified in 4744:ref:`amdgpu-memory-scopes`. 4745 4746The code sequences used to implement the memory model specify the order of 4747instructions that a single thread must execute. The ``s_waitcnt`` and cache 4748management instructions such as ``buffer_wbinvl1_vol`` are defined with respect 4749to other memory instructions executed by the same thread. This allows them to be 4750moved earlier or later which can allow them to be combined with other instances 4751of the same instruction, or hoisted/sunk out of loops to improve performance. 4752Only the instructions related to the memory model are given; additional 4753``s_waitcnt`` instructions are required to ensure registers are defined before 4754being used. These may be able to be combined with the memory model ``s_waitcnt`` 4755instructions as described above. 4756 4757The AMDGPU backend supports the following memory models: 4758 4759 HSA Memory Model [HSA]_ 4760 The HSA memory model uses a single happens-before relation for all address 4761 spaces (see :ref:`amdgpu-address-spaces`). 4762 OpenCL Memory Model [OpenCL]_ 4763 The OpenCL memory model which has separate happens-before relations for the 4764 global and local address spaces. Only a fence specifying both global and 4765 local address space, and seq_cst instructions join the relationships. Since 4766 the LLVM ``memfence`` instruction does not allow an address space to be 4767 specified the OpenCL fence has to conservatively assume both local and 4768 global address space was specified. However, optimizations can often be 4769 done to eliminate the additional ``s_waitcnt`` instructions when there are 4770 no intervening memory instructions which access the corresponding address 4771 space. The code sequences in the table indicate what can be omitted for the 4772 OpenCL memory. The target triple environment is used to determine if the 4773 source language is OpenCL (see :ref:`amdgpu-opencl`). 4774 4775``ds/flat_load/store/atomic`` instructions to local memory are termed LDS 4776operations. 4777 4778``buffer/global/flat_load/store/atomic`` instructions to global memory are 4779termed vector memory operations. 4780 4781Private address space uses ``buffer_load/store`` using the scratch V# 4782(GFX6-GFX8), or ``scratch_load/store`` (GFX9-GFX10). Since only a single thread 4783is accessing the memory, atomic memory orderings are not meaningful, and all 4784accesses are treated as non-atomic. 4785 4786Constant address space uses ``buffer/global_load`` instructions (or equivalent 4787scalar memory instructions). Since the constant address space contents do not 4788change during the execution of a kernel dispatch it is not legal to perform 4789stores, and atomic memory orderings are not meaningful, and all accesses are 4790treated as non-atomic. 4791 4792A memory synchronization scope wider than work-group is not meaningful for the 4793group (LDS) address space and is treated as work-group. 4794 4795The memory model does not support the region address space which is treated as 4796non-atomic. 4797 4798Acquire memory ordering is not meaningful on store atomic instructions and is 4799treated as non-atomic. 4800 4801Release memory ordering is not meaningful on load atomic instructions and is 4802treated a non-atomic. 4803 4804Acquire-release memory ordering is not meaningful on load or store atomic 4805instructions and is treated as acquire and release respectively. 4806 4807The memory order also adds the single thread optimization constraints defined in 4808table 4809:ref:`amdgpu-amdhsa-memory-model-single-thread-optimization-constraints-table`. 4810 4811 .. table:: AMDHSA Memory Model Single Thread Optimization Constraints 4812 :name: amdgpu-amdhsa-memory-model-single-thread-optimization-constraints-table 4813 4814 ============ ============================================================== 4815 LLVM Memory Optimization Constraints 4816 Ordering 4817 ============ ============================================================== 4818 unordered *none* 4819 monotonic *none* 4820 acquire - If a load atomic/atomicrmw then no following load/load 4821 atomic/store/store atomic/atomicrmw/fence instruction can be 4822 moved before the acquire. 4823 - If a fence then same as load atomic, plus no preceding 4824 associated fence-paired-atomic can be moved after the fence. 4825 release - If a store atomic/atomicrmw then no preceding load/load 4826 atomic/store/store atomic/atomicrmw/fence instruction can be 4827 moved after the release. 4828 - If a fence then same as store atomic, plus no following 4829 associated fence-paired-atomic can be moved before the 4830 fence. 4831 acq_rel Same constraints as both acquire and release. 4832 seq_cst - If a load atomic then same constraints as acquire, plus no 4833 preceding sequentially consistent load atomic/store 4834 atomic/atomicrmw/fence instruction can be moved after the 4835 seq_cst. 4836 - If a store atomic then the same constraints as release, plus 4837 no following sequentially consistent load atomic/store 4838 atomic/atomicrmw/fence instruction can be moved before the 4839 seq_cst. 4840 - If an atomicrmw/fence then same constraints as acq_rel. 4841 ============ ============================================================== 4842 4843The code sequences used to implement the memory model are defined in the 4844following sections: 4845 4846* :ref:`amdgpu-amdhsa-memory-model-gfx6-gfx9` 4847* :ref:`amdgpu-amdhsa-memory-model-gfx90a` 4848* :ref:`amdgpu-amdhsa-memory-model-gfx10` 4849 4850.. _amdgpu-amdhsa-memory-model-gfx6-gfx9: 4851 4852Memory Model GFX6-GFX9 4853++++++++++++++++++++++ 4854 4855For GFX6-GFX9: 4856 4857* Each agent has multiple shader arrays (SA). 4858* Each SA has multiple compute units (CU). 4859* Each CU has multiple SIMDs that execute wavefronts. 4860* The wavefronts for a single work-group are executed in the same CU but may be 4861 executed by different SIMDs. 4862* Each CU has a single LDS memory shared by the wavefronts of the work-groups 4863 executing on it. 4864* All LDS operations of a CU are performed as wavefront wide operations in a 4865 global order and involve no caching. Completion is reported to a wavefront in 4866 execution order. 4867* The LDS memory has multiple request queues shared by the SIMDs of a 4868 CU. Therefore, the LDS operations performed by different wavefronts of a 4869 work-group can be reordered relative to each other, which can result in 4870 reordering the visibility of vector memory operations with respect to LDS 4871 operations of other wavefronts in the same work-group. A ``s_waitcnt 4872 lgkmcnt(0)`` is required to ensure synchronization between LDS operations and 4873 vector memory operations between wavefronts of a work-group, but not between 4874 operations performed by the same wavefront. 4875* The vector memory operations are performed as wavefront wide operations and 4876 completion is reported to a wavefront in execution order. The exception is 4877 that for GFX7-GFX9 ``flat_load/store/atomic`` instructions can report out of 4878 vector memory order if they access LDS memory, and out of LDS operation order 4879 if they access global memory. 4880* The vector memory operations access a single vector L1 cache shared by all 4881 SIMDs a CU. Therefore, no special action is required for coherence between the 4882 lanes of a single wavefront, or for coherence between wavefronts in the same 4883 work-group. A ``buffer_wbinvl1_vol`` is required for coherence between 4884 wavefronts executing in different work-groups as they may be executing on 4885 different CUs. 4886* The scalar memory operations access a scalar L1 cache shared by all wavefronts 4887 on a group of CUs. The scalar and vector L1 caches are not coherent. However, 4888 scalar operations are used in a restricted way so do not impact the memory 4889 model. See :ref:`amdgpu-amdhsa-memory-spaces`. 4890* The vector and scalar memory operations use an L2 cache shared by all CUs on 4891 the same agent. 4892* The L2 cache has independent channels to service disjoint ranges of virtual 4893 addresses. 4894* Each CU has a separate request queue per channel. Therefore, the vector and 4895 scalar memory operations performed by wavefronts executing in different 4896 work-groups (which may be executing on different CUs) of an agent can be 4897 reordered relative to each other. A ``s_waitcnt vmcnt(0)`` is required to 4898 ensure synchronization between vector memory operations of different CUs. It 4899 ensures a previous vector memory operation has completed before executing a 4900 subsequent vector memory or LDS operation and so can be used to meet the 4901 requirements of acquire and release. 4902* The L2 cache can be kept coherent with other agents on some targets, or ranges 4903 of virtual addresses can be set up to bypass it to ensure system coherence. 4904 4905Scalar memory operations are only used to access memory that is proven to not 4906change during the execution of the kernel dispatch. This includes constant 4907address space and global address space for program scope ``const`` variables. 4908Therefore, the kernel machine code does not have to maintain the scalar cache to 4909ensure it is coherent with the vector caches. The scalar and vector caches are 4910invalidated between kernel dispatches by CP since constant address space data 4911may change between kernel dispatch executions. See 4912:ref:`amdgpu-amdhsa-memory-spaces`. 4913 4914The one exception is if scalar writes are used to spill SGPR registers. In this 4915case the AMDGPU backend ensures the memory location used to spill is never 4916accessed by vector memory operations at the same time. If scalar writes are used 4917then a ``s_dcache_wb`` is inserted before the ``s_endpgm`` and before a function 4918return since the locations may be used for vector memory instructions by a 4919future wavefront that uses the same scratch area, or a function call that 4920creates a frame at the same address, respectively. There is no need for a 4921``s_dcache_inv`` as all scalar writes are write-before-read in the same thread. 4922 4923For kernarg backing memory: 4924 4925* CP invalidates the L1 cache at the start of each kernel dispatch. 4926* On dGPU the kernarg backing memory is allocated in host memory accessed as 4927 MTYPE UC (uncached) to avoid needing to invalidate the L2 cache. This also 4928 causes it to be treated as non-volatile and so is not invalidated by 4929 ``*_vol``. 4930* On APU the kernarg backing memory it is accessed as MTYPE CC (cache coherent) 4931 and so the L2 cache will be coherent with the CPU and other agents. 4932 4933Scratch backing memory (which is used for the private address space) is accessed 4934with MTYPE NC_NV (non-coherent non-volatile). Since the private address space is 4935only accessed by a single thread, and is always write-before-read, there is 4936never a need to invalidate these entries from the L1 cache. Hence all cache 4937invalidates are done as ``*_vol`` to only invalidate the volatile cache lines. 4938 4939The code sequences used to implement the memory model for GFX6-GFX9 are defined 4940in table :ref:`amdgpu-amdhsa-memory-model-code-sequences-gfx6-gfx9-table`. 4941 4942 .. table:: AMDHSA Memory Model Code Sequences GFX6-GFX9 4943 :name: amdgpu-amdhsa-memory-model-code-sequences-gfx6-gfx9-table 4944 4945 ============ ============ ============== ========== ================================ 4946 LLVM Instr LLVM Memory LLVM Memory AMDGPU AMDGPU Machine Code 4947 Ordering Sync Scope Address GFX6-GFX9 4948 Space 4949 ============ ============ ============== ========== ================================ 4950 **Non-Atomic** 4951 ------------------------------------------------------------------------------------ 4952 load *none* *none* - global - !volatile & !nontemporal 4953 - generic 4954 - private 1. buffer/global/flat_load 4955 - constant 4956 - !volatile & nontemporal 4957 4958 1. buffer/global/flat_load 4959 glc=1 slc=1 4960 4961 - volatile 4962 4963 1. buffer/global/flat_load 4964 glc=1 4965 2. s_waitcnt vmcnt(0) 4966 4967 - Must happen before 4968 any following volatile 4969 global/generic 4970 load/store. 4971 - Ensures that 4972 volatile 4973 operations to 4974 different 4975 addresses will not 4976 be reordered by 4977 hardware. 4978 4979 load *none* *none* - local 1. ds_load 4980 store *none* *none* - global - !volatile & !nontemporal 4981 - generic 4982 - private 1. buffer/global/flat_store 4983 - constant 4984 - !volatile & nontemporal 4985 4986 1. buffer/global/flat_store 4987 glc=1 slc=1 4988 4989 - volatile 4990 4991 1. buffer/global/flat_store 4992 2. s_waitcnt vmcnt(0) 4993 4994 - Must happen before 4995 any following volatile 4996 global/generic 4997 load/store. 4998 - Ensures that 4999 volatile 5000 operations to 5001 different 5002 addresses will not 5003 be reordered by 5004 hardware. 5005 5006 store *none* *none* - local 1. ds_store 5007 **Unordered Atomic** 5008 ------------------------------------------------------------------------------------ 5009 load atomic unordered *any* *any* *Same as non-atomic*. 5010 store atomic unordered *any* *any* *Same as non-atomic*. 5011 atomicrmw unordered *any* *any* *Same as monotonic atomic*. 5012 **Monotonic Atomic** 5013 ------------------------------------------------------------------------------------ 5014 load atomic monotonic - singlethread - global 1. buffer/global/ds/flat_load 5015 - wavefront - local 5016 - workgroup - generic 5017 load atomic monotonic - agent - global 1. buffer/global/flat_load 5018 - system - generic glc=1 5019 store atomic monotonic - singlethread - global 1. buffer/global/flat_store 5020 - wavefront - generic 5021 - workgroup 5022 - agent 5023 - system 5024 store atomic monotonic - singlethread - local 1. ds_store 5025 - wavefront 5026 - workgroup 5027 atomicrmw monotonic - singlethread - global 1. buffer/global/flat_atomic 5028 - wavefront - generic 5029 - workgroup 5030 - agent 5031 - system 5032 atomicrmw monotonic - singlethread - local 1. ds_atomic 5033 - wavefront 5034 - workgroup 5035 **Acquire Atomic** 5036 ------------------------------------------------------------------------------------ 5037 load atomic acquire - singlethread - global 1. buffer/global/ds/flat_load 5038 - wavefront - local 5039 - generic 5040 load atomic acquire - workgroup - global 1. buffer/global_load 5041 load atomic acquire - workgroup - local 1. ds/flat_load 5042 - generic 2. s_waitcnt lgkmcnt(0) 5043 5044 - If OpenCL, omit. 5045 - Must happen before 5046 any following 5047 global/generic 5048 load/load 5049 atomic/store/store 5050 atomic/atomicrmw. 5051 - Ensures any 5052 following global 5053 data read is no 5054 older than a local load 5055 atomic value being 5056 acquired. 5057 5058 load atomic acquire - agent - global 1. buffer/global_load 5059 - system glc=1 5060 2. s_waitcnt vmcnt(0) 5061 5062 - Must happen before 5063 following 5064 buffer_wbinvl1_vol. 5065 - Ensures the load 5066 has completed 5067 before invalidating 5068 the cache. 5069 5070 3. buffer_wbinvl1_vol 5071 5072 - Must happen before 5073 any following 5074 global/generic 5075 load/load 5076 atomic/atomicrmw. 5077 - Ensures that 5078 following 5079 loads will not see 5080 stale global data. 5081 5082 load atomic acquire - agent - generic 1. flat_load glc=1 5083 - system 2. s_waitcnt vmcnt(0) & 5084 lgkmcnt(0) 5085 5086 - If OpenCL omit 5087 lgkmcnt(0). 5088 - Must happen before 5089 following 5090 buffer_wbinvl1_vol. 5091 - Ensures the flat_load 5092 has completed 5093 before invalidating 5094 the cache. 5095 5096 3. buffer_wbinvl1_vol 5097 5098 - Must happen before 5099 any following 5100 global/generic 5101 load/load 5102 atomic/atomicrmw. 5103 - Ensures that 5104 following loads 5105 will not see stale 5106 global data. 5107 5108 atomicrmw acquire - singlethread - global 1. buffer/global/ds/flat_atomic 5109 - wavefront - local 5110 - generic 5111 atomicrmw acquire - workgroup - global 1. buffer/global_atomic 5112 atomicrmw acquire - workgroup - local 1. ds/flat_atomic 5113 - generic 2. s_waitcnt lgkmcnt(0) 5114 5115 - If OpenCL, omit. 5116 - Must happen before 5117 any following 5118 global/generic 5119 load/load 5120 atomic/store/store 5121 atomic/atomicrmw. 5122 - Ensures any 5123 following global 5124 data read is no 5125 older than a local 5126 atomicrmw value 5127 being acquired. 5128 5129 atomicrmw acquire - agent - global 1. buffer/global_atomic 5130 - system 2. s_waitcnt vmcnt(0) 5131 5132 - Must happen before 5133 following 5134 buffer_wbinvl1_vol. 5135 - Ensures the 5136 atomicrmw has 5137 completed before 5138 invalidating the 5139 cache. 5140 5141 3. buffer_wbinvl1_vol 5142 5143 - Must happen before 5144 any following 5145 global/generic 5146 load/load 5147 atomic/atomicrmw. 5148 - Ensures that 5149 following loads 5150 will not see stale 5151 global data. 5152 5153 atomicrmw acquire - agent - generic 1. flat_atomic 5154 - system 2. s_waitcnt vmcnt(0) & 5155 lgkmcnt(0) 5156 5157 - If OpenCL, omit 5158 lgkmcnt(0). 5159 - Must happen before 5160 following 5161 buffer_wbinvl1_vol. 5162 - Ensures the 5163 atomicrmw has 5164 completed before 5165 invalidating the 5166 cache. 5167 5168 3. buffer_wbinvl1_vol 5169 5170 - Must happen before 5171 any following 5172 global/generic 5173 load/load 5174 atomic/atomicrmw. 5175 - Ensures that 5176 following loads 5177 will not see stale 5178 global data. 5179 5180 fence acquire - singlethread *none* *none* 5181 - wavefront 5182 fence acquire - workgroup *none* 1. s_waitcnt lgkmcnt(0) 5183 5184 - If OpenCL and 5185 address space is 5186 not generic, omit. 5187 - However, since LLVM 5188 currently has no 5189 address space on 5190 the fence need to 5191 conservatively 5192 always generate. If 5193 fence had an 5194 address space then 5195 set to address 5196 space of OpenCL 5197 fence flag, or to 5198 generic if both 5199 local and global 5200 flags are 5201 specified. 5202 - Must happen after 5203 any preceding 5204 local/generic load 5205 atomic/atomicrmw 5206 with an equal or 5207 wider sync scope 5208 and memory ordering 5209 stronger than 5210 unordered (this is 5211 termed the 5212 fence-paired-atomic). 5213 - Must happen before 5214 any following 5215 global/generic 5216 load/load 5217 atomic/store/store 5218 atomic/atomicrmw. 5219 - Ensures any 5220 following global 5221 data read is no 5222 older than the 5223 value read by the 5224 fence-paired-atomic. 5225 5226 fence acquire - agent *none* 1. s_waitcnt lgkmcnt(0) & 5227 - system vmcnt(0) 5228 5229 - If OpenCL and 5230 address space is 5231 not generic, omit 5232 lgkmcnt(0). 5233 - However, since LLVM 5234 currently has no 5235 address space on 5236 the fence need to 5237 conservatively 5238 always generate 5239 (see comment for 5240 previous fence). 5241 - Could be split into 5242 separate s_waitcnt 5243 vmcnt(0) and 5244 s_waitcnt 5245 lgkmcnt(0) to allow 5246 them to be 5247 independently moved 5248 according to the 5249 following rules. 5250 - s_waitcnt vmcnt(0) 5251 must happen after 5252 any preceding 5253 global/generic load 5254 atomic/atomicrmw 5255 with an equal or 5256 wider sync scope 5257 and memory ordering 5258 stronger than 5259 unordered (this is 5260 termed the 5261 fence-paired-atomic). 5262 - s_waitcnt lgkmcnt(0) 5263 must happen after 5264 any preceding 5265 local/generic load 5266 atomic/atomicrmw 5267 with an equal or 5268 wider sync scope 5269 and memory ordering 5270 stronger than 5271 unordered (this is 5272 termed the 5273 fence-paired-atomic). 5274 - Must happen before 5275 the following 5276 buffer_wbinvl1_vol. 5277 - Ensures that the 5278 fence-paired atomic 5279 has completed 5280 before invalidating 5281 the 5282 cache. Therefore 5283 any following 5284 locations read must 5285 be no older than 5286 the value read by 5287 the 5288 fence-paired-atomic. 5289 5290 2. buffer_wbinvl1_vol 5291 5292 - Must happen before any 5293 following global/generic 5294 load/load 5295 atomic/store/store 5296 atomic/atomicrmw. 5297 - Ensures that 5298 following loads 5299 will not see stale 5300 global data. 5301 5302 **Release Atomic** 5303 ------------------------------------------------------------------------------------ 5304 store atomic release - singlethread - global 1. buffer/global/ds/flat_store 5305 - wavefront - local 5306 - generic 5307 store atomic release - workgroup - global 1. s_waitcnt lgkmcnt(0) 5308 - generic 5309 - If OpenCL, omit. 5310 - Must happen after 5311 any preceding 5312 local/generic 5313 load/store/load 5314 atomic/store 5315 atomic/atomicrmw. 5316 - Must happen before 5317 the following 5318 store. 5319 - Ensures that all 5320 memory operations 5321 to local have 5322 completed before 5323 performing the 5324 store that is being 5325 released. 5326 5327 2. buffer/global/flat_store 5328 store atomic release - workgroup - local 1. ds_store 5329 store atomic release - agent - global 1. s_waitcnt lgkmcnt(0) & 5330 - system - generic vmcnt(0) 5331 5332 - If OpenCL and 5333 address space is 5334 not generic, omit 5335 lgkmcnt(0). 5336 - Could be split into 5337 separate s_waitcnt 5338 vmcnt(0) and 5339 s_waitcnt 5340 lgkmcnt(0) to allow 5341 them to be 5342 independently moved 5343 according to the 5344 following rules. 5345 - s_waitcnt vmcnt(0) 5346 must happen after 5347 any preceding 5348 global/generic 5349 load/store/load 5350 atomic/store 5351 atomic/atomicrmw. 5352 - s_waitcnt lgkmcnt(0) 5353 must happen after 5354 any preceding 5355 local/generic 5356 load/store/load 5357 atomic/store 5358 atomic/atomicrmw. 5359 - Must happen before 5360 the following 5361 store. 5362 - Ensures that all 5363 memory operations 5364 to memory have 5365 completed before 5366 performing the 5367 store that is being 5368 released. 5369 5370 2. buffer/global/flat_store 5371 atomicrmw release - singlethread - global 1. buffer/global/ds/flat_atomic 5372 - wavefront - local 5373 - generic 5374 atomicrmw release - workgroup - global 1. s_waitcnt lgkmcnt(0) 5375 - generic 5376 - If OpenCL, omit. 5377 - Must happen after 5378 any preceding 5379 local/generic 5380 load/store/load 5381 atomic/store 5382 atomic/atomicrmw. 5383 - Must happen before 5384 the following 5385 atomicrmw. 5386 - Ensures that all 5387 memory operations 5388 to local have 5389 completed before 5390 performing the 5391 atomicrmw that is 5392 being released. 5393 5394 2. buffer/global/flat_atomic 5395 atomicrmw release - workgroup - local 1. ds_atomic 5396 atomicrmw release - agent - global 1. s_waitcnt lgkmcnt(0) & 5397 - system - generic vmcnt(0) 5398 5399 - If OpenCL, omit 5400 lgkmcnt(0). 5401 - Could be split into 5402 separate s_waitcnt 5403 vmcnt(0) and 5404 s_waitcnt 5405 lgkmcnt(0) to allow 5406 them to be 5407 independently moved 5408 according to the 5409 following rules. 5410 - s_waitcnt vmcnt(0) 5411 must happen after 5412 any preceding 5413 global/generic 5414 load/store/load 5415 atomic/store 5416 atomic/atomicrmw. 5417 - s_waitcnt lgkmcnt(0) 5418 must happen after 5419 any preceding 5420 local/generic 5421 load/store/load 5422 atomic/store 5423 atomic/atomicrmw. 5424 - Must happen before 5425 the following 5426 atomicrmw. 5427 - Ensures that all 5428 memory operations 5429 to global and local 5430 have completed 5431 before performing 5432 the atomicrmw that 5433 is being released. 5434 5435 2. buffer/global/flat_atomic 5436 fence release - singlethread *none* *none* 5437 - wavefront 5438 fence release - workgroup *none* 1. s_waitcnt lgkmcnt(0) 5439 5440 - If OpenCL and 5441 address space is 5442 not generic, omit. 5443 - However, since LLVM 5444 currently has no 5445 address space on 5446 the fence need to 5447 conservatively 5448 always generate. If 5449 fence had an 5450 address space then 5451 set to address 5452 space of OpenCL 5453 fence flag, or to 5454 generic if both 5455 local and global 5456 flags are 5457 specified. 5458 - Must happen after 5459 any preceding 5460 local/generic 5461 load/load 5462 atomic/store/store 5463 atomic/atomicrmw. 5464 - Must happen before 5465 any following store 5466 atomic/atomicrmw 5467 with an equal or 5468 wider sync scope 5469 and memory ordering 5470 stronger than 5471 unordered (this is 5472 termed the 5473 fence-paired-atomic). 5474 - Ensures that all 5475 memory operations 5476 to local have 5477 completed before 5478 performing the 5479 following 5480 fence-paired-atomic. 5481 5482 fence release - agent *none* 1. s_waitcnt lgkmcnt(0) & 5483 - system vmcnt(0) 5484 5485 - If OpenCL and 5486 address space is 5487 not generic, omit 5488 lgkmcnt(0). 5489 - If OpenCL and 5490 address space is 5491 local, omit 5492 vmcnt(0). 5493 - However, since LLVM 5494 currently has no 5495 address space on 5496 the fence need to 5497 conservatively 5498 always generate. If 5499 fence had an 5500 address space then 5501 set to address 5502 space of OpenCL 5503 fence flag, or to 5504 generic if both 5505 local and global 5506 flags are 5507 specified. 5508 - Could be split into 5509 separate s_waitcnt 5510 vmcnt(0) and 5511 s_waitcnt 5512 lgkmcnt(0) to allow 5513 them to be 5514 independently moved 5515 according to the 5516 following rules. 5517 - s_waitcnt vmcnt(0) 5518 must happen after 5519 any preceding 5520 global/generic 5521 load/store/load 5522 atomic/store 5523 atomic/atomicrmw. 5524 - s_waitcnt lgkmcnt(0) 5525 must happen after 5526 any preceding 5527 local/generic 5528 load/store/load 5529 atomic/store 5530 atomic/atomicrmw. 5531 - Must happen before 5532 any following store 5533 atomic/atomicrmw 5534 with an equal or 5535 wider sync scope 5536 and memory ordering 5537 stronger than 5538 unordered (this is 5539 termed the 5540 fence-paired-atomic). 5541 - Ensures that all 5542 memory operations 5543 have 5544 completed before 5545 performing the 5546 following 5547 fence-paired-atomic. 5548 5549 **Acquire-Release Atomic** 5550 ------------------------------------------------------------------------------------ 5551 atomicrmw acq_rel - singlethread - global 1. buffer/global/ds/flat_atomic 5552 - wavefront - local 5553 - generic 5554 atomicrmw acq_rel - workgroup - global 1. s_waitcnt lgkmcnt(0) 5555 5556 - If OpenCL, omit. 5557 - Must happen after 5558 any preceding 5559 local/generic 5560 load/store/load 5561 atomic/store 5562 atomic/atomicrmw. 5563 - Must happen before 5564 the following 5565 atomicrmw. 5566 - Ensures that all 5567 memory operations 5568 to local have 5569 completed before 5570 performing the 5571 atomicrmw that is 5572 being released. 5573 5574 2. buffer/global_atomic 5575 5576 atomicrmw acq_rel - workgroup - local 1. ds_atomic 5577 2. s_waitcnt lgkmcnt(0) 5578 5579 - If OpenCL, omit. 5580 - Must happen before 5581 any following 5582 global/generic 5583 load/load 5584 atomic/store/store 5585 atomic/atomicrmw. 5586 - Ensures any 5587 following global 5588 data read is no 5589 older than the local load 5590 atomic value being 5591 acquired. 5592 5593 atomicrmw acq_rel - workgroup - generic 1. s_waitcnt lgkmcnt(0) 5594 5595 - If OpenCL, omit. 5596 - Must happen after 5597 any preceding 5598 local/generic 5599 load/store/load 5600 atomic/store 5601 atomic/atomicrmw. 5602 - Must happen before 5603 the following 5604 atomicrmw. 5605 - Ensures that all 5606 memory operations 5607 to local have 5608 completed before 5609 performing the 5610 atomicrmw that is 5611 being released. 5612 5613 2. flat_atomic 5614 3. s_waitcnt lgkmcnt(0) 5615 5616 - If OpenCL, omit. 5617 - Must happen before 5618 any following 5619 global/generic 5620 load/load 5621 atomic/store/store 5622 atomic/atomicrmw. 5623 - Ensures any 5624 following global 5625 data read is no 5626 older than a local load 5627 atomic value being 5628 acquired. 5629 5630 atomicrmw acq_rel - agent - global 1. s_waitcnt lgkmcnt(0) & 5631 - system vmcnt(0) 5632 5633 - If OpenCL, omit 5634 lgkmcnt(0). 5635 - Could be split into 5636 separate s_waitcnt 5637 vmcnt(0) and 5638 s_waitcnt 5639 lgkmcnt(0) to allow 5640 them to be 5641 independently moved 5642 according to the 5643 following rules. 5644 - s_waitcnt vmcnt(0) 5645 must happen after 5646 any preceding 5647 global/generic 5648 load/store/load 5649 atomic/store 5650 atomic/atomicrmw. 5651 - s_waitcnt lgkmcnt(0) 5652 must happen after 5653 any preceding 5654 local/generic 5655 load/store/load 5656 atomic/store 5657 atomic/atomicrmw. 5658 - Must happen before 5659 the following 5660 atomicrmw. 5661 - Ensures that all 5662 memory operations 5663 to global have 5664 completed before 5665 performing the 5666 atomicrmw that is 5667 being released. 5668 5669 2. buffer/global_atomic 5670 3. s_waitcnt vmcnt(0) 5671 5672 - Must happen before 5673 following 5674 buffer_wbinvl1_vol. 5675 - Ensures the 5676 atomicrmw has 5677 completed before 5678 invalidating the 5679 cache. 5680 5681 4. buffer_wbinvl1_vol 5682 5683 - Must happen before 5684 any following 5685 global/generic 5686 load/load 5687 atomic/atomicrmw. 5688 - Ensures that 5689 following loads 5690 will not see stale 5691 global data. 5692 5693 atomicrmw acq_rel - agent - generic 1. s_waitcnt lgkmcnt(0) & 5694 - system vmcnt(0) 5695 5696 - If OpenCL, omit 5697 lgkmcnt(0). 5698 - Could be split into 5699 separate s_waitcnt 5700 vmcnt(0) and 5701 s_waitcnt 5702 lgkmcnt(0) to allow 5703 them to be 5704 independently moved 5705 according to the 5706 following rules. 5707 - s_waitcnt vmcnt(0) 5708 must happen after 5709 any preceding 5710 global/generic 5711 load/store/load 5712 atomic/store 5713 atomic/atomicrmw. 5714 - s_waitcnt lgkmcnt(0) 5715 must happen after 5716 any preceding 5717 local/generic 5718 load/store/load 5719 atomic/store 5720 atomic/atomicrmw. 5721 - Must happen before 5722 the following 5723 atomicrmw. 5724 - Ensures that all 5725 memory operations 5726 to global have 5727 completed before 5728 performing the 5729 atomicrmw that is 5730 being released. 5731 5732 2. flat_atomic 5733 3. s_waitcnt vmcnt(0) & 5734 lgkmcnt(0) 5735 5736 - If OpenCL, omit 5737 lgkmcnt(0). 5738 - Must happen before 5739 following 5740 buffer_wbinvl1_vol. 5741 - Ensures the 5742 atomicrmw has 5743 completed before 5744 invalidating the 5745 cache. 5746 5747 4. buffer_wbinvl1_vol 5748 5749 - Must happen before 5750 any following 5751 global/generic 5752 load/load 5753 atomic/atomicrmw. 5754 - Ensures that 5755 following loads 5756 will not see stale 5757 global data. 5758 5759 fence acq_rel - singlethread *none* *none* 5760 - wavefront 5761 fence acq_rel - workgroup *none* 1. s_waitcnt lgkmcnt(0) 5762 5763 - If OpenCL and 5764 address space is 5765 not generic, omit. 5766 - However, 5767 since LLVM 5768 currently has no 5769 address space on 5770 the fence need to 5771 conservatively 5772 always generate 5773 (see comment for 5774 previous fence). 5775 - Must happen after 5776 any preceding 5777 local/generic 5778 load/load 5779 atomic/store/store 5780 atomic/atomicrmw. 5781 - Must happen before 5782 any following 5783 global/generic 5784 load/load 5785 atomic/store/store 5786 atomic/atomicrmw. 5787 - Ensures that all 5788 memory operations 5789 to local have 5790 completed before 5791 performing any 5792 following global 5793 memory operations. 5794 - Ensures that the 5795 preceding 5796 local/generic load 5797 atomic/atomicrmw 5798 with an equal or 5799 wider sync scope 5800 and memory ordering 5801 stronger than 5802 unordered (this is 5803 termed the 5804 acquire-fence-paired-atomic) 5805 has completed 5806 before following 5807 global memory 5808 operations. This 5809 satisfies the 5810 requirements of 5811 acquire. 5812 - Ensures that all 5813 previous memory 5814 operations have 5815 completed before a 5816 following 5817 local/generic store 5818 atomic/atomicrmw 5819 with an equal or 5820 wider sync scope 5821 and memory ordering 5822 stronger than 5823 unordered (this is 5824 termed the 5825 release-fence-paired-atomic). 5826 This satisfies the 5827 requirements of 5828 release. 5829 5830 fence acq_rel - agent *none* 1. s_waitcnt lgkmcnt(0) & 5831 - system vmcnt(0) 5832 5833 - If OpenCL and 5834 address space is 5835 not generic, omit 5836 lgkmcnt(0). 5837 - However, since LLVM 5838 currently has no 5839 address space on 5840 the fence need to 5841 conservatively 5842 always generate 5843 (see comment for 5844 previous fence). 5845 - Could be split into 5846 separate s_waitcnt 5847 vmcnt(0) and 5848 s_waitcnt 5849 lgkmcnt(0) to allow 5850 them to be 5851 independently moved 5852 according to the 5853 following rules. 5854 - s_waitcnt vmcnt(0) 5855 must happen after 5856 any preceding 5857 global/generic 5858 load/store/load 5859 atomic/store 5860 atomic/atomicrmw. 5861 - s_waitcnt lgkmcnt(0) 5862 must happen after 5863 any preceding 5864 local/generic 5865 load/store/load 5866 atomic/store 5867 atomic/atomicrmw. 5868 - Must happen before 5869 the following 5870 buffer_wbinvl1_vol. 5871 - Ensures that the 5872 preceding 5873 global/local/generic 5874 load 5875 atomic/atomicrmw 5876 with an equal or 5877 wider sync scope 5878 and memory ordering 5879 stronger than 5880 unordered (this is 5881 termed the 5882 acquire-fence-paired-atomic) 5883 has completed 5884 before invalidating 5885 the cache. This 5886 satisfies the 5887 requirements of 5888 acquire. 5889 - Ensures that all 5890 previous memory 5891 operations have 5892 completed before a 5893 following 5894 global/local/generic 5895 store 5896 atomic/atomicrmw 5897 with an equal or 5898 wider sync scope 5899 and memory ordering 5900 stronger than 5901 unordered (this is 5902 termed the 5903 release-fence-paired-atomic). 5904 This satisfies the 5905 requirements of 5906 release. 5907 5908 2. buffer_wbinvl1_vol 5909 5910 - Must happen before 5911 any following 5912 global/generic 5913 load/load 5914 atomic/store/store 5915 atomic/atomicrmw. 5916 - Ensures that 5917 following loads 5918 will not see stale 5919 global data. This 5920 satisfies the 5921 requirements of 5922 acquire. 5923 5924 **Sequential Consistent Atomic** 5925 ------------------------------------------------------------------------------------ 5926 load atomic seq_cst - singlethread - global *Same as corresponding 5927 - wavefront - local load atomic acquire, 5928 - generic except must generated 5929 all instructions even 5930 for OpenCL.* 5931 load atomic seq_cst - workgroup - global 1. s_waitcnt lgkmcnt(0) 5932 - generic 5933 5934 - Must 5935 happen after 5936 preceding 5937 local/generic load 5938 atomic/store 5939 atomic/atomicrmw 5940 with memory 5941 ordering of seq_cst 5942 and with equal or 5943 wider sync scope. 5944 (Note that seq_cst 5945 fences have their 5946 own s_waitcnt 5947 lgkmcnt(0) and so do 5948 not need to be 5949 considered.) 5950 - Ensures any 5951 preceding 5952 sequential 5953 consistent local 5954 memory instructions 5955 have completed 5956 before executing 5957 this sequentially 5958 consistent 5959 instruction. This 5960 prevents reordering 5961 a seq_cst store 5962 followed by a 5963 seq_cst load. (Note 5964 that seq_cst is 5965 stronger than 5966 acquire/release as 5967 the reordering of 5968 load acquire 5969 followed by a store 5970 release is 5971 prevented by the 5972 s_waitcnt of 5973 the release, but 5974 there is nothing 5975 preventing a store 5976 release followed by 5977 load acquire from 5978 completing out of 5979 order. The s_waitcnt 5980 could be placed after 5981 seq_store or before 5982 the seq_load. We 5983 choose the load to 5984 make the s_waitcnt be 5985 as late as possible 5986 so that the store 5987 may have already 5988 completed.) 5989 5990 2. *Following 5991 instructions same as 5992 corresponding load 5993 atomic acquire, 5994 except must generated 5995 all instructions even 5996 for OpenCL.* 5997 load atomic seq_cst - workgroup - local *Same as corresponding 5998 load atomic acquire, 5999 except must generated 6000 all instructions even 6001 for OpenCL.* 6002 6003 load atomic seq_cst - agent - global 1. s_waitcnt lgkmcnt(0) & 6004 - system - generic vmcnt(0) 6005 6006 - Could be split into 6007 separate s_waitcnt 6008 vmcnt(0) 6009 and s_waitcnt 6010 lgkmcnt(0) to allow 6011 them to be 6012 independently moved 6013 according to the 6014 following rules. 6015 - s_waitcnt lgkmcnt(0) 6016 must happen after 6017 preceding 6018 global/generic load 6019 atomic/store 6020 atomic/atomicrmw 6021 with memory 6022 ordering of seq_cst 6023 and with equal or 6024 wider sync scope. 6025 (Note that seq_cst 6026 fences have their 6027 own s_waitcnt 6028 lgkmcnt(0) and so do 6029 not need to be 6030 considered.) 6031 - s_waitcnt vmcnt(0) 6032 must happen after 6033 preceding 6034 global/generic load 6035 atomic/store 6036 atomic/atomicrmw 6037 with memory 6038 ordering of seq_cst 6039 and with equal or 6040 wider sync scope. 6041 (Note that seq_cst 6042 fences have their 6043 own s_waitcnt 6044 vmcnt(0) and so do 6045 not need to be 6046 considered.) 6047 - Ensures any 6048 preceding 6049 sequential 6050 consistent global 6051 memory instructions 6052 have completed 6053 before executing 6054 this sequentially 6055 consistent 6056 instruction. This 6057 prevents reordering 6058 a seq_cst store 6059 followed by a 6060 seq_cst load. (Note 6061 that seq_cst is 6062 stronger than 6063 acquire/release as 6064 the reordering of 6065 load acquire 6066 followed by a store 6067 release is 6068 prevented by the 6069 s_waitcnt of 6070 the release, but 6071 there is nothing 6072 preventing a store 6073 release followed by 6074 load acquire from 6075 completing out of 6076 order. The s_waitcnt 6077 could be placed after 6078 seq_store or before 6079 the seq_load. We 6080 choose the load to 6081 make the s_waitcnt be 6082 as late as possible 6083 so that the store 6084 may have already 6085 completed.) 6086 6087 2. *Following 6088 instructions same as 6089 corresponding load 6090 atomic acquire, 6091 except must generated 6092 all instructions even 6093 for OpenCL.* 6094 store atomic seq_cst - singlethread - global *Same as corresponding 6095 - wavefront - local store atomic release, 6096 - workgroup - generic except must generated 6097 - agent all instructions even 6098 - system for OpenCL.* 6099 atomicrmw seq_cst - singlethread - global *Same as corresponding 6100 - wavefront - local atomicrmw acq_rel, 6101 - workgroup - generic except must generated 6102 - agent all instructions even 6103 - system for OpenCL.* 6104 fence seq_cst - singlethread *none* *Same as corresponding 6105 - wavefront fence acq_rel, 6106 - workgroup except must generated 6107 - agent all instructions even 6108 - system for OpenCL.* 6109 ============ ============ ============== ========== ================================ 6110 6111.. _amdgpu-amdhsa-memory-model-gfx90a: 6112 6113Memory Model GFX90A 6114+++++++++++++++++++ 6115 6116For GFX90A: 6117 6118* Each agent has multiple shader arrays (SA). 6119* Each SA has multiple compute units (CU). 6120* Each CU has multiple SIMDs that execute wavefronts. 6121* The wavefronts for a single work-group are executed in the same CU but may be 6122 executed by different SIMDs. The exception is when in tgsplit execution mode 6123 when the wavefronts may be executed by different SIMDs in different CUs. 6124* Each CU has a single LDS memory shared by the wavefronts of the work-groups 6125 executing on it. The exception is when in tgsplit execution mode when no LDS 6126 is allocated as wavefronts of the same work-group can be in different CUs. 6127* All LDS operations of a CU are performed as wavefront wide operations in a 6128 global order and involve no caching. Completion is reported to a wavefront in 6129 execution order. 6130* The LDS memory has multiple request queues shared by the SIMDs of a 6131 CU. Therefore, the LDS operations performed by different wavefronts of a 6132 work-group can be reordered relative to each other, which can result in 6133 reordering the visibility of vector memory operations with respect to LDS 6134 operations of other wavefronts in the same work-group. A ``s_waitcnt 6135 lgkmcnt(0)`` is required to ensure synchronization between LDS operations and 6136 vector memory operations between wavefronts of a work-group, but not between 6137 operations performed by the same wavefront. 6138* The vector memory operations are performed as wavefront wide operations and 6139 completion is reported to a wavefront in execution order. The exception is 6140 that ``flat_load/store/atomic`` instructions can report out of vector memory 6141 order if they access LDS memory, and out of LDS operation order if they access 6142 global memory. 6143* The vector memory operations access a single vector L1 cache shared by all 6144 SIMDs a CU. Therefore: 6145 6146 * No special action is required for coherence between the lanes of a single 6147 wavefront. 6148 6149 * No special action is required for coherence between wavefronts in the same 6150 work-group since they execute on the same CU. The exception is when in 6151 tgsplit execution mode as wavefronts of the same work-group can be in 6152 different CUs and so a ``buffer_wbinvl1_vol`` is required as described in 6153 the following item. 6154 6155 * A ``buffer_wbinvl1_vol`` is required for coherence between wavefronts 6156 executing in different work-groups as they may be executing on different 6157 CUs. 6158 6159* The scalar memory operations access a scalar L1 cache shared by all wavefronts 6160 on a group of CUs. The scalar and vector L1 caches are not coherent. However, 6161 scalar operations are used in a restricted way so do not impact the memory 6162 model. See :ref:`amdgpu-amdhsa-memory-spaces`. 6163* The vector and scalar memory operations use an L2 cache shared by all CUs on 6164 the same agent. 6165 6166 * The L2 cache has independent channels to service disjoint ranges of virtual 6167 addresses. 6168 * Each CU has a separate request queue per channel. Therefore, the vector and 6169 scalar memory operations performed by wavefronts executing in different 6170 work-groups (which may be executing on different CUs), or the same 6171 work-group if executing in tgsplit mode, of an agent can be reordered 6172 relative to each other. A ``s_waitcnt vmcnt(0)`` is required to ensure 6173 synchronization between vector memory operations of different CUs. It 6174 ensures a previous vector memory operation has completed before executing a 6175 subsequent vector memory or LDS operation and so can be used to meet the 6176 requirements of acquire and release. 6177 * The L2 cache of one agent can be kept coherent with other agents by: 6178 using the MTYPE RW (read-write) or MTYPE CC (cache-coherent) with the PTE 6179 C-bit for memory local to the L2; and using the MTYPE NC (non-coherent) with 6180 the PTE C-bit set or MTYPE UC (uncached) for memory not local to the L2. 6181 6182 * Any local memory cache lines will be automatically invalidated by writes 6183 from CUs associated with other L2 caches, or writes from the CPU, due to 6184 the cache probe caused by coherent requests. Coherent requests are caused 6185 by GPU accesses to pages with the PTE C-bit set, by CPU accesses over 6186 XGMI, and by PCIe requests that are configured to be coherent requests. 6187 * XGMI accesses from the CPU to local memory may be cached on the CPU. 6188 Subsequent access from the GPU will automatically invalidate or writeback 6189 the CPU cache due to the L2 probe filter and and the PTE C-bit being set. 6190 * Since all work-groups on the same agent share the same L2, no L2 6191 invalidation or writeback is required for coherence. 6192 * To ensure coherence of local and remote memory writes of work-groups in 6193 different agents a ``buffer_wbl2`` is required. It will writeback dirty L2 6194 cache lines of MTYPE RW (used for local coarse grain memory) and MTYPE NC 6195 ()used for remote coarse grain memory). Note that MTYPE CC (used for local 6196 fine grain memory) causes write through to DRAM, and MTYPE UC (used for 6197 remote fine grain memory) bypasses the L2, so both will never result in 6198 dirty L2 cache lines. 6199 * To ensure coherence of local and remote memory reads of work-groups in 6200 different agents a ``buffer_invl2`` is required. It will invalidate L2 6201 cache lines with MTYPE NC (used for remote coarse grain memory). Note that 6202 MTYPE CC (used for local fine grain memory) and MTYPE RW (used for local 6203 coarse memory) cause local reads to be invalidated by remote writes with 6204 with the PTE C-bit so these cache lines are not invalidated. Note that 6205 MTYPE UC (used for remote fine grain memory) bypasses the L2, so will 6206 never result in L2 cache lines that need to be invalidated. 6207 6208 * PCIe access from the GPU to the CPU memory is kept coherent by using the 6209 MTYPE UC (uncached) which bypasses the L2. 6210 6211Scalar memory operations are only used to access memory that is proven to not 6212change during the execution of the kernel dispatch. This includes constant 6213address space and global address space for program scope ``const`` variables. 6214Therefore, the kernel machine code does not have to maintain the scalar cache to 6215ensure it is coherent with the vector caches. The scalar and vector caches are 6216invalidated between kernel dispatches by CP since constant address space data 6217may change between kernel dispatch executions. See 6218:ref:`amdgpu-amdhsa-memory-spaces`. 6219 6220The one exception is if scalar writes are used to spill SGPR registers. In this 6221case the AMDGPU backend ensures the memory location used to spill is never 6222accessed by vector memory operations at the same time. If scalar writes are used 6223then a ``s_dcache_wb`` is inserted before the ``s_endpgm`` and before a function 6224return since the locations may be used for vector memory instructions by a 6225future wavefront that uses the same scratch area, or a function call that 6226creates a frame at the same address, respectively. There is no need for a 6227``s_dcache_inv`` as all scalar writes are write-before-read in the same thread. 6228 6229For kernarg backing memory: 6230 6231* CP invalidates the L1 cache at the start of each kernel dispatch. 6232* On dGPU over XGMI or PCIe the kernarg backing memory is allocated in host 6233 memory accessed as MTYPE UC (uncached) to avoid needing to invalidate the L2 6234 cache. This also causes it to be treated as non-volatile and so is not 6235 invalidated by ``*_vol``. 6236* On APU the kernarg backing memory is accessed as MTYPE CC (cache coherent) and 6237 so the L2 cache will be coherent with the CPU and other agents. 6238 6239Scratch backing memory (which is used for the private address space) is accessed 6240with MTYPE NC_NV (non-coherent non-volatile). Since the private address space is 6241only accessed by a single thread, and is always write-before-read, there is 6242never a need to invalidate these entries from the L1 cache. Hence all cache 6243invalidates are done as ``*_vol`` to only invalidate the volatile cache lines. 6244 6245The code sequences used to implement the memory model for GFX90A are defined 6246in table :ref:`amdgpu-amdhsa-memory-model-code-sequences-gfx90a-table`. 6247 6248 .. table:: AMDHSA Memory Model Code Sequences GFX90A 6249 :name: amdgpu-amdhsa-memory-model-code-sequences-gfx90a-table 6250 6251 ============ ============ ============== ========== ================================ 6252 LLVM Instr LLVM Memory LLVM Memory AMDGPU AMDGPU Machine Code 6253 Ordering Sync Scope Address GFX90A 6254 Space 6255 ============ ============ ============== ========== ================================ 6256 **Non-Atomic** 6257 ------------------------------------------------------------------------------------ 6258 load *none* *none* - global - !volatile & !nontemporal 6259 - generic 6260 - private 1. buffer/global/flat_load 6261 - constant 6262 - !volatile & nontemporal 6263 6264 1. buffer/global/flat_load 6265 glc=1 slc=1 6266 6267 - volatile 6268 6269 1. buffer/global/flat_load 6270 glc=1 6271 2. s_waitcnt vmcnt(0) 6272 6273 - Must happen before 6274 any following volatile 6275 global/generic 6276 load/store. 6277 - Ensures that 6278 volatile 6279 operations to 6280 different 6281 addresses will not 6282 be reordered by 6283 hardware. 6284 6285 load *none* *none* - local 1. ds_load 6286 store *none* *none* - global - !volatile & !nontemporal 6287 - generic 6288 - private 1. buffer/global/flat_store 6289 - constant 6290 - !volatile & nontemporal 6291 6292 1. buffer/global/flat_store 6293 glc=1 slc=1 6294 6295 - volatile 6296 6297 1. buffer/global/flat_store 6298 2. s_waitcnt vmcnt(0) 6299 6300 - Must happen before 6301 any following volatile 6302 global/generic 6303 load/store. 6304 - Ensures that 6305 volatile 6306 operations to 6307 different 6308 addresses will not 6309 be reordered by 6310 hardware. 6311 6312 store *none* *none* - local 1. ds_store 6313 **Unordered Atomic** 6314 ------------------------------------------------------------------------------------ 6315 load atomic unordered *any* *any* *Same as non-atomic*. 6316 store atomic unordered *any* *any* *Same as non-atomic*. 6317 atomicrmw unordered *any* *any* *Same as monotonic atomic*. 6318 **Monotonic Atomic** 6319 ------------------------------------------------------------------------------------ 6320 load atomic monotonic - singlethread - global 1. buffer/global/flat_load 6321 - wavefront - generic 6322 load atomic monotonic - workgroup - global 1. buffer/global/flat_load 6323 - generic glc=1 6324 6325 - If not TgSplit execution 6326 mode, omit glc=1. 6327 6328 load atomic monotonic - singlethread - local *If TgSplit execution mode, 6329 - wavefront local address space cannot 6330 - workgroup be used.* 6331 6332 1. ds_load 6333 load atomic monotonic - agent - global 1. buffer/global/flat_load 6334 - generic glc=1 6335 load atomic monotonic - system - global 1. buffer/global/flat_load 6336 - generic glc=1 6337 store atomic monotonic - singlethread - global 1. buffer/global/flat_store 6338 - wavefront - generic 6339 - workgroup 6340 - agent 6341 store atomic monotonic - system - global 1. buffer/global/flat_store 6342 - generic 6343 store atomic monotonic - singlethread - local *If TgSplit execution mode, 6344 - wavefront local address space cannot 6345 - workgroup be used.* 6346 6347 1. ds_store 6348 atomicrmw monotonic - singlethread - global 1. buffer/global/flat_atomic 6349 - wavefront - generic 6350 - workgroup 6351 - agent 6352 atomicrmw monotonic - system - global 1. buffer/global/flat_atomic 6353 - generic 6354 atomicrmw monotonic - singlethread - local *If TgSplit execution mode, 6355 - wavefront local address space cannot 6356 - workgroup be used.* 6357 6358 1. ds_atomic 6359 **Acquire Atomic** 6360 ------------------------------------------------------------------------------------ 6361 load atomic acquire - singlethread - global 1. buffer/global/ds/flat_load 6362 - wavefront - local 6363 - generic 6364 load atomic acquire - workgroup - global 1. buffer/global_load glc=1 6365 6366 - If not TgSplit execution 6367 mode, omit glc=1. 6368 6369 2. s_waitcnt vmcnt(0) 6370 6371 - If not TgSplit execution 6372 mode, omit. 6373 - Must happen before the 6374 following buffer_wbinvl1_vol. 6375 6376 3. buffer_wbinvl1_vol 6377 6378 - If not TgSplit execution 6379 mode, omit. 6380 - Must happen before 6381 any following 6382 global/generic 6383 load/load 6384 atomic/store/store 6385 atomic/atomicrmw. 6386 - Ensures that 6387 following 6388 loads will not see 6389 stale data. 6390 6391 load atomic acquire - workgroup - local *If TgSplit execution mode, 6392 local address space cannot 6393 be used.* 6394 6395 1. ds_load 6396 2. s_waitcnt lgkmcnt(0) 6397 6398 - If OpenCL, omit. 6399 - Must happen before 6400 any following 6401 global/generic 6402 load/load 6403 atomic/store/store 6404 atomic/atomicrmw. 6405 - Ensures any 6406 following global 6407 data read is no 6408 older than the local load 6409 atomic value being 6410 acquired. 6411 6412 load atomic acquire - workgroup - generic 1. flat_load glc=1 6413 6414 - If not TgSplit execution 6415 mode, omit glc=1. 6416 6417 2. s_waitcnt lgkm/vmcnt(0) 6418 6419 - Use lgkmcnt(0) if not 6420 TgSplit execution mode 6421 and vmcnt(0) if TgSplit 6422 execution mode. 6423 - If OpenCL, omit lgkmcnt(0). 6424 - Must happen before 6425 the following 6426 buffer_wbinvl1_vol and any 6427 following global/generic 6428 load/load 6429 atomic/store/store 6430 atomic/atomicrmw. 6431 - Ensures any 6432 following global 6433 data read is no 6434 older than a local load 6435 atomic value being 6436 acquired. 6437 6438 3. buffer_wbinvl1_vol 6439 6440 - If not TgSplit execution 6441 mode, omit. 6442 - Ensures that 6443 following 6444 loads will not see 6445 stale data. 6446 6447 load atomic acquire - agent - global 1. buffer/global_load 6448 glc=1 6449 2. s_waitcnt vmcnt(0) 6450 6451 - Must happen before 6452 following 6453 buffer_wbinvl1_vol. 6454 - Ensures the load 6455 has completed 6456 before invalidating 6457 the cache. 6458 6459 3. buffer_wbinvl1_vol 6460 6461 - Must happen before 6462 any following 6463 global/generic 6464 load/load 6465 atomic/atomicrmw. 6466 - Ensures that 6467 following 6468 loads will not see 6469 stale global data. 6470 6471 load atomic acquire - system - global 1. buffer/global/flat_load 6472 glc=1 6473 2. s_waitcnt vmcnt(0) 6474 6475 - Must happen before 6476 following buffer_invl2 and 6477 buffer_wbinvl1_vol. 6478 - Ensures the load 6479 has completed 6480 before invalidating 6481 the cache. 6482 6483 3. buffer_invl2; 6484 buffer_wbinvl1_vol 6485 6486 - Must happen before 6487 any following 6488 global/generic 6489 load/load 6490 atomic/atomicrmw. 6491 - Ensures that 6492 following 6493 loads will not see 6494 stale L1 global data, 6495 nor see stale L2 MTYPE 6496 NC global data. 6497 MTYPE RW and CC memory will 6498 never be stale in L2 due to 6499 the memory probes. 6500 6501 load atomic acquire - agent - generic 1. flat_load glc=1 6502 2. s_waitcnt vmcnt(0) & 6503 lgkmcnt(0) 6504 6505 - If TgSplit execution mode, 6506 omit lgkmcnt(0). 6507 - If OpenCL omit 6508 lgkmcnt(0). 6509 - Must happen before 6510 following 6511 buffer_wbinvl1_vol. 6512 - Ensures the flat_load 6513 has completed 6514 before invalidating 6515 the cache. 6516 6517 3. buffer_wbinvl1_vol 6518 6519 - Must happen before 6520 any following 6521 global/generic 6522 load/load 6523 atomic/atomicrmw. 6524 - Ensures that 6525 following loads 6526 will not see stale 6527 global data. 6528 6529 load atomic acquire - system - generic 1. flat_load glc=1 6530 2. s_waitcnt vmcnt(0) & 6531 lgkmcnt(0) 6532 6533 - If TgSplit execution mode, 6534 omit lgkmcnt(0). 6535 - If OpenCL omit 6536 lgkmcnt(0). 6537 - Must happen before 6538 following 6539 buffer_invl2 and 6540 buffer_wbinvl1_vol. 6541 - Ensures the flat_load 6542 has completed 6543 before invalidating 6544 the caches. 6545 6546 3. buffer_invl2; 6547 buffer_wbinvl1_vol 6548 6549 - Must happen before 6550 any following 6551 global/generic 6552 load/load 6553 atomic/atomicrmw. 6554 - Ensures that 6555 following 6556 loads will not see 6557 stale L1 global data, 6558 nor see stale L2 MTYPE 6559 NC global data. 6560 MTYPE RW and CC memory will 6561 never be stale in L2 due to 6562 the memory probes. 6563 6564 atomicrmw acquire - singlethread - global 1. buffer/global/flat_atomic 6565 - wavefront - generic 6566 atomicrmw acquire - singlethread - local *If TgSplit execution mode, 6567 - wavefront local address space cannot 6568 be used.* 6569 6570 1. ds_atomic 6571 atomicrmw acquire - workgroup - global 1. buffer/global_atomic 6572 2. s_waitcnt vmcnt(0) 6573 6574 - If not TgSplit execution 6575 mode, omit. 6576 - Must happen before the 6577 following buffer_wbinvl1_vol. 6578 - Ensures the atomicrmw 6579 has completed 6580 before invalidating 6581 the cache. 6582 6583 3. buffer_wbinvl1_vol 6584 6585 - If not TgSplit execution 6586 mode, omit. 6587 - Must happen before 6588 any following 6589 global/generic 6590 load/load 6591 atomic/atomicrmw. 6592 - Ensures that 6593 following loads 6594 will not see stale 6595 global data. 6596 6597 atomicrmw acquire - workgroup - local *If TgSplit execution mode, 6598 local address space cannot 6599 be used.* 6600 6601 1. ds_atomic 6602 2. s_waitcnt lgkmcnt(0) 6603 6604 - If OpenCL, omit. 6605 - Must happen before 6606 any following 6607 global/generic 6608 load/load 6609 atomic/store/store 6610 atomic/atomicrmw. 6611 - Ensures any 6612 following global 6613 data read is no 6614 older than the local 6615 atomicrmw value 6616 being acquired. 6617 6618 atomicrmw acquire - workgroup - generic 1. flat_atomic 6619 2. s_waitcnt lgkm/vmcnt(0) 6620 6621 - Use lgkmcnt(0) if not 6622 TgSplit execution mode 6623 and vmcnt(0) if TgSplit 6624 execution mode. 6625 - If OpenCL, omit lgkmcnt(0). 6626 - Must happen before 6627 the following 6628 buffer_wbinvl1_vol and 6629 any following 6630 global/generic 6631 load/load 6632 atomic/store/store 6633 atomic/atomicrmw. 6634 - Ensures any 6635 following global 6636 data read is no 6637 older than a local 6638 atomicrmw value 6639 being acquired. 6640 6641 3. buffer_wbinvl1_vol 6642 6643 - If not TgSplit execution 6644 mode, omit. 6645 - Ensures that 6646 following 6647 loads will not see 6648 stale data. 6649 6650 atomicrmw acquire - agent - global 1. buffer/global_atomic 6651 2. s_waitcnt vmcnt(0) 6652 6653 - Must happen before 6654 following 6655 buffer_wbinvl1_vol. 6656 - Ensures the 6657 atomicrmw has 6658 completed before 6659 invalidating the 6660 cache. 6661 6662 3. buffer_wbinvl1_vol 6663 6664 - Must happen before 6665 any following 6666 global/generic 6667 load/load 6668 atomic/atomicrmw. 6669 - Ensures that 6670 following loads 6671 will not see stale 6672 global data. 6673 6674 atomicrmw acquire - system - global 1. buffer/global_atomic 6675 2. s_waitcnt vmcnt(0) 6676 6677 - Must happen before 6678 following buffer_invl2 and 6679 buffer_wbinvl1_vol. 6680 - Ensures the 6681 atomicrmw has 6682 completed before 6683 invalidating the 6684 caches. 6685 6686 3. buffer_invl2; 6687 buffer_wbinvl1_vol 6688 6689 - Must happen before 6690 any following 6691 global/generic 6692 load/load 6693 atomic/atomicrmw. 6694 - Ensures that 6695 following 6696 loads will not see 6697 stale L1 global data, 6698 nor see stale L2 MTYPE 6699 NC global data. 6700 MTYPE RW and CC memory will 6701 never be stale in L2 due to 6702 the memory probes. 6703 6704 atomicrmw acquire - agent - generic 1. flat_atomic 6705 2. s_waitcnt vmcnt(0) & 6706 lgkmcnt(0) 6707 6708 - If TgSplit execution mode, 6709 omit lgkmcnt(0). 6710 - If OpenCL, omit 6711 lgkmcnt(0). 6712 - Must happen before 6713 following 6714 buffer_wbinvl1_vol. 6715 - Ensures the 6716 atomicrmw has 6717 completed before 6718 invalidating the 6719 cache. 6720 6721 3. buffer_wbinvl1_vol 6722 6723 - Must happen before 6724 any following 6725 global/generic 6726 load/load 6727 atomic/atomicrmw. 6728 - Ensures that 6729 following loads 6730 will not see stale 6731 global data. 6732 6733 atomicrmw acquire - system - generic 1. flat_atomic 6734 2. s_waitcnt vmcnt(0) & 6735 lgkmcnt(0) 6736 6737 - If TgSplit execution mode, 6738 omit lgkmcnt(0). 6739 - If OpenCL, omit 6740 lgkmcnt(0). 6741 - Must happen before 6742 following 6743 buffer_invl2 and 6744 buffer_wbinvl1_vol. 6745 - Ensures the 6746 atomicrmw has 6747 completed before 6748 invalidating the 6749 caches. 6750 6751 3. buffer_invl2; 6752 buffer_wbinvl1_vol 6753 6754 - Must happen before 6755 any following 6756 global/generic 6757 load/load 6758 atomic/atomicrmw. 6759 - Ensures that 6760 following 6761 loads will not see 6762 stale L1 global data, 6763 nor see stale L2 MTYPE 6764 NC global data. 6765 MTYPE RW and CC memory will 6766 never be stale in L2 due to 6767 the memory probes. 6768 6769 fence acquire - singlethread *none* *none* 6770 - wavefront 6771 fence acquire - workgroup *none* 1. s_waitcnt lgkm/vmcnt(0) 6772 6773 - Use lgkmcnt(0) if not 6774 TgSplit execution mode 6775 and vmcnt(0) if TgSplit 6776 execution mode. 6777 - If OpenCL and 6778 address space is 6779 not generic, omit 6780 lgkmcnt(0). 6781 - If OpenCL and 6782 address space is 6783 local, omit 6784 vmcnt(0). 6785 - However, since LLVM 6786 currently has no 6787 address space on 6788 the fence need to 6789 conservatively 6790 always generate. If 6791 fence had an 6792 address space then 6793 set to address 6794 space of OpenCL 6795 fence flag, or to 6796 generic if both 6797 local and global 6798 flags are 6799 specified. 6800 - s_waitcnt vmcnt(0) 6801 must happen after 6802 any preceding 6803 global/generic load 6804 atomic/ 6805 atomicrmw 6806 with an equal or 6807 wider sync scope 6808 and memory ordering 6809 stronger than 6810 unordered (this is 6811 termed the 6812 fence-paired-atomic). 6813 - s_waitcnt lgkmcnt(0) 6814 must happen after 6815 any preceding 6816 local/generic load 6817 atomic/atomicrmw 6818 with an equal or 6819 wider sync scope 6820 and memory ordering 6821 stronger than 6822 unordered (this is 6823 termed the 6824 fence-paired-atomic). 6825 - Must happen before 6826 the following 6827 buffer_wbinvl1_vol and 6828 any following 6829 global/generic 6830 load/load 6831 atomic/store/store 6832 atomic/atomicrmw. 6833 - Ensures any 6834 following global 6835 data read is no 6836 older than the 6837 value read by the 6838 fence-paired-atomic. 6839 6840 2. buffer_wbinvl1_vol 6841 6842 - If not TgSplit execution 6843 mode, omit. 6844 - Ensures that 6845 following 6846 loads will not see 6847 stale data. 6848 6849 fence acquire - agent *none* 1. s_waitcnt lgkmcnt(0) & 6850 vmcnt(0) 6851 6852 - If TgSplit execution mode, 6853 omit lgkmcnt(0). 6854 - If OpenCL and 6855 address space is 6856 not generic, omit 6857 lgkmcnt(0). 6858 - However, since LLVM 6859 currently has no 6860 address space on 6861 the fence need to 6862 conservatively 6863 always generate 6864 (see comment for 6865 previous fence). 6866 - Could be split into 6867 separate s_waitcnt 6868 vmcnt(0) and 6869 s_waitcnt 6870 lgkmcnt(0) to allow 6871 them to be 6872 independently moved 6873 according to the 6874 following rules. 6875 - s_waitcnt vmcnt(0) 6876 must happen after 6877 any preceding 6878 global/generic load 6879 atomic/atomicrmw 6880 with an equal or 6881 wider sync scope 6882 and memory ordering 6883 stronger than 6884 unordered (this is 6885 termed the 6886 fence-paired-atomic). 6887 - s_waitcnt lgkmcnt(0) 6888 must happen after 6889 any preceding 6890 local/generic load 6891 atomic/atomicrmw 6892 with an equal or 6893 wider sync scope 6894 and memory ordering 6895 stronger than 6896 unordered (this is 6897 termed the 6898 fence-paired-atomic). 6899 - Must happen before 6900 the following 6901 buffer_wbinvl1_vol. 6902 - Ensures that the 6903 fence-paired atomic 6904 has completed 6905 before invalidating 6906 the 6907 cache. Therefore 6908 any following 6909 locations read must 6910 be no older than 6911 the value read by 6912 the 6913 fence-paired-atomic. 6914 6915 2. buffer_wbinvl1_vol 6916 6917 - Must happen before any 6918 following global/generic 6919 load/load 6920 atomic/store/store 6921 atomic/atomicrmw. 6922 - Ensures that 6923 following loads 6924 will not see stale 6925 global data. 6926 6927 fence acquire - system *none* 1. s_waitcnt lgkmcnt(0) & 6928 vmcnt(0) 6929 6930 - If TgSplit execution mode, 6931 omit lgkmcnt(0). 6932 - If OpenCL and 6933 address space is 6934 not generic, omit 6935 lgkmcnt(0). 6936 - However, since LLVM 6937 currently has no 6938 address space on 6939 the fence need to 6940 conservatively 6941 always generate 6942 (see comment for 6943 previous fence). 6944 - Could be split into 6945 separate s_waitcnt 6946 vmcnt(0) and 6947 s_waitcnt 6948 lgkmcnt(0) to allow 6949 them to be 6950 independently moved 6951 according to the 6952 following rules. 6953 - s_waitcnt vmcnt(0) 6954 must happen after 6955 any preceding 6956 global/generic load 6957 atomic/atomicrmw 6958 with an equal or 6959 wider sync scope 6960 and memory ordering 6961 stronger than 6962 unordered (this is 6963 termed the 6964 fence-paired-atomic). 6965 - s_waitcnt lgkmcnt(0) 6966 must happen after 6967 any preceding 6968 local/generic load 6969 atomic/atomicrmw 6970 with an equal or 6971 wider sync scope 6972 and memory ordering 6973 stronger than 6974 unordered (this is 6975 termed the 6976 fence-paired-atomic). 6977 - Must happen before 6978 the following buffer_invl2 and 6979 buffer_wbinvl1_vol. 6980 - Ensures that the 6981 fence-paired atomic 6982 has completed 6983 before invalidating 6984 the 6985 cache. Therefore 6986 any following 6987 locations read must 6988 be no older than 6989 the value read by 6990 the 6991 fence-paired-atomic. 6992 6993 2. buffer_invl2; 6994 buffer_wbinvl1_vol 6995 6996 - Must happen before any 6997 following global/generic 6998 load/load 6999 atomic/store/store 7000 atomic/atomicrmw. 7001 - Ensures that 7002 following 7003 loads will not see 7004 stale L1 global data, 7005 nor see stale L2 MTYPE 7006 NC global data. 7007 MTYPE RW and CC memory will 7008 never be stale in L2 due to 7009 the memory probes. 7010 **Release Atomic** 7011 ------------------------------------------------------------------------------------ 7012 store atomic release - singlethread - global 1. buffer/global/flat_store 7013 - wavefront - generic 7014 store atomic release - singlethread - local *If TgSplit execution mode, 7015 - wavefront local address space cannot 7016 be used.* 7017 7018 1. ds_store 7019 store atomic release - workgroup - global 1. s_waitcnt lgkm/vmcnt(0) 7020 - generic 7021 - Use lgkmcnt(0) if not 7022 TgSplit execution mode 7023 and vmcnt(0) if TgSplit 7024 execution mode. 7025 - If OpenCL, omit lgkmcnt(0). 7026 - s_waitcnt vmcnt(0) 7027 must happen after 7028 any preceding 7029 global/generic load/store/ 7030 load atomic/store atomic/ 7031 atomicrmw. 7032 - s_waitcnt lgkmcnt(0) 7033 must happen after 7034 any preceding 7035 local/generic 7036 load/store/load 7037 atomic/store 7038 atomic/atomicrmw. 7039 - Must happen before 7040 the following 7041 store. 7042 - Ensures that all 7043 memory operations 7044 have 7045 completed before 7046 performing the 7047 store that is being 7048 released. 7049 7050 2. buffer/global/flat_store 7051 store atomic release - workgroup - local *If TgSplit execution mode, 7052 local address space cannot 7053 be used.* 7054 7055 1. ds_store 7056 store atomic release - agent - global 1. s_waitcnt lgkmcnt(0) & 7057 - generic vmcnt(0) 7058 7059 - If TgSplit execution mode, 7060 omit lgkmcnt(0). 7061 - If OpenCL and 7062 address space is 7063 not generic, omit 7064 lgkmcnt(0). 7065 - Could be split into 7066 separate s_waitcnt 7067 vmcnt(0) and 7068 s_waitcnt 7069 lgkmcnt(0) to allow 7070 them to be 7071 independently moved 7072 according to the 7073 following rules. 7074 - s_waitcnt vmcnt(0) 7075 must happen after 7076 any preceding 7077 global/generic 7078 load/store/load 7079 atomic/store 7080 atomic/atomicrmw. 7081 - s_waitcnt lgkmcnt(0) 7082 must happen after 7083 any preceding 7084 local/generic 7085 load/store/load 7086 atomic/store 7087 atomic/atomicrmw. 7088 - Must happen before 7089 the following 7090 store. 7091 - Ensures that all 7092 memory operations 7093 to memory have 7094 completed before 7095 performing the 7096 store that is being 7097 released. 7098 7099 2. buffer/global/flat_store 7100 store atomic release - system - global 1. buffer_wbl2 7101 - generic 7102 - Must happen before 7103 following s_waitcnt. 7104 - Performs L2 writeback to 7105 ensure previous 7106 global/generic 7107 store/atomicrmw are 7108 visible at system scope. 7109 7110 2. s_waitcnt lgkmcnt(0) & 7111 vmcnt(0) 7112 7113 - If TgSplit execution mode, 7114 omit lgkmcnt(0). 7115 - If OpenCL and 7116 address space is 7117 not generic, omit 7118 lgkmcnt(0). 7119 - Could be split into 7120 separate s_waitcnt 7121 vmcnt(0) and 7122 s_waitcnt 7123 lgkmcnt(0) to allow 7124 them to be 7125 independently moved 7126 according to the 7127 following rules. 7128 - s_waitcnt vmcnt(0) 7129 must happen after any 7130 preceding 7131 global/generic 7132 load/store/load 7133 atomic/store 7134 atomic/atomicrmw. 7135 - s_waitcnt lgkmcnt(0) 7136 must happen after any 7137 preceding 7138 local/generic 7139 load/store/load 7140 atomic/store 7141 atomic/atomicrmw. 7142 - Must happen before 7143 the following 7144 store. 7145 - Ensures that all 7146 memory operations 7147 to memory and the L2 7148 writeback have 7149 completed before 7150 performing the 7151 store that is being 7152 released. 7153 7154 3. buffer/global/flat_store 7155 atomicrmw release - singlethread - global 1. buffer/global/flat_atomic 7156 - wavefront - generic 7157 atomicrmw release - singlethread - local *If TgSplit execution mode, 7158 - wavefront local address space cannot 7159 be used.* 7160 7161 1. ds_atomic 7162 atomicrmw release - workgroup - global 1. s_waitcnt lgkm/vmcnt(0) 7163 - generic 7164 - Use lgkmcnt(0) if not 7165 TgSplit execution mode 7166 and vmcnt(0) if TgSplit 7167 execution mode. 7168 - If OpenCL, omit 7169 lgkmcnt(0). 7170 - s_waitcnt vmcnt(0) 7171 must happen after 7172 any preceding 7173 global/generic load/store/ 7174 load atomic/store atomic/ 7175 atomicrmw. 7176 - s_waitcnt lgkmcnt(0) 7177 must happen after 7178 any preceding 7179 local/generic 7180 load/store/load 7181 atomic/store 7182 atomic/atomicrmw. 7183 - Must happen before 7184 the following 7185 atomicrmw. 7186 - Ensures that all 7187 memory operations 7188 have 7189 completed before 7190 performing the 7191 atomicrmw that is 7192 being released. 7193 7194 2. buffer/global/flat_atomic 7195 atomicrmw release - workgroup - local *If TgSplit execution mode, 7196 local address space cannot 7197 be used.* 7198 7199 1. ds_atomic 7200 atomicrmw release - agent - global 1. s_waitcnt lgkmcnt(0) & 7201 - generic vmcnt(0) 7202 7203 - If TgSplit execution mode, 7204 omit lgkmcnt(0). 7205 - If OpenCL, omit 7206 lgkmcnt(0). 7207 - Could be split into 7208 separate s_waitcnt 7209 vmcnt(0) and 7210 s_waitcnt 7211 lgkmcnt(0) to allow 7212 them to be 7213 independently moved 7214 according to the 7215 following rules. 7216 - s_waitcnt vmcnt(0) 7217 must happen after 7218 any preceding 7219 global/generic 7220 load/store/load 7221 atomic/store 7222 atomic/atomicrmw. 7223 - s_waitcnt lgkmcnt(0) 7224 must happen after 7225 any preceding 7226 local/generic 7227 load/store/load 7228 atomic/store 7229 atomic/atomicrmw. 7230 - Must happen before 7231 the following 7232 atomicrmw. 7233 - Ensures that all 7234 memory operations 7235 to global and local 7236 have completed 7237 before performing 7238 the atomicrmw that 7239 is being released. 7240 7241 2. buffer/global/flat_atomic 7242 atomicrmw release - system - global 1. buffer_wbl2 7243 - generic 7244 - Must happen before 7245 following s_waitcnt. 7246 - Performs L2 writeback to 7247 ensure previous 7248 global/generic 7249 store/atomicrmw are 7250 visible at system scope. 7251 7252 2. s_waitcnt lgkmcnt(0) & 7253 vmcnt(0) 7254 7255 - If TgSplit execution mode, 7256 omit lgkmcnt(0). 7257 - If OpenCL, omit 7258 lgkmcnt(0). 7259 - Could be split into 7260 separate s_waitcnt 7261 vmcnt(0) and 7262 s_waitcnt 7263 lgkmcnt(0) to allow 7264 them to be 7265 independently moved 7266 according to the 7267 following rules. 7268 - s_waitcnt vmcnt(0) 7269 must happen after 7270 any preceding 7271 global/generic 7272 load/store/load 7273 atomic/store 7274 atomic/atomicrmw. 7275 - s_waitcnt lgkmcnt(0) 7276 must happen after 7277 any preceding 7278 local/generic 7279 load/store/load 7280 atomic/store 7281 atomic/atomicrmw. 7282 - Must happen before 7283 the following 7284 atomicrmw. 7285 - Ensures that all 7286 memory operations 7287 to memory and the L2 7288 writeback have 7289 completed before 7290 performing the 7291 store that is being 7292 released. 7293 7294 3. buffer/global/flat_atomic 7295 fence release - singlethread *none* *none* 7296 - wavefront 7297 fence release - workgroup *none* 1. s_waitcnt lgkm/vmcnt(0) 7298 7299 - Use lgkmcnt(0) if not 7300 TgSplit execution mode 7301 and vmcnt(0) if TgSplit 7302 execution mode. 7303 - If OpenCL and 7304 address space is 7305 not generic, omit 7306 lgkmcnt(0). 7307 - If OpenCL and 7308 address space is 7309 local, omit 7310 vmcnt(0). 7311 - However, since LLVM 7312 currently has no 7313 address space on 7314 the fence need to 7315 conservatively 7316 always generate. If 7317 fence had an 7318 address space then 7319 set to address 7320 space of OpenCL 7321 fence flag, or to 7322 generic if both 7323 local and global 7324 flags are 7325 specified. 7326 - s_waitcnt vmcnt(0) 7327 must happen after 7328 any preceding 7329 global/generic 7330 load/store/ 7331 load atomic/store atomic/ 7332 atomicrmw. 7333 - s_waitcnt lgkmcnt(0) 7334 must happen after 7335 any preceding 7336 local/generic 7337 load/load 7338 atomic/store/store 7339 atomic/atomicrmw. 7340 - Must happen before 7341 any following store 7342 atomic/atomicrmw 7343 with an equal or 7344 wider sync scope 7345 and memory ordering 7346 stronger than 7347 unordered (this is 7348 termed the 7349 fence-paired-atomic). 7350 - Ensures that all 7351 memory operations 7352 have 7353 completed before 7354 performing the 7355 following 7356 fence-paired-atomic. 7357 7358 fence release - agent *none* 1. s_waitcnt lgkmcnt(0) & 7359 vmcnt(0) 7360 7361 - If TgSplit execution mode, 7362 omit lgkmcnt(0). 7363 - If OpenCL and 7364 address space is 7365 not generic, omit 7366 lgkmcnt(0). 7367 - If OpenCL and 7368 address space is 7369 local, omit 7370 vmcnt(0). 7371 - However, since LLVM 7372 currently has no 7373 address space on 7374 the fence need to 7375 conservatively 7376 always generate. If 7377 fence had an 7378 address space then 7379 set to address 7380 space of OpenCL 7381 fence flag, or to 7382 generic if both 7383 local and global 7384 flags are 7385 specified. 7386 - Could be split into 7387 separate s_waitcnt 7388 vmcnt(0) and 7389 s_waitcnt 7390 lgkmcnt(0) to allow 7391 them to be 7392 independently moved 7393 according to the 7394 following rules. 7395 - s_waitcnt vmcnt(0) 7396 must happen after 7397 any preceding 7398 global/generic 7399 load/store/load 7400 atomic/store 7401 atomic/atomicrmw. 7402 - s_waitcnt lgkmcnt(0) 7403 must happen after 7404 any preceding 7405 local/generic 7406 load/store/load 7407 atomic/store 7408 atomic/atomicrmw. 7409 - Must happen before 7410 any following store 7411 atomic/atomicrmw 7412 with an equal or 7413 wider sync scope 7414 and memory ordering 7415 stronger than 7416 unordered (this is 7417 termed the 7418 fence-paired-atomic). 7419 - Ensures that all 7420 memory operations 7421 have 7422 completed before 7423 performing the 7424 following 7425 fence-paired-atomic. 7426 7427 fence release - system *none* 1. buffer_wbl2 7428 7429 - If OpenCL and 7430 address space is 7431 local, omit. 7432 - Must happen before 7433 following s_waitcnt. 7434 - Performs L2 writeback to 7435 ensure previous 7436 global/generic 7437 store/atomicrmw are 7438 visible at system scope. 7439 7440 2. s_waitcnt lgkmcnt(0) & 7441 vmcnt(0) 7442 7443 - If TgSplit execution mode, 7444 omit lgkmcnt(0). 7445 - If OpenCL and 7446 address space is 7447 not generic, omit 7448 lgkmcnt(0). 7449 - If OpenCL and 7450 address space is 7451 local, omit 7452 vmcnt(0). 7453 - However, since LLVM 7454 currently has no 7455 address space on 7456 the fence need to 7457 conservatively 7458 always generate. If 7459 fence had an 7460 address space then 7461 set to address 7462 space of OpenCL 7463 fence flag, or to 7464 generic if both 7465 local and global 7466 flags are 7467 specified. 7468 - Could be split into 7469 separate s_waitcnt 7470 vmcnt(0) and 7471 s_waitcnt 7472 lgkmcnt(0) to allow 7473 them to be 7474 independently moved 7475 according to the 7476 following rules. 7477 - s_waitcnt vmcnt(0) 7478 must happen after 7479 any preceding 7480 global/generic 7481 load/store/load 7482 atomic/store 7483 atomic/atomicrmw. 7484 - s_waitcnt lgkmcnt(0) 7485 must happen after 7486 any preceding 7487 local/generic 7488 load/store/load 7489 atomic/store 7490 atomic/atomicrmw. 7491 - Must happen before 7492 any following store 7493 atomic/atomicrmw 7494 with an equal or 7495 wider sync scope 7496 and memory ordering 7497 stronger than 7498 unordered (this is 7499 termed the 7500 fence-paired-atomic). 7501 - Ensures that all 7502 memory operations 7503 have 7504 completed before 7505 performing the 7506 following 7507 fence-paired-atomic. 7508 7509 **Acquire-Release Atomic** 7510 ------------------------------------------------------------------------------------ 7511 atomicrmw acq_rel - singlethread - global 1. buffer/global/flat_atomic 7512 - wavefront - generic 7513 atomicrmw acq_rel - singlethread - local *If TgSplit execution mode, 7514 - wavefront local address space cannot 7515 be used.* 7516 7517 1. ds_atomic 7518 atomicrmw acq_rel - workgroup - global 1. s_waitcnt lgkm/vmcnt(0) 7519 7520 - Use lgkmcnt(0) if not 7521 TgSplit execution mode 7522 and vmcnt(0) if TgSplit 7523 execution mode. 7524 - If OpenCL, omit 7525 lgkmcnt(0). 7526 - Must happen after 7527 any preceding 7528 local/generic 7529 load/store/load 7530 atomic/store 7531 atomic/atomicrmw. 7532 - s_waitcnt vmcnt(0) 7533 must happen after 7534 any preceding 7535 global/generic load/store/ 7536 load atomic/store atomic/ 7537 atomicrmw. 7538 - s_waitcnt lgkmcnt(0) 7539 must happen after 7540 any preceding 7541 local/generic 7542 load/store/load 7543 atomic/store 7544 atomic/atomicrmw. 7545 - Must happen before 7546 the following 7547 atomicrmw. 7548 - Ensures that all 7549 memory operations 7550 have 7551 completed before 7552 performing the 7553 atomicrmw that is 7554 being released. 7555 7556 2. buffer/global_atomic 7557 3. s_waitcnt vmcnt(0) 7558 7559 - If not TgSplit execution 7560 mode, omit. 7561 - Must happen before 7562 the following 7563 buffer_wbinvl1_vol. 7564 - Ensures any 7565 following global 7566 data read is no 7567 older than the 7568 atomicrmw value 7569 being acquired. 7570 7571 4. buffer_wbinvl1_vol 7572 7573 - If not TgSplit execution 7574 mode, omit. 7575 - Ensures that 7576 following 7577 loads will not see 7578 stale data. 7579 7580 atomicrmw acq_rel - workgroup - local *If TgSplit execution mode, 7581 local address space cannot 7582 be used.* 7583 7584 1. ds_atomic 7585 2. s_waitcnt lgkmcnt(0) 7586 7587 - If OpenCL, omit. 7588 - Must happen before 7589 any following 7590 global/generic 7591 load/load 7592 atomic/store/store 7593 atomic/atomicrmw. 7594 - Ensures any 7595 following global 7596 data read is no 7597 older than the local load 7598 atomic value being 7599 acquired. 7600 7601 atomicrmw acq_rel - workgroup - generic 1. s_waitcnt lgkm/vmcnt(0) 7602 7603 - Use lgkmcnt(0) if not 7604 TgSplit execution mode 7605 and vmcnt(0) if TgSplit 7606 execution mode. 7607 - If OpenCL, omit 7608 lgkmcnt(0). 7609 - s_waitcnt vmcnt(0) 7610 must happen after 7611 any preceding 7612 global/generic load/store/ 7613 load atomic/store atomic/ 7614 atomicrmw. 7615 - s_waitcnt lgkmcnt(0) 7616 must happen after 7617 any preceding 7618 local/generic 7619 load/store/load 7620 atomic/store 7621 atomic/atomicrmw. 7622 - Must happen before 7623 the following 7624 atomicrmw. 7625 - Ensures that all 7626 memory operations 7627 have 7628 completed before 7629 performing the 7630 atomicrmw that is 7631 being released. 7632 7633 2. flat_atomic 7634 3. s_waitcnt lgkmcnt(0) & 7635 vmcnt(0) 7636 7637 - If not TgSplit execution 7638 mode, omit vmcnt(0). 7639 - If OpenCL, omit 7640 lgkmcnt(0). 7641 - Must happen before 7642 the following 7643 buffer_wbinvl1_vol and 7644 any following 7645 global/generic 7646 load/load 7647 atomic/store/store 7648 atomic/atomicrmw. 7649 - Ensures any 7650 following global 7651 data read is no 7652 older than a local load 7653 atomic value being 7654 acquired. 7655 7656 3. buffer_wbinvl1_vol 7657 7658 - If not TgSplit execution 7659 mode, omit. 7660 - Ensures that 7661 following 7662 loads will not see 7663 stale data. 7664 7665 atomicrmw acq_rel - agent - global 1. s_waitcnt lgkmcnt(0) & 7666 vmcnt(0) 7667 7668 - If TgSplit execution mode, 7669 omit lgkmcnt(0). 7670 - If OpenCL, omit 7671 lgkmcnt(0). 7672 - Could be split into 7673 separate s_waitcnt 7674 vmcnt(0) and 7675 s_waitcnt 7676 lgkmcnt(0) to allow 7677 them to be 7678 independently moved 7679 according to the 7680 following rules. 7681 - s_waitcnt vmcnt(0) 7682 must happen after 7683 any preceding 7684 global/generic 7685 load/store/load 7686 atomic/store 7687 atomic/atomicrmw. 7688 - s_waitcnt lgkmcnt(0) 7689 must happen after 7690 any preceding 7691 local/generic 7692 load/store/load 7693 atomic/store 7694 atomic/atomicrmw. 7695 - Must happen before 7696 the following 7697 atomicrmw. 7698 - Ensures that all 7699 memory operations 7700 to global have 7701 completed before 7702 performing the 7703 atomicrmw that is 7704 being released. 7705 7706 2. buffer/global_atomic 7707 3. s_waitcnt vmcnt(0) 7708 7709 - Must happen before 7710 following 7711 buffer_wbinvl1_vol. 7712 - Ensures the 7713 atomicrmw has 7714 completed before 7715 invalidating the 7716 cache. 7717 7718 4. buffer_wbinvl1_vol 7719 7720 - Must happen before 7721 any following 7722 global/generic 7723 load/load 7724 atomic/atomicrmw. 7725 - Ensures that 7726 following loads 7727 will not see stale 7728 global data. 7729 7730 atomicrmw acq_rel - system - global 1. buffer_wbl2 7731 7732 - Must happen before 7733 following s_waitcnt. 7734 - Performs L2 writeback to 7735 ensure previous 7736 global/generic 7737 store/atomicrmw are 7738 visible at system scope. 7739 7740 2. s_waitcnt lgkmcnt(0) & 7741 vmcnt(0) 7742 7743 - If TgSplit execution mode, 7744 omit lgkmcnt(0). 7745 - If OpenCL, omit 7746 lgkmcnt(0). 7747 - Could be split into 7748 separate s_waitcnt 7749 vmcnt(0) and 7750 s_waitcnt 7751 lgkmcnt(0) to allow 7752 them to be 7753 independently moved 7754 according to the 7755 following rules. 7756 - s_waitcnt vmcnt(0) 7757 must happen after 7758 any preceding 7759 global/generic 7760 load/store/load 7761 atomic/store 7762 atomic/atomicrmw. 7763 - s_waitcnt lgkmcnt(0) 7764 must happen after 7765 any preceding 7766 local/generic 7767 load/store/load 7768 atomic/store 7769 atomic/atomicrmw. 7770 - Must happen before 7771 the following 7772 atomicrmw. 7773 - Ensures that all 7774 memory operations 7775 to global and L2 writeback 7776 have completed before 7777 performing the 7778 atomicrmw that is 7779 being released. 7780 7781 3. buffer/global_atomic 7782 4. s_waitcnt vmcnt(0) 7783 7784 - Must happen before 7785 following buffer_invl2 and 7786 buffer_wbinvl1_vol. 7787 - Ensures the 7788 atomicrmw has 7789 completed before 7790 invalidating the 7791 caches. 7792 7793 5. buffer_invl2; 7794 buffer_wbinvl1_vol 7795 7796 - Must happen before 7797 any following 7798 global/generic 7799 load/load 7800 atomic/atomicrmw. 7801 - Ensures that 7802 following 7803 loads will not see 7804 stale L1 global data, 7805 nor see stale L2 MTYPE 7806 NC global data. 7807 MTYPE RW and CC memory will 7808 never be stale in L2 due to 7809 the memory probes. 7810 7811 atomicrmw acq_rel - agent - generic 1. s_waitcnt lgkmcnt(0) & 7812 vmcnt(0) 7813 7814 - If TgSplit execution mode, 7815 omit lgkmcnt(0). 7816 - If OpenCL, omit 7817 lgkmcnt(0). 7818 - Could be split into 7819 separate s_waitcnt 7820 vmcnt(0) and 7821 s_waitcnt 7822 lgkmcnt(0) to allow 7823 them to be 7824 independently moved 7825 according to the 7826 following rules. 7827 - s_waitcnt vmcnt(0) 7828 must happen after 7829 any preceding 7830 global/generic 7831 load/store/load 7832 atomic/store 7833 atomic/atomicrmw. 7834 - s_waitcnt lgkmcnt(0) 7835 must happen after 7836 any preceding 7837 local/generic 7838 load/store/load 7839 atomic/store 7840 atomic/atomicrmw. 7841 - Must happen before 7842 the following 7843 atomicrmw. 7844 - Ensures that all 7845 memory operations 7846 to global have 7847 completed before 7848 performing the 7849 atomicrmw that is 7850 being released. 7851 7852 2. flat_atomic 7853 3. s_waitcnt vmcnt(0) & 7854 lgkmcnt(0) 7855 7856 - If TgSplit execution mode, 7857 omit lgkmcnt(0). 7858 - If OpenCL, omit 7859 lgkmcnt(0). 7860 - Must happen before 7861 following 7862 buffer_wbinvl1_vol. 7863 - Ensures the 7864 atomicrmw has 7865 completed before 7866 invalidating the 7867 cache. 7868 7869 4. buffer_wbinvl1_vol 7870 7871 - Must happen before 7872 any following 7873 global/generic 7874 load/load 7875 atomic/atomicrmw. 7876 - Ensures that 7877 following loads 7878 will not see stale 7879 global data. 7880 7881 atomicrmw acq_rel - system - generic 1. buffer_wbl2 7882 7883 - Must happen before 7884 following s_waitcnt. 7885 - Performs L2 writeback to 7886 ensure previous 7887 global/generic 7888 store/atomicrmw are 7889 visible at system scope. 7890 7891 2. s_waitcnt lgkmcnt(0) & 7892 vmcnt(0) 7893 7894 - If TgSplit execution mode, 7895 omit lgkmcnt(0). 7896 - If OpenCL, omit 7897 lgkmcnt(0). 7898 - Could be split into 7899 separate s_waitcnt 7900 vmcnt(0) and 7901 s_waitcnt 7902 lgkmcnt(0) to allow 7903 them to be 7904 independently moved 7905 according to the 7906 following rules. 7907 - s_waitcnt vmcnt(0) 7908 must happen after 7909 any preceding 7910 global/generic 7911 load/store/load 7912 atomic/store 7913 atomic/atomicrmw. 7914 - s_waitcnt lgkmcnt(0) 7915 must happen after 7916 any preceding 7917 local/generic 7918 load/store/load 7919 atomic/store 7920 atomic/atomicrmw. 7921 - Must happen before 7922 the following 7923 atomicrmw. 7924 - Ensures that all 7925 memory operations 7926 to global and L2 writeback 7927 have completed before 7928 performing the 7929 atomicrmw that is 7930 being released. 7931 7932 3. flat_atomic 7933 4. s_waitcnt vmcnt(0) & 7934 lgkmcnt(0) 7935 7936 - If TgSplit execution mode, 7937 omit lgkmcnt(0). 7938 - If OpenCL, omit 7939 lgkmcnt(0). 7940 - Must happen before 7941 following buffer_invl2 and 7942 buffer_wbinvl1_vol. 7943 - Ensures the 7944 atomicrmw has 7945 completed before 7946 invalidating the 7947 caches. 7948 7949 5. buffer_invl2; 7950 buffer_wbinvl1_vol 7951 7952 - Must happen before 7953 any following 7954 global/generic 7955 load/load 7956 atomic/atomicrmw. 7957 - Ensures that 7958 following 7959 loads will not see 7960 stale L1 global data, 7961 nor see stale L2 MTYPE 7962 NC global data. 7963 MTYPE RW and CC memory will 7964 never be stale in L2 due to 7965 the memory probes. 7966 7967 fence acq_rel - singlethread *none* *none* 7968 - wavefront 7969 fence acq_rel - workgroup *none* 1. s_waitcnt lgkm/vmcnt(0) 7970 7971 - Use lgkmcnt(0) if not 7972 TgSplit execution mode 7973 and vmcnt(0) if TgSplit 7974 execution mode. 7975 - If OpenCL and 7976 address space is 7977 not generic, omit 7978 lgkmcnt(0). 7979 - If OpenCL and 7980 address space is 7981 local, omit 7982 vmcnt(0). 7983 - However, 7984 since LLVM 7985 currently has no 7986 address space on 7987 the fence need to 7988 conservatively 7989 always generate 7990 (see comment for 7991 previous fence). 7992 - s_waitcnt vmcnt(0) 7993 must happen after 7994 any preceding 7995 global/generic 7996 load/store/ 7997 load atomic/store atomic/ 7998 atomicrmw. 7999 - s_waitcnt lgkmcnt(0) 8000 must happen after 8001 any preceding 8002 local/generic 8003 load/load 8004 atomic/store/store 8005 atomic/atomicrmw. 8006 - Must happen before 8007 any following 8008 global/generic 8009 load/load 8010 atomic/store/store 8011 atomic/atomicrmw. 8012 - Ensures that all 8013 memory operations 8014 have 8015 completed before 8016 performing any 8017 following global 8018 memory operations. 8019 - Ensures that the 8020 preceding 8021 local/generic load 8022 atomic/atomicrmw 8023 with an equal or 8024 wider sync scope 8025 and memory ordering 8026 stronger than 8027 unordered (this is 8028 termed the 8029 acquire-fence-paired-atomic) 8030 has completed 8031 before following 8032 global memory 8033 operations. This 8034 satisfies the 8035 requirements of 8036 acquire. 8037 - Ensures that all 8038 previous memory 8039 operations have 8040 completed before a 8041 following 8042 local/generic store 8043 atomic/atomicrmw 8044 with an equal or 8045 wider sync scope 8046 and memory ordering 8047 stronger than 8048 unordered (this is 8049 termed the 8050 release-fence-paired-atomic). 8051 This satisfies the 8052 requirements of 8053 release. 8054 - Must happen before 8055 the following 8056 buffer_wbinvl1_vol. 8057 - Ensures that the 8058 acquire-fence-paired 8059 atomic has completed 8060 before invalidating 8061 the 8062 cache. Therefore 8063 any following 8064 locations read must 8065 be no older than 8066 the value read by 8067 the 8068 acquire-fence-paired-atomic. 8069 8070 2. buffer_wbinvl1_vol 8071 8072 - If not TgSplit execution 8073 mode, omit. 8074 - Ensures that 8075 following 8076 loads will not see 8077 stale data. 8078 8079 fence acq_rel - agent *none* 1. s_waitcnt lgkmcnt(0) & 8080 vmcnt(0) 8081 8082 - If TgSplit execution mode, 8083 omit lgkmcnt(0). 8084 - If OpenCL and 8085 address space is 8086 not generic, omit 8087 lgkmcnt(0). 8088 - However, since LLVM 8089 currently has no 8090 address space on 8091 the fence need to 8092 conservatively 8093 always generate 8094 (see comment for 8095 previous fence). 8096 - Could be split into 8097 separate s_waitcnt 8098 vmcnt(0) and 8099 s_waitcnt 8100 lgkmcnt(0) to allow 8101 them to be 8102 independently moved 8103 according to the 8104 following rules. 8105 - s_waitcnt vmcnt(0) 8106 must happen after 8107 any preceding 8108 global/generic 8109 load/store/load 8110 atomic/store 8111 atomic/atomicrmw. 8112 - s_waitcnt lgkmcnt(0) 8113 must happen after 8114 any preceding 8115 local/generic 8116 load/store/load 8117 atomic/store 8118 atomic/atomicrmw. 8119 - Must happen before 8120 the following 8121 buffer_wbinvl1_vol. 8122 - Ensures that the 8123 preceding 8124 global/local/generic 8125 load 8126 atomic/atomicrmw 8127 with an equal or 8128 wider sync scope 8129 and memory ordering 8130 stronger than 8131 unordered (this is 8132 termed the 8133 acquire-fence-paired-atomic) 8134 has completed 8135 before invalidating 8136 the cache. This 8137 satisfies the 8138 requirements of 8139 acquire. 8140 - Ensures that all 8141 previous memory 8142 operations have 8143 completed before a 8144 following 8145 global/local/generic 8146 store 8147 atomic/atomicrmw 8148 with an equal or 8149 wider sync scope 8150 and memory ordering 8151 stronger than 8152 unordered (this is 8153 termed the 8154 release-fence-paired-atomic). 8155 This satisfies the 8156 requirements of 8157 release. 8158 8159 2. buffer_wbinvl1_vol 8160 8161 - Must happen before 8162 any following 8163 global/generic 8164 load/load 8165 atomic/store/store 8166 atomic/atomicrmw. 8167 - Ensures that 8168 following loads 8169 will not see stale 8170 global data. This 8171 satisfies the 8172 requirements of 8173 acquire. 8174 8175 fence acq_rel - system *none* 1. buffer_wbl2 8176 8177 - If OpenCL and 8178 address space is 8179 local, omit. 8180 - Must happen before 8181 following s_waitcnt. 8182 - Performs L2 writeback to 8183 ensure previous 8184 global/generic 8185 store/atomicrmw are 8186 visible at system scope. 8187 8188 2. s_waitcnt lgkmcnt(0) & 8189 vmcnt(0) 8190 8191 - If TgSplit execution mode, 8192 omit lgkmcnt(0). 8193 - If OpenCL and 8194 address space is 8195 not generic, omit 8196 lgkmcnt(0). 8197 - However, since LLVM 8198 currently has no 8199 address space on 8200 the fence need to 8201 conservatively 8202 always generate 8203 (see comment for 8204 previous fence). 8205 - Could be split into 8206 separate s_waitcnt 8207 vmcnt(0) and 8208 s_waitcnt 8209 lgkmcnt(0) to allow 8210 them to be 8211 independently moved 8212 according to the 8213 following rules. 8214 - s_waitcnt vmcnt(0) 8215 must happen after 8216 any preceding 8217 global/generic 8218 load/store/load 8219 atomic/store 8220 atomic/atomicrmw. 8221 - s_waitcnt lgkmcnt(0) 8222 must happen after 8223 any preceding 8224 local/generic 8225 load/store/load 8226 atomic/store 8227 atomic/atomicrmw. 8228 - Must happen before 8229 the following buffer_invl2 and 8230 buffer_wbinvl1_vol. 8231 - Ensures that the 8232 preceding 8233 global/local/generic 8234 load 8235 atomic/atomicrmw 8236 with an equal or 8237 wider sync scope 8238 and memory ordering 8239 stronger than 8240 unordered (this is 8241 termed the 8242 acquire-fence-paired-atomic) 8243 has completed 8244 before invalidating 8245 the cache. This 8246 satisfies the 8247 requirements of 8248 acquire. 8249 - Ensures that all 8250 previous memory 8251 operations have 8252 completed before a 8253 following 8254 global/local/generic 8255 store 8256 atomic/atomicrmw 8257 with an equal or 8258 wider sync scope 8259 and memory ordering 8260 stronger than 8261 unordered (this is 8262 termed the 8263 release-fence-paired-atomic). 8264 This satisfies the 8265 requirements of 8266 release. 8267 8268 3. buffer_invl2; 8269 buffer_wbinvl1_vol 8270 8271 - Must happen before 8272 any following 8273 global/generic 8274 load/load 8275 atomic/store/store 8276 atomic/atomicrmw. 8277 - Ensures that 8278 following 8279 loads will not see 8280 stale L1 global data, 8281 nor see stale L2 MTYPE 8282 NC global data. 8283 MTYPE RW and CC memory will 8284 never be stale in L2 due to 8285 the memory probes. 8286 8287 **Sequential Consistent Atomic** 8288 ------------------------------------------------------------------------------------ 8289 load atomic seq_cst - singlethread - global *Same as corresponding 8290 - wavefront - local load atomic acquire, 8291 - generic except must generated 8292 all instructions even 8293 for OpenCL.* 8294 load atomic seq_cst - workgroup - global 1. s_waitcnt lgkm/vmcnt(0) 8295 - generic 8296 - Use lgkmcnt(0) if not 8297 TgSplit execution mode 8298 and vmcnt(0) if TgSplit 8299 execution mode. 8300 - s_waitcnt lgkmcnt(0) must 8301 happen after 8302 preceding 8303 local/generic load 8304 atomic/store 8305 atomic/atomicrmw 8306 with memory 8307 ordering of seq_cst 8308 and with equal or 8309 wider sync scope. 8310 (Note that seq_cst 8311 fences have their 8312 own s_waitcnt 8313 lgkmcnt(0) and so do 8314 not need to be 8315 considered.) 8316 - s_waitcnt vmcnt(0) 8317 must happen after 8318 preceding 8319 global/generic load 8320 atomic/store 8321 atomic/atomicrmw 8322 with memory 8323 ordering of seq_cst 8324 and with equal or 8325 wider sync scope. 8326 (Note that seq_cst 8327 fences have their 8328 own s_waitcnt 8329 vmcnt(0) and so do 8330 not need to be 8331 considered.) 8332 - Ensures any 8333 preceding 8334 sequential 8335 consistent global/local 8336 memory instructions 8337 have completed 8338 before executing 8339 this sequentially 8340 consistent 8341 instruction. This 8342 prevents reordering 8343 a seq_cst store 8344 followed by a 8345 seq_cst load. (Note 8346 that seq_cst is 8347 stronger than 8348 acquire/release as 8349 the reordering of 8350 load acquire 8351 followed by a store 8352 release is 8353 prevented by the 8354 s_waitcnt of 8355 the release, but 8356 there is nothing 8357 preventing a store 8358 release followed by 8359 load acquire from 8360 completing out of 8361 order. The s_waitcnt 8362 could be placed after 8363 seq_store or before 8364 the seq_load. We 8365 choose the load to 8366 make the s_waitcnt be 8367 as late as possible 8368 so that the store 8369 may have already 8370 completed.) 8371 8372 2. *Following 8373 instructions same as 8374 corresponding load 8375 atomic acquire, 8376 except must generated 8377 all instructions even 8378 for OpenCL.* 8379 load atomic seq_cst - workgroup - local *If TgSplit execution mode, 8380 local address space cannot 8381 be used.* 8382 8383 *Same as corresponding 8384 load atomic acquire, 8385 except must generated 8386 all instructions even 8387 for OpenCL.* 8388 8389 load atomic seq_cst - agent - global 1. s_waitcnt lgkmcnt(0) & 8390 - system - generic vmcnt(0) 8391 8392 - If TgSplit execution mode, 8393 omit lgkmcnt(0). 8394 - Could be split into 8395 separate s_waitcnt 8396 vmcnt(0) 8397 and s_waitcnt 8398 lgkmcnt(0) to allow 8399 them to be 8400 independently moved 8401 according to the 8402 following rules. 8403 - s_waitcnt lgkmcnt(0) 8404 must happen after 8405 preceding 8406 global/generic load 8407 atomic/store 8408 atomic/atomicrmw 8409 with memory 8410 ordering of seq_cst 8411 and with equal or 8412 wider sync scope. 8413 (Note that seq_cst 8414 fences have their 8415 own s_waitcnt 8416 lgkmcnt(0) and so do 8417 not need to be 8418 considered.) 8419 - s_waitcnt vmcnt(0) 8420 must happen after 8421 preceding 8422 global/generic load 8423 atomic/store 8424 atomic/atomicrmw 8425 with memory 8426 ordering of seq_cst 8427 and with equal or 8428 wider sync scope. 8429 (Note that seq_cst 8430 fences have their 8431 own s_waitcnt 8432 vmcnt(0) and so do 8433 not need to be 8434 considered.) 8435 - Ensures any 8436 preceding 8437 sequential 8438 consistent global 8439 memory instructions 8440 have completed 8441 before executing 8442 this sequentially 8443 consistent 8444 instruction. This 8445 prevents reordering 8446 a seq_cst store 8447 followed by a 8448 seq_cst load. (Note 8449 that seq_cst is 8450 stronger than 8451 acquire/release as 8452 the reordering of 8453 load acquire 8454 followed by a store 8455 release is 8456 prevented by the 8457 s_waitcnt of 8458 the release, but 8459 there is nothing 8460 preventing a store 8461 release followed by 8462 load acquire from 8463 completing out of 8464 order. The s_waitcnt 8465 could be placed after 8466 seq_store or before 8467 the seq_load. We 8468 choose the load to 8469 make the s_waitcnt be 8470 as late as possible 8471 so that the store 8472 may have already 8473 completed.) 8474 8475 2. *Following 8476 instructions same as 8477 corresponding load 8478 atomic acquire, 8479 except must generated 8480 all instructions even 8481 for OpenCL.* 8482 store atomic seq_cst - singlethread - global *Same as corresponding 8483 - wavefront - local store atomic release, 8484 - workgroup - generic except must generated 8485 - agent all instructions even 8486 - system for OpenCL.* 8487 atomicrmw seq_cst - singlethread - global *Same as corresponding 8488 - wavefront - local atomicrmw acq_rel, 8489 - workgroup - generic except must generated 8490 - agent all instructions even 8491 - system for OpenCL.* 8492 fence seq_cst - singlethread *none* *Same as corresponding 8493 - wavefront fence acq_rel, 8494 - workgroup except must generated 8495 - agent all instructions even 8496 - system for OpenCL.* 8497 ============ ============ ============== ========== ================================ 8498 8499.. _amdgpu-amdhsa-memory-model-gfx10: 8500 8501Memory Model GFX10 8502++++++++++++++++++ 8503 8504For GFX10: 8505 8506* Each agent has multiple shader arrays (SA). 8507* Each SA has multiple work-group processors (WGP). 8508* Each WGP has multiple compute units (CU). 8509* Each CU has multiple SIMDs that execute wavefronts. 8510* The wavefronts for a single work-group are executed in the same 8511 WGP. In CU wavefront execution mode the wavefronts may be executed by 8512 different SIMDs in the same CU. In WGP wavefront execution mode the 8513 wavefronts may be executed by different SIMDs in different CUs in the same 8514 WGP. 8515* Each WGP has a single LDS memory shared by the wavefronts of the work-groups 8516 executing on it. 8517* All LDS operations of a WGP are performed as wavefront wide operations in a 8518 global order and involve no caching. Completion is reported to a wavefront in 8519 execution order. 8520* The LDS memory has multiple request queues shared by the SIMDs of a 8521 WGP. Therefore, the LDS operations performed by different wavefronts of a 8522 work-group can be reordered relative to each other, which can result in 8523 reordering the visibility of vector memory operations with respect to LDS 8524 operations of other wavefronts in the same work-group. A ``s_waitcnt 8525 lgkmcnt(0)`` is required to ensure synchronization between LDS operations and 8526 vector memory operations between wavefronts of a work-group, but not between 8527 operations performed by the same wavefront. 8528* The vector memory operations are performed as wavefront wide operations. 8529 Completion of load/store/sample operations are reported to a wavefront in 8530 execution order of other load/store/sample operations performed by that 8531 wavefront. 8532* The vector memory operations access a vector L0 cache. There is a single L0 8533 cache per CU. Each SIMD of a CU accesses the same L0 cache. Therefore, no 8534 special action is required for coherence between the lanes of a single 8535 wavefront. However, a ``buffer_gl0_inv`` is required for coherence between 8536 wavefronts executing in the same work-group as they may be executing on SIMDs 8537 of different CUs that access different L0s. A ``buffer_gl0_inv`` is also 8538 required for coherence between wavefronts executing in different work-groups 8539 as they may be executing on different WGPs. 8540* The scalar memory operations access a scalar L0 cache shared by all wavefronts 8541 on a WGP. The scalar and vector L0 caches are not coherent. However, scalar 8542 operations are used in a restricted way so do not impact the memory model. See 8543 :ref:`amdgpu-amdhsa-memory-spaces`. 8544* The vector and scalar memory L0 caches use an L1 cache shared by all WGPs on 8545 the same SA. Therefore, no special action is required for coherence between 8546 the wavefronts of a single work-group. However, a ``buffer_gl1_inv`` is 8547 required for coherence between wavefronts executing in different work-groups 8548 as they may be executing on different SAs that access different L1s. 8549* The L1 caches have independent quadrants to service disjoint ranges of virtual 8550 addresses. 8551* Each L0 cache has a separate request queue per L1 quadrant. Therefore, the 8552 vector and scalar memory operations performed by different wavefronts, whether 8553 executing in the same or different work-groups (which may be executing on 8554 different CUs accessing different L0s), can be reordered relative to each 8555 other. A ``s_waitcnt vmcnt(0) & vscnt(0)`` is required to ensure 8556 synchronization between vector memory operations of different wavefronts. It 8557 ensures a previous vector memory operation has completed before executing a 8558 subsequent vector memory or LDS operation and so can be used to meet the 8559 requirements of acquire, release and sequential consistency. 8560* The L1 caches use an L2 cache shared by all SAs on the same agent. 8561* The L2 cache has independent channels to service disjoint ranges of virtual 8562 addresses. 8563* Each L1 quadrant of a single SA accesses a different L2 channel. Each L1 8564 quadrant has a separate request queue per L2 channel. Therefore, the vector 8565 and scalar memory operations performed by wavefronts executing in different 8566 work-groups (which may be executing on different SAs) of an agent can be 8567 reordered relative to each other. A ``s_waitcnt vmcnt(0) & vscnt(0)`` is 8568 required to ensure synchronization between vector memory operations of 8569 different SAs. It ensures a previous vector memory operation has completed 8570 before executing a subsequent vector memory and so can be used to meet the 8571 requirements of acquire, release and sequential consistency. 8572* The L2 cache can be kept coherent with other agents on some targets, or ranges 8573 of virtual addresses can be set up to bypass it to ensure system coherence. 8574 8575Scalar memory operations are only used to access memory that is proven to not 8576change during the execution of the kernel dispatch. This includes constant 8577address space and global address space for program scope ``const`` variables. 8578Therefore, the kernel machine code does not have to maintain the scalar cache to 8579ensure it is coherent with the vector caches. The scalar and vector caches are 8580invalidated between kernel dispatches by CP since constant address space data 8581may change between kernel dispatch executions. See 8582:ref:`amdgpu-amdhsa-memory-spaces`. 8583 8584The one exception is if scalar writes are used to spill SGPR registers. In this 8585case the AMDGPU backend ensures the memory location used to spill is never 8586accessed by vector memory operations at the same time. If scalar writes are used 8587then a ``s_dcache_wb`` is inserted before the ``s_endpgm`` and before a function 8588return since the locations may be used for vector memory instructions by a 8589future wavefront that uses the same scratch area, or a function call that 8590creates a frame at the same address, respectively. There is no need for a 8591``s_dcache_inv`` as all scalar writes are write-before-read in the same thread. 8592 8593For kernarg backing memory: 8594 8595* CP invalidates the L0 and L1 caches at the start of each kernel dispatch. 8596* On dGPU the kernarg backing memory is accessed as MTYPE UC (uncached) to avoid 8597 needing to invalidate the L2 cache. 8598* On APU the kernarg backing memory is accessed as MTYPE CC (cache coherent) and 8599 so the L2 cache will be coherent with the CPU and other agents. 8600 8601Scratch backing memory (which is used for the private address space) is accessed 8602with MTYPE NC (non-coherent). Since the private address space is only accessed 8603by a single thread, and is always write-before-read, there is never a need to 8604invalidate these entries from the L0 or L1 caches. 8605 8606Wavefronts are executed in native mode with in-order reporting of loads and 8607sample instructions. In this mode vmcnt reports completion of load, atomic with 8608return and sample instructions in order, and the vscnt reports the completion of 8609store and atomic without return in order. See ``MEM_ORDERED`` field in 8610:ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 8611 8612Wavefronts can be executed in WGP or CU wavefront execution mode: 8613 8614* In WGP wavefront execution mode the wavefronts of a work-group are executed 8615 on the SIMDs of both CUs of the WGP. Therefore, explicit management of the per 8616 CU L0 caches is required for work-group synchronization. Also accesses to L1 8617 at work-group scope need to be explicitly ordered as the accesses from 8618 different CUs are not ordered. 8619* In CU wavefront execution mode the wavefronts of a work-group are executed on 8620 the SIMDs of a single CU of the WGP. Therefore, all global memory access by 8621 the work-group access the same L0 which in turn ensures L1 accesses are 8622 ordered and so do not require explicit management of the caches for 8623 work-group synchronization. 8624 8625See ``WGP_MODE`` field in 8626:ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table` and 8627:ref:`amdgpu-target-features`. 8628 8629The code sequences used to implement the memory model for GFX10 are defined in 8630table :ref:`amdgpu-amdhsa-memory-model-code-sequences-gfx10-table`. 8631 8632 .. table:: AMDHSA Memory Model Code Sequences GFX10 8633 :name: amdgpu-amdhsa-memory-model-code-sequences-gfx10-table 8634 8635 ============ ============ ============== ========== ================================ 8636 LLVM Instr LLVM Memory LLVM Memory AMDGPU AMDGPU Machine Code 8637 Ordering Sync Scope Address GFX10 8638 Space 8639 ============ ============ ============== ========== ================================ 8640 **Non-Atomic** 8641 ------------------------------------------------------------------------------------ 8642 load *none* *none* - global - !volatile & !nontemporal 8643 - generic 8644 - private 1. buffer/global/flat_load 8645 - constant 8646 - !volatile & nontemporal 8647 8648 1. buffer/global/flat_load 8649 slc=1 8650 8651 - volatile 8652 8653 1. buffer/global/flat_load 8654 glc=1 dlc=1 8655 2. s_waitcnt vmcnt(0) 8656 8657 - Must happen before 8658 any following volatile 8659 global/generic 8660 load/store. 8661 - Ensures that 8662 volatile 8663 operations to 8664 different 8665 addresses will not 8666 be reordered by 8667 hardware. 8668 8669 load *none* *none* - local 1. ds_load 8670 store *none* *none* - global - !volatile & !nontemporal 8671 - generic 8672 - private 1. buffer/global/flat_store 8673 - constant 8674 - !volatile & nontemporal 8675 8676 1. buffer/global/flat_store 8677 slc=1 8678 8679 - volatile 8680 8681 1. buffer/global/flat_store 8682 2. s_waitcnt vscnt(0) 8683 8684 - Must happen before 8685 any following volatile 8686 global/generic 8687 load/store. 8688 - Ensures that 8689 volatile 8690 operations to 8691 different 8692 addresses will not 8693 be reordered by 8694 hardware. 8695 8696 store *none* *none* - local 1. ds_store 8697 **Unordered Atomic** 8698 ------------------------------------------------------------------------------------ 8699 load atomic unordered *any* *any* *Same as non-atomic*. 8700 store atomic unordered *any* *any* *Same as non-atomic*. 8701 atomicrmw unordered *any* *any* *Same as monotonic atomic*. 8702 **Monotonic Atomic** 8703 ------------------------------------------------------------------------------------ 8704 load atomic monotonic - singlethread - global 1. buffer/global/flat_load 8705 - wavefront - generic 8706 load atomic monotonic - workgroup - global 1. buffer/global/flat_load 8707 - generic glc=1 8708 8709 - If CU wavefront execution 8710 mode, omit glc=1. 8711 8712 load atomic monotonic - singlethread - local 1. ds_load 8713 - wavefront 8714 - workgroup 8715 load atomic monotonic - agent - global 1. buffer/global/flat_load 8716 - system - generic glc=1 dlc=1 8717 store atomic monotonic - singlethread - global 1. buffer/global/flat_store 8718 - wavefront - generic 8719 - workgroup 8720 - agent 8721 - system 8722 store atomic monotonic - singlethread - local 1. ds_store 8723 - wavefront 8724 - workgroup 8725 atomicrmw monotonic - singlethread - global 1. buffer/global/flat_atomic 8726 - wavefront - generic 8727 - workgroup 8728 - agent 8729 - system 8730 atomicrmw monotonic - singlethread - local 1. ds_atomic 8731 - wavefront 8732 - workgroup 8733 **Acquire Atomic** 8734 ------------------------------------------------------------------------------------ 8735 load atomic acquire - singlethread - global 1. buffer/global/ds/flat_load 8736 - wavefront - local 8737 - generic 8738 load atomic acquire - workgroup - global 1. buffer/global_load glc=1 8739 8740 - If CU wavefront execution 8741 mode, omit glc=1. 8742 8743 2. s_waitcnt vmcnt(0) 8744 8745 - If CU wavefront execution 8746 mode, omit. 8747 - Must happen before 8748 the following buffer_gl0_inv 8749 and before any following 8750 global/generic 8751 load/load 8752 atomic/store/store 8753 atomic/atomicrmw. 8754 8755 3. buffer_gl0_inv 8756 8757 - If CU wavefront execution 8758 mode, omit. 8759 - Ensures that 8760 following 8761 loads will not see 8762 stale data. 8763 8764 load atomic acquire - workgroup - local 1. ds_load 8765 2. s_waitcnt lgkmcnt(0) 8766 8767 - If OpenCL, omit. 8768 - Must happen before 8769 the following buffer_gl0_inv 8770 and before any following 8771 global/generic load/load 8772 atomic/store/store 8773 atomic/atomicrmw. 8774 - Ensures any 8775 following global 8776 data read is no 8777 older than the local load 8778 atomic value being 8779 acquired. 8780 8781 3. buffer_gl0_inv 8782 8783 - If CU wavefront execution 8784 mode, omit. 8785 - If OpenCL, omit. 8786 - Ensures that 8787 following 8788 loads will not see 8789 stale data. 8790 8791 load atomic acquire - workgroup - generic 1. flat_load glc=1 8792 8793 - If CU wavefront execution 8794 mode, omit glc=1. 8795 8796 2. s_waitcnt lgkmcnt(0) & 8797 vmcnt(0) 8798 8799 - If CU wavefront execution 8800 mode, omit vmcnt(0). 8801 - If OpenCL, omit 8802 lgkmcnt(0). 8803 - Must happen before 8804 the following 8805 buffer_gl0_inv and any 8806 following global/generic 8807 load/load 8808 atomic/store/store 8809 atomic/atomicrmw. 8810 - Ensures any 8811 following global 8812 data read is no 8813 older than a local load 8814 atomic value being 8815 acquired. 8816 8817 3. buffer_gl0_inv 8818 8819 - If CU wavefront execution 8820 mode, omit. 8821 - Ensures that 8822 following 8823 loads will not see 8824 stale data. 8825 8826 load atomic acquire - agent - global 1. buffer/global_load 8827 - system glc=1 dlc=1 8828 2. s_waitcnt vmcnt(0) 8829 8830 - Must happen before 8831 following 8832 buffer_gl*_inv. 8833 - Ensures the load 8834 has completed 8835 before invalidating 8836 the caches. 8837 8838 3. buffer_gl0_inv; 8839 buffer_gl1_inv 8840 8841 - Must happen before 8842 any following 8843 global/generic 8844 load/load 8845 atomic/atomicrmw. 8846 - Ensures that 8847 following 8848 loads will not see 8849 stale global data. 8850 8851 load atomic acquire - agent - generic 1. flat_load glc=1 dlc=1 8852 - system 2. s_waitcnt vmcnt(0) & 8853 lgkmcnt(0) 8854 8855 - If OpenCL omit 8856 lgkmcnt(0). 8857 - Must happen before 8858 following 8859 buffer_gl*_invl. 8860 - Ensures the flat_load 8861 has completed 8862 before invalidating 8863 the caches. 8864 8865 3. buffer_gl0_inv; 8866 buffer_gl1_inv 8867 8868 - Must happen before 8869 any following 8870 global/generic 8871 load/load 8872 atomic/atomicrmw. 8873 - Ensures that 8874 following loads 8875 will not see stale 8876 global data. 8877 8878 atomicrmw acquire - singlethread - global 1. buffer/global/ds/flat_atomic 8879 - wavefront - local 8880 - generic 8881 atomicrmw acquire - workgroup - global 1. buffer/global_atomic 8882 2. s_waitcnt vm/vscnt(0) 8883 8884 - If CU wavefront execution 8885 mode, omit. 8886 - Use vmcnt(0) if atomic with 8887 return and vscnt(0) if 8888 atomic with no-return. 8889 - Must happen before 8890 the following buffer_gl0_inv 8891 and before any following 8892 global/generic 8893 load/load 8894 atomic/store/store 8895 atomic/atomicrmw. 8896 8897 3. buffer_gl0_inv 8898 8899 - If CU wavefront execution 8900 mode, omit. 8901 - Ensures that 8902 following 8903 loads will not see 8904 stale data. 8905 8906 atomicrmw acquire - workgroup - local 1. ds_atomic 8907 2. s_waitcnt lgkmcnt(0) 8908 8909 - If OpenCL, omit. 8910 - Must happen before 8911 the following 8912 buffer_gl0_inv. 8913 - Ensures any 8914 following global 8915 data read is no 8916 older than the local 8917 atomicrmw value 8918 being acquired. 8919 8920 3. buffer_gl0_inv 8921 8922 - If OpenCL omit. 8923 - Ensures that 8924 following 8925 loads will not see 8926 stale data. 8927 8928 atomicrmw acquire - workgroup - generic 1. flat_atomic 8929 2. s_waitcnt lgkmcnt(0) & 8930 vm/vscnt(0) 8931 8932 - If CU wavefront execution 8933 mode, omit vm/vscnt(0). 8934 - If OpenCL, omit lgkmcnt(0). 8935 - Use vmcnt(0) if atomic with 8936 return and vscnt(0) if 8937 atomic with no-return. 8938 - Must happen before 8939 the following 8940 buffer_gl0_inv. 8941 - Ensures any 8942 following global 8943 data read is no 8944 older than a local 8945 atomicrmw value 8946 being acquired. 8947 8948 3. buffer_gl0_inv 8949 8950 - If CU wavefront execution 8951 mode, omit. 8952 - Ensures that 8953 following 8954 loads will not see 8955 stale data. 8956 8957 atomicrmw acquire - agent - global 1. buffer/global_atomic 8958 - system 2. s_waitcnt vm/vscnt(0) 8959 8960 - Use vmcnt(0) if atomic with 8961 return and vscnt(0) if 8962 atomic with no-return. 8963 - Must happen before 8964 following 8965 buffer_gl*_inv. 8966 - Ensures the 8967 atomicrmw has 8968 completed before 8969 invalidating the 8970 caches. 8971 8972 3. buffer_gl0_inv; 8973 buffer_gl1_inv 8974 8975 - Must happen before 8976 any following 8977 global/generic 8978 load/load 8979 atomic/atomicrmw. 8980 - Ensures that 8981 following loads 8982 will not see stale 8983 global data. 8984 8985 atomicrmw acquire - agent - generic 1. flat_atomic 8986 - system 2. s_waitcnt vm/vscnt(0) & 8987 lgkmcnt(0) 8988 8989 - If OpenCL, omit 8990 lgkmcnt(0). 8991 - Use vmcnt(0) if atomic with 8992 return and vscnt(0) if 8993 atomic with no-return. 8994 - Must happen before 8995 following 8996 buffer_gl*_inv. 8997 - Ensures the 8998 atomicrmw has 8999 completed before 9000 invalidating the 9001 caches. 9002 9003 3. buffer_gl0_inv; 9004 buffer_gl1_inv 9005 9006 - Must happen before 9007 any following 9008 global/generic 9009 load/load 9010 atomic/atomicrmw. 9011 - Ensures that 9012 following loads 9013 will not see stale 9014 global data. 9015 9016 fence acquire - singlethread *none* *none* 9017 - wavefront 9018 fence acquire - workgroup *none* 1. s_waitcnt lgkmcnt(0) & 9019 vmcnt(0) & vscnt(0) 9020 9021 - If CU wavefront execution 9022 mode, omit vmcnt(0) and 9023 vscnt(0). 9024 - If OpenCL and 9025 address space is 9026 not generic, omit 9027 lgkmcnt(0). 9028 - If OpenCL and 9029 address space is 9030 local, omit 9031 vmcnt(0) and vscnt(0). 9032 - However, since LLVM 9033 currently has no 9034 address space on 9035 the fence need to 9036 conservatively 9037 always generate. If 9038 fence had an 9039 address space then 9040 set to address 9041 space of OpenCL 9042 fence flag, or to 9043 generic if both 9044 local and global 9045 flags are 9046 specified. 9047 - Could be split into 9048 separate s_waitcnt 9049 vmcnt(0), s_waitcnt 9050 vscnt(0) and s_waitcnt 9051 lgkmcnt(0) to allow 9052 them to be 9053 independently moved 9054 according to the 9055 following rules. 9056 - s_waitcnt vmcnt(0) 9057 must happen after 9058 any preceding 9059 global/generic load 9060 atomic/ 9061 atomicrmw-with-return-value 9062 with an equal or 9063 wider sync scope 9064 and memory ordering 9065 stronger than 9066 unordered (this is 9067 termed the 9068 fence-paired-atomic). 9069 - s_waitcnt vscnt(0) 9070 must happen after 9071 any preceding 9072 global/generic 9073 atomicrmw-no-return-value 9074 with an equal or 9075 wider sync scope 9076 and memory ordering 9077 stronger than 9078 unordered (this is 9079 termed the 9080 fence-paired-atomic). 9081 - s_waitcnt lgkmcnt(0) 9082 must happen after 9083 any preceding 9084 local/generic load 9085 atomic/atomicrmw 9086 with an equal or 9087 wider sync scope 9088 and memory ordering 9089 stronger than 9090 unordered (this is 9091 termed the 9092 fence-paired-atomic). 9093 - Must happen before 9094 the following 9095 buffer_gl0_inv. 9096 - Ensures that the 9097 fence-paired atomic 9098 has completed 9099 before invalidating 9100 the 9101 cache. Therefore 9102 any following 9103 locations read must 9104 be no older than 9105 the value read by 9106 the 9107 fence-paired-atomic. 9108 9109 3. buffer_gl0_inv 9110 9111 - If CU wavefront execution 9112 mode, omit. 9113 - Ensures that 9114 following 9115 loads will not see 9116 stale data. 9117 9118 fence acquire - agent *none* 1. s_waitcnt lgkmcnt(0) & 9119 - system vmcnt(0) & vscnt(0) 9120 9121 - If OpenCL and 9122 address space is 9123 not generic, omit 9124 lgkmcnt(0). 9125 - If OpenCL and 9126 address space is 9127 local, omit 9128 vmcnt(0) and vscnt(0). 9129 - However, since LLVM 9130 currently has no 9131 address space on 9132 the fence need to 9133 conservatively 9134 always generate 9135 (see comment for 9136 previous fence). 9137 - Could be split into 9138 separate s_waitcnt 9139 vmcnt(0), s_waitcnt 9140 vscnt(0) and s_waitcnt 9141 lgkmcnt(0) to allow 9142 them to be 9143 independently moved 9144 according to the 9145 following rules. 9146 - s_waitcnt vmcnt(0) 9147 must happen after 9148 any preceding 9149 global/generic load 9150 atomic/ 9151 atomicrmw-with-return-value 9152 with an equal or 9153 wider sync scope 9154 and memory ordering 9155 stronger than 9156 unordered (this is 9157 termed the 9158 fence-paired-atomic). 9159 - s_waitcnt vscnt(0) 9160 must happen after 9161 any preceding 9162 global/generic 9163 atomicrmw-no-return-value 9164 with an equal or 9165 wider sync scope 9166 and memory ordering 9167 stronger than 9168 unordered (this is 9169 termed the 9170 fence-paired-atomic). 9171 - s_waitcnt lgkmcnt(0) 9172 must happen after 9173 any preceding 9174 local/generic load 9175 atomic/atomicrmw 9176 with an equal or 9177 wider sync scope 9178 and memory ordering 9179 stronger than 9180 unordered (this is 9181 termed the 9182 fence-paired-atomic). 9183 - Must happen before 9184 the following 9185 buffer_gl*_inv. 9186 - Ensures that the 9187 fence-paired atomic 9188 has completed 9189 before invalidating 9190 the 9191 caches. Therefore 9192 any following 9193 locations read must 9194 be no older than 9195 the value read by 9196 the 9197 fence-paired-atomic. 9198 9199 2. buffer_gl0_inv; 9200 buffer_gl1_inv 9201 9202 - Must happen before any 9203 following global/generic 9204 load/load 9205 atomic/store/store 9206 atomic/atomicrmw. 9207 - Ensures that 9208 following loads 9209 will not see stale 9210 global data. 9211 9212 **Release Atomic** 9213 ------------------------------------------------------------------------------------ 9214 store atomic release - singlethread - global 1. buffer/global/ds/flat_store 9215 - wavefront - local 9216 - generic 9217 store atomic release - workgroup - global 1. s_waitcnt lgkmcnt(0) & 9218 - generic vmcnt(0) & vscnt(0) 9219 9220 - If CU wavefront execution 9221 mode, omit vmcnt(0) and 9222 vscnt(0). 9223 - If OpenCL, omit 9224 lgkmcnt(0). 9225 - Could be split into 9226 separate s_waitcnt 9227 vmcnt(0), s_waitcnt 9228 vscnt(0) and s_waitcnt 9229 lgkmcnt(0) to allow 9230 them to be 9231 independently moved 9232 according to the 9233 following rules. 9234 - s_waitcnt vmcnt(0) 9235 must happen after 9236 any preceding 9237 global/generic load/load 9238 atomic/ 9239 atomicrmw-with-return-value. 9240 - s_waitcnt vscnt(0) 9241 must happen after 9242 any preceding 9243 global/generic 9244 store/store 9245 atomic/ 9246 atomicrmw-no-return-value. 9247 - s_waitcnt lgkmcnt(0) 9248 must happen after 9249 any preceding 9250 local/generic 9251 load/store/load 9252 atomic/store 9253 atomic/atomicrmw. 9254 - Must happen before 9255 the following 9256 store. 9257 - Ensures that all 9258 memory operations 9259 have 9260 completed before 9261 performing the 9262 store that is being 9263 released. 9264 9265 2. buffer/global/flat_store 9266 store atomic release - workgroup - local 1. s_waitcnt vmcnt(0) & vscnt(0) 9267 9268 - If CU wavefront execution 9269 mode, omit. 9270 - If OpenCL, omit. 9271 - Could be split into 9272 separate s_waitcnt 9273 vmcnt(0) and s_waitcnt 9274 vscnt(0) to allow 9275 them to be 9276 independently moved 9277 according to the 9278 following rules. 9279 - s_waitcnt vmcnt(0) 9280 must happen after 9281 any preceding 9282 global/generic load/load 9283 atomic/ 9284 atomicrmw-with-return-value. 9285 - s_waitcnt vscnt(0) 9286 must happen after 9287 any preceding 9288 global/generic 9289 store/store atomic/ 9290 atomicrmw-no-return-value. 9291 - Must happen before 9292 the following 9293 store. 9294 - Ensures that all 9295 global memory 9296 operations have 9297 completed before 9298 performing the 9299 store that is being 9300 released. 9301 9302 2. ds_store 9303 store atomic release - agent - global 1. s_waitcnt lgkmcnt(0) & 9304 - system - generic vmcnt(0) & vscnt(0) 9305 9306 - If OpenCL and 9307 address space is 9308 not generic, omit 9309 lgkmcnt(0). 9310 - Could be split into 9311 separate s_waitcnt 9312 vmcnt(0), s_waitcnt vscnt(0) 9313 and s_waitcnt 9314 lgkmcnt(0) to allow 9315 them to be 9316 independently moved 9317 according to the 9318 following rules. 9319 - s_waitcnt vmcnt(0) 9320 must happen after 9321 any preceding 9322 global/generic 9323 load/load 9324 atomic/ 9325 atomicrmw-with-return-value. 9326 - s_waitcnt vscnt(0) 9327 must happen after 9328 any preceding 9329 global/generic 9330 store/store atomic/ 9331 atomicrmw-no-return-value. 9332 - s_waitcnt lgkmcnt(0) 9333 must happen after 9334 any preceding 9335 local/generic 9336 load/store/load 9337 atomic/store 9338 atomic/atomicrmw. 9339 - Must happen before 9340 the following 9341 store. 9342 - Ensures that all 9343 memory operations 9344 have 9345 completed before 9346 performing the 9347 store that is being 9348 released. 9349 9350 2. buffer/global/flat_store 9351 atomicrmw release - singlethread - global 1. buffer/global/ds/flat_atomic 9352 - wavefront - local 9353 - generic 9354 atomicrmw release - workgroup - global 1. s_waitcnt lgkmcnt(0) & 9355 - generic vmcnt(0) & vscnt(0) 9356 9357 - If CU wavefront execution 9358 mode, omit vmcnt(0) and 9359 vscnt(0). 9360 - If OpenCL, omit lgkmcnt(0). 9361 - Could be split into 9362 separate s_waitcnt 9363 vmcnt(0), s_waitcnt 9364 vscnt(0) and s_waitcnt 9365 lgkmcnt(0) to allow 9366 them to be 9367 independently moved 9368 according to the 9369 following rules. 9370 - s_waitcnt vmcnt(0) 9371 must happen after 9372 any preceding 9373 global/generic load/load 9374 atomic/ 9375 atomicrmw-with-return-value. 9376 - s_waitcnt vscnt(0) 9377 must happen after 9378 any preceding 9379 global/generic 9380 store/store 9381 atomic/ 9382 atomicrmw-no-return-value. 9383 - s_waitcnt lgkmcnt(0) 9384 must happen after 9385 any preceding 9386 local/generic 9387 load/store/load 9388 atomic/store 9389 atomic/atomicrmw. 9390 - Must happen before 9391 the following 9392 atomicrmw. 9393 - Ensures that all 9394 memory operations 9395 have 9396 completed before 9397 performing the 9398 atomicrmw that is 9399 being released. 9400 9401 2. buffer/global/flat_atomic 9402 atomicrmw release - workgroup - local 1. s_waitcnt vmcnt(0) & vscnt(0) 9403 9404 - If CU wavefront execution 9405 mode, omit. 9406 - If OpenCL, omit. 9407 - Could be split into 9408 separate s_waitcnt 9409 vmcnt(0) and s_waitcnt 9410 vscnt(0) to allow 9411 them to be 9412 independently moved 9413 according to the 9414 following rules. 9415 - s_waitcnt vmcnt(0) 9416 must happen after 9417 any preceding 9418 global/generic load/load 9419 atomic/ 9420 atomicrmw-with-return-value. 9421 - s_waitcnt vscnt(0) 9422 must happen after 9423 any preceding 9424 global/generic 9425 store/store atomic/ 9426 atomicrmw-no-return-value. 9427 - Must happen before 9428 the following 9429 store. 9430 - Ensures that all 9431 global memory 9432 operations have 9433 completed before 9434 performing the 9435 store that is being 9436 released. 9437 9438 2. ds_atomic 9439 atomicrmw release - agent - global 1. s_waitcnt lgkmcnt(0) & 9440 - system - generic vmcnt(0) & vscnt(0) 9441 9442 - If OpenCL, omit 9443 lgkmcnt(0). 9444 - Could be split into 9445 separate s_waitcnt 9446 vmcnt(0), s_waitcnt 9447 vscnt(0) and s_waitcnt 9448 lgkmcnt(0) to allow 9449 them to be 9450 independently moved 9451 according to the 9452 following rules. 9453 - s_waitcnt vmcnt(0) 9454 must happen after 9455 any preceding 9456 global/generic 9457 load/load atomic/ 9458 atomicrmw-with-return-value. 9459 - s_waitcnt vscnt(0) 9460 must happen after 9461 any preceding 9462 global/generic 9463 store/store atomic/ 9464 atomicrmw-no-return-value. 9465 - s_waitcnt lgkmcnt(0) 9466 must happen after 9467 any preceding 9468 local/generic 9469 load/store/load 9470 atomic/store 9471 atomic/atomicrmw. 9472 - Must happen before 9473 the following 9474 atomicrmw. 9475 - Ensures that all 9476 memory operations 9477 to global and local 9478 have completed 9479 before performing 9480 the atomicrmw that 9481 is being released. 9482 9483 2. buffer/global/flat_atomic 9484 fence release - singlethread *none* *none* 9485 - wavefront 9486 fence release - workgroup *none* 1. s_waitcnt lgkmcnt(0) & 9487 vmcnt(0) & vscnt(0) 9488 9489 - If CU wavefront execution 9490 mode, omit vmcnt(0) and 9491 vscnt(0). 9492 - If OpenCL and 9493 address space is 9494 not generic, omit 9495 lgkmcnt(0). 9496 - If OpenCL and 9497 address space is 9498 local, omit 9499 vmcnt(0) and vscnt(0). 9500 - However, since LLVM 9501 currently has no 9502 address space on 9503 the fence need to 9504 conservatively 9505 always generate. If 9506 fence had an 9507 address space then 9508 set to address 9509 space of OpenCL 9510 fence flag, or to 9511 generic if both 9512 local and global 9513 flags are 9514 specified. 9515 - Could be split into 9516 separate s_waitcnt 9517 vmcnt(0), s_waitcnt 9518 vscnt(0) and s_waitcnt 9519 lgkmcnt(0) to allow 9520 them to be 9521 independently moved 9522 according to the 9523 following rules. 9524 - s_waitcnt vmcnt(0) 9525 must happen after 9526 any preceding 9527 global/generic 9528 load/load 9529 atomic/ 9530 atomicrmw-with-return-value. 9531 - s_waitcnt vscnt(0) 9532 must happen after 9533 any preceding 9534 global/generic 9535 store/store atomic/ 9536 atomicrmw-no-return-value. 9537 - s_waitcnt lgkmcnt(0) 9538 must happen after 9539 any preceding 9540 local/generic 9541 load/store/load 9542 atomic/store atomic/ 9543 atomicrmw. 9544 - Must happen before 9545 any following store 9546 atomic/atomicrmw 9547 with an equal or 9548 wider sync scope 9549 and memory ordering 9550 stronger than 9551 unordered (this is 9552 termed the 9553 fence-paired-atomic). 9554 - Ensures that all 9555 memory operations 9556 have 9557 completed before 9558 performing the 9559 following 9560 fence-paired-atomic. 9561 9562 fence release - agent *none* 1. s_waitcnt lgkmcnt(0) & 9563 - system vmcnt(0) & vscnt(0) 9564 9565 - If OpenCL and 9566 address space is 9567 not generic, omit 9568 lgkmcnt(0). 9569 - If OpenCL and 9570 address space is 9571 local, omit 9572 vmcnt(0) and vscnt(0). 9573 - However, since LLVM 9574 currently has no 9575 address space on 9576 the fence need to 9577 conservatively 9578 always generate. If 9579 fence had an 9580 address space then 9581 set to address 9582 space of OpenCL 9583 fence flag, or to 9584 generic if both 9585 local and global 9586 flags are 9587 specified. 9588 - Could be split into 9589 separate s_waitcnt 9590 vmcnt(0), s_waitcnt 9591 vscnt(0) and s_waitcnt 9592 lgkmcnt(0) to allow 9593 them to be 9594 independently moved 9595 according to the 9596 following rules. 9597 - s_waitcnt vmcnt(0) 9598 must happen after 9599 any preceding 9600 global/generic 9601 load/load atomic/ 9602 atomicrmw-with-return-value. 9603 - s_waitcnt vscnt(0) 9604 must happen after 9605 any preceding 9606 global/generic 9607 store/store atomic/ 9608 atomicrmw-no-return-value. 9609 - s_waitcnt lgkmcnt(0) 9610 must happen after 9611 any preceding 9612 local/generic 9613 load/store/load 9614 atomic/store 9615 atomic/atomicrmw. 9616 - Must happen before 9617 any following store 9618 atomic/atomicrmw 9619 with an equal or 9620 wider sync scope 9621 and memory ordering 9622 stronger than 9623 unordered (this is 9624 termed the 9625 fence-paired-atomic). 9626 - Ensures that all 9627 memory operations 9628 have 9629 completed before 9630 performing the 9631 following 9632 fence-paired-atomic. 9633 9634 **Acquire-Release Atomic** 9635 ------------------------------------------------------------------------------------ 9636 atomicrmw acq_rel - singlethread - global 1. buffer/global/ds/flat_atomic 9637 - wavefront - local 9638 - generic 9639 atomicrmw acq_rel - workgroup - global 1. s_waitcnt lgkmcnt(0) & 9640 vmcnt(0) & vscnt(0) 9641 9642 - If CU wavefront execution 9643 mode, omit vmcnt(0) and 9644 vscnt(0). 9645 - If OpenCL, omit 9646 lgkmcnt(0). 9647 - Must happen after 9648 any preceding 9649 local/generic 9650 load/store/load 9651 atomic/store 9652 atomic/atomicrmw. 9653 - Could be split into 9654 separate s_waitcnt 9655 vmcnt(0), s_waitcnt 9656 vscnt(0), and s_waitcnt 9657 lgkmcnt(0) to allow 9658 them to be 9659 independently moved 9660 according to the 9661 following rules. 9662 - s_waitcnt vmcnt(0) 9663 must happen after 9664 any preceding 9665 global/generic load/load 9666 atomic/ 9667 atomicrmw-with-return-value. 9668 - s_waitcnt vscnt(0) 9669 must happen after 9670 any preceding 9671 global/generic 9672 store/store 9673 atomic/ 9674 atomicrmw-no-return-value. 9675 - s_waitcnt lgkmcnt(0) 9676 must happen after 9677 any preceding 9678 local/generic 9679 load/store/load 9680 atomic/store 9681 atomic/atomicrmw. 9682 - Must happen before 9683 the following 9684 atomicrmw. 9685 - Ensures that all 9686 memory operations 9687 have 9688 completed before 9689 performing the 9690 atomicrmw that is 9691 being released. 9692 9693 2. buffer/global_atomic 9694 3. s_waitcnt vm/vscnt(0) 9695 9696 - If CU wavefront execution 9697 mode, omit. 9698 - Use vmcnt(0) if atomic with 9699 return and vscnt(0) if 9700 atomic with no-return. 9701 - Must happen before 9702 the following 9703 buffer_gl0_inv. 9704 - Ensures any 9705 following global 9706 data read is no 9707 older than the 9708 atomicrmw value 9709 being acquired. 9710 9711 4. buffer_gl0_inv 9712 9713 - If CU wavefront execution 9714 mode, omit. 9715 - Ensures that 9716 following 9717 loads will not see 9718 stale data. 9719 9720 atomicrmw acq_rel - workgroup - local 1. s_waitcnt vmcnt(0) & vscnt(0) 9721 9722 - If CU wavefront execution 9723 mode, omit. 9724 - If OpenCL, omit. 9725 - Could be split into 9726 separate s_waitcnt 9727 vmcnt(0) and s_waitcnt 9728 vscnt(0) to allow 9729 them to be 9730 independently moved 9731 according to the 9732 following rules. 9733 - s_waitcnt vmcnt(0) 9734 must happen after 9735 any preceding 9736 global/generic load/load 9737 atomic/ 9738 atomicrmw-with-return-value. 9739 - s_waitcnt vscnt(0) 9740 must happen after 9741 any preceding 9742 global/generic 9743 store/store atomic/ 9744 atomicrmw-no-return-value. 9745 - Must happen before 9746 the following 9747 store. 9748 - Ensures that all 9749 global memory 9750 operations have 9751 completed before 9752 performing the 9753 store that is being 9754 released. 9755 9756 2. ds_atomic 9757 3. s_waitcnt lgkmcnt(0) 9758 9759 - If OpenCL, omit. 9760 - Must happen before 9761 the following 9762 buffer_gl0_inv. 9763 - Ensures any 9764 following global 9765 data read is no 9766 older than the local load 9767 atomic value being 9768 acquired. 9769 9770 4. buffer_gl0_inv 9771 9772 - If CU wavefront execution 9773 mode, omit. 9774 - If OpenCL omit. 9775 - Ensures that 9776 following 9777 loads will not see 9778 stale data. 9779 9780 atomicrmw acq_rel - workgroup - generic 1. s_waitcnt lgkmcnt(0) & 9781 vmcnt(0) & vscnt(0) 9782 9783 - If CU wavefront execution 9784 mode, omit vmcnt(0) and 9785 vscnt(0). 9786 - If OpenCL, omit lgkmcnt(0). 9787 - Could be split into 9788 separate s_waitcnt 9789 vmcnt(0), s_waitcnt 9790 vscnt(0) and s_waitcnt 9791 lgkmcnt(0) to allow 9792 them to be 9793 independently moved 9794 according to the 9795 following rules. 9796 - s_waitcnt vmcnt(0) 9797 must happen after 9798 any preceding 9799 global/generic load/load 9800 atomic/ 9801 atomicrmw-with-return-value. 9802 - s_waitcnt vscnt(0) 9803 must happen after 9804 any preceding 9805 global/generic 9806 store/store 9807 atomic/ 9808 atomicrmw-no-return-value. 9809 - s_waitcnt lgkmcnt(0) 9810 must happen after 9811 any preceding 9812 local/generic 9813 load/store/load 9814 atomic/store 9815 atomic/atomicrmw. 9816 - Must happen before 9817 the following 9818 atomicrmw. 9819 - Ensures that all 9820 memory operations 9821 have 9822 completed before 9823 performing the 9824 atomicrmw that is 9825 being released. 9826 9827 2. flat_atomic 9828 3. s_waitcnt lgkmcnt(0) & 9829 vmcnt(0) & vscnt(0) 9830 9831 - If CU wavefront execution 9832 mode, omit vmcnt(0) and 9833 vscnt(0). 9834 - If OpenCL, omit lgkmcnt(0). 9835 - Must happen before 9836 the following 9837 buffer_gl0_inv. 9838 - Ensures any 9839 following global 9840 data read is no 9841 older than the load 9842 atomic value being 9843 acquired. 9844 9845 3. buffer_gl0_inv 9846 9847 - If CU wavefront execution 9848 mode, omit. 9849 - Ensures that 9850 following 9851 loads will not see 9852 stale data. 9853 9854 atomicrmw acq_rel - agent - global 1. s_waitcnt lgkmcnt(0) & 9855 - system vmcnt(0) & vscnt(0) 9856 9857 - If OpenCL, omit 9858 lgkmcnt(0). 9859 - Could be split into 9860 separate s_waitcnt 9861 vmcnt(0), s_waitcnt 9862 vscnt(0) and s_waitcnt 9863 lgkmcnt(0) to allow 9864 them to be 9865 independently moved 9866 according to the 9867 following rules. 9868 - s_waitcnt vmcnt(0) 9869 must happen after 9870 any preceding 9871 global/generic 9872 load/load atomic/ 9873 atomicrmw-with-return-value. 9874 - s_waitcnt vscnt(0) 9875 must happen after 9876 any preceding 9877 global/generic 9878 store/store atomic/ 9879 atomicrmw-no-return-value. 9880 - s_waitcnt lgkmcnt(0) 9881 must happen after 9882 any preceding 9883 local/generic 9884 load/store/load 9885 atomic/store 9886 atomic/atomicrmw. 9887 - Must happen before 9888 the following 9889 atomicrmw. 9890 - Ensures that all 9891 memory operations 9892 to global have 9893 completed before 9894 performing the 9895 atomicrmw that is 9896 being released. 9897 9898 2. buffer/global_atomic 9899 3. s_waitcnt vm/vscnt(0) 9900 9901 - Use vmcnt(0) if atomic with 9902 return and vscnt(0) if 9903 atomic with no-return. 9904 - Must happen before 9905 following 9906 buffer_gl*_inv. 9907 - Ensures the 9908 atomicrmw has 9909 completed before 9910 invalidating the 9911 caches. 9912 9913 4. buffer_gl0_inv; 9914 buffer_gl1_inv 9915 9916 - Must happen before 9917 any following 9918 global/generic 9919 load/load 9920 atomic/atomicrmw. 9921 - Ensures that 9922 following loads 9923 will not see stale 9924 global data. 9925 9926 atomicrmw acq_rel - agent - generic 1. s_waitcnt lgkmcnt(0) & 9927 - system vmcnt(0) & vscnt(0) 9928 9929 - If OpenCL, omit 9930 lgkmcnt(0). 9931 - Could be split into 9932 separate s_waitcnt 9933 vmcnt(0), s_waitcnt 9934 vscnt(0), and s_waitcnt 9935 lgkmcnt(0) to allow 9936 them to be 9937 independently moved 9938 according to the 9939 following rules. 9940 - s_waitcnt vmcnt(0) 9941 must happen after 9942 any preceding 9943 global/generic 9944 load/load atomic 9945 atomicrmw-with-return-value. 9946 - s_waitcnt vscnt(0) 9947 must happen after 9948 any preceding 9949 global/generic 9950 store/store atomic/ 9951 atomicrmw-no-return-value. 9952 - s_waitcnt lgkmcnt(0) 9953 must happen after 9954 any preceding 9955 local/generic 9956 load/store/load 9957 atomic/store 9958 atomic/atomicrmw. 9959 - Must happen before 9960 the following 9961 atomicrmw. 9962 - Ensures that all 9963 memory operations 9964 have 9965 completed before 9966 performing the 9967 atomicrmw that is 9968 being released. 9969 9970 2. flat_atomic 9971 3. s_waitcnt vm/vscnt(0) & 9972 lgkmcnt(0) 9973 9974 - If OpenCL, omit 9975 lgkmcnt(0). 9976 - Use vmcnt(0) if atomic with 9977 return and vscnt(0) if 9978 atomic with no-return. 9979 - Must happen before 9980 following 9981 buffer_gl*_inv. 9982 - Ensures the 9983 atomicrmw has 9984 completed before 9985 invalidating the 9986 caches. 9987 9988 4. buffer_gl0_inv; 9989 buffer_gl1_inv 9990 9991 - Must happen before 9992 any following 9993 global/generic 9994 load/load 9995 atomic/atomicrmw. 9996 - Ensures that 9997 following loads 9998 will not see stale 9999 global data. 10000 10001 fence acq_rel - singlethread *none* *none* 10002 - wavefront 10003 fence acq_rel - workgroup *none* 1. s_waitcnt lgkmcnt(0) & 10004 vmcnt(0) & vscnt(0) 10005 10006 - If CU wavefront execution 10007 mode, omit vmcnt(0) and 10008 vscnt(0). 10009 - If OpenCL and 10010 address space is 10011 not generic, omit 10012 lgkmcnt(0). 10013 - If OpenCL and 10014 address space is 10015 local, omit 10016 vmcnt(0) and vscnt(0). 10017 - However, 10018 since LLVM 10019 currently has no 10020 address space on 10021 the fence need to 10022 conservatively 10023 always generate 10024 (see comment for 10025 previous fence). 10026 - Could be split into 10027 separate s_waitcnt 10028 vmcnt(0), s_waitcnt 10029 vscnt(0) and s_waitcnt 10030 lgkmcnt(0) to allow 10031 them to be 10032 independently moved 10033 according to the 10034 following rules. 10035 - s_waitcnt vmcnt(0) 10036 must happen after 10037 any preceding 10038 global/generic 10039 load/load 10040 atomic/ 10041 atomicrmw-with-return-value. 10042 - s_waitcnt vscnt(0) 10043 must happen after 10044 any preceding 10045 global/generic 10046 store/store atomic/ 10047 atomicrmw-no-return-value. 10048 - s_waitcnt lgkmcnt(0) 10049 must happen after 10050 any preceding 10051 local/generic 10052 load/store/load 10053 atomic/store atomic/ 10054 atomicrmw. 10055 - Must happen before 10056 any following 10057 global/generic 10058 load/load 10059 atomic/store/store 10060 atomic/atomicrmw. 10061 - Ensures that all 10062 memory operations 10063 have 10064 completed before 10065 performing any 10066 following global 10067 memory operations. 10068 - Ensures that the 10069 preceding 10070 local/generic load 10071 atomic/atomicrmw 10072 with an equal or 10073 wider sync scope 10074 and memory ordering 10075 stronger than 10076 unordered (this is 10077 termed the 10078 acquire-fence-paired-atomic) 10079 has completed 10080 before following 10081 global memory 10082 operations. This 10083 satisfies the 10084 requirements of 10085 acquire. 10086 - Ensures that all 10087 previous memory 10088 operations have 10089 completed before a 10090 following 10091 local/generic store 10092 atomic/atomicrmw 10093 with an equal or 10094 wider sync scope 10095 and memory ordering 10096 stronger than 10097 unordered (this is 10098 termed the 10099 release-fence-paired-atomic). 10100 This satisfies the 10101 requirements of 10102 release. 10103 - Must happen before 10104 the following 10105 buffer_gl0_inv. 10106 - Ensures that the 10107 acquire-fence-paired 10108 atomic has completed 10109 before invalidating 10110 the 10111 cache. Therefore 10112 any following 10113 locations read must 10114 be no older than 10115 the value read by 10116 the 10117 acquire-fence-paired-atomic. 10118 10119 3. buffer_gl0_inv 10120 10121 - If CU wavefront execution 10122 mode, omit. 10123 - Ensures that 10124 following 10125 loads will not see 10126 stale data. 10127 10128 fence acq_rel - agent *none* 1. s_waitcnt lgkmcnt(0) & 10129 - system vmcnt(0) & vscnt(0) 10130 10131 - If OpenCL and 10132 address space is 10133 not generic, omit 10134 lgkmcnt(0). 10135 - If OpenCL and 10136 address space is 10137 local, omit 10138 vmcnt(0) and vscnt(0). 10139 - However, since LLVM 10140 currently has no 10141 address space on 10142 the fence need to 10143 conservatively 10144 always generate 10145 (see comment for 10146 previous fence). 10147 - Could be split into 10148 separate s_waitcnt 10149 vmcnt(0), s_waitcnt 10150 vscnt(0) and s_waitcnt 10151 lgkmcnt(0) to allow 10152 them to be 10153 independently moved 10154 according to the 10155 following rules. 10156 - s_waitcnt vmcnt(0) 10157 must happen after 10158 any preceding 10159 global/generic 10160 load/load 10161 atomic/ 10162 atomicrmw-with-return-value. 10163 - s_waitcnt vscnt(0) 10164 must happen after 10165 any preceding 10166 global/generic 10167 store/store atomic/ 10168 atomicrmw-no-return-value. 10169 - s_waitcnt lgkmcnt(0) 10170 must happen after 10171 any preceding 10172 local/generic 10173 load/store/load 10174 atomic/store 10175 atomic/atomicrmw. 10176 - Must happen before 10177 the following 10178 buffer_gl*_inv. 10179 - Ensures that the 10180 preceding 10181 global/local/generic 10182 load 10183 atomic/atomicrmw 10184 with an equal or 10185 wider sync scope 10186 and memory ordering 10187 stronger than 10188 unordered (this is 10189 termed the 10190 acquire-fence-paired-atomic) 10191 has completed 10192 before invalidating 10193 the caches. This 10194 satisfies the 10195 requirements of 10196 acquire. 10197 - Ensures that all 10198 previous memory 10199 operations have 10200 completed before a 10201 following 10202 global/local/generic 10203 store 10204 atomic/atomicrmw 10205 with an equal or 10206 wider sync scope 10207 and memory ordering 10208 stronger than 10209 unordered (this is 10210 termed the 10211 release-fence-paired-atomic). 10212 This satisfies the 10213 requirements of 10214 release. 10215 10216 2. buffer_gl0_inv; 10217 buffer_gl1_inv 10218 10219 - Must happen before 10220 any following 10221 global/generic 10222 load/load 10223 atomic/store/store 10224 atomic/atomicrmw. 10225 - Ensures that 10226 following loads 10227 will not see stale 10228 global data. This 10229 satisfies the 10230 requirements of 10231 acquire. 10232 10233 **Sequential Consistent Atomic** 10234 ------------------------------------------------------------------------------------ 10235 load atomic seq_cst - singlethread - global *Same as corresponding 10236 - wavefront - local load atomic acquire, 10237 - generic except must generated 10238 all instructions even 10239 for OpenCL.* 10240 load atomic seq_cst - workgroup - global 1. s_waitcnt lgkmcnt(0) & 10241 - generic vmcnt(0) & vscnt(0) 10242 10243 - If CU wavefront execution 10244 mode, omit vmcnt(0) and 10245 vscnt(0). 10246 - Could be split into 10247 separate s_waitcnt 10248 vmcnt(0), s_waitcnt 10249 vscnt(0), and s_waitcnt 10250 lgkmcnt(0) to allow 10251 them to be 10252 independently moved 10253 according to the 10254 following rules. 10255 - s_waitcnt lgkmcnt(0) must 10256 happen after 10257 preceding 10258 local/generic load 10259 atomic/store 10260 atomic/atomicrmw 10261 with memory 10262 ordering of seq_cst 10263 and with equal or 10264 wider sync scope. 10265 (Note that seq_cst 10266 fences have their 10267 own s_waitcnt 10268 lgkmcnt(0) and so do 10269 not need to be 10270 considered.) 10271 - s_waitcnt vmcnt(0) 10272 must happen after 10273 preceding 10274 global/generic load 10275 atomic/ 10276 atomicrmw-with-return-value 10277 with memory 10278 ordering of seq_cst 10279 and with equal or 10280 wider sync scope. 10281 (Note that seq_cst 10282 fences have their 10283 own s_waitcnt 10284 vmcnt(0) and so do 10285 not need to be 10286 considered.) 10287 - s_waitcnt vscnt(0) 10288 Must happen after 10289 preceding 10290 global/generic store 10291 atomic/ 10292 atomicrmw-no-return-value 10293 with memory 10294 ordering of seq_cst 10295 and with equal or 10296 wider sync scope. 10297 (Note that seq_cst 10298 fences have their 10299 own s_waitcnt 10300 vscnt(0) and so do 10301 not need to be 10302 considered.) 10303 - Ensures any 10304 preceding 10305 sequential 10306 consistent global/local 10307 memory instructions 10308 have completed 10309 before executing 10310 this sequentially 10311 consistent 10312 instruction. This 10313 prevents reordering 10314 a seq_cst store 10315 followed by a 10316 seq_cst load. (Note 10317 that seq_cst is 10318 stronger than 10319 acquire/release as 10320 the reordering of 10321 load acquire 10322 followed by a store 10323 release is 10324 prevented by the 10325 s_waitcnt of 10326 the release, but 10327 there is nothing 10328 preventing a store 10329 release followed by 10330 load acquire from 10331 completing out of 10332 order. The s_waitcnt 10333 could be placed after 10334 seq_store or before 10335 the seq_load. We 10336 choose the load to 10337 make the s_waitcnt be 10338 as late as possible 10339 so that the store 10340 may have already 10341 completed.) 10342 10343 2. *Following 10344 instructions same as 10345 corresponding load 10346 atomic acquire, 10347 except must generated 10348 all instructions even 10349 for OpenCL.* 10350 load atomic seq_cst - workgroup - local 10351 10352 1. s_waitcnt vmcnt(0) & vscnt(0) 10353 10354 - If CU wavefront execution 10355 mode, omit. 10356 - Could be split into 10357 separate s_waitcnt 10358 vmcnt(0) and s_waitcnt 10359 vscnt(0) to allow 10360 them to be 10361 independently moved 10362 according to the 10363 following rules. 10364 - s_waitcnt vmcnt(0) 10365 Must happen after 10366 preceding 10367 global/generic load 10368 atomic/ 10369 atomicrmw-with-return-value 10370 with memory 10371 ordering of seq_cst 10372 and with equal or 10373 wider sync scope. 10374 (Note that seq_cst 10375 fences have their 10376 own s_waitcnt 10377 vmcnt(0) and so do 10378 not need to be 10379 considered.) 10380 - s_waitcnt vscnt(0) 10381 Must happen after 10382 preceding 10383 global/generic store 10384 atomic/ 10385 atomicrmw-no-return-value 10386 with memory 10387 ordering of seq_cst 10388 and with equal or 10389 wider sync scope. 10390 (Note that seq_cst 10391 fences have their 10392 own s_waitcnt 10393 vscnt(0) and so do 10394 not need to be 10395 considered.) 10396 - Ensures any 10397 preceding 10398 sequential 10399 consistent global 10400 memory instructions 10401 have completed 10402 before executing 10403 this sequentially 10404 consistent 10405 instruction. This 10406 prevents reordering 10407 a seq_cst store 10408 followed by a 10409 seq_cst load. (Note 10410 that seq_cst is 10411 stronger than 10412 acquire/release as 10413 the reordering of 10414 load acquire 10415 followed by a store 10416 release is 10417 prevented by the 10418 s_waitcnt of 10419 the release, but 10420 there is nothing 10421 preventing a store 10422 release followed by 10423 load acquire from 10424 completing out of 10425 order. The s_waitcnt 10426 could be placed after 10427 seq_store or before 10428 the seq_load. We 10429 choose the load to 10430 make the s_waitcnt be 10431 as late as possible 10432 so that the store 10433 may have already 10434 completed.) 10435 10436 2. *Following 10437 instructions same as 10438 corresponding load 10439 atomic acquire, 10440 except must generated 10441 all instructions even 10442 for OpenCL.* 10443 10444 load atomic seq_cst - agent - global 1. s_waitcnt lgkmcnt(0) & 10445 - system - generic vmcnt(0) & vscnt(0) 10446 10447 - Could be split into 10448 separate s_waitcnt 10449 vmcnt(0), s_waitcnt 10450 vscnt(0) and s_waitcnt 10451 lgkmcnt(0) to allow 10452 them to be 10453 independently moved 10454 according to the 10455 following rules. 10456 - s_waitcnt lgkmcnt(0) 10457 must happen after 10458 preceding 10459 local load 10460 atomic/store 10461 atomic/atomicrmw 10462 with memory 10463 ordering of seq_cst 10464 and with equal or 10465 wider sync scope. 10466 (Note that seq_cst 10467 fences have their 10468 own s_waitcnt 10469 lgkmcnt(0) and so do 10470 not need to be 10471 considered.) 10472 - s_waitcnt vmcnt(0) 10473 must happen after 10474 preceding 10475 global/generic load 10476 atomic/ 10477 atomicrmw-with-return-value 10478 with memory 10479 ordering of seq_cst 10480 and with equal or 10481 wider sync scope. 10482 (Note that seq_cst 10483 fences have their 10484 own s_waitcnt 10485 vmcnt(0) and so do 10486 not need to be 10487 considered.) 10488 - s_waitcnt vscnt(0) 10489 Must happen after 10490 preceding 10491 global/generic store 10492 atomic/ 10493 atomicrmw-no-return-value 10494 with memory 10495 ordering of seq_cst 10496 and with equal or 10497 wider sync scope. 10498 (Note that seq_cst 10499 fences have their 10500 own s_waitcnt 10501 vscnt(0) and so do 10502 not need to be 10503 considered.) 10504 - Ensures any 10505 preceding 10506 sequential 10507 consistent global 10508 memory instructions 10509 have completed 10510 before executing 10511 this sequentially 10512 consistent 10513 instruction. This 10514 prevents reordering 10515 a seq_cst store 10516 followed by a 10517 seq_cst load. (Note 10518 that seq_cst is 10519 stronger than 10520 acquire/release as 10521 the reordering of 10522 load acquire 10523 followed by a store 10524 release is 10525 prevented by the 10526 s_waitcnt of 10527 the release, but 10528 there is nothing 10529 preventing a store 10530 release followed by 10531 load acquire from 10532 completing out of 10533 order. The s_waitcnt 10534 could be placed after 10535 seq_store or before 10536 the seq_load. We 10537 choose the load to 10538 make the s_waitcnt be 10539 as late as possible 10540 so that the store 10541 may have already 10542 completed.) 10543 10544 2. *Following 10545 instructions same as 10546 corresponding load 10547 atomic acquire, 10548 except must generated 10549 all instructions even 10550 for OpenCL.* 10551 store atomic seq_cst - singlethread - global *Same as corresponding 10552 - wavefront - local store atomic release, 10553 - workgroup - generic except must generated 10554 - agent all instructions even 10555 - system for OpenCL.* 10556 atomicrmw seq_cst - singlethread - global *Same as corresponding 10557 - wavefront - local atomicrmw acq_rel, 10558 - workgroup - generic except must generated 10559 - agent all instructions even 10560 - system for OpenCL.* 10561 fence seq_cst - singlethread *none* *Same as corresponding 10562 - wavefront fence acq_rel, 10563 - workgroup except must generated 10564 - agent all instructions even 10565 - system for OpenCL.* 10566 ============ ============ ============== ========== ================================ 10567 10568Trap Handler ABI 10569~~~~~~~~~~~~~~~~ 10570 10571For code objects generated by the AMDGPU backend for HSA [HSA]_ compatible 10572runtimes (see :ref:`amdgpu-os`), the runtime installs a trap handler that 10573supports the ``s_trap`` instruction. For usage see: 10574 10575- :ref:`amdgpu-trap-handler-for-amdhsa-os-v2-table` 10576- :ref:`amdgpu-trap-handler-for-amdhsa-os-v3-table` 10577- :ref:`amdgpu-trap-handler-for-amdhsa-os-v4-table` 10578 10579 .. table:: AMDGPU Trap Handler for AMDHSA OS Code Object V2 10580 :name: amdgpu-trap-handler-for-amdhsa-os-v2-table 10581 10582 =================== =============== =============== ======================================= 10583 Usage Code Sequence Trap Handler Description 10584 Inputs 10585 =================== =============== =============== ======================================= 10586 reserved ``s_trap 0x00`` Reserved by hardware. 10587 ``debugtrap(arg)`` ``s_trap 0x01`` ``SGPR0-1``: Reserved for Finalizer HSA ``debugtrap`` 10588 ``queue_ptr`` intrinsic (not implemented). 10589 ``VGPR0``: 10590 ``arg`` 10591 ``llvm.trap`` ``s_trap 0x02`` ``SGPR0-1``: Causes wave to be halted with the PC at 10592 ``queue_ptr`` the trap instruction. The associated 10593 queue is signalled to put it into the 10594 error state. When the queue is put in 10595 the error state, the waves executing 10596 dispatches on the queue will be 10597 terminated. 10598 ``llvm.debugtrap`` ``s_trap 0x03`` *none* - If debugger not enabled then behaves 10599 as a no-operation. The trap handler 10600 is entered and immediately returns to 10601 continue execution of the wavefront. 10602 - If the debugger is enabled, causes 10603 the debug trap to be reported by the 10604 debugger and the wavefront is put in 10605 the halt state with the PC at the 10606 instruction. The debugger must 10607 increment the PC and resume the wave. 10608 reserved ``s_trap 0x04`` Reserved. 10609 reserved ``s_trap 0x05`` Reserved. 10610 reserved ``s_trap 0x06`` Reserved. 10611 reserved ``s_trap 0x07`` Reserved. 10612 reserved ``s_trap 0x08`` Reserved. 10613 reserved ``s_trap 0xfe`` Reserved. 10614 reserved ``s_trap 0xff`` Reserved. 10615 =================== =============== =============== ======================================= 10616 10617.. 10618 10619 .. table:: AMDGPU Trap Handler for AMDHSA OS Code Object V3 10620 :name: amdgpu-trap-handler-for-amdhsa-os-v3-table 10621 10622 =================== =============== =============== ======================================= 10623 Usage Code Sequence Trap Handler Description 10624 Inputs 10625 =================== =============== =============== ======================================= 10626 reserved ``s_trap 0x00`` Reserved by hardware. 10627 debugger breakpoint ``s_trap 0x01`` *none* Reserved for debugger to use for 10628 breakpoints. Causes wave to be halted 10629 with the PC at the trap instruction. 10630 The debugger is responsible to resume 10631 the wave, including the instruction 10632 that the breakpoint overwrote. 10633 ``llvm.trap`` ``s_trap 0x02`` ``SGPR0-1``: Causes wave to be halted with the PC at 10634 ``queue_ptr`` the trap instruction. The associated 10635 queue is signalled to put it into the 10636 error state. When the queue is put in 10637 the error state, the waves executing 10638 dispatches on the queue will be 10639 terminated. 10640 ``llvm.debugtrap`` ``s_trap 0x03`` *none* - If debugger not enabled then behaves 10641 as a no-operation. The trap handler 10642 is entered and immediately returns to 10643 continue execution of the wavefront. 10644 - If the debugger is enabled, causes 10645 the debug trap to be reported by the 10646 debugger and the wavefront is put in 10647 the halt state with the PC at the 10648 instruction. The debugger must 10649 increment the PC and resume the wave. 10650 reserved ``s_trap 0x04`` Reserved. 10651 reserved ``s_trap 0x05`` Reserved. 10652 reserved ``s_trap 0x06`` Reserved. 10653 reserved ``s_trap 0x07`` Reserved. 10654 reserved ``s_trap 0x08`` Reserved. 10655 reserved ``s_trap 0xfe`` Reserved. 10656 reserved ``s_trap 0xff`` Reserved. 10657 =================== =============== =============== ======================================= 10658 10659.. 10660 10661 .. table:: AMDGPU Trap Handler for AMDHSA OS Code Object V4 10662 :name: amdgpu-trap-handler-for-amdhsa-os-v4-table 10663 10664 =================== =============== ================ ================= ======================================= 10665 Usage Code Sequence GFX6-GFX8 Inputs GFX9-GFX10 Inputs Description 10666 =================== =============== ================ ================= ======================================= 10667 reserved ``s_trap 0x00`` Reserved by hardware. 10668 debugger breakpoint ``s_trap 0x01`` *none* *none* Reserved for debugger to use for 10669 breakpoints. Causes wave to be halted 10670 with the PC at the trap instruction. 10671 The debugger is responsible to resume 10672 the wave, including the instruction 10673 that the breakpoint overwrote. 10674 ``llvm.trap`` ``s_trap 0x02`` ``SGPR0-1``: *none* Causes wave to be halted with the PC at 10675 ``queue_ptr`` the trap instruction. The associated 10676 queue is signalled to put it into the 10677 error state. When the queue is put in 10678 the error state, the waves executing 10679 dispatches on the queue will be 10680 terminated. 10681 ``llvm.debugtrap`` ``s_trap 0x03`` *none* *none* - If debugger not enabled then behaves 10682 as a no-operation. The trap handler 10683 is entered and immediately returns to 10684 continue execution of the wavefront. 10685 - If the debugger is enabled, causes 10686 the debug trap to be reported by the 10687 debugger and the wavefront is put in 10688 the halt state with the PC at the 10689 instruction. The debugger must 10690 increment the PC and resume the wave. 10691 reserved ``s_trap 0x04`` Reserved. 10692 reserved ``s_trap 0x05`` Reserved. 10693 reserved ``s_trap 0x06`` Reserved. 10694 reserved ``s_trap 0x07`` Reserved. 10695 reserved ``s_trap 0x08`` Reserved. 10696 reserved ``s_trap 0xfe`` Reserved. 10697 reserved ``s_trap 0xff`` Reserved. 10698 =================== =============== ================ ================= ======================================= 10699 10700.. _amdgpu-amdhsa-function-call-convention: 10701 10702Call Convention 10703~~~~~~~~~~~~~~~ 10704 10705.. note:: 10706 10707 This section is currently incomplete and has inaccuracies. It is WIP that will 10708 be updated as information is determined. 10709 10710See :ref:`amdgpu-dwarf-address-space-identifier` for information on swizzled 10711addresses. Unswizzled addresses are normal linear addresses. 10712 10713.. _amdgpu-amdhsa-function-call-convention-kernel-functions: 10714 10715Kernel Functions 10716++++++++++++++++ 10717 10718This section describes the call convention ABI for the outer kernel function. 10719 10720See :ref:`amdgpu-amdhsa-initial-kernel-execution-state` for the kernel call 10721convention. 10722 10723The following is not part of the AMDGPU kernel calling convention but describes 10724how the AMDGPU implements function calls: 10725 107261. Clang decides the kernarg layout to match the *HSA Programmer's Language 10727 Reference* [HSA]_. 10728 10729 - All structs are passed directly. 10730 - Lambda values are passed *TBA*. 10731 10732 .. TODO:: 10733 10734 - Does this really follow HSA rules? Or are structs >16 bytes passed 10735 by-value struct? 10736 - What is ABI for lambda values? 10737 107384. The kernel performs certain setup in its prolog, as described in 10739 :ref:`amdgpu-amdhsa-kernel-prolog`. 10740 10741.. _amdgpu-amdhsa-function-call-convention-non-kernel-functions: 10742 10743Non-Kernel Functions 10744++++++++++++++++++++ 10745 10746This section describes the call convention ABI for functions other than the 10747outer kernel function. 10748 10749If a kernel has function calls then scratch is always allocated and used for 10750the call stack which grows from low address to high address using the swizzled 10751scratch address space. 10752 10753On entry to a function: 10754 107551. SGPR0-3 contain a V# with the following properties (see 10756 :ref:`amdgpu-amdhsa-kernel-prolog-private-segment-buffer`): 10757 10758 * Base address pointing to the beginning of the wavefront scratch backing 10759 memory. 10760 * Swizzled with dword element size and stride of wavefront size elements. 10761 107622. The FLAT_SCRATCH register pair is setup. See 10763 :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`. 107643. GFX6-GFX8: M0 register set to the size of LDS in bytes. See 10765 :ref:`amdgpu-amdhsa-kernel-prolog-m0`. 107664. The EXEC register is set to the lanes active on entry to the function. 107675. MODE register: *TBD* 107686. VGPR0-31 and SGPR4-29 are used to pass function input arguments as described 10769 below. 107707. SGPR30-31 return address (RA). The code address that the function must 10771 return to when it completes. The value is undefined if the function is *no 10772 return*. 107738. SGPR32 is used for the stack pointer (SP). It is an unswizzled scratch 10774 offset relative to the beginning of the wavefront scratch backing memory. 10775 10776 The unswizzled SP can be used with buffer instructions as an unswizzled SGPR 10777 offset with the scratch V# in SGPR0-3 to access the stack in a swizzled 10778 manner. 10779 10780 The unswizzled SP value can be converted into the swizzled SP value by: 10781 10782 | swizzled SP = unswizzled SP / wavefront size 10783 10784 This may be used to obtain the private address space address of stack 10785 objects and to convert this address to a flat address by adding the flat 10786 scratch aperture base address. 10787 10788 The swizzled SP value is always 4 bytes aligned for the ``r600`` 10789 architecture and 16 byte aligned for the ``amdgcn`` architecture. 10790 10791 .. note:: 10792 10793 The ``amdgcn`` value is selected to avoid dynamic stack alignment for the 10794 OpenCL language which has the largest base type defined as 16 bytes. 10795 10796 On entry, the swizzled SP value is the address of the first function 10797 argument passed on the stack. Other stack passed arguments are positive 10798 offsets from the entry swizzled SP value. 10799 10800 The function may use positive offsets beyond the last stack passed argument 10801 for stack allocated local variables and register spill slots. If necessary, 10802 the function may align these to greater alignment than 16 bytes. After these 10803 the function may dynamically allocate space for such things as runtime sized 10804 ``alloca`` local allocations. 10805 10806 If the function calls another function, it will place any stack allocated 10807 arguments after the last local allocation and adjust SGPR32 to the address 10808 after the last local allocation. 10809 108109. All other registers are unspecified. 1081110. Any necessary ``s_waitcnt`` has been performed to ensure memory is available 10812 to the function. 10813 10814On exit from a function: 10815 108161. VGPR0-31 and SGPR4-29 are used to pass function result arguments as 10817 described below. Any registers used are considered clobbered registers. 108182. The following registers are preserved and have the same value as on entry: 10819 10820 * FLAT_SCRATCH 10821 * EXEC 10822 * GFX6-GFX8: M0 10823 * All SGPR registers except the clobbered registers of SGPR4-31. 10824 * VGPR40-47 10825 * VGPR56-63 10826 * VGPR72-79 10827 * VGPR88-95 10828 * VGPR104-111 10829 * VGPR120-127 10830 * VGPR136-143 10831 * VGPR152-159 10832 * VGPR168-175 10833 * VGPR184-191 10834 * VGPR200-207 10835 * VGPR216-223 10836 * VGPR232-239 10837 * VGPR248-255 10838 10839 .. note:: 10840 10841 Except the argument registers, the VGPRs clobbered and the preserved 10842 registers are intermixed at regular intervals in order to keep a 10843 similar ratio independent of the number of allocated VGPRs. 10844 10845 * GFX90A: All AGPR registers except the clobbered registers AGPR0-31. 10846 * Lanes of all VGPRs that are inactive at the call site. 10847 10848 For the AMDGPU backend, an inter-procedural register allocation (IPRA) 10849 optimization may mark some of clobbered SGPR and VGPR registers as 10850 preserved if it can be determined that the called function does not change 10851 their value. 10852 108532. The PC is set to the RA provided on entry. 108543. MODE register: *TBD*. 108554. All other registers are clobbered. 108565. Any necessary ``s_waitcnt`` has been performed to ensure memory accessed by 10857 function is available to the caller. 10858 10859.. TODO:: 10860 10861 - How are function results returned? The address of structured types is passed 10862 by reference, but what about other types? 10863 10864The function input arguments are made up of the formal arguments explicitly 10865declared by the source language function plus the implicit input arguments used 10866by the implementation. 10867 10868The source language input arguments are: 10869 108701. Any source language implicit ``this`` or ``self`` argument comes first as a 10871 pointer type. 108722. Followed by the function formal arguments in left to right source order. 10873 10874The source language result arguments are: 10875 108761. The function result argument. 10877 10878The source language input or result struct type arguments that are less than or 10879equal to 16 bytes, are decomposed recursively into their base type fields, and 10880each field is passed as if a separate argument. For input arguments, if the 10881called function requires the struct to be in memory, for example because its 10882address is taken, then the function body is responsible for allocating a stack 10883location and copying the field arguments into it. Clang terms this *direct 10884struct*. 10885 10886The source language input struct type arguments that are greater than 16 bytes, 10887are passed by reference. The caller is responsible for allocating a stack 10888location to make a copy of the struct value and pass the address as the input 10889argument. The called function is responsible to perform the dereference when 10890accessing the input argument. Clang terms this *by-value struct*. 10891 10892A source language result struct type argument that is greater than 16 bytes, is 10893returned by reference. The caller is responsible for allocating a stack location 10894to hold the result value and passes the address as the last input argument 10895(before the implicit input arguments). In this case there are no result 10896arguments. The called function is responsible to perform the dereference when 10897storing the result value. Clang terms this *structured return (sret)*. 10898 10899*TODO: correct the ``sret`` definition.* 10900 10901.. TODO:: 10902 10903 Is this definition correct? Or is ``sret`` only used if passing in registers, and 10904 pass as non-decomposed struct as stack argument? Or something else? Is the 10905 memory location in the caller stack frame, or a stack memory argument and so 10906 no address is passed as the caller can directly write to the argument stack 10907 location? But then the stack location is still live after return. If an 10908 argument stack location is it the first stack argument or the last one? 10909 10910Lambda argument types are treated as struct types with an implementation defined 10911set of fields. 10912 10913.. TODO:: 10914 10915 Need to specify the ABI for lambda types for AMDGPU. 10916 10917For AMDGPU backend all source language arguments (including the decomposed 10918struct type arguments) are passed in VGPRs unless marked ``inreg`` in which case 10919they are passed in SGPRs. 10920 10921The AMDGPU backend walks the function call graph from the leaves to determine 10922which implicit input arguments are used, propagating to each caller of the 10923function. The used implicit arguments are appended to the function arguments 10924after the source language arguments in the following order: 10925 10926.. TODO:: 10927 10928 Is recursion or external functions supported? 10929 109301. Work-Item ID (1 VGPR) 10931 10932 The X, Y and Z work-item ID are packed into a single VGRP with the following 10933 layout. Only fields actually used by the function are set. The other bits 10934 are undefined. 10935 10936 The values come from the initial kernel execution state. See 10937 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`. 10938 10939 .. table:: Work-item implicit argument layout 10940 :name: amdgpu-amdhsa-workitem-implicit-argument-layout-table 10941 10942 ======= ======= ============== 10943 Bits Size Field Name 10944 ======= ======= ============== 10945 9:0 10 bits X Work-Item ID 10946 19:10 10 bits Y Work-Item ID 10947 29:20 10 bits Z Work-Item ID 10948 31:30 2 bits Unused 10949 ======= ======= ============== 10950 109512. Dispatch Ptr (2 SGPRs) 10952 10953 The value comes from the initial kernel execution state. See 10954 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 10955 109563. Queue Ptr (2 SGPRs) 10957 10958 The value comes from the initial kernel execution state. See 10959 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 10960 109614. Kernarg Segment Ptr (2 SGPRs) 10962 10963 The value comes from the initial kernel execution state. See 10964 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 10965 109665. Dispatch id (2 SGPRs) 10967 10968 The value comes from the initial kernel execution state. See 10969 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 10970 109716. Work-Group ID X (1 SGPR) 10972 10973 The value comes from the initial kernel execution state. See 10974 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 10975 109767. Work-Group ID Y (1 SGPR) 10977 10978 The value comes from the initial kernel execution state. See 10979 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 10980 109818. Work-Group ID Z (1 SGPR) 10982 10983 The value comes from the initial kernel execution state. See 10984 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 10985 109869. Implicit Argument Ptr (2 SGPRs) 10987 10988 The value is computed by adding an offset to Kernarg Segment Ptr to get the 10989 global address space pointer to the first kernarg implicit argument. 10990 10991The input and result arguments are assigned in order in the following manner: 10992 10993.. note:: 10994 10995 There are likely some errors and omissions in the following description that 10996 need correction. 10997 10998 .. TODO:: 10999 11000 Check the Clang source code to decipher how function arguments and return 11001 results are handled. Also see the AMDGPU specific values used. 11002 11003* VGPR arguments are assigned to consecutive VGPRs starting at VGPR0 up to 11004 VGPR31. 11005 11006 If there are more arguments than will fit in these registers, the remaining 11007 arguments are allocated on the stack in order on naturally aligned 11008 addresses. 11009 11010 .. TODO:: 11011 11012 How are overly aligned structures allocated on the stack? 11013 11014* SGPR arguments are assigned to consecutive SGPRs starting at SGPR0 up to 11015 SGPR29. 11016 11017 If there are more arguments than will fit in these registers, the remaining 11018 arguments are allocated on the stack in order on naturally aligned 11019 addresses. 11020 11021Note that decomposed struct type arguments may have some fields passed in 11022registers and some in memory. 11023 11024.. TODO:: 11025 11026 So, a struct which can pass some fields as decomposed register arguments, will 11027 pass the rest as decomposed stack elements? But an argument that will not start 11028 in registers will not be decomposed and will be passed as a non-decomposed 11029 stack value? 11030 11031The following is not part of the AMDGPU function calling convention but 11032describes how the AMDGPU implements function calls: 11033 110341. SGPR33 is used as a frame pointer (FP) if necessary. Like the SP it is an 11035 unswizzled scratch address. It is only needed if runtime sized ``alloca`` 11036 are used, or for the reasons defined in ``SIFrameLowering``. 110372. Runtime stack alignment is supported. SGPR34 is used as a base pointer (BP) 11038 to access the incoming stack arguments in the function. The BP is needed 11039 only when the function requires the runtime stack alignment. 11040 110413. Allocating SGPR arguments on the stack are not supported. 11042 110434. No CFI is currently generated. See 11044 :ref:`amdgpu-dwarf-call-frame-information`. 11045 11046 .. note:: 11047 11048 CFI will be generated that defines the CFA as the unswizzled address 11049 relative to the wave scratch base in the unswizzled private address space 11050 of the lowest address stack allocated local variable. 11051 11052 ``DW_AT_frame_base`` will be defined as the swizzled address in the 11053 swizzled private address space by dividing the CFA by the wavefront size 11054 (since CFA is always at least dword aligned which matches the scratch 11055 swizzle element size). 11056 11057 If no dynamic stack alignment was performed, the stack allocated arguments 11058 are accessed as negative offsets relative to ``DW_AT_frame_base``, and the 11059 local variables and register spill slots are accessed as positive offsets 11060 relative to ``DW_AT_frame_base``. 11061 110625. Function argument passing is implemented by copying the input physical 11063 registers to virtual registers on entry. The register allocator can spill if 11064 necessary. These are copied back to physical registers at call sites. The 11065 net effect is that each function call can have these values in entirely 11066 distinct locations. The IPRA can help avoid shuffling argument registers. 110676. Call sites are implemented by setting up the arguments at positive offsets 11068 from SP. Then SP is incremented to account for the known frame size before 11069 the call and decremented after the call. 11070 11071 .. note:: 11072 11073 The CFI will reflect the changed calculation needed to compute the CFA 11074 from SP. 11075 110767. 4 byte spill slots are used in the stack frame. One slot is allocated for an 11077 emergency spill slot. Buffer instructions are used for stack accesses and 11078 not the ``flat_scratch`` instruction. 11079 11080 .. TODO:: 11081 11082 Explain when the emergency spill slot is used. 11083 11084.. TODO:: 11085 11086 Possible broken issues: 11087 11088 - Stack arguments must be aligned to required alignment. 11089 - Stack is aligned to max(16, max formal argument alignment) 11090 - Direct argument < 64 bits should check register budget. 11091 - Register budget calculation should respect ``inreg`` for SGPR. 11092 - SGPR overflow is not handled. 11093 - struct with 1 member unpeeling is not checking size of member. 11094 - ``sret`` is after ``this`` pointer. 11095 - Caller is not implementing stack realignment: need an extra pointer. 11096 - Should say AMDGPU passes FP rather than SP. 11097 - Should CFI define CFA as address of locals or arguments. Difference is 11098 apparent when have implemented dynamic alignment. 11099 - If ``SCRATCH`` instruction could allow negative offsets, then can make FP be 11100 highest address of stack frame and use negative offset for locals. Would 11101 allow SP to be the same as FP and could support signal-handler-like as now 11102 have a real SP for the top of the stack. 11103 - How is ``sret`` passed on the stack? In argument stack area? Can it overlay 11104 arguments? 11105 11106AMDPAL 11107------ 11108 11109This section provides code conventions used when the target triple OS is 11110``amdpal`` (see :ref:`amdgpu-target-triples`). 11111 11112.. _amdgpu-amdpal-code-object-metadata-section: 11113 11114Code Object Metadata 11115~~~~~~~~~~~~~~~~~~~~ 11116 11117.. note:: 11118 11119 The metadata is currently in development and is subject to major 11120 changes. Only the current version is supported. *When this document 11121 was generated the version was 2.6.* 11122 11123Code object metadata is specified by the ``NT_AMDGPU_METADATA`` note 11124record (see :ref:`amdgpu-note-records-v3-v4`). 11125 11126The metadata is represented as Message Pack formatted binary data (see 11127[MsgPack]_). The top level is a Message Pack map that includes the keys 11128defined in table :ref:`amdgpu-amdpal-code-object-metadata-map-table` 11129and referenced tables. 11130 11131Additional information can be added to the maps. To avoid conflicts, any 11132key names should be prefixed by "*vendor-name*." where ``vendor-name`` 11133can be the name of the vendor and specific vendor tool that generates the 11134information. The prefix is abbreviated to simply "." when it appears 11135within a map that has been added by the same *vendor-name*. 11136 11137 .. table:: AMDPAL Code Object Metadata Map 11138 :name: amdgpu-amdpal-code-object-metadata-map-table 11139 11140 =================== ============== ========= ====================================================================== 11141 String Key Value Type Required? Description 11142 =================== ============== ========= ====================================================================== 11143 "amdpal.version" sequence of Required PAL code object metadata (major, minor) version. The current values 11144 2 integers are defined by *Util::Abi::PipelineMetadata(Major|Minor)Version*. 11145 "amdpal.pipelines" sequence of Required Per-pipeline metadata. See 11146 map :ref:`amdgpu-amdpal-code-object-pipeline-metadata-map-table` for the 11147 definition of the keys included in that map. 11148 =================== ============== ========= ====================================================================== 11149 11150.. 11151 11152 .. table:: AMDPAL Code Object Pipeline Metadata Map 11153 :name: amdgpu-amdpal-code-object-pipeline-metadata-map-table 11154 11155 ====================================== ============== ========= =================================================== 11156 String Key Value Type Required? Description 11157 ====================================== ============== ========= =================================================== 11158 ".name" string Source name of the pipeline. 11159 ".type" string Pipeline type, e.g. VsPs. Values include: 11160 11161 - "VsPs" 11162 - "Gs" 11163 - "Cs" 11164 - "Ngg" 11165 - "Tess" 11166 - "GsTess" 11167 - "NggTess" 11168 11169 ".internal_pipeline_hash" sequence of Required Internal compiler hash for this pipeline. Lower 11170 2 integers 64 bits is the "stable" portion of the hash, used 11171 for e.g. shader replacement lookup. Upper 64 bits 11172 is the "unique" portion of the hash, used for 11173 e.g. pipeline cache lookup. The value is 11174 implementation defined, and can not be relied on 11175 between different builds of the compiler. 11176 ".shaders" map Per-API shader metadata. See 11177 :ref:`amdgpu-amdpal-code-object-shader-map-table` 11178 for the definition of the keys included in that 11179 map. 11180 ".hardware_stages" map Per-hardware stage metadata. See 11181 :ref:`amdgpu-amdpal-code-object-hardware-stage-map-table` 11182 for the definition of the keys included in that 11183 map. 11184 ".shader_functions" map Per-shader function metadata. See 11185 :ref:`amdgpu-amdpal-code-object-shader-function-map-table` 11186 for the definition of the keys included in that 11187 map. 11188 ".registers" map Required Hardware register configuration. See 11189 :ref:`amdgpu-amdpal-code-object-register-map-table` 11190 for the definition of the keys included in that 11191 map. 11192 ".user_data_limit" integer Number of user data entries accessed by this 11193 pipeline. 11194 ".spill_threshold" integer The user data spill threshold. 0xFFFF for 11195 NoUserDataSpilling. 11196 ".uses_viewport_array_index" boolean Indicates whether or not the pipeline uses the 11197 viewport array index feature. Pipelines which use 11198 this feature can render into all 16 viewports, 11199 whereas pipelines which do not use it are 11200 restricted to viewport #0. 11201 ".es_gs_lds_size" integer Size in bytes of LDS space used internally for 11202 handling data-passing between the ES and GS 11203 shader stages. This can be zero if the data is 11204 passed using off-chip buffers. This value should 11205 be used to program all user-SGPRs which have been 11206 marked with "UserDataMapping::EsGsLdsSize" 11207 (typically only the GS and VS HW stages will ever 11208 have a user-SGPR so marked). 11209 ".nggSubgroupSize" integer Explicit maximum subgroup size for NGG shaders 11210 (maximum number of threads in a subgroup). 11211 ".num_interpolants" integer Graphics only. Number of PS interpolants. 11212 ".mesh_scratch_memory_size" integer Max mesh shader scratch memory used. 11213 ".api" string Name of the client graphics API. 11214 ".api_create_info" binary Graphics API shader create info binary blob. Can 11215 be defined by the driver using the compiler if 11216 they want to be able to correlate API-specific 11217 information used during creation at a later time. 11218 ====================================== ============== ========= =================================================== 11219 11220.. 11221 11222 .. table:: AMDPAL Code Object Shader Map 11223 :name: amdgpu-amdpal-code-object-shader-map-table 11224 11225 11226 +-------------+--------------+-------------------------------------------------------------------+ 11227 |String Key |Value Type |Description | 11228 +=============+==============+===================================================================+ 11229 |- ".compute" |map |See :ref:`amdgpu-amdpal-code-object-api-shader-metadata-map-table` | 11230 |- ".vertex" | |for the definition of the keys included in that map. | 11231 |- ".hull" | | | 11232 |- ".domain" | | | 11233 |- ".geometry"| | | 11234 |- ".pixel" | | | 11235 +-------------+--------------+-------------------------------------------------------------------+ 11236 11237.. 11238 11239 .. table:: AMDPAL Code Object API Shader Metadata Map 11240 :name: amdgpu-amdpal-code-object-api-shader-metadata-map-table 11241 11242 ==================== ============== ========= ===================================================================== 11243 String Key Value Type Required? Description 11244 ==================== ============== ========= ===================================================================== 11245 ".api_shader_hash" sequence of Required Input shader hash, typically passed in from the client. The value 11246 2 integers is implementation defined, and can not be relied on between 11247 different builds of the compiler. 11248 ".hardware_mapping" sequence of Required Flags indicating the HW stages this API shader maps to. Values 11249 string include: 11250 11251 - ".ls" 11252 - ".hs" 11253 - ".es" 11254 - ".gs" 11255 - ".vs" 11256 - ".ps" 11257 - ".cs" 11258 11259 ==================== ============== ========= ===================================================================== 11260 11261.. 11262 11263 .. table:: AMDPAL Code Object Hardware Stage Map 11264 :name: amdgpu-amdpal-code-object-hardware-stage-map-table 11265 11266 +-------------+--------------+-----------------------------------------------------------------------+ 11267 |String Key |Value Type |Description | 11268 +=============+==============+=======================================================================+ 11269 |- ".ls" |map |See :ref:`amdgpu-amdpal-code-object-hardware-stage-metadata-map-table` | 11270 |- ".hs" | |for the definition of the keys included in that map. | 11271 |- ".es" | | | 11272 |- ".gs" | | | 11273 |- ".vs" | | | 11274 |- ".ps" | | | 11275 |- ".cs" | | | 11276 +-------------+--------------+-----------------------------------------------------------------------+ 11277 11278.. 11279 11280 .. table:: AMDPAL Code Object Hardware Stage Metadata Map 11281 :name: amdgpu-amdpal-code-object-hardware-stage-metadata-map-table 11282 11283 ========================== ============== ========= =============================================================== 11284 String Key Value Type Required? Description 11285 ========================== ============== ========= =============================================================== 11286 ".entry_point" string The ELF symbol pointing to this pipeline's stage entry point. 11287 ".scratch_memory_size" integer Scratch memory size in bytes. 11288 ".lds_size" integer Local Data Share size in bytes. 11289 ".perf_data_buffer_size" integer Performance data buffer size in bytes. 11290 ".vgpr_count" integer Number of VGPRs used. 11291 ".sgpr_count" integer Number of SGPRs used. 11292 ".vgpr_limit" integer If non-zero, indicates the shader was compiled with a 11293 directive to instruct the compiler to limit the VGPR usage to 11294 be less than or equal to the specified value (only set if 11295 different from HW default). 11296 ".sgpr_limit" integer SGPR count upper limit (only set if different from HW 11297 default). 11298 ".threadgroup_dimensions" sequence of Thread-group X/Y/Z dimensions (Compute only). 11299 3 integers 11300 ".wavefront_size" integer Wavefront size (only set if different from HW default). 11301 ".uses_uavs" boolean The shader reads or writes UAVs. 11302 ".uses_rovs" boolean The shader reads or writes ROVs. 11303 ".writes_uavs" boolean The shader writes to one or more UAVs. 11304 ".writes_depth" boolean The shader writes out a depth value. 11305 ".uses_append_consume" boolean The shader uses append and/or consume operations, either 11306 memory or GDS. 11307 ".uses_prim_id" boolean The shader uses PrimID. 11308 ========================== ============== ========= =============================================================== 11309 11310.. 11311 11312 .. table:: AMDPAL Code Object Shader Function Map 11313 :name: amdgpu-amdpal-code-object-shader-function-map-table 11314 11315 =============== ============== ==================================================================== 11316 String Key Value Type Description 11317 =============== ============== ==================================================================== 11318 *symbol name* map *symbol name* is the ELF symbol name of the shader function code 11319 entry address. The value is the function's metadata. See 11320 :ref:`amdgpu-amdpal-code-object-shader-function-metadata-map-table`. 11321 =============== ============== ==================================================================== 11322 11323.. 11324 11325 .. table:: AMDPAL Code Object Shader Function Metadata Map 11326 :name: amdgpu-amdpal-code-object-shader-function-metadata-map-table 11327 11328 ============================= ============== ================================================================= 11329 String Key Value Type Description 11330 ============================= ============== ================================================================= 11331 ".api_shader_hash" sequence of Input shader hash, typically passed in from the client. The value 11332 2 integers is implementation defined, and can not be relied on between 11333 different builds of the compiler. 11334 ".scratch_memory_size" integer Size in bytes of scratch memory used by the shader. 11335 ".lds_size" integer Size in bytes of LDS memory. 11336 ".vgpr_count" integer Number of VGPRs used by the shader. 11337 ".sgpr_count" integer Number of SGPRs used by the shader. 11338 ".stack_frame_size_in_bytes" integer Amount of stack size used by the shader. 11339 ".shader_subtype" string Shader subtype/kind. Values include: 11340 11341 - "Unknown" 11342 11343 ============================= ============== ================================================================= 11344 11345.. 11346 11347 .. table:: AMDPAL Code Object Register Map 11348 :name: amdgpu-amdpal-code-object-register-map-table 11349 11350 ========================== ============== ==================================================================== 11351 32-bit Integer Key Value Type Description 11352 ========================== ============== ==================================================================== 11353 ``reg offset`` 32-bit integer ``reg offset`` is the dword offset into the GFXIP register space of 11354 a GRBM register (i.e., driver accessible GPU register number, not 11355 shader GPR register number). The driver is required to program each 11356 specified register to the corresponding specified value when 11357 executing this pipeline. Typically, the ``reg offsets`` are the 11358 ``uint16_t`` offsets to each register as defined by the hardware 11359 chip headers. The register is set to the provided value. However, a 11360 ``reg offset`` that specifies a user data register (e.g., 11361 COMPUTE_USER_DATA_0) needs special treatment. See 11362 :ref:`amdgpu-amdpal-code-object-user-data-section` section for more 11363 information. 11364 ========================== ============== ==================================================================== 11365 11366.. _amdgpu-amdpal-code-object-user-data-section: 11367 11368User Data 11369+++++++++ 11370 11371Each hardware stage has a set of 32-bit physical SPI *user data registers* 11372(either 16 or 32 based on graphics IP and the stage) which can be 11373written from a command buffer and then loaded into SGPRs when waves are 11374launched via a subsequent dispatch or draw operation. This is the way 11375most arguments are passed from the application/runtime to a hardware 11376shader. 11377 11378PAL abstracts this functionality by exposing a set of 128 *user data 11379entries* per pipeline a client can use to pass arguments from a command 11380buffer to one or more shaders in that pipeline. The ELF code object must 11381specify a mapping from virtualized *user data entries* to physical *user 11382data registers*, and PAL is responsible for implementing that mapping, 11383including spilling overflow *user data entries* to memory if needed. 11384 11385Since the *user data registers* are GRBM-accessible SPI registers, this 11386mapping is actually embedded in the ``.registers`` metadata entry. For 11387most registers, the value in that map is a literal 32-bit value that 11388should be written to the register by the driver. However, when the 11389register is a *user data register* (any USER_DATA register e.g., 11390SPI_SHADER_USER_DATA_PS_5), the value is instead an encoding that tells 11391the driver to write either a *user data entry* value or one of several 11392driver-internal values to the register. This encoding is described in 11393the following table: 11394 11395.. note:: 11396 11397 Currently, *user data registers* 0 and 1 (e.g., SPI_SHADER_USER_DATA_PS_0, 11398 and SPI_SHADER_USER_DATA_PS_1) are reserved. *User data register* 0 must 11399 always be programmed to the address of the GlobalTable, and *user data 11400 register* 1 must always be programmed to the address of the PerShaderTable. 11401 11402.. 11403 11404 .. table:: AMDPAL User Data Mapping 11405 :name: amdgpu-amdpal-code-object-metadata-user-data-mapping-table 11406 11407 ========== ================= =============================================================================== 11408 Value Name Description 11409 ========== ================= =============================================================================== 11410 0..127 *User Data Entry* 32-bit value of user_data_entry[N] as specified via *CmdSetUserData()* 11411 0x10000000 GlobalTable 32-bit pointer to GPU memory containing the global internal table (should 11412 always point to *user data register* 0). 11413 0x10000001 PerShaderTable 32-bit pointer to GPU memory containing the per-shader internal table. See 11414 :ref:`amdgpu-amdpal-code-object-metadata-user-data-per-shader-table-section` 11415 for more detail (should always point to *user data register* 1). 11416 0x10000002 SpillTable 32-bit pointer to GPU memory containing the user data spill table. See 11417 :ref:`amdgpu-amdpal-code-object-metadata-user-data-spill-table-section` for 11418 more detail. 11419 0x10000003 BaseVertex Vertex offset (32-bit unsigned integer). Not needed if the pipeline doesn't 11420 reference the draw index in the vertex shader. Only supported by the first 11421 stage in a graphics pipeline. 11422 0x10000004 BaseInstance Instance offset (32-bit unsigned integer). Only supported by the first stage in 11423 a graphics pipeline. 11424 0x10000005 DrawIndex Draw index (32-bit unsigned integer). Only supported by the first stage in a 11425 graphics pipeline. 11426 0x10000006 Workgroup Thread group count (32-bit unsigned integer). Low half of a 64-bit address of 11427 a buffer containing the grid dimensions for a Compute dispatch operation. The 11428 high half of the address is stored in the next sequential user-SGPR. Only 11429 supported by compute pipelines. 11430 0x1000000A EsGsLdsSize Indicates that PAL will program this user-SGPR to contain the amount of LDS 11431 space used for the ES/GS pseudo-ring-buffer for passing data between shader 11432 stages. 11433 0x1000000B ViewId View id (32-bit unsigned integer) identifies a view of graphic 11434 pipeline instancing. 11435 0x1000000C StreamOutTable 32-bit pointer to GPU memory containing the stream out target SRD table. This 11436 can only appear for one shader stage per pipeline. 11437 0x1000000D PerShaderPerfData 32-bit pointer to GPU memory containing the per-shader performance data buffer. 11438 0x1000000F VertexBufferTable 32-bit pointer to GPU memory containing the vertex buffer SRD table. This can 11439 only appear for one shader stage per pipeline. 11440 0x10000010 UavExportTable 32-bit pointer to GPU memory containing the UAV export SRD table. This can 11441 only appear for one shader stage per pipeline (PS). These replace color targets 11442 and are completely separate from any UAVs used by the shader. This is optional, 11443 and only used by the PS when UAV exports are used to replace color-target 11444 exports to optimize specific shaders. 11445 0x10000011 NggCullingData 64-bit pointer to GPU memory containing the hardware register data needed by 11446 some NGG pipelines to perform culling. This value contains the address of the 11447 first of two consecutive registers which provide the full GPU address. 11448 0x10000015 FetchShaderPtr 64-bit pointer to GPU memory containing the fetch shader subroutine. 11449 ========== ================= =============================================================================== 11450 11451.. _amdgpu-amdpal-code-object-metadata-user-data-per-shader-table-section: 11452 11453Per-Shader Table 11454################ 11455 11456Low 32 bits of the GPU address for an optional buffer in the ``.data`` 11457section of the ELF. The high 32 bits of the address match the high 32 bits 11458of the shader's program counter. 11459 11460The buffer can be anything the shader compiler needs it for, and 11461allows each shader to have its own region of the ``.data`` section. 11462Typically, this could be a table of buffer SRD's and the data pointed to 11463by the buffer SRD's, but it could be a flat-address region of memory as 11464well. Its layout and usage are defined by the shader compiler. 11465 11466Each shader's table in the ``.data`` section is referenced by the symbol 11467``_amdgpu_``\ *xs*\ ``_shdr_intrl_data`` where *xs* corresponds with the 11468hardware shader stage the data is for. E.g., 11469``_amdgpu_cs_shdr_intrl_data`` for the compute shader hardware stage. 11470 11471.. _amdgpu-amdpal-code-object-metadata-user-data-spill-table-section: 11472 11473Spill Table 11474########### 11475 11476It is possible for a hardware shader to need access to more *user data 11477entries* than there are slots available in user data registers for one 11478or more hardware shader stages. In that case, the PAL runtime expects 11479the necessary *user data entries* to be spilled to GPU memory and use 11480one user data register to point to the spilled user data memory. The 11481value of the *user data entry* must then represent the location where 11482a shader expects to read the low 32-bits of the table's GPU virtual 11483address. The *spill table* itself represents a set of 32-bit values 11484managed by the PAL runtime in GPU-accessible memory that can be made 11485indirectly accessible to a hardware shader. 11486 11487Unspecified OS 11488-------------- 11489 11490This section provides code conventions used when the target triple OS is 11491empty (see :ref:`amdgpu-target-triples`). 11492 11493Trap Handler ABI 11494~~~~~~~~~~~~~~~~ 11495 11496For code objects generated by AMDGPU backend for non-amdhsa OS, the runtime does 11497not install a trap handler. The ``llvm.trap`` and ``llvm.debugtrap`` 11498instructions are handled as follows: 11499 11500 .. table:: AMDGPU Trap Handler for Non-AMDHSA OS 11501 :name: amdgpu-trap-handler-for-non-amdhsa-os-table 11502 11503 =============== =============== =========================================== 11504 Usage Code Sequence Description 11505 =============== =============== =========================================== 11506 llvm.trap s_endpgm Causes wavefront to be terminated. 11507 llvm.debugtrap *none* Compiler warning given that there is no 11508 trap handler installed. 11509 =============== =============== =========================================== 11510 11511Source Languages 11512================ 11513 11514.. _amdgpu-opencl: 11515 11516OpenCL 11517------ 11518 11519When the language is OpenCL the following differences occur: 11520 115211. The OpenCL memory model is used (see :ref:`amdgpu-amdhsa-memory-model`). 115222. The AMDGPU backend appends additional arguments to the kernel's explicit 11523 arguments for the AMDHSA OS (see 11524 :ref:`opencl-kernel-implicit-arguments-appended-for-amdhsa-os-table`). 115253. Additional metadata is generated 11526 (see :ref:`amdgpu-amdhsa-code-object-metadata`). 11527 11528 .. table:: OpenCL kernel implicit arguments appended for AMDHSA OS 11529 :name: opencl-kernel-implicit-arguments-appended-for-amdhsa-os-table 11530 11531 ======== ==== ========= =========================================== 11532 Position Byte Byte Description 11533 Size Alignment 11534 ======== ==== ========= =========================================== 11535 1 8 8 OpenCL Global Offset X 11536 2 8 8 OpenCL Global Offset Y 11537 3 8 8 OpenCL Global Offset Z 11538 4 8 8 OpenCL address of printf buffer 11539 5 8 8 OpenCL address of virtual queue used by 11540 enqueue_kernel. 11541 6 8 8 OpenCL address of AqlWrap struct used by 11542 enqueue_kernel. 11543 7 8 8 Pointer argument used for Multi-gird 11544 synchronization. 11545 ======== ==== ========= =========================================== 11546 11547.. _amdgpu-hcc: 11548 11549HCC 11550--- 11551 11552When the language is HCC the following differences occur: 11553 115541. The HSA memory model is used (see :ref:`amdgpu-amdhsa-memory-model`). 11555 11556.. _amdgpu-assembler: 11557 11558Assembler 11559--------- 11560 11561AMDGPU backend has LLVM-MC based assembler which is currently in development. 11562It supports AMDGCN GFX6-GFX10. 11563 11564This section describes general syntax for instructions and operands. 11565 11566Instructions 11567~~~~~~~~~~~~ 11568 11569An instruction has the following :doc:`syntax<AMDGPUInstructionSyntax>`: 11570 11571 | ``<``\ *opcode*\ ``> <``\ *operand0*\ ``>, <``\ *operand1*\ ``>,... 11572 <``\ *modifier0*\ ``> <``\ *modifier1*\ ``>...`` 11573 11574:doc:`Operands<AMDGPUOperandSyntax>` are comma-separated while 11575:doc:`modifiers<AMDGPUModifierSyntax>` are space-separated. 11576 11577The order of operands and modifiers is fixed. 11578Most modifiers are optional and may be omitted. 11579 11580Links to detailed instruction syntax description may be found in the following 11581table. Note that features under development are not included 11582in this description. 11583 11584 =================================== ======================================= 11585 Core ISA ISA Extensions 11586 =================================== ======================================= 11587 :doc:`GFX7<AMDGPU/AMDGPUAsmGFX7>` \- 11588 :doc:`GFX8<AMDGPU/AMDGPUAsmGFX8>` \- 11589 :doc:`GFX9<AMDGPU/AMDGPUAsmGFX9>` :doc:`gfx900<AMDGPU/AMDGPUAsmGFX900>` 11590 11591 :doc:`gfx902<AMDGPU/AMDGPUAsmGFX900>` 11592 11593 :doc:`gfx904<AMDGPU/AMDGPUAsmGFX904>` 11594 11595 :doc:`gfx906<AMDGPU/AMDGPUAsmGFX906>` 11596 11597 :doc:`gfx908<AMDGPU/AMDGPUAsmGFX908>` 11598 11599 :doc:`gfx909<AMDGPU/AMDGPUAsmGFX900>` 11600 11601 :doc:`gfx90a<AMDGPU/AMDGPUAsmGFX90a>` 11602 11603 :doc:`GFX10<AMDGPU/AMDGPUAsmGFX10>` :doc:`gfx1011<AMDGPU/AMDGPUAsmGFX1011>` 11604 11605 :doc:`gfx1012<AMDGPU/AMDGPUAsmGFX1011>` 11606 =================================== ======================================= 11607 11608For more information about instructions, their semantics and supported 11609combinations of operands, refer to one of instruction set architecture manuals 11610[AMD-GCN-GFX6]_, [AMD-GCN-GFX7]_, [AMD-GCN-GFX8]_, 11611[AMD-GCN-GFX900-GFX904-VEGA]_, [AMD-GCN-GFX906-VEGA7NM]_ 11612[AMD-GCN-GFX908-CDNA1]_, [AMD-GCN-GFX10-RDNA1]_ and [AMD-GCN-GFX10-RDNA2]_. 11613 11614Operands 11615~~~~~~~~ 11616 11617Detailed description of operands may be found :doc:`here<AMDGPUOperandSyntax>`. 11618 11619Modifiers 11620~~~~~~~~~ 11621 11622Detailed description of modifiers may be found 11623:doc:`here<AMDGPUModifierSyntax>`. 11624 11625Instruction Examples 11626~~~~~~~~~~~~~~~~~~~~ 11627 11628DS 11629++ 11630 11631.. code-block:: nasm 11632 11633 ds_add_u32 v2, v4 offset:16 11634 ds_write_src2_b64 v2 offset0:4 offset1:8 11635 ds_cmpst_f32 v2, v4, v6 11636 ds_min_rtn_f64 v[8:9], v2, v[4:5] 11637 11638For full list of supported instructions, refer to "LDS/GDS instructions" in ISA 11639Manual. 11640 11641FLAT 11642++++ 11643 11644.. code-block:: nasm 11645 11646 flat_load_dword v1, v[3:4] 11647 flat_store_dwordx3 v[3:4], v[5:7] 11648 flat_atomic_swap v1, v[3:4], v5 glc 11649 flat_atomic_cmpswap v1, v[3:4], v[5:6] glc slc 11650 flat_atomic_fmax_x2 v[1:2], v[3:4], v[5:6] glc 11651 11652For full list of supported instructions, refer to "FLAT instructions" in ISA 11653Manual. 11654 11655MUBUF 11656+++++ 11657 11658.. code-block:: nasm 11659 11660 buffer_load_dword v1, off, s[4:7], s1 11661 buffer_store_dwordx4 v[1:4], v2, ttmp[4:7], s1 offen offset:4 glc tfe 11662 buffer_store_format_xy v[1:2], off, s[4:7], s1 11663 buffer_wbinvl1 11664 buffer_atomic_inc v1, v2, s[8:11], s4 idxen offset:4 slc 11665 11666For full list of supported instructions, refer to "MUBUF Instructions" in ISA 11667Manual. 11668 11669SMRD/SMEM 11670+++++++++ 11671 11672.. code-block:: nasm 11673 11674 s_load_dword s1, s[2:3], 0xfc 11675 s_load_dwordx8 s[8:15], s[2:3], s4 11676 s_load_dwordx16 s[88:103], s[2:3], s4 11677 s_dcache_inv_vol 11678 s_memtime s[4:5] 11679 11680For full list of supported instructions, refer to "Scalar Memory Operations" in 11681ISA Manual. 11682 11683SOP1 11684++++ 11685 11686.. code-block:: nasm 11687 11688 s_mov_b32 s1, s2 11689 s_mov_b64 s[0:1], 0x80000000 11690 s_cmov_b32 s1, 200 11691 s_wqm_b64 s[2:3], s[4:5] 11692 s_bcnt0_i32_b64 s1, s[2:3] 11693 s_swappc_b64 s[2:3], s[4:5] 11694 s_cbranch_join s[4:5] 11695 11696For full list of supported instructions, refer to "SOP1 Instructions" in ISA 11697Manual. 11698 11699SOP2 11700++++ 11701 11702.. code-block:: nasm 11703 11704 s_add_u32 s1, s2, s3 11705 s_and_b64 s[2:3], s[4:5], s[6:7] 11706 s_cselect_b32 s1, s2, s3 11707 s_andn2_b32 s2, s4, s6 11708 s_lshr_b64 s[2:3], s[4:5], s6 11709 s_ashr_i32 s2, s4, s6 11710 s_bfm_b64 s[2:3], s4, s6 11711 s_bfe_i64 s[2:3], s[4:5], s6 11712 s_cbranch_g_fork s[4:5], s[6:7] 11713 11714For full list of supported instructions, refer to "SOP2 Instructions" in ISA 11715Manual. 11716 11717SOPC 11718++++ 11719 11720.. code-block:: nasm 11721 11722 s_cmp_eq_i32 s1, s2 11723 s_bitcmp1_b32 s1, s2 11724 s_bitcmp0_b64 s[2:3], s4 11725 s_setvskip s3, s5 11726 11727For full list of supported instructions, refer to "SOPC Instructions" in ISA 11728Manual. 11729 11730SOPP 11731++++ 11732 11733.. code-block:: nasm 11734 11735 s_barrier 11736 s_nop 2 11737 s_endpgm 11738 s_waitcnt 0 ; Wait for all counters to be 0 11739 s_waitcnt vmcnt(0) & expcnt(0) & lgkmcnt(0) ; Equivalent to above 11740 s_waitcnt vmcnt(1) ; Wait for vmcnt counter to be 1. 11741 s_sethalt 9 11742 s_sleep 10 11743 s_sendmsg 0x1 11744 s_sendmsg sendmsg(MSG_INTERRUPT) 11745 s_trap 1 11746 11747For full list of supported instructions, refer to "SOPP Instructions" in ISA 11748Manual. 11749 11750Unless otherwise mentioned, little verification is performed on the operands 11751of SOPP Instructions, so it is up to the programmer to be familiar with the 11752range or acceptable values. 11753 11754VALU 11755++++ 11756 11757For vector ALU instruction opcodes (VOP1, VOP2, VOP3, VOPC, VOP_DPP, VOP_SDWA), 11758the assembler will automatically use optimal encoding based on its operands. To 11759force specific encoding, one can add a suffix to the opcode of the instruction: 11760 11761* _e32 for 32-bit VOP1/VOP2/VOPC 11762* _e64 for 64-bit VOP3 11763* _dpp for VOP_DPP 11764* _sdwa for VOP_SDWA 11765 11766VOP1/VOP2/VOP3/VOPC examples: 11767 11768.. code-block:: nasm 11769 11770 v_mov_b32 v1, v2 11771 v_mov_b32_e32 v1, v2 11772 v_nop 11773 v_cvt_f64_i32_e32 v[1:2], v2 11774 v_floor_f32_e32 v1, v2 11775 v_bfrev_b32_e32 v1, v2 11776 v_add_f32_e32 v1, v2, v3 11777 v_mul_i32_i24_e64 v1, v2, 3 11778 v_mul_i32_i24_e32 v1, -3, v3 11779 v_mul_i32_i24_e32 v1, -100, v3 11780 v_addc_u32 v1, s[0:1], v2, v3, s[2:3] 11781 v_max_f16_e32 v1, v2, v3 11782 11783VOP_DPP examples: 11784 11785.. code-block:: nasm 11786 11787 v_mov_b32 v0, v0 quad_perm:[0,2,1,1] 11788 v_sin_f32 v0, v0 row_shl:1 row_mask:0xa bank_mask:0x1 bound_ctrl:0 11789 v_mov_b32 v0, v0 wave_shl:1 11790 v_mov_b32 v0, v0 row_mirror 11791 v_mov_b32 v0, v0 row_bcast:31 11792 v_mov_b32 v0, v0 quad_perm:[1,3,0,1] row_mask:0xa bank_mask:0x1 bound_ctrl:0 11793 v_add_f32 v0, v0, |v0| row_shl:1 row_mask:0xa bank_mask:0x1 bound_ctrl:0 11794 v_max_f16 v1, v2, v3 row_shl:1 row_mask:0xa bank_mask:0x1 bound_ctrl:0 11795 11796VOP_SDWA examples: 11797 11798.. code-block:: nasm 11799 11800 v_mov_b32 v1, v2 dst_sel:BYTE_0 dst_unused:UNUSED_PRESERVE src0_sel:DWORD 11801 v_min_u32 v200, v200, v1 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:BYTE_1 src1_sel:DWORD 11802 v_sin_f32 v0, v0 dst_unused:UNUSED_PAD src0_sel:WORD_1 11803 v_fract_f32 v0, |v0| dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 11804 v_cmpx_le_u32 vcc, v1, v2 src0_sel:BYTE_2 src1_sel:WORD_0 11805 11806For full list of supported instructions, refer to "Vector ALU instructions". 11807 11808.. _amdgpu-amdhsa-assembler-predefined-symbols-v2: 11809 11810Code Object V2 Predefined Symbols 11811~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 11812 11813.. warning:: 11814 Code object V2 is not the default code object version emitted by 11815 this version of LLVM. 11816 11817The AMDGPU assembler defines and updates some symbols automatically. These 11818symbols do not affect code generation. 11819 11820.option.machine_version_major 11821+++++++++++++++++++++++++++++ 11822 11823Set to the GFX major generation number of the target being assembled for. For 11824example, when assembling for a "GFX9" target this will be set to the integer 11825value "9". The possible GFX major generation numbers are presented in 11826:ref:`amdgpu-processors`. 11827 11828.option.machine_version_minor 11829+++++++++++++++++++++++++++++ 11830 11831Set to the GFX minor generation number of the target being assembled for. For 11832example, when assembling for a "GFX810" target this will be set to the integer 11833value "1". The possible GFX minor generation numbers are presented in 11834:ref:`amdgpu-processors`. 11835 11836.option.machine_version_stepping 11837++++++++++++++++++++++++++++++++ 11838 11839Set to the GFX stepping generation number of the target being assembled for. 11840For example, when assembling for a "GFX704" target this will be set to the 11841integer value "4". The possible GFX stepping generation numbers are presented 11842in :ref:`amdgpu-processors`. 11843 11844.kernel.vgpr_count 11845++++++++++++++++++ 11846 11847Set to zero each time a 11848:ref:`amdgpu-amdhsa-assembler-directive-amdgpu_hsa_kernel` directive is 11849encountered. At each instruction, if the current value of this symbol is less 11850than or equal to the maximum VGPR number explicitly referenced within that 11851instruction then the symbol value is updated to equal that VGPR number plus 11852one. 11853 11854.kernel.sgpr_count 11855++++++++++++++++++ 11856 11857Set to zero each time a 11858:ref:`amdgpu-amdhsa-assembler-directive-amdgpu_hsa_kernel` directive is 11859encountered. At each instruction, if the current value of this symbol is less 11860than or equal to the maximum VGPR number explicitly referenced within that 11861instruction then the symbol value is updated to equal that SGPR number plus 11862one. 11863 11864.. _amdgpu-amdhsa-assembler-directives-v2: 11865 11866Code Object V2 Directives 11867~~~~~~~~~~~~~~~~~~~~~~~~~ 11868 11869.. warning:: 11870 Code object V2 is not the default code object version emitted by 11871 this version of LLVM. 11872 11873AMDGPU ABI defines auxiliary data in output code object. In assembly source, 11874one can specify them with assembler directives. 11875 11876.hsa_code_object_version major, minor 11877+++++++++++++++++++++++++++++++++++++ 11878 11879*major* and *minor* are integers that specify the version of the HSA code 11880object that will be generated by the assembler. 11881 11882.hsa_code_object_isa [major, minor, stepping, vendor, arch] 11883+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 11884 11885 11886*major*, *minor*, and *stepping* are all integers that describe the instruction 11887set architecture (ISA) version of the assembly program. 11888 11889*vendor* and *arch* are quoted strings. *vendor* should always be equal to 11890"AMD" and *arch* should always be equal to "AMDGPU". 11891 11892By default, the assembler will derive the ISA version, *vendor*, and *arch* 11893from the value of the -mcpu option that is passed to the assembler. 11894 11895.. _amdgpu-amdhsa-assembler-directive-amdgpu_hsa_kernel: 11896 11897.amdgpu_hsa_kernel (name) 11898+++++++++++++++++++++++++ 11899 11900This directives specifies that the symbol with given name is a kernel entry 11901point (label) and the object should contain corresponding symbol of type 11902STT_AMDGPU_HSA_KERNEL. 11903 11904.amd_kernel_code_t 11905++++++++++++++++++ 11906 11907This directive marks the beginning of a list of key / value pairs that are used 11908to specify the amd_kernel_code_t object that will be emitted by the assembler. 11909The list must be terminated by the *.end_amd_kernel_code_t* directive. For any 11910amd_kernel_code_t values that are unspecified a default value will be used. The 11911default value for all keys is 0, with the following exceptions: 11912 11913- *amd_code_version_major* defaults to 1. 11914- *amd_kernel_code_version_minor* defaults to 2. 11915- *amd_machine_kind* defaults to 1. 11916- *amd_machine_version_major*, *machine_version_minor*, and 11917 *amd_machine_version_stepping* are derived from the value of the -mcpu option 11918 that is passed to the assembler. 11919- *kernel_code_entry_byte_offset* defaults to 256. 11920- *wavefront_size* defaults 6 for all targets before GFX10. For GFX10 onwards 11921 defaults to 6 if target feature ``wavefrontsize64`` is enabled, otherwise 5. 11922 Note that wavefront size is specified as a power of two, so a value of **n** 11923 means a size of 2^ **n**. 11924- *call_convention* defaults to -1. 11925- *kernarg_segment_alignment*, *group_segment_alignment*, and 11926 *private_segment_alignment* default to 4. Note that alignments are specified 11927 as a power of 2, so a value of **n** means an alignment of 2^ **n**. 11928- *enable_tg_split* defaults to 1 if target feature ``tgsplit`` is enabled for 11929 GFX90A onwards. 11930- *enable_wgp_mode* defaults to 1 if target feature ``cumode`` is disabled for 11931 GFX10 onwards. 11932- *enable_mem_ordered* defaults to 1 for GFX10 onwards. 11933 11934The *.amd_kernel_code_t* directive must be placed immediately after the 11935function label and before any instructions. 11936 11937For a full list of amd_kernel_code_t keys, refer to AMDGPU ABI document, 11938comments in lib/Target/AMDGPU/AmdKernelCodeT.h and test/CodeGen/AMDGPU/hsa.s. 11939 11940.. _amdgpu-amdhsa-assembler-example-v2: 11941 11942Code Object V2 Example Source Code 11943~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 11944 11945.. warning:: 11946 Code Object V2 is not the default code object version emitted by 11947 this version of LLVM. 11948 11949Here is an example of a minimal assembly source file, defining one HSA kernel: 11950 11951.. code:: 11952 :number-lines: 11953 11954 .hsa_code_object_version 1,0 11955 .hsa_code_object_isa 11956 11957 .hsatext 11958 .globl hello_world 11959 .p2align 8 11960 .amdgpu_hsa_kernel hello_world 11961 11962 hello_world: 11963 11964 .amd_kernel_code_t 11965 enable_sgpr_kernarg_segment_ptr = 1 11966 is_ptr64 = 1 11967 compute_pgm_rsrc1_vgprs = 0 11968 compute_pgm_rsrc1_sgprs = 0 11969 compute_pgm_rsrc2_user_sgpr = 2 11970 compute_pgm_rsrc1_wgp_mode = 0 11971 compute_pgm_rsrc1_mem_ordered = 0 11972 compute_pgm_rsrc1_fwd_progress = 1 11973 .end_amd_kernel_code_t 11974 11975 s_load_dwordx2 s[0:1], s[0:1] 0x0 11976 v_mov_b32 v0, 3.14159 11977 s_waitcnt lgkmcnt(0) 11978 v_mov_b32 v1, s0 11979 v_mov_b32 v2, s1 11980 flat_store_dword v[1:2], v0 11981 s_endpgm 11982 .Lfunc_end0: 11983 .size hello_world, .Lfunc_end0-hello_world 11984 11985.. _amdgpu-amdhsa-assembler-predefined-symbols-v3-v4: 11986 11987Code Object V3 to V4 Predefined Symbols 11988~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 11989 11990The AMDGPU assembler defines and updates some symbols automatically. These 11991symbols do not affect code generation. 11992 11993.amdgcn.gfx_generation_number 11994+++++++++++++++++++++++++++++ 11995 11996Set to the GFX major generation number of the target being assembled for. For 11997example, when assembling for a "GFX9" target this will be set to the integer 11998value "9". The possible GFX major generation numbers are presented in 11999:ref:`amdgpu-processors`. 12000 12001.amdgcn.gfx_generation_minor 12002++++++++++++++++++++++++++++ 12003 12004Set to the GFX minor generation number of the target being assembled for. For 12005example, when assembling for a "GFX810" target this will be set to the integer 12006value "1". The possible GFX minor generation numbers are presented in 12007:ref:`amdgpu-processors`. 12008 12009.amdgcn.gfx_generation_stepping 12010+++++++++++++++++++++++++++++++ 12011 12012Set to the GFX stepping generation number of the target being assembled for. 12013For example, when assembling for a "GFX704" target this will be set to the 12014integer value "4". The possible GFX stepping generation numbers are presented 12015in :ref:`amdgpu-processors`. 12016 12017.. _amdgpu-amdhsa-assembler-symbol-next_free_vgpr: 12018 12019.amdgcn.next_free_vgpr 12020++++++++++++++++++++++ 12021 12022Set to zero before assembly begins. At each instruction, if the current value 12023of this symbol is less than or equal to the maximum VGPR number explicitly 12024referenced within that instruction then the symbol value is updated to equal 12025that VGPR number plus one. 12026 12027May be used to set the `.amdhsa_next_free_vgpr` directive in 12028:ref:`amdhsa-kernel-directives-table`. 12029 12030May be set at any time, e.g. manually set to zero at the start of each kernel. 12031 12032.. _amdgpu-amdhsa-assembler-symbol-next_free_sgpr: 12033 12034.amdgcn.next_free_sgpr 12035++++++++++++++++++++++ 12036 12037Set to zero before assembly begins. At each instruction, if the current value 12038of this symbol is less than or equal the maximum SGPR number explicitly 12039referenced within that instruction then the symbol value is updated to equal 12040that SGPR number plus one. 12041 12042May be used to set the `.amdhsa_next_free_spgr` directive in 12043:ref:`amdhsa-kernel-directives-table`. 12044 12045May be set at any time, e.g. manually set to zero at the start of each kernel. 12046 12047.. _amdgpu-amdhsa-assembler-directives-v3-v4: 12048 12049Code Object V3 to V4 Directives 12050~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 12051 12052Directives which begin with ``.amdgcn`` are valid for all ``amdgcn`` 12053architecture processors, and are not OS-specific. Directives which begin with 12054``.amdhsa`` are specific to ``amdgcn`` architecture processors when the 12055``amdhsa`` OS is specified. See :ref:`amdgpu-target-triples` and 12056:ref:`amdgpu-processors`. 12057 12058.. _amdgpu-assembler-directive-amdgcn-target: 12059 12060.amdgcn_target <target-triple> "-" <target-id> 12061++++++++++++++++++++++++++++++++++++++++++++++ 12062 12063Optional directive which declares the ``<target-triple>-<target-id>`` supported 12064by the containing assembler source file. Used by the assembler to validate 12065command-line options such as ``-triple``, ``-mcpu``, and 12066``--offload-arch=<target-id>``. A non-canonical target ID is allowed. See 12067:ref:`amdgpu-target-triples` and :ref:`amdgpu-target-id`. 12068 12069.. note:: 12070 12071 The target ID syntax used for code object V2 to V3 for this directive differs 12072 from that used elsewhere. See :ref:`amdgpu-target-id-v2-v3`. 12073 12074.amdhsa_kernel <name> 12075+++++++++++++++++++++ 12076 12077Creates a correctly aligned AMDHSA kernel descriptor and a symbol, 12078``<name>.kd``, in the current location of the current section. Only valid when 12079the OS is ``amdhsa``. ``<name>`` must be a symbol that labels the first 12080instruction to execute, and does not need to be previously defined. 12081 12082Marks the beginning of a list of directives used to generate the bytes of a 12083kernel descriptor, as described in :ref:`amdgpu-amdhsa-kernel-descriptor`. 12084Directives which may appear in this list are described in 12085:ref:`amdhsa-kernel-directives-table`. Directives may appear in any order, must 12086be valid for the target being assembled for, and cannot be repeated. Directives 12087support the range of values specified by the field they reference in 12088:ref:`amdgpu-amdhsa-kernel-descriptor`. If a directive is not specified, it is 12089assumed to have its default value, unless it is marked as "Required", in which 12090case it is an error to omit the directive. This list of directives is 12091terminated by an ``.end_amdhsa_kernel`` directive. 12092 12093 .. table:: AMDHSA Kernel Assembler Directives 12094 :name: amdhsa-kernel-directives-table 12095 12096 ======================================================== =================== ============ =================== 12097 Directive Default Supported On Description 12098 ======================================================== =================== ============ =================== 12099 ``.amdhsa_group_segment_fixed_size`` 0 GFX6-GFX10 Controls GROUP_SEGMENT_FIXED_SIZE in 12100 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 12101 ``.amdhsa_private_segment_fixed_size`` 0 GFX6-GFX10 Controls PRIVATE_SEGMENT_FIXED_SIZE in 12102 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 12103 ``.amdhsa_kernarg_size`` 0 GFX6-GFX10 Controls KERNARG_SIZE in 12104 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 12105 ``.amdhsa_user_sgpr_private_segment_buffer`` 0 GFX6-GFX10 Controls ENABLE_SGPR_PRIVATE_SEGMENT_BUFFER in 12106 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 12107 ``.amdhsa_user_sgpr_dispatch_ptr`` 0 GFX6-GFX10 Controls ENABLE_SGPR_DISPATCH_PTR in 12108 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 12109 ``.amdhsa_user_sgpr_queue_ptr`` 0 GFX6-GFX10 Controls ENABLE_SGPR_QUEUE_PTR in 12110 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 12111 ``.amdhsa_user_sgpr_kernarg_segment_ptr`` 0 GFX6-GFX10 Controls ENABLE_SGPR_KERNARG_SEGMENT_PTR in 12112 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 12113 ``.amdhsa_user_sgpr_dispatch_id`` 0 GFX6-GFX10 Controls ENABLE_SGPR_DISPATCH_ID in 12114 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 12115 ``.amdhsa_user_sgpr_flat_scratch_init`` 0 GFX6-GFX10 Controls ENABLE_SGPR_FLAT_SCRATCH_INIT in 12116 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 12117 ``.amdhsa_user_sgpr_private_segment_size`` 0 GFX6-GFX10 Controls ENABLE_SGPR_PRIVATE_SEGMENT_SIZE in 12118 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 12119 ``.amdhsa_wavefront_size32`` Target GFX10 Controls ENABLE_WAVEFRONT_SIZE32 in 12120 Feature :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 12121 Specific 12122 (wavefrontsize64) 12123 ``.amdhsa_system_sgpr_private_segment_wavefront_offset`` 0 GFX6-GFX10 Controls ENABLE_PRIVATE_SEGMENT in 12124 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 12125 ``.amdhsa_system_sgpr_workgroup_id_x`` 1 GFX6-GFX10 Controls ENABLE_SGPR_WORKGROUP_ID_X in 12126 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 12127 ``.amdhsa_system_sgpr_workgroup_id_y`` 0 GFX6-GFX10 Controls ENABLE_SGPR_WORKGROUP_ID_Y in 12128 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 12129 ``.amdhsa_system_sgpr_workgroup_id_z`` 0 GFX6-GFX10 Controls ENABLE_SGPR_WORKGROUP_ID_Z in 12130 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 12131 ``.amdhsa_system_sgpr_workgroup_info`` 0 GFX6-GFX10 Controls ENABLE_SGPR_WORKGROUP_INFO in 12132 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 12133 ``.amdhsa_system_vgpr_workitem_id`` 0 GFX6-GFX10 Controls ENABLE_VGPR_WORKITEM_ID in 12134 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 12135 Possible values are defined in 12136 :ref:`amdgpu-amdhsa-system-vgpr-work-item-id-enumeration-values-table`. 12137 ``.amdhsa_next_free_vgpr`` Required GFX6-GFX10 Maximum VGPR number explicitly referenced, plus one. 12138 Used to calculate GRANULATED_WORKITEM_VGPR_COUNT in 12139 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 12140 ``.amdhsa_next_free_sgpr`` Required GFX6-GFX10 Maximum SGPR number explicitly referenced, plus one. 12141 Used to calculate GRANULATED_WAVEFRONT_SGPR_COUNT in 12142 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 12143 ``.amdhsa_accum_offset`` Required GFX90A Offset of a first AccVGPR in the unified register file. 12144 Used to calculate ACCUM_OFFSET in 12145 :ref:`amdgpu-amdhsa-compute_pgm_rsrc3-gfx90a-table`. 12146 ``.amdhsa_reserve_vcc`` 1 GFX6-GFX10 Whether the kernel may use the special VCC SGPR. 12147 Used to calculate GRANULATED_WAVEFRONT_SGPR_COUNT in 12148 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 12149 ``.amdhsa_reserve_flat_scratch`` 1 GFX7-GFX10 Whether the kernel may use flat instructions to access 12150 scratch memory. Used to calculate 12151 GRANULATED_WAVEFRONT_SGPR_COUNT in 12152 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 12153 ``.amdhsa_reserve_xnack_mask`` Target GFX8-GFX10 Whether the kernel may trigger XNACK replay. 12154 Feature Used to calculate GRANULATED_WAVEFRONT_SGPR_COUNT in 12155 Specific :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 12156 (xnack) 12157 ``.amdhsa_float_round_mode_32`` 0 GFX6-GFX10 Controls FLOAT_ROUND_MODE_32 in 12158 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 12159 Possible values are defined in 12160 :ref:`amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table`. 12161 ``.amdhsa_float_round_mode_16_64`` 0 GFX6-GFX10 Controls FLOAT_ROUND_MODE_16_64 in 12162 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 12163 Possible values are defined in 12164 :ref:`amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table`. 12165 ``.amdhsa_float_denorm_mode_32`` 0 GFX6-GFX10 Controls FLOAT_DENORM_MODE_32 in 12166 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 12167 Possible values are defined in 12168 :ref:`amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table`. 12169 ``.amdhsa_float_denorm_mode_16_64`` 3 GFX6-GFX10 Controls FLOAT_DENORM_MODE_16_64 in 12170 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 12171 Possible values are defined in 12172 :ref:`amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table`. 12173 ``.amdhsa_dx10_clamp`` 1 GFX6-GFX10 Controls ENABLE_DX10_CLAMP in 12174 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 12175 ``.amdhsa_ieee_mode`` 1 GFX6-GFX10 Controls ENABLE_IEEE_MODE in 12176 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 12177 ``.amdhsa_fp16_overflow`` 0 GFX9-GFX10 Controls FP16_OVFL in 12178 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 12179 ``.amdhsa_tg_split`` Target GFX90A Controls TG_SPLIT in 12180 Feature :ref:`amdgpu-amdhsa-compute_pgm_rsrc3-gfx90a-table`. 12181 Specific 12182 (tgsplit) 12183 ``.amdhsa_workgroup_processor_mode`` Target GFX10 Controls ENABLE_WGP_MODE in 12184 Feature :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 12185 Specific 12186 (cumode) 12187 ``.amdhsa_memory_ordered`` 1 GFX10 Controls MEM_ORDERED in 12188 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 12189 ``.amdhsa_forward_progress`` 0 GFX10 Controls FWD_PROGRESS in 12190 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 12191 ``.amdhsa_exception_fp_ieee_invalid_op`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_INVALID_OPERATION in 12192 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 12193 ``.amdhsa_exception_fp_denorm_src`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_FP_DENORMAL_SOURCE in 12194 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 12195 ``.amdhsa_exception_fp_ieee_div_zero`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_DIVISION_BY_ZERO in 12196 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 12197 ``.amdhsa_exception_fp_ieee_overflow`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_OVERFLOW in 12198 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 12199 ``.amdhsa_exception_fp_ieee_underflow`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_UNDERFLOW in 12200 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 12201 ``.amdhsa_exception_fp_ieee_inexact`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_INEXACT in 12202 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 12203 ``.amdhsa_exception_int_div_zero`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_INT_DIVIDE_BY_ZERO in 12204 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 12205 ======================================================== =================== ============ =================== 12206 12207.amdgpu_metadata 12208++++++++++++++++ 12209 12210Optional directive which declares the contents of the ``NT_AMDGPU_METADATA`` 12211note record (see :ref:`amdgpu-elf-note-records-table-v3-v4`). 12212 12213The contents must be in the [YAML]_ markup format, with the same structure and 12214semantics described in :ref:`amdgpu-amdhsa-code-object-metadata-v3` or 12215:ref:`amdgpu-amdhsa-code-object-metadata-v4`. 12216 12217This directive is terminated by an ``.end_amdgpu_metadata`` directive. 12218 12219.. _amdgpu-amdhsa-assembler-example-v3-v4: 12220 12221Code Object V3 to V4 Example Source Code 12222~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 12223 12224Here is an example of a minimal assembly source file, defining one HSA kernel: 12225 12226.. code:: 12227 :number-lines: 12228 12229 .amdgcn_target "amdgcn-amd-amdhsa--gfx900+xnack" // optional 12230 12231 .text 12232 .globl hello_world 12233 .p2align 8 12234 .type hello_world,@function 12235 hello_world: 12236 s_load_dwordx2 s[0:1], s[0:1] 0x0 12237 v_mov_b32 v0, 3.14159 12238 s_waitcnt lgkmcnt(0) 12239 v_mov_b32 v1, s0 12240 v_mov_b32 v2, s1 12241 flat_store_dword v[1:2], v0 12242 s_endpgm 12243 .Lfunc_end0: 12244 .size hello_world, .Lfunc_end0-hello_world 12245 12246 .rodata 12247 .p2align 6 12248 .amdhsa_kernel hello_world 12249 .amdhsa_user_sgpr_kernarg_segment_ptr 1 12250 .amdhsa_next_free_vgpr .amdgcn.next_free_vgpr 12251 .amdhsa_next_free_sgpr .amdgcn.next_free_sgpr 12252 .end_amdhsa_kernel 12253 12254 .amdgpu_metadata 12255 --- 12256 amdhsa.version: 12257 - 1 12258 - 0 12259 amdhsa.kernels: 12260 - .name: hello_world 12261 .symbol: hello_world.kd 12262 .kernarg_segment_size: 48 12263 .group_segment_fixed_size: 0 12264 .private_segment_fixed_size: 0 12265 .kernarg_segment_align: 4 12266 .wavefront_size: 64 12267 .sgpr_count: 2 12268 .vgpr_count: 3 12269 .max_flat_workgroup_size: 256 12270 .args: 12271 - .size: 8 12272 .offset: 0 12273 .value_kind: global_buffer 12274 .address_space: global 12275 .actual_access: write_only 12276 //... 12277 .end_amdgpu_metadata 12278 12279This kernel is equivalent to the following HIP program: 12280 12281.. code:: 12282 :number-lines: 12283 12284 __global__ void hello_world(float *p) { 12285 *p = 3.14159f; 12286 } 12287 12288If an assembly source file contains multiple kernels and/or functions, the 12289:ref:`amdgpu-amdhsa-assembler-symbol-next_free_vgpr` and 12290:ref:`amdgpu-amdhsa-assembler-symbol-next_free_sgpr` symbols may be reset using 12291the ``.set <symbol>, <expression>`` directive. For example, in the case of two 12292kernels, where ``function1`` is only called from ``kernel1`` it is sufficient 12293to group the function with the kernel that calls it and reset the symbols 12294between the two connected components: 12295 12296.. code:: 12297 :number-lines: 12298 12299 .amdgcn_target "amdgcn-amd-amdhsa--gfx900+xnack" // optional 12300 12301 // gpr tracking symbols are implicitly set to zero 12302 12303 .text 12304 .globl kern0 12305 .p2align 8 12306 .type kern0,@function 12307 kern0: 12308 // ... 12309 s_endpgm 12310 .Lkern0_end: 12311 .size kern0, .Lkern0_end-kern0 12312 12313 .rodata 12314 .p2align 6 12315 .amdhsa_kernel kern0 12316 // ... 12317 .amdhsa_next_free_vgpr .amdgcn.next_free_vgpr 12318 .amdhsa_next_free_sgpr .amdgcn.next_free_sgpr 12319 .end_amdhsa_kernel 12320 12321 // reset symbols to begin tracking usage in func1 and kern1 12322 .set .amdgcn.next_free_vgpr, 0 12323 .set .amdgcn.next_free_sgpr, 0 12324 12325 .text 12326 .hidden func1 12327 .global func1 12328 .p2align 2 12329 .type func1,@function 12330 func1: 12331 // ... 12332 s_setpc_b64 s[30:31] 12333 .Lfunc1_end: 12334 .size func1, .Lfunc1_end-func1 12335 12336 .globl kern1 12337 .p2align 8 12338 .type kern1,@function 12339 kern1: 12340 // ... 12341 s_getpc_b64 s[4:5] 12342 s_add_u32 s4, s4, func1@rel32@lo+4 12343 s_addc_u32 s5, s5, func1@rel32@lo+4 12344 s_swappc_b64 s[30:31], s[4:5] 12345 // ... 12346 s_endpgm 12347 .Lkern1_end: 12348 .size kern1, .Lkern1_end-kern1 12349 12350 .rodata 12351 .p2align 6 12352 .amdhsa_kernel kern1 12353 // ... 12354 .amdhsa_next_free_vgpr .amdgcn.next_free_vgpr 12355 .amdhsa_next_free_sgpr .amdgcn.next_free_sgpr 12356 .end_amdhsa_kernel 12357 12358These symbols cannot identify connected components in order to automatically 12359track the usage for each kernel. However, in some cases careful organization of 12360the kernels and functions in the source file means there is minimal additional 12361effort required to accurately calculate GPR usage. 12362 12363Additional Documentation 12364======================== 12365 12366.. [AMD-GCN-GFX6] `AMD Southern Islands Series ISA <http://developer.amd.com/wordpress/media/2012/12/AMD_Southern_Islands_Instruction_Set_Architecture.pdf>`__ 12367.. [AMD-GCN-GFX7] `AMD Sea Islands Series ISA <http://developer.amd.com/wordpress/media/2013/07/AMD_Sea_Islands_Instruction_Set_Architecture.pdf>`_ 12368.. [AMD-GCN-GFX8] `AMD GCN3 Instruction Set Architecture <http://amd-dev.wpengine.netdna-cdn.com/wordpress/media/2013/12/AMD_GCN3_Instruction_Set_Architecture_rev1.1.pdf>`__ 12369.. [AMD-GCN-GFX900-GFX904-VEGA] `AMD Vega Instruction Set Architecture <http://developer.amd.com/wordpress/media/2013/12/Vega_Shader_ISA_28July2017.pdf>`__ 12370.. [AMD-GCN-GFX906-VEGA7NM] `AMD Vega 7nm Instruction Set Architecture <https://gpuopen.com/wp-content/uploads/2019/11/Vega_7nm_Shader_ISA_26November2019.pdf>`__ 12371.. [AMD-GCN-GFX908-CDNA1] `AMD Instinct MI100 Instruction Set Architecture <https://developer.amd.com/wp-content/resources/CDNA1_Shader_ISA_14December2020.pdf>`__ 12372.. [AMD-GCN-GFX10-RDNA1] `AMD RDNA 1.0 Instruction Set Architecture <https://gpuopen.com/wp-content/uploads/2019/08/RDNA_Shader_ISA_5August2019.pdf>`__ 12373.. [AMD-GCN-GFX10-RDNA2] `AMD RDNA 2 Instruction Set Architecture <https://developer.amd.com/wp-content/resources/RDNA2_Shader_ISA_November2020.pdf>`__ 12374.. [AMD-RADEON-HD-2000-3000] `AMD R6xx shader ISA <http://developer.amd.com/wordpress/media/2012/10/R600_Instruction_Set_Architecture.pdf>`__ 12375.. [AMD-RADEON-HD-4000] `AMD R7xx shader ISA <http://developer.amd.com/wordpress/media/2012/10/R700-Family_Instruction_Set_Architecture.pdf>`__ 12376.. [AMD-RADEON-HD-5000] `AMD Evergreen shader ISA <http://developer.amd.com/wordpress/media/2012/10/AMD_Evergreen-Family_Instruction_Set_Architecture.pdf>`__ 12377.. [AMD-RADEON-HD-6000] `AMD Cayman/Trinity shader ISA <http://developer.amd.com/wordpress/media/2012/10/AMD_HD_6900_Series_Instruction_Set_Architecture.pdf>`__ 12378.. [AMD-ROCm] `AMD ROCm™ Platform <https://rocmdocs.amd.com/>`__ 12379.. [AMD-ROCm-github] `AMD ROCm™ github <http://github.com/RadeonOpenCompute>`__ 12380.. [AMD-ROCm-Release-Notes] `AMD ROCm Release Notes <https://github.com/RadeonOpenCompute/ROCm>`__ 12381.. [CLANG-ATTR] `Attributes in Clang <https://clang.llvm.org/docs/AttributeReference.html>`__ 12382.. [DWARF] `DWARF Debugging Information Format <http://dwarfstd.org/>`__ 12383.. [ELF] `Executable and Linkable Format (ELF) <http://www.sco.com/developers/gabi/>`__ 12384.. [HRF] `Heterogeneous-race-free Memory Models <http://benedictgaster.org/wp-content/uploads/2014/01/asplos269-FINAL.pdf>`__ 12385.. [HSA] `Heterogeneous System Architecture (HSA) Foundation <http://www.hsafoundation.com/>`__ 12386.. [MsgPack] `Message Pack <http://www.msgpack.org/>`__ 12387.. [OpenCL] `The OpenCL Specification Version 2.0 <http://www.khronos.org/registry/cl/specs/opencl-2.0.pdf>`__ 12388.. [SEMVER] `Semantic Versioning <https://semver.org/>`__ 12389.. [YAML] `YAML Ain't Markup Language (YAML™) Version 1.2 <http://www.yaml.org/spec/1.2/spec.html>`__ 12390