1============================= 2User Guide for AMDGPU Backend 3============================= 4 5.. contents:: 6 :local: 7 8.. toctree:: 9 :hidden: 10 11 AMDGPU/AMDGPUAsmGFX7 12 AMDGPU/AMDGPUAsmGFX8 13 AMDGPU/AMDGPUAsmGFX9 14 AMDGPU/AMDGPUAsmGFX900 15 AMDGPU/AMDGPUAsmGFX904 16 AMDGPU/AMDGPUAsmGFX906 17 AMDGPU/AMDGPUAsmGFX908 18 AMDGPU/AMDGPUAsmGFX90a 19 AMDGPU/AMDGPUAsmGFX10 20 AMDGPU/AMDGPUAsmGFX1011 21 AMDGPUModifierSyntax 22 AMDGPUOperandSyntax 23 AMDGPUInstructionSyntax 24 AMDGPUInstructionNotation 25 AMDGPUDwarfExtensionsForHeterogeneousDebugging 26 27Introduction 28============ 29 30The AMDGPU backend provides ISA code generation for AMD GPUs, starting with the 31R600 family up until the current GCN families. It lives in the 32``llvm/lib/Target/AMDGPU`` directory. 33 34LLVM 35==== 36 37.. _amdgpu-target-triples: 38 39Target Triples 40-------------- 41 42Use the Clang option ``-target <Architecture>-<Vendor>-<OS>-<Environment>`` 43to specify the target triple: 44 45 .. table:: AMDGPU Architectures 46 :name: amdgpu-architecture-table 47 48 ============ ============================================================== 49 Architecture Description 50 ============ ============================================================== 51 ``r600`` AMD GPUs HD2XXX-HD6XXX for graphics and compute shaders. 52 ``amdgcn`` AMD GPUs GCN GFX6 onwards for graphics and compute shaders. 53 ============ ============================================================== 54 55 .. table:: AMDGPU Vendors 56 :name: amdgpu-vendor-table 57 58 ============ ============================================================== 59 Vendor Description 60 ============ ============================================================== 61 ``amd`` Can be used for all AMD GPU usage. 62 ``mesa3d`` Can be used if the OS is ``mesa3d``. 63 ============ ============================================================== 64 65 .. table:: AMDGPU Operating Systems 66 :name: amdgpu-os 67 68 ============== ============================================================ 69 OS Description 70 ============== ============================================================ 71 *<empty>* Defaults to the *unknown* OS. 72 ``amdhsa`` Compute kernels executed on HSA [HSA]_ compatible runtimes 73 such as: 74 75 - AMD's ROCm™ runtime [AMD-ROCm]_ using the *rocm-amdhsa* 76 loader on Linux. See *AMD ROCm Platform Release Notes* 77 [AMD-ROCm-Release-Notes]_ for supported hardware and 78 software. 79 - AMD's PAL runtime using the *pal-amdhsa* loader on 80 Windows. 81 82 ``amdpal`` Graphic shaders and compute kernels executed on AMD's PAL 83 runtime using the *pal-amdpal* loader on Windows and Linux 84 Pro. 85 ``mesa3d`` Graphic shaders and compute kernels executed on AMD's Mesa 86 3D runtime using the *mesa-mesa3d* loader on Linux. 87 ============== ============================================================ 88 89 .. table:: AMDGPU Environments 90 :name: amdgpu-environment-table 91 92 ============ ============================================================== 93 Environment Description 94 ============ ============================================================== 95 *<empty>* Default. 96 ============ ============================================================== 97 98.. _amdgpu-processors: 99 100Processors 101---------- 102 103Use the Clang options ``-mcpu=<target-id>`` or ``--offload-arch=<target-id>`` to 104specify the AMDGPU processor together with optional target features. See 105:ref:`amdgpu-target-id` and :ref:`amdgpu-target-features` for AMD GPU target 106specific information. 107 108Every processor supports every OS ABI (see :ref:`amdgpu-os`) with the following exceptions: 109 110* ``amdhsa`` is not supported in ``r600`` architecture (see :ref:`amdgpu-architecture-table`). 111 112 113 .. table:: AMDGPU Processors 114 :name: amdgpu-processor-table 115 116 =========== =============== ============ ===== ================= =============== =============== ====================== 117 Processor Alternative Target dGPU/ Target Target OS Support Example 118 Processor Triple APU Features Properties *(see* Products 119 Architecture Supported `amdgpu-os`_ 120 *and 121 corresponding 122 runtime release 123 notes for 124 current 125 information and 126 level of 127 support)* 128 =========== =============== ============ ===== ================= =============== =============== ====================== 129 **Radeon HD 2000/3000 Series (R600)** [AMD-RADEON-HD-2000-3000]_ 130 ----------------------------------------------------------------------------------------------------------------------- 131 ``r600`` ``r600`` dGPU - Does not 132 support 133 generic 134 address 135 space 136 ``r630`` ``r600`` dGPU - Does not 137 support 138 generic 139 address 140 space 141 ``rs880`` ``r600`` dGPU - Does not 142 support 143 generic 144 address 145 space 146 ``rv670`` ``r600`` dGPU - Does not 147 support 148 generic 149 address 150 space 151 **Radeon HD 4000 Series (R700)** [AMD-RADEON-HD-4000]_ 152 ----------------------------------------------------------------------------------------------------------------------- 153 ``rv710`` ``r600`` dGPU - Does not 154 support 155 generic 156 address 157 space 158 ``rv730`` ``r600`` dGPU - Does not 159 support 160 generic 161 address 162 space 163 ``rv770`` ``r600`` dGPU - Does not 164 support 165 generic 166 address 167 space 168 **Radeon HD 5000 Series (Evergreen)** [AMD-RADEON-HD-5000]_ 169 ----------------------------------------------------------------------------------------------------------------------- 170 ``cedar`` ``r600`` dGPU - Does not 171 support 172 generic 173 address 174 space 175 ``cypress`` ``r600`` dGPU - Does not 176 support 177 generic 178 address 179 space 180 ``juniper`` ``r600`` dGPU - Does not 181 support 182 generic 183 address 184 space 185 ``redwood`` ``r600`` dGPU - Does not 186 support 187 generic 188 address 189 space 190 ``sumo`` ``r600`` dGPU - Does not 191 support 192 generic 193 address 194 space 195 **Radeon HD 6000 Series (Northern Islands)** [AMD-RADEON-HD-6000]_ 196 ----------------------------------------------------------------------------------------------------------------------- 197 ``barts`` ``r600`` dGPU - Does not 198 support 199 generic 200 address 201 space 202 ``caicos`` ``r600`` dGPU - Does not 203 support 204 generic 205 address 206 space 207 ``cayman`` ``r600`` dGPU - Does not 208 support 209 generic 210 address 211 space 212 ``turks`` ``r600`` dGPU - Does not 213 support 214 generic 215 address 216 space 217 **GCN GFX6 (Southern Islands (SI))** [AMD-GCN-GFX6]_ 218 ----------------------------------------------------------------------------------------------------------------------- 219 ``gfx600`` - ``tahiti`` ``amdgcn`` dGPU - Does not - *pal-amdpal* 220 support 221 generic 222 address 223 space 224 ``gfx601`` - ``pitcairn`` ``amdgcn`` dGPU - Does not - *pal-amdpal* 225 - ``verde`` support 226 generic 227 address 228 space 229 ``gfx602`` - ``hainan`` ``amdgcn`` dGPU - Does not - *pal-amdpal* 230 - ``oland`` support 231 generic 232 address 233 space 234 **GCN GFX7 (Sea Islands (CI))** [AMD-GCN-GFX7]_ 235 ----------------------------------------------------------------------------------------------------------------------- 236 ``gfx700`` - ``kaveri`` ``amdgcn`` APU - Offset - *rocm-amdhsa* - A6-7000 237 flat - *pal-amdhsa* - A6 Pro-7050B 238 scratch - *pal-amdpal* - A8-7100 239 - A8 Pro-7150B 240 - A10-7300 241 - A10 Pro-7350B 242 - FX-7500 243 - A8-7200P 244 - A10-7400P 245 - FX-7600P 246 ``gfx701`` - ``hawaii`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - FirePro W8100 247 flat - *pal-amdhsa* - FirePro W9100 248 scratch - *pal-amdpal* - FirePro S9150 249 - FirePro S9170 250 ``gfx702`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - Radeon R9 290 251 flat - *pal-amdhsa* - Radeon R9 290x 252 scratch - *pal-amdpal* - Radeon R390 253 - Radeon R390x 254 ``gfx703`` - ``kabini`` ``amdgcn`` APU - Offset - *pal-amdhsa* - E1-2100 255 - ``mullins`` flat - *pal-amdpal* - E1-2200 256 scratch - E1-2500 257 - E2-3000 258 - E2-3800 259 - A4-5000 260 - A4-5100 261 - A6-5200 262 - A4 Pro-3340B 263 ``gfx704`` - ``bonaire`` ``amdgcn`` dGPU - Offset - *pal-amdhsa* - Radeon HD 7790 264 flat - *pal-amdpal* - Radeon HD 8770 265 scratch - R7 260 266 - R7 260X 267 ``gfx705`` ``amdgcn`` APU - Offset - *pal-amdhsa* *TBA* 268 flat - *pal-amdpal* 269 scratch .. TODO:: 270 271 Add product 272 names. 273 274 **GCN GFX8 (Volcanic Islands (VI))** [AMD-GCN-GFX8]_ 275 ----------------------------------------------------------------------------------------------------------------------- 276 ``gfx801`` - ``carrizo`` ``amdgcn`` APU - xnack - Offset - *rocm-amdhsa* - A6-8500P 277 flat - *pal-amdhsa* - Pro A6-8500B 278 scratch - *pal-amdpal* - A8-8600P 279 - Pro A8-8600B 280 - FX-8800P 281 - Pro A12-8800B 282 - A10-8700P 283 - Pro A10-8700B 284 - A10-8780P 285 - A10-9600P 286 - A10-9630P 287 - A12-9700P 288 - A12-9730P 289 - FX-9800P 290 - FX-9830P 291 - E2-9010 292 - A6-9210 293 - A9-9410 294 ``gfx802`` - ``iceland`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - Radeon R9 285 295 - ``tonga`` flat - *pal-amdhsa* - Radeon R9 380 296 scratch - *pal-amdpal* - Radeon R9 385 297 ``gfx803`` - ``fiji`` ``amdgcn`` dGPU - *rocm-amdhsa* - Radeon R9 Nano 298 - *pal-amdhsa* - Radeon R9 Fury 299 - *pal-amdpal* - Radeon R9 FuryX 300 - Radeon Pro Duo 301 - FirePro S9300x2 302 - Radeon Instinct MI8 303 \ - ``polaris10`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - Radeon RX 470 304 flat - *pal-amdhsa* - Radeon RX 480 305 scratch - *pal-amdpal* - Radeon Instinct MI6 306 \ - ``polaris11`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - Radeon RX 460 307 flat - *pal-amdhsa* 308 scratch - *pal-amdpal* 309 ``gfx805`` - ``tongapro`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - FirePro S7150 310 flat - *pal-amdhsa* - FirePro S7100 311 scratch - *pal-amdpal* - FirePro W7100 312 - Mobile FirePro 313 M7170 314 ``gfx810`` - ``stoney`` ``amdgcn`` APU - xnack - Offset - *rocm-amdhsa* *TBA* 315 flat - *pal-amdhsa* 316 scratch - *pal-amdpal* .. TODO:: 317 318 Add product 319 names. 320 321 **GCN GFX9 (Vega)** [AMD-GCN-GFX900-GFX904-VEGA]_ [AMD-GCN-GFX906-VEGA7NM]_ [AMD-GCN-GFX908-CDNA1]_ 322 ----------------------------------------------------------------------------------------------------------------------- 323 ``gfx900`` ``amdgcn`` dGPU - xnack - Absolute - *rocm-amdhsa* - Radeon Vega 324 flat - *pal-amdhsa* Frontier Edition 325 scratch - *pal-amdpal* - Radeon RX Vega 56 326 - Radeon RX Vega 64 327 - Radeon RX Vega 64 328 Liquid 329 - Radeon Instinct MI25 330 ``gfx902`` ``amdgcn`` APU - xnack - Absolute - *rocm-amdhsa* - Ryzen 3 2200G 331 flat - *pal-amdhsa* - Ryzen 5 2400G 332 scratch - *pal-amdpal* 333 ``gfx904`` ``amdgcn`` dGPU - xnack - *rocm-amdhsa* *TBA* 334 - *pal-amdhsa* 335 - *pal-amdpal* .. TODO:: 336 337 Add product 338 names. 339 340 ``gfx906`` ``amdgcn`` dGPU - sramecc - Absolute - *rocm-amdhsa* - Radeon Instinct MI50 341 - xnack flat - *pal-amdhsa* - Radeon Instinct MI60 342 scratch - *pal-amdpal* - Radeon VII 343 - Radeon Pro VII 344 ``gfx908`` ``amdgcn`` dGPU - sramecc - *rocm-amdhsa* - AMD Instinct MI100 Accelerator 345 - xnack - Absolute 346 flat 347 scratch 348 ``gfx909`` ``amdgcn`` APU - xnack - Absolute - *pal-amdpal* *TBA* 349 flat 350 scratch .. TODO:: 351 352 Add product 353 names. 354 355 ``gfx90a`` ``amdgcn`` dGPU - sramecc - Absolute - *rocm-amdhsa* *TBA* 356 - tgsplit flat 357 - xnack scratch .. TODO:: 358 - Packed 359 work-item Add product 360 IDs names. 361 362 ``gfx90c`` ``amdgcn`` APU - xnack - Absolute - *pal-amdpal* - Ryzen 7 4700G 363 flat - Ryzen 7 4700GE 364 scratch - Ryzen 5 4600G 365 - Ryzen 5 4600GE 366 - Ryzen 3 4300G 367 - Ryzen 3 4300GE 368 - Ryzen Pro 4000G 369 - Ryzen 7 Pro 4700G 370 - Ryzen 7 Pro 4750GE 371 - Ryzen 5 Pro 4650G 372 - Ryzen 5 Pro 4650GE 373 - Ryzen 3 Pro 4350G 374 - Ryzen 3 Pro 4350GE 375 376 **GCN GFX10 (RDNA 1)** [AMD-GCN-GFX10-RDNA1]_ 377 ----------------------------------------------------------------------------------------------------------------------- 378 ``gfx1010`` ``amdgcn`` dGPU - cumode - Absolute - *rocm-amdhsa* - Radeon RX 5700 379 - wavefrontsize64 flat - *pal-amdhsa* - Radeon RX 5700 XT 380 - xnack scratch - *pal-amdpal* - Radeon Pro 5600 XT 381 - Radeon Pro 5600M 382 ``gfx1011`` ``amdgcn`` dGPU - cumode - *rocm-amdhsa* - Radeon Pro V520 383 - wavefrontsize64 - Absolute - *pal-amdhsa* 384 - xnack flat - *pal-amdpal* 385 scratch 386 ``gfx1012`` ``amdgcn`` dGPU - cumode - Absolute - *rocm-amdhsa* - Radeon RX 5500 387 - wavefrontsize64 flat - *pal-amdhsa* - Radeon RX 5500 XT 388 - xnack scratch - *pal-amdpal* 389 ``gfx1013`` ``amdgcn`` APU - cumode - Absolute - *rocm-amdhsa* *TBA* 390 - wavefrontsize64 flat - *pal-amdhsa* 391 - xnack scratch - *pal-amdpal* .. TODO:: 392 393 Add product 394 names. 395 396 **GCN GFX10 (RDNA 2)** [AMD-GCN-GFX10-RDNA2]_ 397 ----------------------------------------------------------------------------------------------------------------------- 398 ``gfx1030`` ``amdgcn`` dGPU - cumode - Absolute - *rocm-amdhsa* - Radeon RX 6800 399 - wavefrontsize64 flat - *pal-amdhsa* - Radeon RX 6800 XT 400 scratch - *pal-amdpal* - Radeon RX 6900 XT 401 ``gfx1031`` ``amdgcn`` dGPU - cumode - Absolute - *rocm-amdhsa* - Radeon RX 6700 XT 402 - wavefrontsize64 flat - *pal-amdhsa* 403 scratch - *pal-amdpal* 404 ``gfx1032`` ``amdgcn`` dGPU - cumode - Absolute - *rocm-amdhsa* *TBA* 405 - wavefrontsize64 flat - *pal-amdhsa* 406 scratch - *pal-amdpal* .. TODO:: 407 408 Add product 409 names. 410 411 ``gfx1033`` ``amdgcn`` APU - cumode - Absolute - *pal-amdpal* *TBA* 412 - wavefrontsize64 flat 413 scratch .. TODO:: 414 415 Add product 416 names. 417 ``gfx1034`` ``amdgcn`` dGPU - cumode - Absolute - *pal-amdpal* *TBA* 418 - wavefrontsize64 flat 419 scratch .. TODO:: 420 421 Add product 422 names. 423 424 ``gfx1035`` ``amdgcn`` APU - cumode - Absolute - *pal-amdpal* *TBA* 425 - wavefrontsize64 flat 426 scratch .. TODO:: 427 Add product 428 names. 429 430 =========== =============== ============ ===== ================= =============== =============== ====================== 431 432.. _amdgpu-target-features: 433 434Target Features 435--------------- 436 437Target features control how code is generated to support certain 438processor specific features. Not all target features are supported by 439all processors. The runtime must ensure that the features supported by 440the device used to execute the code match the features enabled when 441generating the code. A mismatch of features may result in incorrect 442execution, or a reduction in performance. 443 444The target features supported by each processor is listed in 445:ref:`amdgpu-processor-table`. 446 447Target features are controlled by exactly one of the following Clang 448options: 449 450``-mcpu=<target-id>`` or ``--offload-arch=<target-id>`` 451 452 The ``-mcpu`` and ``--offload-arch`` can specify the target feature as 453 optional components of the target ID. If omitted, the target feature has the 454 ``any`` value. See :ref:`amdgpu-target-id`. 455 456``-m[no-]<target-feature>`` 457 458 Target features not specified by the target ID are specified using a 459 separate option. These target features can have an ``on`` or ``off`` 460 value. ``on`` is specified by omitting the ``no-`` prefix, and 461 ``off`` is specified by including the ``no-`` prefix. The default 462 if not specified is ``off``. 463 464For example: 465 466``-mcpu=gfx908:xnack+`` 467 Enable the ``xnack`` feature. 468``-mcpu=gfx908:xnack-`` 469 Disable the ``xnack`` feature. 470``-mcumode`` 471 Enable the ``cumode`` feature. 472``-mno-cumode`` 473 Disable the ``cumode`` feature. 474 475 .. table:: AMDGPU Target Features 476 :name: amdgpu-target-features-table 477 478 =============== ============================ ================================================== 479 Target Feature Clang Option to Control Description 480 Name 481 =============== ============================ ================================================== 482 cumode - ``-m[no-]cumode`` Control the wavefront execution mode used 483 when generating code for kernels. When disabled 484 native WGP wavefront execution mode is used, 485 when enabled CU wavefront execution mode is used 486 (see :ref:`amdgpu-amdhsa-memory-model`). 487 488 sramecc - ``-mcpu`` If specified, generate code that can only be 489 - ``--offload-arch`` loaded and executed in a process that has a 490 matching setting for SRAMECC. 491 492 If not specified for code object V2 to V3, generate 493 code that can be loaded and executed in a process 494 with SRAMECC enabled. 495 496 If not specified for code object V4, generate 497 code that can be loaded and executed in a process 498 with either setting of SRAMECC. 499 500 tgsplit ``-m[no-]tgsplit`` Enable/disable generating code that assumes 501 work-groups are launched in threadgroup split mode. 502 When enabled the waves of a work-group may be 503 launched in different CUs. 504 505 wavefrontsize64 - ``-m[no-]wavefrontsize64`` Control the wavefront size used when 506 generating code for kernels. When disabled 507 native wavefront size 32 is used, when enabled 508 wavefront size 64 is used. 509 510 xnack - ``-mcpu`` If specified, generate code that can only be 511 - ``--offload-arch`` loaded and executed in a process that has a 512 matching setting for XNACK replay. 513 514 If not specified for code object V2 to V3, generate 515 code that can be loaded and executed in a process 516 with XNACK replay enabled. 517 518 If not specified for code object V4, generate 519 code that can be loaded and executed in a process 520 with either setting of XNACK replay. 521 522 XNACK replay can be used for demand paging and 523 page migration. If enabled in the device, then if 524 a page fault occurs the code may execute 525 incorrectly unless generated with XNACK replay 526 enabled, or generated for code object V4 without 527 specifying XNACK replay. Executing code that was 528 generated with XNACK replay enabled, or generated 529 for code object V4 without specifying XNACK replay, 530 on a device that does not have XNACK replay 531 enabled will execute correctly but may be less 532 performant than code generated for XNACK replay 533 disabled. 534 =============== ============================ ================================================== 535 536.. _amdgpu-target-id: 537 538Target ID 539--------- 540 541AMDGPU supports target IDs. See `Clang Offload Bundler 542<https://clang.llvm.org/docs/ClangOffloadBundler.html>`_ for a general 543description. The AMDGPU target specific information is: 544 545**processor** 546 Is an AMDGPU processor or alternative processor name specified in 547 :ref:`amdgpu-processor-table`. The non-canonical form target ID allows both 548 the primary processor and alternative processor names. The canonical form 549 target ID only allow the primary processor name. 550 551**target-feature** 552 Is a target feature name specified in :ref:`amdgpu-target-features-table` that 553 is supported by the processor. The target features supported by each processor 554 is specified in :ref:`amdgpu-processor-table`. Those that can be specified in 555 a target ID are marked as being controlled by ``-mcpu`` and 556 ``--offload-arch``. Each target feature must appear at most once in a target 557 ID. The non-canonical form target ID allows the target features to be 558 specified in any order. The canonical form target ID requires the target 559 features to be specified in alphabetic order. 560 561.. _amdgpu-target-id-v2-v3: 562 563Code Object V2 to V3 Target ID 564~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 565 566The target ID syntax for code object V2 to V3 is the same as defined in `Clang 567Offload Bundler <https://clang.llvm.org/docs/ClangOffloadBundler.html>`_ except 568when used in the :ref:`amdgpu-assembler-directive-amdgcn-target` assembler 569directive and the bundle entry ID. In those cases it has the following BNF 570syntax: 571 572.. code:: 573 574 <target-id> ::== <processor> ( "+" <target-feature> )* 575 576Where a target feature is omitted if *Off* and present if *On* or *Any*. 577 578.. note:: 579 580 The code object V2 to V3 cannot represent *Any* and treats it the same as 581 *On*. 582 583.. _amdgpu-embedding-bundled-objects: 584 585Embedding Bundled Code Objects 586------------------------------ 587 588AMDGPU supports the HIP and OpenMP languages that perform code object embedding 589as described in `Clang Offload Bundler 590<https://clang.llvm.org/docs/ClangOffloadBundler.html>`_. 591 592.. note:: 593 594 The target ID syntax used for code object V2 to V3 for a bundle entry ID 595 differs from that used elsewhere. See :ref:`amdgpu-target-id-v2-v3`. 596 597.. _amdgpu-address-spaces: 598 599Address Spaces 600-------------- 601 602The AMDGPU architecture supports a number of memory address spaces. The address 603space names use the OpenCL standard names, with some additions. 604 605The AMDGPU address spaces correspond to target architecture specific LLVM 606address space numbers used in LLVM IR. 607 608The AMDGPU address spaces are described in 609:ref:`amdgpu-address-spaces-table`. Only 64-bit process address spaces are 610supported for the ``amdgcn`` target. 611 612 .. table:: AMDGPU Address Spaces 613 :name: amdgpu-address-spaces-table 614 615 ================================= =============== =========== ================ ======= ============================ 616 .. 64-Bit Process Address Space 617 --------------------------------- --------------- ----------- ---------------- ------------------------------------ 618 Address Space Name LLVM IR Address HSA Segment Hardware Address NULL Value 619 Space Number Name Name Size 620 ================================= =============== =========== ================ ======= ============================ 621 Generic 0 flat flat 64 0x0000000000000000 622 Global 1 global global 64 0x0000000000000000 623 Region 2 N/A GDS 32 *not implemented for AMDHSA* 624 Local 3 group LDS 32 0xFFFFFFFF 625 Constant 4 constant *same as global* 64 0x0000000000000000 626 Private 5 private scratch 32 0xFFFFFFFF 627 Constant 32-bit 6 *TODO* 0x00000000 628 Buffer Fat Pointer (experimental) 7 *TODO* 629 ================================= =============== =========== ================ ======= ============================ 630 631**Generic** 632 The generic address space is supported unless the *Target Properties* column 633 of :ref:`amdgpu-processor-table` specifies *Does not support generic address 634 space*. 635 636 The generic address space uses the hardware flat address support for two fixed 637 ranges of virtual addresses (the private and local apertures), that are 638 outside the range of addressable global memory, to map from a flat address to 639 a private or local address. This uses FLAT instructions that can take a flat 640 address and access global, private (scratch), and group (LDS) memory depending 641 on if the address is within one of the aperture ranges. 642 643 Flat access to scratch requires hardware aperture setup and setup in the 644 kernel prologue (see :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`). Flat 645 access to LDS requires hardware aperture setup and M0 (GFX7-GFX8) register 646 setup (see :ref:`amdgpu-amdhsa-kernel-prolog-m0`). 647 648 To convert between a private or group address space address (termed a segment 649 address) and a flat address the base address of the corresponding aperture 650 can be used. For GFX7-GFX8 these are available in the 651 :ref:`amdgpu-amdhsa-hsa-aql-queue` the address of which can be obtained with 652 Queue Ptr SGPR (see :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). For 653 GFX9-GFX10 the aperture base addresses are directly available as inline 654 constant registers ``SRC_SHARED_BASE/LIMIT`` and ``SRC_PRIVATE_BASE/LIMIT``. 655 In 64-bit address mode the aperture sizes are 2^32 bytes and the base is 656 aligned to 2^32 which makes it easier to convert from flat to segment or 657 segment to flat. 658 659 A global address space address has the same value when used as a flat address 660 so no conversion is needed. 661 662**Global and Constant** 663 The global and constant address spaces both use global virtual addresses, 664 which are the same virtual address space used by the CPU. However, some 665 virtual addresses may only be accessible to the CPU, some only accessible 666 by the GPU, and some by both. 667 668 Using the constant address space indicates that the data will not change 669 during the execution of the kernel. This allows scalar read instructions to 670 be used. As the constant address space could only be modified on the host 671 side, a generic pointer loaded from the constant address space is safe to be 672 assumed as a global pointer since only the device global memory is visible 673 and managed on the host side. The vector and scalar L1 caches are invalidated 674 of volatile data before each kernel dispatch execution to allow constant 675 memory to change values between kernel dispatches. 676 677**Region** 678 The region address space uses the hardware Global Data Store (GDS). All 679 wavefronts executing on the same device will access the same memory for any 680 given region address. However, the same region address accessed by wavefronts 681 executing on different devices will access different memory. It is higher 682 performance than global memory. It is allocated by the runtime. The data 683 store (DS) instructions can be used to access it. 684 685**Local** 686 The local address space uses the hardware Local Data Store (LDS) which is 687 automatically allocated when the hardware creates the wavefronts of a 688 work-group, and freed when all the wavefronts of a work-group have 689 terminated. All wavefronts belonging to the same work-group will access the 690 same memory for any given local address. However, the same local address 691 accessed by wavefronts belonging to different work-groups will access 692 different memory. It is higher performance than global memory. The data store 693 (DS) instructions can be used to access it. 694 695**Private** 696 The private address space uses the hardware scratch memory support which 697 automatically allocates memory when it creates a wavefront and frees it when 698 a wavefronts terminates. The memory accessed by a lane of a wavefront for any 699 given private address will be different to the memory accessed by another lane 700 of the same or different wavefront for the same private address. 701 702 If a kernel dispatch uses scratch, then the hardware allocates memory from a 703 pool of backing memory allocated by the runtime for each wavefront. The lanes 704 of the wavefront access this using dword (4 byte) interleaving. The mapping 705 used from private address to backing memory address is: 706 707 ``wavefront-scratch-base + 708 ((private-address / 4) * wavefront-size * 4) + 709 (wavefront-lane-id * 4) + (private-address % 4)`` 710 711 If each lane of a wavefront accesses the same private address, the 712 interleaving results in adjacent dwords being accessed and hence requires 713 fewer cache lines to be fetched. 714 715 There are different ways that the wavefront scratch base address is 716 determined by a wavefront (see 717 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 718 719 Scratch memory can be accessed in an interleaved manner using buffer 720 instructions with the scratch buffer descriptor and per wavefront scratch 721 offset, by the scratch instructions, or by flat instructions. Multi-dword 722 access is not supported except by flat and scratch instructions in 723 GFX9-GFX10. 724 725**Constant 32-bit** 726 *TODO* 727 728**Buffer Fat Pointer** 729 The buffer fat pointer is an experimental address space that is currently 730 unsupported in the backend. It exposes a non-integral pointer that is in 731 the future intended to support the modelling of 128-bit buffer descriptors 732 plus a 32-bit offset into the buffer (in total encapsulating a 160-bit 733 *pointer*), allowing normal LLVM load/store/atomic operations to be used to 734 model the buffer descriptors used heavily in graphics workloads targeting 735 the backend. 736 737.. _amdgpu-memory-scopes: 738 739Memory Scopes 740------------- 741 742This section provides LLVM memory synchronization scopes supported by the AMDGPU 743backend memory model when the target triple OS is ``amdhsa`` (see 744:ref:`amdgpu-amdhsa-memory-model` and :ref:`amdgpu-target-triples`). 745 746The memory model supported is based on the HSA memory model [HSA]_ which is 747based in turn on HRF-indirect with scope inclusion [HRF]_. The happens-before 748relation is transitive over the synchronizes-with relation independent of scope 749and synchronizes-with allows the memory scope instances to be inclusive (see 750table :ref:`amdgpu-amdhsa-llvm-sync-scopes-table`). 751 752This is different to the OpenCL [OpenCL]_ memory model which does not have scope 753inclusion and requires the memory scopes to exactly match. However, this 754is conservatively correct for OpenCL. 755 756 .. table:: AMDHSA LLVM Sync Scopes 757 :name: amdgpu-amdhsa-llvm-sync-scopes-table 758 759 ======================= =================================================== 760 LLVM Sync Scope Description 761 ======================= =================================================== 762 *none* The default: ``system``. 763 764 Synchronizes with, and participates in modification 765 and seq_cst total orderings with, other operations 766 (except image operations) for all address spaces 767 (except private, or generic that accesses private) 768 provided the other operation's sync scope is: 769 770 - ``system``. 771 - ``agent`` and executed by a thread on the same 772 agent. 773 - ``workgroup`` and executed by a thread in the 774 same work-group. 775 - ``wavefront`` and executed by a thread in the 776 same wavefront. 777 778 ``agent`` Synchronizes with, and participates in modification 779 and seq_cst total orderings with, other operations 780 (except image operations) for all address spaces 781 (except private, or generic that accesses private) 782 provided the other operation's sync scope is: 783 784 - ``system`` or ``agent`` and executed by a thread 785 on the same agent. 786 - ``workgroup`` and executed by a thread in the 787 same work-group. 788 - ``wavefront`` and executed by a thread in the 789 same wavefront. 790 791 ``workgroup`` Synchronizes with, and participates in modification 792 and seq_cst total orderings with, other operations 793 (except image operations) for all address spaces 794 (except private, or generic that accesses private) 795 provided the other operation's sync scope is: 796 797 - ``system``, ``agent`` or ``workgroup`` and 798 executed by a thread in the same work-group. 799 - ``wavefront`` and executed by a thread in the 800 same wavefront. 801 802 ``wavefront`` Synchronizes with, and participates in modification 803 and seq_cst total orderings with, other operations 804 (except image operations) for all address spaces 805 (except private, or generic that accesses private) 806 provided the other operation's sync scope is: 807 808 - ``system``, ``agent``, ``workgroup`` or 809 ``wavefront`` and executed by a thread in the 810 same wavefront. 811 812 ``singlethread`` Only synchronizes with and participates in 813 modification and seq_cst total orderings with, 814 other operations (except image operations) running 815 in the same thread for all address spaces (for 816 example, in signal handlers). 817 818 ``one-as`` Same as ``system`` but only synchronizes with other 819 operations within the same address space. 820 821 ``agent-one-as`` Same as ``agent`` but only synchronizes with other 822 operations within the same address space. 823 824 ``workgroup-one-as`` Same as ``workgroup`` but only synchronizes with 825 other operations within the same address space. 826 827 ``wavefront-one-as`` Same as ``wavefront`` but only synchronizes with 828 other operations within the same address space. 829 830 ``singlethread-one-as`` Same as ``singlethread`` but only synchronizes with 831 other operations within the same address space. 832 ======================= =================================================== 833 834LLVM IR Intrinsics 835------------------ 836 837The AMDGPU backend implements the following LLVM IR intrinsics. 838 839*This section is WIP.* 840 841.. TODO:: 842 843 List AMDGPU intrinsics. 844 845LLVM IR Attributes 846------------------ 847 848The AMDGPU backend supports the following LLVM IR attributes. 849 850 .. table:: AMDGPU LLVM IR Attributes 851 :name: amdgpu-llvm-ir-attributes-table 852 853 ======================================= ========================================================== 854 LLVM Attribute Description 855 ======================================= ========================================================== 856 "amdgpu-flat-work-group-size"="min,max" Specify the minimum and maximum flat work group sizes that 857 will be specified when the kernel is dispatched. Generated 858 by the ``amdgpu_flat_work_group_size`` CLANG attribute [CLANG-ATTR]_. 859 "amdgpu-implicitarg-num-bytes"="n" Number of kernel argument bytes to add to the kernel 860 argument block size for the implicit arguments. This 861 varies by OS and language (for OpenCL see 862 :ref:`opencl-kernel-implicit-arguments-appended-for-amdhsa-os-table`). 863 "amdgpu-num-sgpr"="n" Specifies the number of SGPRs to use. Generated by 864 the ``amdgpu_num_sgpr`` CLANG attribute [CLANG-ATTR]_. 865 "amdgpu-num-vgpr"="n" Specifies the number of VGPRs to use. Generated by the 866 ``amdgpu_num_vgpr`` CLANG attribute [CLANG-ATTR]_. 867 "amdgpu-waves-per-eu"="m,n" Specify the minimum and maximum number of waves per 868 execution unit. Generated by the ``amdgpu_waves_per_eu`` 869 CLANG attribute [CLANG-ATTR]_. 870 "amdgpu-ieee" true/false. Specify whether the function expects the IEEE field of the 871 mode register to be set on entry. Overrides the default for 872 the calling convention. 873 "amdgpu-dx10-clamp" true/false. Specify whether the function expects the DX10_CLAMP field of 874 the mode register to be set on entry. Overrides the default 875 for the calling convention. 876 877 "amdgpu-no-workitem-id-x" Indicates the function does not depend on the value of the 878 llvm.amdgcn.workitem.id.x intrinsic. If a function is marked with this 879 attribute, or reached through a call site marked with this attribute, 880 the value returned by the intrinsic is undefined. The backend can 881 generally infer this during code generation, so typically there is no 882 benefit to frontends marking functions with this. 883 884 "amdgpu-no-workitem-id-y" The same as amdgpu-no-workitem-id-x, except for the 885 llvm.amdgcn.workitem.id.y intrinsic. 886 887 "amdgpu-no-workitem-id-z" The same as amdgpu-no-workitem-id-x, except for the 888 llvm.amdgcn.workitem.id.z intrinsic. 889 890 "amdgpu-no-workgroup-id-x" The same as amdgpu-no-workitem-id-x, except for the 891 llvm.amdgcn.workgroup.id.x intrinsic. 892 893 "amdgpu-no-workgroup-id-y" The same as amdgpu-no-workitem-id-x, except for the 894 llvm.amdgcn.workgroup.id.y intrinsic. 895 896 "amdgpu-no-workgroup-id-z" The same as amdgpu-no-workitem-id-x, except for the 897 llvm.amdgcn.workgroup.id.z intrinsic. 898 899 "amdgpu-no-dispatch-ptr" The same as amdgpu-no-workitem-id-x, except for the 900 llvm.amdgcn.dispatch.ptr intrinsic. 901 902 "amdgpu-no-implicitarg-ptr" The same as amdgpu-no-workitem-id-x, except for the 903 llvm.amdgcn.implicitarg.ptr intrinsic. 904 905 "amdgpu-no-dispatch-id" The same as amdgpu-no-workitem-id-x, except for the 906 llvm.amdgcn.dispatch.id intrinsic. 907 908 "amdgpu-no-queue-ptr" Similar to amdgpu-no-workitem-id-x, except for the 909 llvm.amdgcn.queue.ptr intrinsic. Note that unlike the other ABI hint 910 attributes, the queue pointer may be required in situations where the 911 intrinsic call does not directly appear in the program. Some subtargets 912 require the queue pointer for to handle some addrspacecasts, as well 913 as the llvm.amdgcn.is.shared, llvm.amdgcn.is.private, llvm.trap, and 914 llvm.debug intrinsics. 915 916 ======================================= ========================================================== 917 918.. _amdgpu-elf-code-object: 919 920ELF Code Object 921=============== 922 923The AMDGPU backend generates a standard ELF [ELF]_ relocatable code object that 924can be linked by ``lld`` to produce a standard ELF shared code object which can 925be loaded and executed on an AMDGPU target. 926 927.. _amdgpu-elf-header: 928 929Header 930------ 931 932The AMDGPU backend uses the following ELF header: 933 934 .. table:: AMDGPU ELF Header 935 :name: amdgpu-elf-header-table 936 937 ========================== =============================== 938 Field Value 939 ========================== =============================== 940 ``e_ident[EI_CLASS]`` ``ELFCLASS64`` 941 ``e_ident[EI_DATA]`` ``ELFDATA2LSB`` 942 ``e_ident[EI_OSABI]`` - ``ELFOSABI_NONE`` 943 - ``ELFOSABI_AMDGPU_HSA`` 944 - ``ELFOSABI_AMDGPU_PAL`` 945 - ``ELFOSABI_AMDGPU_MESA3D`` 946 ``e_ident[EI_ABIVERSION]`` - ``ELFABIVERSION_AMDGPU_HSA_V2`` 947 - ``ELFABIVERSION_AMDGPU_HSA_V3`` 948 - ``ELFABIVERSION_AMDGPU_HSA_V4`` 949 - ``ELFABIVERSION_AMDGPU_PAL`` 950 - ``ELFABIVERSION_AMDGPU_MESA3D`` 951 ``e_type`` - ``ET_REL`` 952 - ``ET_DYN`` 953 ``e_machine`` ``EM_AMDGPU`` 954 ``e_entry`` 0 955 ``e_flags`` See :ref:`amdgpu-elf-header-e_flags-v2-table`, 956 :ref:`amdgpu-elf-header-e_flags-table-v3`, 957 and :ref:`amdgpu-elf-header-e_flags-table-v4` 958 ========================== =============================== 959 960.. 961 962 .. table:: AMDGPU ELF Header Enumeration Values 963 :name: amdgpu-elf-header-enumeration-values-table 964 965 =============================== ===== 966 Name Value 967 =============================== ===== 968 ``EM_AMDGPU`` 224 969 ``ELFOSABI_NONE`` 0 970 ``ELFOSABI_AMDGPU_HSA`` 64 971 ``ELFOSABI_AMDGPU_PAL`` 65 972 ``ELFOSABI_AMDGPU_MESA3D`` 66 973 ``ELFABIVERSION_AMDGPU_HSA_V2`` 0 974 ``ELFABIVERSION_AMDGPU_HSA_V3`` 1 975 ``ELFABIVERSION_AMDGPU_HSA_V4`` 2 976 ``ELFABIVERSION_AMDGPU_PAL`` 0 977 ``ELFABIVERSION_AMDGPU_MESA3D`` 0 978 =============================== ===== 979 980``e_ident[EI_CLASS]`` 981 The ELF class is: 982 983 * ``ELFCLASS32`` for ``r600`` architecture. 984 985 * ``ELFCLASS64`` for ``amdgcn`` architecture which only supports 64-bit 986 process address space applications. 987 988``e_ident[EI_DATA]`` 989 All AMDGPU targets use ``ELFDATA2LSB`` for little-endian byte ordering. 990 991``e_ident[EI_OSABI]`` 992 One of the following AMDGPU target architecture specific OS ABIs 993 (see :ref:`amdgpu-os`): 994 995 * ``ELFOSABI_NONE`` for *unknown* OS. 996 997 * ``ELFOSABI_AMDGPU_HSA`` for ``amdhsa`` OS. 998 999 * ``ELFOSABI_AMDGPU_PAL`` for ``amdpal`` OS. 1000 1001 * ``ELFOSABI_AMDGPU_MESA3D`` for ``mesa3D`` OS. 1002 1003``e_ident[EI_ABIVERSION]`` 1004 The ABI version of the AMDGPU target architecture specific OS ABI to which the code 1005 object conforms: 1006 1007 * ``ELFABIVERSION_AMDGPU_HSA_V2`` is used to specify the version of AMD HSA 1008 runtime ABI for code object V2. Specify using the Clang option 1009 ``-mcode-object-version=2``. 1010 1011 * ``ELFABIVERSION_AMDGPU_HSA_V3`` is used to specify the version of AMD HSA 1012 runtime ABI for code object V3. Specify using the Clang option 1013 ``-mcode-object-version=3``. This is the default code object 1014 version if not specified. 1015 1016 * ``ELFABIVERSION_AMDGPU_HSA_V4`` is used to specify the version of AMD HSA 1017 runtime ABI for code object V4. Specify using the Clang option 1018 ``-mcode-object-version=4``. 1019 1020 * ``ELFABIVERSION_AMDGPU_PAL`` is used to specify the version of AMD PAL 1021 runtime ABI. 1022 1023 * ``ELFABIVERSION_AMDGPU_MESA3D`` is used to specify the version of AMD MESA 1024 3D runtime ABI. 1025 1026``e_type`` 1027 Can be one of the following values: 1028 1029 1030 ``ET_REL`` 1031 The type produced by the AMDGPU backend compiler as it is relocatable code 1032 object. 1033 1034 ``ET_DYN`` 1035 The type produced by the linker as it is a shared code object. 1036 1037 The AMD HSA runtime loader requires a ``ET_DYN`` code object. 1038 1039``e_machine`` 1040 The value ``EM_AMDGPU`` is used for the machine for all processors supported 1041 by the ``r600`` and ``amdgcn`` architectures (see 1042 :ref:`amdgpu-processor-table`). The specific processor is specified in the 1043 ``NT_AMD_HSA_ISA_VERSION`` note record for code object V2 (see 1044 :ref:`amdgpu-note-records-v2`) and in the ``EF_AMDGPU_MACH`` bit field of the 1045 ``e_flags`` for code object V3 to V4 (see 1046 :ref:`amdgpu-elf-header-e_flags-table-v3` and 1047 :ref:`amdgpu-elf-header-e_flags-table-v4`). 1048 1049``e_entry`` 1050 The entry point is 0 as the entry points for individual kernels must be 1051 selected in order to invoke them through AQL packets. 1052 1053``e_flags`` 1054 The AMDGPU backend uses the following ELF header flags: 1055 1056 .. table:: AMDGPU ELF Header ``e_flags`` for Code Object V2 1057 :name: amdgpu-elf-header-e_flags-v2-table 1058 1059 ===================================== ===== ============================= 1060 Name Value Description 1061 ===================================== ===== ============================= 1062 ``EF_AMDGPU_FEATURE_XNACK_V2`` 0x01 Indicates if the ``xnack`` 1063 target feature is 1064 enabled for all code 1065 contained in the code object. 1066 If the processor 1067 does not support the 1068 ``xnack`` target 1069 feature then must 1070 be 0. 1071 See 1072 :ref:`amdgpu-target-features`. 1073 ``EF_AMDGPU_FEATURE_TRAP_HANDLER_V2`` 0x02 Indicates if the trap 1074 handler is enabled for all 1075 code contained in the code 1076 object. If the processor 1077 does not support a trap 1078 handler then must be 0. 1079 See 1080 :ref:`amdgpu-target-features`. 1081 ===================================== ===== ============================= 1082 1083 .. table:: AMDGPU ELF Header ``e_flags`` for Code Object V3 1084 :name: amdgpu-elf-header-e_flags-table-v3 1085 1086 ================================= ===== ============================= 1087 Name Value Description 1088 ================================= ===== ============================= 1089 ``EF_AMDGPU_MACH`` 0x0ff AMDGPU processor selection 1090 mask for 1091 ``EF_AMDGPU_MACH_xxx`` values 1092 defined in 1093 :ref:`amdgpu-ef-amdgpu-mach-table`. 1094 ``EF_AMDGPU_FEATURE_XNACK_V3`` 0x100 Indicates if the ``xnack`` 1095 target feature is 1096 enabled for all code 1097 contained in the code object. 1098 If the processor 1099 does not support the 1100 ``xnack`` target 1101 feature then must 1102 be 0. 1103 See 1104 :ref:`amdgpu-target-features`. 1105 ``EF_AMDGPU_FEATURE_SRAMECC_V3`` 0x200 Indicates if the ``sramecc`` 1106 target feature is 1107 enabled for all code 1108 contained in the code object. 1109 If the processor 1110 does not support the 1111 ``sramecc`` target 1112 feature then must 1113 be 0. 1114 See 1115 :ref:`amdgpu-target-features`. 1116 ================================= ===== ============================= 1117 1118 .. table:: AMDGPU ELF Header ``e_flags`` for Code Object V4 1119 :name: amdgpu-elf-header-e_flags-table-v4 1120 1121 ============================================ ===== =================================== 1122 Name Value Description 1123 ============================================ ===== =================================== 1124 ``EF_AMDGPU_MACH`` 0x0ff AMDGPU processor selection 1125 mask for 1126 ``EF_AMDGPU_MACH_xxx`` values 1127 defined in 1128 :ref:`amdgpu-ef-amdgpu-mach-table`. 1129 ``EF_AMDGPU_FEATURE_XNACK_V4`` 0x300 XNACK selection mask for 1130 ``EF_AMDGPU_FEATURE_XNACK_*_V4`` 1131 values. 1132 ``EF_AMDGPU_FEATURE_XNACK_UNSUPPORTED_V4`` 0x000 XNACK unsuppored. 1133 ``EF_AMDGPU_FEATURE_XNACK_ANY_V4`` 0x100 XNACK can have any value. 1134 ``EF_AMDGPU_FEATURE_XNACK_OFF_V4`` 0x200 XNACK disabled. 1135 ``EF_AMDGPU_FEATURE_XNACK_ON_V4`` 0x300 XNACK enabled. 1136 ``EF_AMDGPU_FEATURE_SRAMECC_V4`` 0xc00 SRAMECC selection mask for 1137 ``EF_AMDGPU_FEATURE_SRAMECC_*_V4`` 1138 values. 1139 ``EF_AMDGPU_FEATURE_SRAMECC_UNSUPPORTED_V4`` 0x000 SRAMECC unsuppored. 1140 ``EF_AMDGPU_FEATURE_SRAMECC_ANY_V4`` 0x400 SRAMECC can have any value. 1141 ``EF_AMDGPU_FEATURE_SRAMECC_OFF_V4`` 0x800 SRAMECC disabled, 1142 ``EF_AMDGPU_FEATURE_SRAMECC_ON_V4`` 0xc00 SRAMECC enabled. 1143 ============================================ ===== =================================== 1144 1145 .. table:: AMDGPU ``EF_AMDGPU_MACH`` Values 1146 :name: amdgpu-ef-amdgpu-mach-table 1147 1148 ==================================== ========== ============================= 1149 Name Value Description (see 1150 :ref:`amdgpu-processor-table`) 1151 ==================================== ========== ============================= 1152 ``EF_AMDGPU_MACH_NONE`` 0x000 *not specified* 1153 ``EF_AMDGPU_MACH_R600_R600`` 0x001 ``r600`` 1154 ``EF_AMDGPU_MACH_R600_R630`` 0x002 ``r630`` 1155 ``EF_AMDGPU_MACH_R600_RS880`` 0x003 ``rs880`` 1156 ``EF_AMDGPU_MACH_R600_RV670`` 0x004 ``rv670`` 1157 ``EF_AMDGPU_MACH_R600_RV710`` 0x005 ``rv710`` 1158 ``EF_AMDGPU_MACH_R600_RV730`` 0x006 ``rv730`` 1159 ``EF_AMDGPU_MACH_R600_RV770`` 0x007 ``rv770`` 1160 ``EF_AMDGPU_MACH_R600_CEDAR`` 0x008 ``cedar`` 1161 ``EF_AMDGPU_MACH_R600_CYPRESS`` 0x009 ``cypress`` 1162 ``EF_AMDGPU_MACH_R600_JUNIPER`` 0x00a ``juniper`` 1163 ``EF_AMDGPU_MACH_R600_REDWOOD`` 0x00b ``redwood`` 1164 ``EF_AMDGPU_MACH_R600_SUMO`` 0x00c ``sumo`` 1165 ``EF_AMDGPU_MACH_R600_BARTS`` 0x00d ``barts`` 1166 ``EF_AMDGPU_MACH_R600_CAICOS`` 0x00e ``caicos`` 1167 ``EF_AMDGPU_MACH_R600_CAYMAN`` 0x00f ``cayman`` 1168 ``EF_AMDGPU_MACH_R600_TURKS`` 0x010 ``turks`` 1169 *reserved* 0x011 - Reserved for ``r600`` 1170 0x01f architecture processors. 1171 ``EF_AMDGPU_MACH_AMDGCN_GFX600`` 0x020 ``gfx600`` 1172 ``EF_AMDGPU_MACH_AMDGCN_GFX601`` 0x021 ``gfx601`` 1173 ``EF_AMDGPU_MACH_AMDGCN_GFX700`` 0x022 ``gfx700`` 1174 ``EF_AMDGPU_MACH_AMDGCN_GFX701`` 0x023 ``gfx701`` 1175 ``EF_AMDGPU_MACH_AMDGCN_GFX702`` 0x024 ``gfx702`` 1176 ``EF_AMDGPU_MACH_AMDGCN_GFX703`` 0x025 ``gfx703`` 1177 ``EF_AMDGPU_MACH_AMDGCN_GFX704`` 0x026 ``gfx704`` 1178 *reserved* 0x027 Reserved. 1179 ``EF_AMDGPU_MACH_AMDGCN_GFX801`` 0x028 ``gfx801`` 1180 ``EF_AMDGPU_MACH_AMDGCN_GFX802`` 0x029 ``gfx802`` 1181 ``EF_AMDGPU_MACH_AMDGCN_GFX803`` 0x02a ``gfx803`` 1182 ``EF_AMDGPU_MACH_AMDGCN_GFX810`` 0x02b ``gfx810`` 1183 ``EF_AMDGPU_MACH_AMDGCN_GFX900`` 0x02c ``gfx900`` 1184 ``EF_AMDGPU_MACH_AMDGCN_GFX902`` 0x02d ``gfx902`` 1185 ``EF_AMDGPU_MACH_AMDGCN_GFX904`` 0x02e ``gfx904`` 1186 ``EF_AMDGPU_MACH_AMDGCN_GFX906`` 0x02f ``gfx906`` 1187 ``EF_AMDGPU_MACH_AMDGCN_GFX908`` 0x030 ``gfx908`` 1188 ``EF_AMDGPU_MACH_AMDGCN_GFX909`` 0x031 ``gfx909`` 1189 ``EF_AMDGPU_MACH_AMDGCN_GFX90C`` 0x032 ``gfx90c`` 1190 ``EF_AMDGPU_MACH_AMDGCN_GFX1010`` 0x033 ``gfx1010`` 1191 ``EF_AMDGPU_MACH_AMDGCN_GFX1011`` 0x034 ``gfx1011`` 1192 ``EF_AMDGPU_MACH_AMDGCN_GFX1012`` 0x035 ``gfx1012`` 1193 ``EF_AMDGPU_MACH_AMDGCN_GFX1030`` 0x036 ``gfx1030`` 1194 ``EF_AMDGPU_MACH_AMDGCN_GFX1031`` 0x037 ``gfx1031`` 1195 ``EF_AMDGPU_MACH_AMDGCN_GFX1032`` 0x038 ``gfx1032`` 1196 ``EF_AMDGPU_MACH_AMDGCN_GFX1033`` 0x039 ``gfx1033`` 1197 ``EF_AMDGPU_MACH_AMDGCN_GFX602`` 0x03a ``gfx602`` 1198 ``EF_AMDGPU_MACH_AMDGCN_GFX705`` 0x03b ``gfx705`` 1199 ``EF_AMDGPU_MACH_AMDGCN_GFX805`` 0x03c ``gfx805`` 1200 ``EF_AMDGPU_MACH_AMDGCN_GFX1035`` 0x03d ``gfx1035`` 1201 ``EF_AMDGPU_MACH_AMDGCN_GFX1034`` 0x03e ``gfx1034`` 1202 ``EF_AMDGPU_MACH_AMDGCN_GFX90A`` 0x03f ``gfx90a`` 1203 *reserved* 0x040 Reserved. 1204 *reserved* 0x041 Reserved. 1205 ``EF_AMDGPU_MACH_AMDGCN_GFX1013`` 0x042 ``gfx1013`` 1206 *reserved* 0x043 Reserved. 1207 *reserved* 0x044 Reserved. 1208 *reserved* 0x045 Reserved. 1209 ==================================== ========== ============================= 1210 1211Sections 1212-------- 1213 1214An AMDGPU target ELF code object has the standard ELF sections which include: 1215 1216 .. table:: AMDGPU ELF Sections 1217 :name: amdgpu-elf-sections-table 1218 1219 ================== ================ ================================= 1220 Name Type Attributes 1221 ================== ================ ================================= 1222 ``.bss`` ``SHT_NOBITS`` ``SHF_ALLOC`` + ``SHF_WRITE`` 1223 ``.data`` ``SHT_PROGBITS`` ``SHF_ALLOC`` + ``SHF_WRITE`` 1224 ``.debug_``\ *\** ``SHT_PROGBITS`` *none* 1225 ``.dynamic`` ``SHT_DYNAMIC`` ``SHF_ALLOC`` 1226 ``.dynstr`` ``SHT_PROGBITS`` ``SHF_ALLOC`` 1227 ``.dynsym`` ``SHT_PROGBITS`` ``SHF_ALLOC`` 1228 ``.got`` ``SHT_PROGBITS`` ``SHF_ALLOC`` + ``SHF_WRITE`` 1229 ``.hash`` ``SHT_HASH`` ``SHF_ALLOC`` 1230 ``.note`` ``SHT_NOTE`` *none* 1231 ``.rela``\ *name* ``SHT_RELA`` *none* 1232 ``.rela.dyn`` ``SHT_RELA`` *none* 1233 ``.rodata`` ``SHT_PROGBITS`` ``SHF_ALLOC`` 1234 ``.shstrtab`` ``SHT_STRTAB`` *none* 1235 ``.strtab`` ``SHT_STRTAB`` *none* 1236 ``.symtab`` ``SHT_SYMTAB`` *none* 1237 ``.text`` ``SHT_PROGBITS`` ``SHF_ALLOC`` + ``SHF_EXECINSTR`` 1238 ================== ================ ================================= 1239 1240These sections have their standard meanings (see [ELF]_) and are only generated 1241if needed. 1242 1243``.debug``\ *\** 1244 The standard DWARF sections. See :ref:`amdgpu-dwarf-debug-information` for 1245 information on the DWARF produced by the AMDGPU backend. 1246 1247``.dynamic``, ``.dynstr``, ``.dynsym``, ``.hash`` 1248 The standard sections used by a dynamic loader. 1249 1250``.note`` 1251 See :ref:`amdgpu-note-records` for the note records supported by the AMDGPU 1252 backend. 1253 1254``.rela``\ *name*, ``.rela.dyn`` 1255 For relocatable code objects, *name* is the name of the section that the 1256 relocation records apply. For example, ``.rela.text`` is the section name for 1257 relocation records associated with the ``.text`` section. 1258 1259 For linked shared code objects, ``.rela.dyn`` contains all the relocation 1260 records from each of the relocatable code object's ``.rela``\ *name* sections. 1261 1262 See :ref:`amdgpu-relocation-records` for the relocation records supported by 1263 the AMDGPU backend. 1264 1265``.text`` 1266 The executable machine code for the kernels and functions they call. Generated 1267 as position independent code. See :ref:`amdgpu-code-conventions` for 1268 information on conventions used in the isa generation. 1269 1270.. _amdgpu-note-records: 1271 1272Note Records 1273------------ 1274 1275The AMDGPU backend code object contains ELF note records in the ``.note`` 1276section. The set of generated notes and their semantics depend on the code 1277object version; see :ref:`amdgpu-note-records-v2` and 1278:ref:`amdgpu-note-records-v3-v4`. 1279 1280As required by ``ELFCLASS32`` and ``ELFCLASS64``, minimal zero-byte padding 1281must be generated after the ``name`` field to ensure the ``desc`` field is 4 1282byte aligned. In addition, minimal zero-byte padding must be generated to 1283ensure the ``desc`` field size is a multiple of 4 bytes. The ``sh_addralign`` 1284field of the ``.note`` section must be at least 4 to indicate at least 8 byte 1285alignment. 1286 1287.. _amdgpu-note-records-v2: 1288 1289Code Object V2 Note Records 1290~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1291 1292.. warning:: 1293 Code object V2 is not the default code object version emitted by 1294 this version of LLVM. 1295 1296The AMDGPU backend code object uses the following ELF note record in the 1297``.note`` section when compiling for code object V2. 1298 1299The note record vendor field is "AMD". 1300 1301Additional note records may be present, but any which are not documented here 1302are deprecated and should not be used. 1303 1304 .. table:: AMDGPU Code Object V2 ELF Note Records 1305 :name: amdgpu-elf-note-records-v2-table 1306 1307 ===== ===================================== ====================================== 1308 Name Type Description 1309 ===== ===================================== ====================================== 1310 "AMD" ``NT_AMD_HSA_CODE_OBJECT_VERSION`` Code object version. 1311 "AMD" ``NT_AMD_HSA_HSAIL`` HSAIL properties generated by the HSAIL 1312 Finalizer and not the LLVM compiler. 1313 "AMD" ``NT_AMD_HSA_ISA_VERSION`` Target ISA version. 1314 "AMD" ``NT_AMD_HSA_METADATA`` Metadata null terminated string in 1315 YAML [YAML]_ textual format. 1316 "AMD" ``NT_AMD_HSA_ISA_NAME`` Target ISA name. 1317 ===== ===================================== ====================================== 1318 1319.. 1320 1321 .. table:: AMDGPU Code Object V2 ELF Note Record Enumeration Values 1322 :name: amdgpu-elf-note-record-enumeration-values-v2-table 1323 1324 ===================================== ===== 1325 Name Value 1326 ===================================== ===== 1327 ``NT_AMD_HSA_CODE_OBJECT_VERSION`` 1 1328 ``NT_AMD_HSA_HSAIL`` 2 1329 ``NT_AMD_HSA_ISA_VERSION`` 3 1330 *reserved* 4-9 1331 ``NT_AMD_HSA_METADATA`` 10 1332 ``NT_AMD_HSA_ISA_NAME`` 11 1333 ===================================== ===== 1334 1335``NT_AMD_HSA_CODE_OBJECT_VERSION`` 1336 Specifies the code object version number. The description field has the 1337 following layout: 1338 1339 .. code:: c 1340 1341 struct amdgpu_hsa_note_code_object_version_s { 1342 uint32_t major_version; 1343 uint32_t minor_version; 1344 }; 1345 1346 The ``major_version`` has a value less than or equal to 2. 1347 1348``NT_AMD_HSA_HSAIL`` 1349 Specifies the HSAIL properties used by the HSAIL Finalizer. The description 1350 field has the following layout: 1351 1352 .. code:: c 1353 1354 struct amdgpu_hsa_note_hsail_s { 1355 uint32_t hsail_major_version; 1356 uint32_t hsail_minor_version; 1357 uint8_t profile; 1358 uint8_t machine_model; 1359 uint8_t default_float_round; 1360 }; 1361 1362``NT_AMD_HSA_ISA_VERSION`` 1363 Specifies the target ISA version. The description field has the following layout: 1364 1365 .. code:: c 1366 1367 struct amdgpu_hsa_note_isa_s { 1368 uint16_t vendor_name_size; 1369 uint16_t architecture_name_size; 1370 uint32_t major; 1371 uint32_t minor; 1372 uint32_t stepping; 1373 char vendor_and_architecture_name[1]; 1374 }; 1375 1376 ``vendor_name_size`` and ``architecture_name_size`` are the length of the 1377 vendor and architecture names respectively, including the NUL character. 1378 1379 ``vendor_and_architecture_name`` contains the NUL terminates string for the 1380 vendor, immediately followed by the NUL terminated string for the 1381 architecture. 1382 1383 This note record is used by the HSA runtime loader. 1384 1385 Code object V2 only supports a limited number of processors and has fixed 1386 settings for target features. See 1387 :ref:`amdgpu-elf-note-record-supported_processors-v2-table` for a list of 1388 processors and the corresponding target ID. In the table the note record ISA 1389 name is a concatenation of the vendor name, architecture name, major, minor, 1390 and stepping separated by a ":". 1391 1392 The target ID column shows the processor name and fixed target features used 1393 by the LLVM compiler. The LLVM compiler does not generate a 1394 ``NT_AMD_HSA_HSAIL`` note record. 1395 1396 A code object generated by the Finalizer also uses code object V2 and always 1397 generates a ``NT_AMD_HSA_HSAIL`` note record. The processor name and 1398 ``sramecc`` target feature is as shown in 1399 :ref:`amdgpu-elf-note-record-supported_processors-v2-table` but the ``xnack`` 1400 target feature is specified by the ``EF_AMDGPU_FEATURE_XNACK_V2`` ``e_flags`` 1401 bit. 1402 1403``NT_AMD_HSA_ISA_NAME`` 1404 Specifies the target ISA name as a non-NUL terminated string. 1405 1406 This note record is not used by the HSA runtime loader. 1407 1408 See the ``NT_AMD_HSA_ISA_VERSION`` note record description of the code object 1409 V2's limited support of processors and fixed settings for target features. 1410 1411 See :ref:`amdgpu-elf-note-record-supported_processors-v2-table` for a mapping 1412 from the string to the corresponding target ID. If the ``xnack`` target 1413 feature is supported and enabled, the string produced by the LLVM compiler 1414 will may have a ``+xnack`` appended. The Finlizer did not do the appending and 1415 instead used the ``EF_AMDGPU_FEATURE_XNACK_V2`` ``e_flags`` bit. 1416 1417``NT_AMD_HSA_METADATA`` 1418 Specifies extensible metadata associated with the code objects executed on HSA 1419 [HSA]_ compatible runtimes (see :ref:`amdgpu-os`). It is required when the 1420 target triple OS is ``amdhsa`` (see :ref:`amdgpu-target-triples`). See 1421 :ref:`amdgpu-amdhsa-code-object-metadata-v2` for the syntax of the code object 1422 metadata string. 1423 1424 .. table:: AMDGPU Code Object V2 Supported Processors and Fixed Target Feature Settings 1425 :name: amdgpu-elf-note-record-supported_processors-v2-table 1426 1427 ===================== ========================== 1428 Note Record ISA Name Target ID 1429 ===================== ========================== 1430 ``AMD:AMDGPU:6:0:0`` ``gfx600`` 1431 ``AMD:AMDGPU:6:0:1`` ``gfx601`` 1432 ``AMD:AMDGPU:6:0:2`` ``gfx602`` 1433 ``AMD:AMDGPU:7:0:0`` ``gfx700`` 1434 ``AMD:AMDGPU:7:0:1`` ``gfx701`` 1435 ``AMD:AMDGPU:7:0:2`` ``gfx702`` 1436 ``AMD:AMDGPU:7:0:3`` ``gfx703`` 1437 ``AMD:AMDGPU:7:0:4`` ``gfx704`` 1438 ``AMD:AMDGPU:7:0:5`` ``gfx705`` 1439 ``AMD:AMDGPU:8:0:0`` ``gfx802`` 1440 ``AMD:AMDGPU:8:0:1`` ``gfx801:xnack+`` 1441 ``AMD:AMDGPU:8:0:2`` ``gfx802`` 1442 ``AMD:AMDGPU:8:0:3`` ``gfx803`` 1443 ``AMD:AMDGPU:8:0:4`` ``gfx803`` 1444 ``AMD:AMDGPU:8:0:5`` ``gfx805`` 1445 ``AMD:AMDGPU:8:1:0`` ``gfx810:xnack+`` 1446 ``AMD:AMDGPU:9:0:0`` ``gfx900:xnack-`` 1447 ``AMD:AMDGPU:9:0:1`` ``gfx900:xnack+`` 1448 ``AMD:AMDGPU:9:0:2`` ``gfx902:xnack-`` 1449 ``AMD:AMDGPU:9:0:3`` ``gfx902:xnack+`` 1450 ``AMD:AMDGPU:9:0:4`` ``gfx904:xnack-`` 1451 ``AMD:AMDGPU:9:0:5`` ``gfx904:xnack+`` 1452 ``AMD:AMDGPU:9:0:6`` ``gfx906:sramecc-:xnack-`` 1453 ``AMD:AMDGPU:9:0:7`` ``gfx906:sramecc-:xnack+`` 1454 ``AMD:AMDGPU:9:0:12`` ``gfx90c:xnack-`` 1455 ===================== ========================== 1456 1457.. _amdgpu-note-records-v3-v4: 1458 1459Code Object V3 to V4 Note Records 1460~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1461 1462The AMDGPU backend code object uses the following ELF note record in the 1463``.note`` section when compiling for code object V3 to V4. 1464 1465The note record vendor field is "AMDGPU". 1466 1467Additional note records may be present, but any which are not documented here 1468are deprecated and should not be used. 1469 1470 .. table:: AMDGPU Code Object V3 to V4 ELF Note Records 1471 :name: amdgpu-elf-note-records-table-v3-v4 1472 1473 ======== ============================== ====================================== 1474 Name Type Description 1475 ======== ============================== ====================================== 1476 "AMDGPU" ``NT_AMDGPU_METADATA`` Metadata in Message Pack [MsgPack]_ 1477 binary format. 1478 ======== ============================== ====================================== 1479 1480.. 1481 1482 .. table:: AMDGPU Code Object V3 to V4 ELF Note Record Enumeration Values 1483 :name: amdgpu-elf-note-record-enumeration-values-table-v3-v4 1484 1485 ============================== ===== 1486 Name Value 1487 ============================== ===== 1488 *reserved* 0-31 1489 ``NT_AMDGPU_METADATA`` 32 1490 ============================== ===== 1491 1492``NT_AMDGPU_METADATA`` 1493 Specifies extensible metadata associated with an AMDGPU code object. It is 1494 encoded as a map in the Message Pack [MsgPack]_ binary data format. See 1495 :ref:`amdgpu-amdhsa-code-object-metadata-v3` and 1496 :ref:`amdgpu-amdhsa-code-object-metadata-v4` for the map keys defined for the 1497 ``amdhsa`` OS. 1498 1499.. _amdgpu-symbols: 1500 1501Symbols 1502------- 1503 1504Symbols include the following: 1505 1506 .. table:: AMDGPU ELF Symbols 1507 :name: amdgpu-elf-symbols-table 1508 1509 ===================== ================== ================ ================== 1510 Name Type Section Description 1511 ===================== ================== ================ ================== 1512 *link-name* ``STT_OBJECT`` - ``.data`` Global variable 1513 - ``.rodata`` 1514 - ``.bss`` 1515 *link-name*\ ``.kd`` ``STT_OBJECT`` - ``.rodata`` Kernel descriptor 1516 *link-name* ``STT_FUNC`` - ``.text`` Kernel entry point 1517 *link-name* ``STT_OBJECT`` - SHN_AMDGPU_LDS Global variable in LDS 1518 ===================== ================== ================ ================== 1519 1520Global variable 1521 Global variables both used and defined by the compilation unit. 1522 1523 If the symbol is defined in the compilation unit then it is allocated in the 1524 appropriate section according to if it has initialized data or is readonly. 1525 1526 If the symbol is external then its section is ``STN_UNDEF`` and the loader 1527 will resolve relocations using the definition provided by another code object 1528 or explicitly defined by the runtime. 1529 1530 If the symbol resides in local/group memory (LDS) then its section is the 1531 special processor specific section name ``SHN_AMDGPU_LDS``, and the 1532 ``st_value`` field describes alignment requirements as it does for common 1533 symbols. 1534 1535 .. TODO:: 1536 1537 Add description of linked shared object symbols. Seems undefined symbols 1538 are marked as STT_NOTYPE. 1539 1540Kernel descriptor 1541 Every HSA kernel has an associated kernel descriptor. It is the address of the 1542 kernel descriptor that is used in the AQL dispatch packet used to invoke the 1543 kernel, not the kernel entry point. The layout of the HSA kernel descriptor is 1544 defined in :ref:`amdgpu-amdhsa-kernel-descriptor`. 1545 1546Kernel entry point 1547 Every HSA kernel also has a symbol for its machine code entry point. 1548 1549.. _amdgpu-relocation-records: 1550 1551Relocation Records 1552------------------ 1553 1554AMDGPU backend generates ``Elf64_Rela`` relocation records. Supported 1555relocatable fields are: 1556 1557``word32`` 1558 This specifies a 32-bit field occupying 4 bytes with arbitrary byte 1559 alignment. These values use the same byte order as other word values in the 1560 AMDGPU architecture. 1561 1562``word64`` 1563 This specifies a 64-bit field occupying 8 bytes with arbitrary byte 1564 alignment. These values use the same byte order as other word values in the 1565 AMDGPU architecture. 1566 1567Following notations are used for specifying relocation calculations: 1568 1569**A** 1570 Represents the addend used to compute the value of the relocatable field. 1571 1572**G** 1573 Represents the offset into the global offset table at which the relocation 1574 entry's symbol will reside during execution. 1575 1576**GOT** 1577 Represents the address of the global offset table. 1578 1579**P** 1580 Represents the place (section offset for ``et_rel`` or address for ``et_dyn``) 1581 of the storage unit being relocated (computed using ``r_offset``). 1582 1583**S** 1584 Represents the value of the symbol whose index resides in the relocation 1585 entry. Relocations not using this must specify a symbol index of 1586 ``STN_UNDEF``. 1587 1588**B** 1589 Represents the base address of a loaded executable or shared object which is 1590 the difference between the ELF address and the actual load address. 1591 Relocations using this are only valid in executable or shared objects. 1592 1593The following relocation types are supported: 1594 1595 .. table:: AMDGPU ELF Relocation Records 1596 :name: amdgpu-elf-relocation-records-table 1597 1598 ========================== ======= ===== ========== ============================== 1599 Relocation Type Kind Value Field Calculation 1600 ========================== ======= ===== ========== ============================== 1601 ``R_AMDGPU_NONE`` 0 *none* *none* 1602 ``R_AMDGPU_ABS32_LO`` Static, 1 ``word32`` (S + A) & 0xFFFFFFFF 1603 Dynamic 1604 ``R_AMDGPU_ABS32_HI`` Static, 2 ``word32`` (S + A) >> 32 1605 Dynamic 1606 ``R_AMDGPU_ABS64`` Static, 3 ``word64`` S + A 1607 Dynamic 1608 ``R_AMDGPU_REL32`` Static 4 ``word32`` S + A - P 1609 ``R_AMDGPU_REL64`` Static 5 ``word64`` S + A - P 1610 ``R_AMDGPU_ABS32`` Static, 6 ``word32`` S + A 1611 Dynamic 1612 ``R_AMDGPU_GOTPCREL`` Static 7 ``word32`` G + GOT + A - P 1613 ``R_AMDGPU_GOTPCREL32_LO`` Static 8 ``word32`` (G + GOT + A - P) & 0xFFFFFFFF 1614 ``R_AMDGPU_GOTPCREL32_HI`` Static 9 ``word32`` (G + GOT + A - P) >> 32 1615 ``R_AMDGPU_REL32_LO`` Static 10 ``word32`` (S + A - P) & 0xFFFFFFFF 1616 ``R_AMDGPU_REL32_HI`` Static 11 ``word32`` (S + A - P) >> 32 1617 *reserved* 12 1618 ``R_AMDGPU_RELATIVE64`` Dynamic 13 ``word64`` B + A 1619 ``R_AMDGPU_REL16`` Static 14 ``word16`` ((S + A - P) - 4) / 4 1620 ========================== ======= ===== ========== ============================== 1621 1622``R_AMDGPU_ABS32_LO`` and ``R_AMDGPU_ABS32_HI`` are only supported by 1623the ``mesa3d`` OS, which does not support ``R_AMDGPU_ABS64``. 1624 1625There is no current OS loader support for 32-bit programs and so 1626``R_AMDGPU_ABS32`` is not used. 1627 1628.. _amdgpu-loaded-code-object-path-uniform-resource-identifier: 1629 1630Loaded Code Object Path Uniform Resource Identifier (URI) 1631--------------------------------------------------------- 1632 1633The AMD GPU code object loader represents the path of the ELF shared object from 1634which the code object was loaded as a textual Uniform Resource Identifier (URI). 1635Note that the code object is the in memory loaded relocated form of the ELF 1636shared object. Multiple code objects may be loaded at different memory 1637addresses in the same process from the same ELF shared object. 1638 1639The loaded code object path URI syntax is defined by the following BNF syntax: 1640 1641.. code:: 1642 1643 code_object_uri ::== file_uri | memory_uri 1644 file_uri ::== "file://" file_path [ range_specifier ] 1645 memory_uri ::== "memory://" process_id range_specifier 1646 range_specifier ::== [ "#" | "?" ] "offset=" number "&" "size=" number 1647 file_path ::== URI_ENCODED_OS_FILE_PATH 1648 process_id ::== DECIMAL_NUMBER 1649 number ::== HEX_NUMBER | DECIMAL_NUMBER | OCTAL_NUMBER 1650 1651**number** 1652 Is a C integral literal where hexadecimal values are prefixed by "0x" or "0X", 1653 and octal values by "0". 1654 1655**file_path** 1656 Is the file's path specified as a URI encoded UTF-8 string. In URI encoding, 1657 every character that is not in the regular expression ``[a-zA-Z0-9/_.~-]`` is 1658 encoded as two uppercase hexadecimal digits proceeded by "%". Directories in 1659 the path are separated by "/". 1660 1661**offset** 1662 Is a 0-based byte offset to the start of the code object. For a file URI, it 1663 is from the start of the file specified by the ``file_path``, and if omitted 1664 defaults to 0. For a memory URI, it is the memory address and is required. 1665 1666**size** 1667 Is the number of bytes in the code object. For a file URI, if omitted it 1668 defaults to the size of the file. It is required for a memory URI. 1669 1670**process_id** 1671 Is the identity of the process owning the memory. For Linux it is the C 1672 unsigned integral decimal literal for the process ID (PID). 1673 1674For example: 1675 1676.. code:: 1677 1678 file:///dir1/dir2/file1 1679 file:///dir3/dir4/file2#offset=0x2000&size=3000 1680 memory://1234#offset=0x20000&size=3000 1681 1682.. _amdgpu-dwarf-debug-information: 1683 1684DWARF Debug Information 1685======================= 1686 1687.. warning:: 1688 1689 This section describes **provisional support** for AMDGPU DWARF [DWARF]_ that 1690 is not currently fully implemented and is subject to change. 1691 1692AMDGPU generates DWARF [DWARF]_ debugging information ELF sections (see 1693:ref:`amdgpu-elf-code-object`) which contain information that maps the code 1694object executable code and data to the source language constructs. It can be 1695used by tools such as debuggers and profilers. It uses features defined in 1696:doc:`AMDGPUDwarfExtensionsForHeterogeneousDebugging` that are made available in 1697DWARF Version 4 and DWARF Version 5 as an LLVM vendor extension. 1698 1699This section defines the AMDGPU target architecture specific DWARF mappings. 1700 1701.. _amdgpu-dwarf-register-identifier: 1702 1703Register Identifier 1704------------------- 1705 1706This section defines the AMDGPU target architecture register numbers used in 1707DWARF operation expressions (see DWARF Version 5 section 2.5 and 1708:ref:`amdgpu-dwarf-operation-expressions`) and Call Frame Information 1709instructions (see DWARF Version 5 section 6.4 and 1710:ref:`amdgpu-dwarf-call-frame-information`). 1711 1712A single code object can contain code for kernels that have different wavefront 1713sizes. The vector registers and some scalar registers are based on the wavefront 1714size. AMDGPU defines distinct DWARF registers for each wavefront size. This 1715simplifies the consumer of the DWARF so that each register has a fixed size, 1716rather than being dynamic according to the wavefront size mode. Similarly, 1717distinct DWARF registers are defined for those registers that vary in size 1718according to the process address size. This allows a consumer to treat a 1719specific AMDGPU processor as a single architecture regardless of how it is 1720configured at run time. The compiler explicitly specifies the DWARF registers 1721that match the mode in which the code it is generating will be executed. 1722 1723DWARF registers are encoded as numbers, which are mapped to architecture 1724registers. The mapping for AMDGPU is defined in 1725:ref:`amdgpu-dwarf-register-mapping-table`. All AMDGPU targets use the same 1726mapping. 1727 1728.. table:: AMDGPU DWARF Register Mapping 1729 :name: amdgpu-dwarf-register-mapping-table 1730 1731 ============== ================= ======== ================================== 1732 DWARF Register AMDGPU Register Bit Size Description 1733 ============== ================= ======== ================================== 1734 0 PC_32 32 Program Counter (PC) when 1735 executing in a 32-bit process 1736 address space. Used in the CFI to 1737 describe the PC of the calling 1738 frame. 1739 1 EXEC_MASK_32 32 Execution Mask Register when 1740 executing in wavefront 32 mode. 1741 2-15 *Reserved* *Reserved for highly accessed 1742 registers using DWARF shortcut.* 1743 16 PC_64 64 Program Counter (PC) when 1744 executing in a 64-bit process 1745 address space. Used in the CFI to 1746 describe the PC of the calling 1747 frame. 1748 17 EXEC_MASK_64 64 Execution Mask Register when 1749 executing in wavefront 64 mode. 1750 18-31 *Reserved* *Reserved for highly accessed 1751 registers using DWARF shortcut.* 1752 32-95 SGPR0-SGPR63 32 Scalar General Purpose 1753 Registers. 1754 96-127 *Reserved* *Reserved for frequently accessed 1755 registers using DWARF 1-byte ULEB.* 1756 128 STATUS 32 Status Register. 1757 129-511 *Reserved* *Reserved for future Scalar 1758 Architectural Registers.* 1759 512 VCC_32 32 Vector Condition Code Register 1760 when executing in wavefront 32 1761 mode. 1762 513-767 *Reserved* *Reserved for future Vector 1763 Architectural Registers when 1764 executing in wavefront 32 mode.* 1765 768 VCC_64 64 Vector Condition Code Register 1766 when executing in wavefront 64 1767 mode. 1768 769-1023 *Reserved* *Reserved for future Vector 1769 Architectural Registers when 1770 executing in wavefront 64 mode.* 1771 1024-1087 *Reserved* *Reserved for padding.* 1772 1088-1129 SGPR64-SGPR105 32 Scalar General Purpose Registers. 1773 1130-1535 *Reserved* *Reserved for future Scalar 1774 General Purpose Registers.* 1775 1536-1791 VGPR0-VGPR255 32*32 Vector General Purpose Registers 1776 when executing in wavefront 32 1777 mode. 1778 1792-2047 *Reserved* *Reserved for future Vector 1779 General Purpose Registers when 1780 executing in wavefront 32 mode.* 1781 2048-2303 AGPR0-AGPR255 32*32 Vector Accumulation Registers 1782 when executing in wavefront 32 1783 mode. 1784 2304-2559 *Reserved* *Reserved for future Vector 1785 Accumulation Registers when 1786 executing in wavefront 32 mode.* 1787 2560-2815 VGPR0-VGPR255 64*32 Vector General Purpose Registers 1788 when executing in wavefront 64 1789 mode. 1790 2816-3071 *Reserved* *Reserved for future Vector 1791 General Purpose Registers when 1792 executing in wavefront 64 mode.* 1793 3072-3327 AGPR0-AGPR255 64*32 Vector Accumulation Registers 1794 when executing in wavefront 64 1795 mode. 1796 3328-3583 *Reserved* *Reserved for future Vector 1797 Accumulation Registers when 1798 executing in wavefront 64 mode.* 1799 ============== ================= ======== ================================== 1800 1801The vector registers are represented as the full size for the wavefront. They 1802are organized as consecutive dwords (32-bits), one per lane, with the dword at 1803the least significant bit position corresponding to lane 0 and so forth. DWARF 1804location expressions involving the ``DW_OP_LLVM_offset`` and 1805``DW_OP_LLVM_push_lane`` operations are used to select the part of the vector 1806register corresponding to the lane that is executing the current thread of 1807execution in languages that are implemented using a SIMD or SIMT execution 1808model. 1809 1810If the wavefront size is 32 lanes then the wavefront 32 mode register 1811definitions are used. If the wavefront size is 64 lanes then the wavefront 64 1812mode register definitions are used. Some AMDGPU targets support executing in 1813both wavefront 32 and wavefront 64 mode. The register definitions corresponding 1814to the wavefront mode of the generated code will be used. 1815 1816If code is generated to execute in a 32-bit process address space, then the 181732-bit process address space register definitions are used. If code is generated 1818to execute in a 64-bit process address space, then the 64-bit process address 1819space register definitions are used. The ``amdgcn`` target only supports the 182064-bit process address space. 1821 1822.. _amdgpu-dwarf-address-class-identifier: 1823 1824Address Class Identifier 1825------------------------ 1826 1827The DWARF address class represents the source language memory space. See DWARF 1828Version 5 section 2.12 which is updated by the *DWARF Extensions For 1829Heterogeneous Debugging* section :ref:`amdgpu-dwarf-segment_addresses`. 1830 1831The DWARF address class mapping used for AMDGPU is defined in 1832:ref:`amdgpu-dwarf-address-class-mapping-table`. 1833 1834.. table:: AMDGPU DWARF Address Class Mapping 1835 :name: amdgpu-dwarf-address-class-mapping-table 1836 1837 ========================= ====== ================= 1838 DWARF AMDGPU 1839 -------------------------------- ----------------- 1840 Address Class Name Value Address Space 1841 ========================= ====== ================= 1842 ``DW_ADDR_none`` 0x0000 Generic (Flat) 1843 ``DW_ADDR_LLVM_global`` 0x0001 Global 1844 ``DW_ADDR_LLVM_constant`` 0x0002 Global 1845 ``DW_ADDR_LLVM_group`` 0x0003 Local (group/LDS) 1846 ``DW_ADDR_LLVM_private`` 0x0004 Private (Scratch) 1847 ``DW_ADDR_AMDGPU_region`` 0x8000 Region (GDS) 1848 ========================= ====== ================= 1849 1850The DWARF address class values defined in the *DWARF Extensions For 1851Heterogeneous Debugging* section :ref:`amdgpu-dwarf-segment_addresses` are used. 1852 1853In addition, ``DW_ADDR_AMDGPU_region`` is encoded as a vendor extension. This is 1854available for use for the AMD extension for access to the hardware GDS memory 1855which is scratchpad memory allocated per device. 1856 1857For AMDGPU if no ``DW_AT_address_class`` attribute is present, then the default 1858address class of ``DW_ADDR_none`` is used. 1859 1860See :ref:`amdgpu-dwarf-address-space-identifier` for information on the AMDGPU 1861mapping of DWARF address classes to DWARF address spaces, including address size 1862and NULL value. 1863 1864.. _amdgpu-dwarf-address-space-identifier: 1865 1866Address Space Identifier 1867------------------------ 1868 1869DWARF address spaces correspond to target architecture specific linear 1870addressable memory areas. See DWARF Version 5 section 2.12 and *DWARF Extensions 1871For Heterogeneous Debugging* section :ref:`amdgpu-dwarf-segment_addresses`. 1872 1873The DWARF address space mapping used for AMDGPU is defined in 1874:ref:`amdgpu-dwarf-address-space-mapping-table`. 1875 1876.. table:: AMDGPU DWARF Address Space Mapping 1877 :name: amdgpu-dwarf-address-space-mapping-table 1878 1879 ======================================= ===== ======= ======== ================= ======================= 1880 DWARF AMDGPU Notes 1881 --------------------------------------- ----- ---------------- ----------------- ----------------------- 1882 Address Space Name Value Address Bit Size Address Space 1883 --------------------------------------- ----- ------- -------- ----------------- ----------------------- 1884 .. 64-bit 32-bit 1885 process process 1886 address address 1887 space space 1888 ======================================= ===== ======= ======== ================= ======================= 1889 ``DW_ASPACE_none`` 0x00 64 32 Global *default address space* 1890 ``DW_ASPACE_AMDGPU_generic`` 0x01 64 32 Generic (Flat) 1891 ``DW_ASPACE_AMDGPU_region`` 0x02 32 32 Region (GDS) 1892 ``DW_ASPACE_AMDGPU_local`` 0x03 32 32 Local (group/LDS) 1893 *Reserved* 0x04 1894 ``DW_ASPACE_AMDGPU_private_lane`` 0x05 32 32 Private (Scratch) *focused lane* 1895 ``DW_ASPACE_AMDGPU_private_wave`` 0x06 32 32 Private (Scratch) *unswizzled wavefront* 1896 ======================================= ===== ======= ======== ================= ======================= 1897 1898See :ref:`amdgpu-address-spaces` for information on the AMDGPU address spaces 1899including address size and NULL value. 1900 1901The ``DW_ASPACE_none`` address space is the default target architecture address 1902space used in DWARF operations that do not specify an address space. It 1903therefore has to map to the global address space so that the ``DW_OP_addr*`` and 1904related operations can refer to addresses in the program code. 1905 1906The ``DW_ASPACE_AMDGPU_generic`` address space allows location expressions to 1907specify the flat address space. If the address corresponds to an address in the 1908local address space, then it corresponds to the wavefront that is executing the 1909focused thread of execution. If the address corresponds to an address in the 1910private address space, then it corresponds to the lane that is executing the 1911focused thread of execution for languages that are implemented using a SIMD or 1912SIMT execution model. 1913 1914.. note:: 1915 1916 CUDA-like languages such as HIP that do not have address spaces in the 1917 language type system, but do allow variables to be allocated in different 1918 address spaces, need to explicitly specify the ``DW_ASPACE_AMDGPU_generic`` 1919 address space in the DWARF expression operations as the default address space 1920 is the global address space. 1921 1922The ``DW_ASPACE_AMDGPU_local`` address space allows location expressions to 1923specify the local address space corresponding to the wavefront that is executing 1924the focused thread of execution. 1925 1926The ``DW_ASPACE_AMDGPU_private_lane`` address space allows location expressions 1927to specify the private address space corresponding to the lane that is executing 1928the focused thread of execution for languages that are implemented using a SIMD 1929or SIMT execution model. 1930 1931The ``DW_ASPACE_AMDGPU_private_wave`` address space allows location expressions 1932to specify the unswizzled private address space corresponding to the wavefront 1933that is executing the focused thread of execution. The wavefront view of private 1934memory is the per wavefront unswizzled backing memory layout defined in 1935:ref:`amdgpu-address-spaces`, such that address 0 corresponds to the first 1936location for the backing memory of the wavefront (namely the address is not 1937offset by ``wavefront-scratch-base``). The following formula can be used to 1938convert from a ``DW_ASPACE_AMDGPU_private_lane`` address to a 1939``DW_ASPACE_AMDGPU_private_wave`` address: 1940 1941:: 1942 1943 private-address-wavefront = 1944 ((private-address-lane / 4) * wavefront-size * 4) + 1945 (wavefront-lane-id * 4) + (private-address-lane % 4) 1946 1947If the ``DW_ASPACE_AMDGPU_private_lane`` address is dword aligned, and the start 1948of the dwords for each lane starting with lane 0 is required, then this 1949simplifies to: 1950 1951:: 1952 1953 private-address-wavefront = 1954 private-address-lane * wavefront-size 1955 1956A compiler can use the ``DW_ASPACE_AMDGPU_private_wave`` address space to read a 1957complete spilled vector register back into a complete vector register in the 1958CFI. The frame pointer can be a private lane address which is dword aligned, 1959which can be shifted to multiply by the wavefront size, and then used to form a 1960private wavefront address that gives a location for a contiguous set of dwords, 1961one per lane, where the vector register dwords are spilled. The compiler knows 1962the wavefront size since it generates the code. Note that the type of the 1963address may have to be converted as the size of a 1964``DW_ASPACE_AMDGPU_private_lane`` address may be smaller than the size of a 1965``DW_ASPACE_AMDGPU_private_wave`` address. 1966 1967.. _amdgpu-dwarf-lane-identifier: 1968 1969Lane identifier 1970--------------- 1971 1972DWARF lane identifies specify a target architecture lane position for hardware 1973that executes in a SIMD or SIMT manner, and on which a source language maps its 1974threads of execution onto those lanes. The DWARF lane identifier is pushed by 1975the ``DW_OP_LLVM_push_lane`` DWARF expression operation. See DWARF Version 5 1976section 2.5 which is updated by *DWARF Extensions For Heterogeneous Debugging* 1977section :ref:`amdgpu-dwarf-operation-expressions`. 1978 1979For AMDGPU, the lane identifier corresponds to the hardware lane ID of a 1980wavefront. It is numbered from 0 to the wavefront size minus 1. 1981 1982Operation Expressions 1983--------------------- 1984 1985DWARF expressions are used to compute program values and the locations of 1986program objects. See DWARF Version 5 section 2.5 and 1987:ref:`amdgpu-dwarf-operation-expressions`. 1988 1989DWARF location descriptions describe how to access storage which includes memory 1990and registers. When accessing storage on AMDGPU, bytes are ordered with least 1991significant bytes first, and bits are ordered within bytes with least 1992significant bits first. 1993 1994For AMDGPU CFI expressions, ``DW_OP_LLVM_select_bit_piece`` is used to describe 1995unwinding vector registers that are spilled under the execution mask to memory: 1996the zero-single location description is the vector register, and the one-single 1997location description is the spilled memory location description. The 1998``DW_OP_LLVM_form_aspace_address`` is used to specify the address space of the 1999memory location description. 2000 2001In AMDGPU expressions, ``DW_OP_LLVM_select_bit_piece`` is used by the 2002``DW_AT_LLVM_lane_pc`` attribute expression where divergent control flow is 2003controlled by the execution mask. An undefined location description together 2004with ``DW_OP_LLVM_extend`` is used to indicate the lane was not active on entry 2005to the subprogram. See :ref:`amdgpu-dwarf-dw-at-llvm-lane-pc` for an example. 2006 2007Debugger Information Entry Attributes 2008------------------------------------- 2009 2010This section describes how certain debugger information entry attributes are 2011used by AMDGPU. See the sections in DWARF Version 5 section 2 which are updated 2012by *DWARF Extensions For Heterogeneous Debugging* section 2013:ref:`amdgpu-dwarf-debugging-information-entry-attributes`. 2014 2015.. _amdgpu-dwarf-dw-at-llvm-lane-pc: 2016 2017``DW_AT_LLVM_lane_pc`` 2018~~~~~~~~~~~~~~~~~~~~~~ 2019 2020For AMDGPU, the ``DW_AT_LLVM_lane_pc`` attribute is used to specify the program 2021location of the separate lanes of a SIMT thread. 2022 2023If the lane is an active lane then this will be the same as the current program 2024location. 2025 2026If the lane is inactive, but was active on entry to the subprogram, then this is 2027the program location in the subprogram at which execution of the lane is 2028conceptual positioned. 2029 2030If the lane was not active on entry to the subprogram, then this will be the 2031undefined location. A client debugger can check if the lane is part of a valid 2032work-group by checking that the lane is in the range of the associated 2033work-group within the grid, accounting for partial work-groups. If it is not, 2034then the debugger can omit any information for the lane. Otherwise, the debugger 2035may repeatedly unwind the stack and inspect the ``DW_AT_LLVM_lane_pc`` of the 2036calling subprogram until it finds a non-undefined location. Conceptually the 2037lane only has the call frames that it has a non-undefined 2038``DW_AT_LLVM_lane_pc``. 2039 2040The following example illustrates how the AMDGPU backend can generate a DWARF 2041location list expression for the nested ``IF/THEN/ELSE`` structures of the 2042following subprogram pseudo code for a target with 64 lanes per wavefront. 2043 2044.. code:: 2045 :number-lines: 2046 2047 SUBPROGRAM X 2048 BEGIN 2049 a; 2050 IF (c1) THEN 2051 b; 2052 IF (c2) THEN 2053 c; 2054 ELSE 2055 d; 2056 ENDIF 2057 e; 2058 ELSE 2059 f; 2060 ENDIF 2061 g; 2062 END 2063 2064The AMDGPU backend may generate the following pseudo LLVM MIR to manipulate the 2065execution mask (``EXEC``) to linearize the control flow. The condition is 2066evaluated to make a mask of the lanes for which the condition evaluates to true. 2067First the ``THEN`` region is executed by setting the ``EXEC`` mask to the 2068logical ``AND`` of the current ``EXEC`` mask with the condition mask. Then the 2069``ELSE`` region is executed by negating the ``EXEC`` mask and logical ``AND`` of 2070the saved ``EXEC`` mask at the start of the region. After the ``IF/THEN/ELSE`` 2071region the ``EXEC`` mask is restored to the value it had at the beginning of the 2072region. This is shown below. Other approaches are possible, but the basic 2073concept is the same. 2074 2075.. code:: 2076 :number-lines: 2077 2078 $lex_start: 2079 a; 2080 %1 = EXEC 2081 %2 = c1 2082 $lex_1_start: 2083 EXEC = %1 & %2 2084 $if_1_then: 2085 b; 2086 %3 = EXEC 2087 %4 = c2 2088 $lex_1_1_start: 2089 EXEC = %3 & %4 2090 $lex_1_1_then: 2091 c; 2092 EXEC = ~EXEC & %3 2093 $lex_1_1_else: 2094 d; 2095 EXEC = %3 2096 $lex_1_1_end: 2097 e; 2098 EXEC = ~EXEC & %1 2099 $lex_1_else: 2100 f; 2101 EXEC = %1 2102 $lex_1_end: 2103 g; 2104 $lex_end: 2105 2106To create the DWARF location list expression that defines the location 2107description of a vector of lane program locations, the LLVM MIR ``DBG_VALUE`` 2108pseudo instruction can be used to annotate the linearized control flow. This can 2109be done by defining an artificial variable for the lane PC. The DWARF location 2110list expression created for it is used as the value of the 2111``DW_AT_LLVM_lane_pc`` attribute on the subprogram's debugger information entry. 2112 2113A DWARF procedure is defined for each well nested structured control flow region 2114which provides the conceptual lane program location for a lane if it is not 2115active (namely it is divergent). The DWARF operation expression for each region 2116conceptually inherits the value of the immediately enclosing region and modifies 2117it according to the semantics of the region. 2118 2119For an ``IF/THEN/ELSE`` region the divergent program location is at the start of 2120the region for the ``THEN`` region since it is executed first. For the ``ELSE`` 2121region the divergent program location is at the end of the ``IF/THEN/ELSE`` 2122region since the ``THEN`` region has completed. 2123 2124The lane PC artificial variable is assigned at each region transition. It uses 2125the immediately enclosing region's DWARF procedure to compute the program 2126location for each lane assuming they are divergent, and then modifies the result 2127by inserting the current program location for each lane that the ``EXEC`` mask 2128indicates is active. 2129 2130By having separate DWARF procedures for each region, they can be reused to 2131define the value for any nested region. This reduces the total size of the DWARF 2132operation expressions. 2133 2134The following provides an example using pseudo LLVM MIR. 2135 2136.. code:: 2137 :number-lines: 2138 2139 $lex_start: 2140 DEFINE_DWARF %__uint_64 = DW_TAG_base_type[ 2141 DW_AT_name = "__uint64"; 2142 DW_AT_byte_size = 8; 2143 DW_AT_encoding = DW_ATE_unsigned; 2144 ]; 2145 DEFINE_DWARF %__active_lane_pc = DW_TAG_dwarf_procedure[ 2146 DW_AT_name = "__active_lane_pc"; 2147 DW_AT_location = [ 2148 DW_OP_regx PC; 2149 DW_OP_LLVM_extend 64, 64; 2150 DW_OP_regval_type EXEC, %uint_64; 2151 DW_OP_LLVM_select_bit_piece 64, 64; 2152 ]; 2153 ]; 2154 DEFINE_DWARF %__divergent_lane_pc = DW_TAG_dwarf_procedure[ 2155 DW_AT_name = "__divergent_lane_pc"; 2156 DW_AT_location = [ 2157 DW_OP_LLVM_undefined; 2158 DW_OP_LLVM_extend 64, 64; 2159 ]; 2160 ]; 2161 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 2162 DW_OP_call_ref %__divergent_lane_pc; 2163 DW_OP_call_ref %__active_lane_pc; 2164 ]; 2165 a; 2166 %1 = EXEC; 2167 DBG_VALUE %1, $noreg, %__lex_1_save_exec; 2168 %2 = c1; 2169 $lex_1_start: 2170 EXEC = %1 & %2; 2171 $lex_1_then: 2172 DEFINE_DWARF %__divergent_lane_pc_1_then = DW_TAG_dwarf_procedure[ 2173 DW_AT_name = "__divergent_lane_pc_1_then"; 2174 DW_AT_location = DIExpression[ 2175 DW_OP_call_ref %__divergent_lane_pc; 2176 DW_OP_addrx &lex_1_start; 2177 DW_OP_stack_value; 2178 DW_OP_LLVM_extend 64, 64; 2179 DW_OP_call_ref %__lex_1_save_exec; 2180 DW_OP_deref_type 64, %__uint_64; 2181 DW_OP_LLVM_select_bit_piece 64, 64; 2182 ]; 2183 ]; 2184 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 2185 DW_OP_call_ref %__divergent_lane_pc_1_then; 2186 DW_OP_call_ref %__active_lane_pc; 2187 ]; 2188 b; 2189 %3 = EXEC; 2190 DBG_VALUE %3, %__lex_1_1_save_exec; 2191 %4 = c2; 2192 $lex_1_1_start: 2193 EXEC = %3 & %4; 2194 $lex_1_1_then: 2195 DEFINE_DWARF %__divergent_lane_pc_1_1_then = DW_TAG_dwarf_procedure[ 2196 DW_AT_name = "__divergent_lane_pc_1_1_then"; 2197 DW_AT_location = DIExpression[ 2198 DW_OP_call_ref %__divergent_lane_pc_1_then; 2199 DW_OP_addrx &lex_1_1_start; 2200 DW_OP_stack_value; 2201 DW_OP_LLVM_extend 64, 64; 2202 DW_OP_call_ref %__lex_1_1_save_exec; 2203 DW_OP_deref_type 64, %__uint_64; 2204 DW_OP_LLVM_select_bit_piece 64, 64; 2205 ]; 2206 ]; 2207 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 2208 DW_OP_call_ref %__divergent_lane_pc_1_1_then; 2209 DW_OP_call_ref %__active_lane_pc; 2210 ]; 2211 c; 2212 EXEC = ~EXEC & %3; 2213 $lex_1_1_else: 2214 DEFINE_DWARF %__divergent_lane_pc_1_1_else = DW_TAG_dwarf_procedure[ 2215 DW_AT_name = "__divergent_lane_pc_1_1_else"; 2216 DW_AT_location = DIExpression[ 2217 DW_OP_call_ref %__divergent_lane_pc_1_then; 2218 DW_OP_addrx &lex_1_1_end; 2219 DW_OP_stack_value; 2220 DW_OP_LLVM_extend 64, 64; 2221 DW_OP_call_ref %__lex_1_1_save_exec; 2222 DW_OP_deref_type 64, %__uint_64; 2223 DW_OP_LLVM_select_bit_piece 64, 64; 2224 ]; 2225 ]; 2226 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 2227 DW_OP_call_ref %__divergent_lane_pc_1_1_else; 2228 DW_OP_call_ref %__active_lane_pc; 2229 ]; 2230 d; 2231 EXEC = %3; 2232 $lex_1_1_end: 2233 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 2234 DW_OP_call_ref %__divergent_lane_pc; 2235 DW_OP_call_ref %__active_lane_pc; 2236 ]; 2237 e; 2238 EXEC = ~EXEC & %1; 2239 $lex_1_else: 2240 DEFINE_DWARF %__divergent_lane_pc_1_else = DW_TAG_dwarf_procedure[ 2241 DW_AT_name = "__divergent_lane_pc_1_else"; 2242 DW_AT_location = DIExpression[ 2243 DW_OP_call_ref %__divergent_lane_pc; 2244 DW_OP_addrx &lex_1_end; 2245 DW_OP_stack_value; 2246 DW_OP_LLVM_extend 64, 64; 2247 DW_OP_call_ref %__lex_1_save_exec; 2248 DW_OP_deref_type 64, %__uint_64; 2249 DW_OP_LLVM_select_bit_piece 64, 64; 2250 ]; 2251 ]; 2252 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 2253 DW_OP_call_ref %__divergent_lane_pc_1_else; 2254 DW_OP_call_ref %__active_lane_pc; 2255 ]; 2256 f; 2257 EXEC = %1; 2258 $lex_1_end: 2259 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc DIExpression[ 2260 DW_OP_call_ref %__divergent_lane_pc; 2261 DW_OP_call_ref %__active_lane_pc; 2262 ]; 2263 g; 2264 $lex_end: 2265 2266The DWARF procedure ``%__active_lane_pc`` is used to update the lane pc elements 2267that are active, with the current program location. 2268 2269Artificial variables %__lex_1_save_exec and %__lex_1_1_save_exec are created for 2270the execution masks saved on entry to a region. Using the ``DBG_VALUE`` pseudo 2271instruction, location list entries will be created that describe where the 2272artificial variables are allocated at any given program location. The compiler 2273may allocate them to registers or spill them to memory. 2274 2275The DWARF procedures for each region use the values of the saved execution mask 2276artificial variables to only update the lanes that are active on entry to the 2277region. All other lanes retain the value of the enclosing region where they were 2278last active. If they were not active on entry to the subprogram, then will have 2279the undefined location description. 2280 2281Other structured control flow regions can be handled similarly. For example, 2282loops would set the divergent program location for the region at the end of the 2283loop. Any lanes active will be in the loop, and any lanes not active must have 2284exited the loop. 2285 2286An ``IF/THEN/ELSEIF/ELSEIF/...`` region can be treated as a nest of 2287``IF/THEN/ELSE`` regions. 2288 2289The DWARF procedures can use the active lane artificial variable described in 2290:ref:`amdgpu-dwarf-amdgpu-dw-at-llvm-active-lane` rather than the actual 2291``EXEC`` mask in order to support whole or quad wavefront mode. 2292 2293.. _amdgpu-dwarf-amdgpu-dw-at-llvm-active-lane: 2294 2295``DW_AT_LLVM_active_lane`` 2296~~~~~~~~~~~~~~~~~~~~~~~~~~ 2297 2298The ``DW_AT_LLVM_active_lane`` attribute on a subprogram debugger information 2299entry is used to specify the lanes that are conceptually active for a SIMT 2300thread. 2301 2302The execution mask may be modified to implement whole or quad wavefront mode 2303operations. For example, all lanes may need to temporarily be made active to 2304execute a whole wavefront operation. Such regions would save the ``EXEC`` mask, 2305update it to enable the necessary lanes, perform the operations, and then 2306restore the ``EXEC`` mask from the saved value. While executing the whole 2307wavefront region, the conceptual execution mask is the saved value, not the 2308``EXEC`` value. 2309 2310This is handled by defining an artificial variable for the active lane mask. The 2311active lane mask artificial variable would be the actual ``EXEC`` mask for 2312normal regions, and the saved execution mask for regions where the mask is 2313temporarily updated. The location list expression created for this artificial 2314variable is used to define the value of the ``DW_AT_LLVM_active_lane`` 2315attribute. 2316 2317``DW_AT_LLVM_augmentation`` 2318~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2319 2320For AMDGPU, the ``DW_AT_LLVM_augmentation`` attribute of a compilation unit 2321debugger information entry has the following value for the augmentation string: 2322 2323:: 2324 2325 [amdgpu:v0.0] 2326 2327The "vX.Y" specifies the major X and minor Y version number of the AMDGPU 2328extensions used in the DWARF of the compilation unit. The version number 2329conforms to [SEMVER]_. 2330 2331Call Frame Information 2332---------------------- 2333 2334DWARF Call Frame Information (CFI) describes how a consumer can virtually 2335*unwind* call frames in a running process or core dump. See DWARF Version 5 2336section 6.4 and :ref:`amdgpu-dwarf-call-frame-information`. 2337 2338For AMDGPU, the Common Information Entry (CIE) fields have the following values: 2339 23401. ``augmentation`` string contains the following null-terminated UTF-8 string: 2341 2342 :: 2343 2344 [amd:v0.0] 2345 2346 The ``vX.Y`` specifies the major X and minor Y version number of the AMDGPU 2347 extensions used in this CIE or to the FDEs that use it. The version number 2348 conforms to [SEMVER]_. 2349 23502. ``address_size`` for the ``Global`` address space is defined in 2351 :ref:`amdgpu-dwarf-address-space-identifier`. 2352 23533. ``segment_selector_size`` is 0 as AMDGPU does not use a segment selector. 2354 23554. ``code_alignment_factor`` is 4 bytes. 2356 2357 .. TODO:: 2358 2359 Add to :ref:`amdgpu-processor-table` table. 2360 23615. ``data_alignment_factor`` is 4 bytes. 2362 2363 .. TODO:: 2364 2365 Add to :ref:`amdgpu-processor-table` table. 2366 23676. ``return_address_register`` is ``PC_32`` for 32-bit processes and ``PC_64`` 2368 for 64-bit processes defined in :ref:`amdgpu-dwarf-register-identifier`. 2369 23707. ``initial_instructions`` Since a subprogram X with fewer registers can be 2371 called from subprogram Y that has more allocated, X will not change any of 2372 the extra registers as it cannot access them. Therefore, the default rule 2373 for all columns is ``same value``. 2374 2375For AMDGPU the register number follows the numbering defined in 2376:ref:`amdgpu-dwarf-register-identifier`. 2377 2378For AMDGPU the instructions are variable size. A consumer can subtract 1 from 2379the return address to get the address of a byte within the call site 2380instructions. See DWARF Version 5 section 6.4.4. 2381 2382Accelerated Access 2383------------------ 2384 2385See DWARF Version 5 section 6.1. 2386 2387Lookup By Name Section Header 2388~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2389 2390See DWARF Version 5 section 6.1.1.4.1 and :ref:`amdgpu-dwarf-lookup-by-name`. 2391 2392For AMDGPU the lookup by name section header table: 2393 2394``augmentation_string_size`` (uword) 2395 2396 Set to the length of the ``augmentation_string`` value which is always a 2397 multiple of 4. 2398 2399``augmentation_string`` (sequence of UTF-8 characters) 2400 2401 Contains the following UTF-8 string null padded to a multiple of 4 bytes: 2402 2403 :: 2404 2405 [amdgpu:v0.0] 2406 2407 The "vX.Y" specifies the major X and minor Y version number of the AMDGPU 2408 extensions used in the DWARF of this index. The version number conforms to 2409 [SEMVER]_. 2410 2411 .. note:: 2412 2413 This is different to the DWARF Version 5 definition that requires the first 2414 4 characters to be the vendor ID. But this is consistent with the other 2415 augmentation strings and does allow multiple vendor contributions. However, 2416 backwards compatibility may be more desirable. 2417 2418Lookup By Address Section Header 2419~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2420 2421See DWARF Version 5 section 6.1.2. 2422 2423For AMDGPU the lookup by address section header table: 2424 2425``address_size`` (ubyte) 2426 2427 Match the address size for the ``Global`` address space defined in 2428 :ref:`amdgpu-dwarf-address-space-identifier`. 2429 2430``segment_selector_size`` (ubyte) 2431 2432 AMDGPU does not use a segment selector so this is 0. The entries in the 2433 ``.debug_aranges`` do not have a segment selector. 2434 2435Line Number Information 2436----------------------- 2437 2438See DWARF Version 5 section 6.2 and :ref:`amdgpu-dwarf-line-number-information`. 2439 2440AMDGPU does not use the ``isa`` state machine registers and always sets it to 0. 2441The instruction set must be obtained from the ELF file header ``e_flags`` field 2442in the ``EF_AMDGPU_MACH`` bit position (see :ref:`ELF Header 2443<amdgpu-elf-header>`). See DWARF Version 5 section 6.2.2. 2444 2445.. TODO:: 2446 2447 Should the ``isa`` state machine register be used to indicate if the code is 2448 in wavefront32 or wavefront64 mode? Or used to specify the architecture ISA? 2449 2450For AMDGPU the line number program header fields have the following values (see 2451DWARF Version 5 section 6.2.4): 2452 2453``address_size`` (ubyte) 2454 Matches the address size for the ``Global`` address space defined in 2455 :ref:`amdgpu-dwarf-address-space-identifier`. 2456 2457``segment_selector_size`` (ubyte) 2458 AMDGPU does not use a segment selector so this is 0. 2459 2460``minimum_instruction_length`` (ubyte) 2461 For GFX9-GFX10 this is 4. 2462 2463``maximum_operations_per_instruction`` (ubyte) 2464 For GFX9-GFX10 this is 1. 2465 2466Source text for online-compiled programs (for example, those compiled by the 2467OpenCL language runtime) may be embedded into the DWARF Version 5 line table. 2468See DWARF Version 5 section 6.2.4.1 which is updated by *DWARF Extensions For 2469Heterogeneous Debugging* section :ref:`DW_LNCT_LLVM_source 2470<amdgpu-dwarf-line-number-information-dw-lnct-llvm-source>`. 2471 2472The Clang option used to control source embedding in AMDGPU is defined in 2473:ref:`amdgpu-clang-debug-options-table`. 2474 2475 .. table:: AMDGPU Clang Debug Options 2476 :name: amdgpu-clang-debug-options-table 2477 2478 ==================== ================================================== 2479 Debug Flag Description 2480 ==================== ================================================== 2481 -g[no-]embed-source Enable/disable embedding source text in DWARF 2482 debug sections. Useful for environments where 2483 source cannot be written to disk, such as 2484 when performing online compilation. 2485 ==================== ================================================== 2486 2487For example: 2488 2489``-gembed-source`` 2490 Enable the embedded source. 2491 2492``-gno-embed-source`` 2493 Disable the embedded source. 2494 249532-Bit and 64-Bit DWARF Formats 2496------------------------------- 2497 2498See DWARF Version 5 section 7.4 and 2499:ref:`amdgpu-dwarf-32-bit-and-64-bit-dwarf-formats`. 2500 2501For AMDGPU: 2502 2503* For the ``amdgcn`` target architecture only the 64-bit process address space 2504 is supported. 2505 2506* The producer can generate either 32-bit or 64-bit DWARF format. LLVM generates 2507 the 32-bit DWARF format. 2508 2509Unit Headers 2510------------ 2511 2512For AMDGPU the following values apply for each of the unit headers described in 2513DWARF Version 5 sections 7.5.1.1, 7.5.1.2, and 7.5.1.3: 2514 2515``address_size`` (ubyte) 2516 Matches the address size for the ``Global`` address space defined in 2517 :ref:`amdgpu-dwarf-address-space-identifier`. 2518 2519.. _amdgpu-code-conventions: 2520 2521Code Conventions 2522================ 2523 2524This section provides code conventions used for each supported target triple OS 2525(see :ref:`amdgpu-target-triples`). 2526 2527AMDHSA 2528------ 2529 2530This section provides code conventions used when the target triple OS is 2531``amdhsa`` (see :ref:`amdgpu-target-triples`). 2532 2533.. _amdgpu-amdhsa-code-object-metadata: 2534 2535Code Object Metadata 2536~~~~~~~~~~~~~~~~~~~~ 2537 2538The code object metadata specifies extensible metadata associated with the code 2539objects executed on HSA [HSA]_ compatible runtimes (see :ref:`amdgpu-os`). The 2540encoding and semantics of this metadata depends on the code object version; see 2541:ref:`amdgpu-amdhsa-code-object-metadata-v2`, 2542:ref:`amdgpu-amdhsa-code-object-metadata-v3`, and 2543:ref:`amdgpu-amdhsa-code-object-metadata-v4`. 2544 2545Code object metadata is specified in a note record (see 2546:ref:`amdgpu-note-records`) and is required when the target triple OS is 2547``amdhsa`` (see :ref:`amdgpu-target-triples`). It must contain the minimum 2548information necessary to support the HSA compatible runtime kernel queries. For 2549example, the segment sizes needed in a dispatch packet. In addition, a 2550high-level language runtime may require other information to be included. For 2551example, the AMD OpenCL runtime records kernel argument information. 2552 2553.. _amdgpu-amdhsa-code-object-metadata-v2: 2554 2555Code Object V2 Metadata 2556+++++++++++++++++++++++ 2557 2558.. warning:: 2559 Code object V2 is not the default code object version emitted by this version 2560 of LLVM. 2561 2562Code object V2 metadata is specified by the ``NT_AMD_HSA_METADATA`` note record 2563(see :ref:`amdgpu-note-records-v2`). 2564 2565The metadata is specified as a YAML formatted string (see [YAML]_ and 2566:doc:`YamlIO`). 2567 2568.. TODO:: 2569 2570 Is the string null terminated? It probably should not if YAML allows it to 2571 contain null characters, otherwise it should be. 2572 2573The metadata is represented as a single YAML document comprised of the mapping 2574defined in table :ref:`amdgpu-amdhsa-code-object-metadata-map-v2-table` and 2575referenced tables. 2576 2577For boolean values, the string values of ``false`` and ``true`` are used for 2578false and true respectively. 2579 2580Additional information can be added to the mappings. To avoid conflicts, any 2581non-AMD key names should be prefixed by "*vendor-name*.". 2582 2583 .. table:: AMDHSA Code Object V2 Metadata Map 2584 :name: amdgpu-amdhsa-code-object-metadata-map-v2-table 2585 2586 ========== ============== ========= ======================================= 2587 String Key Value Type Required? Description 2588 ========== ============== ========= ======================================= 2589 "Version" sequence of Required - The first integer is the major 2590 2 integers version. Currently 1. 2591 - The second integer is the minor 2592 version. Currently 0. 2593 "Printf" sequence of Each string is encoded information 2594 strings about a printf function call. The 2595 encoded information is organized as 2596 fields separated by colon (':'): 2597 2598 ``ID:N:S[0]:S[1]:...:S[N-1]:FormatString`` 2599 2600 where: 2601 2602 ``ID`` 2603 A 32-bit integer as a unique id for 2604 each printf function call 2605 2606 ``N`` 2607 A 32-bit integer equal to the number 2608 of arguments of printf function call 2609 minus 1 2610 2611 ``S[i]`` (where i = 0, 1, ... , N-1) 2612 32-bit integers for the size in bytes 2613 of the i-th FormatString argument of 2614 the printf function call 2615 2616 FormatString 2617 The format string passed to the 2618 printf function call. 2619 "Kernels" sequence of Required Sequence of the mappings for each 2620 mapping kernel in the code object. See 2621 :ref:`amdgpu-amdhsa-code-object-kernel-metadata-map-v2-table` 2622 for the definition of the mapping. 2623 ========== ============== ========= ======================================= 2624 2625.. 2626 2627 .. table:: AMDHSA Code Object V2 Kernel Metadata Map 2628 :name: amdgpu-amdhsa-code-object-kernel-metadata-map-v2-table 2629 2630 ================= ============== ========= ================================ 2631 String Key Value Type Required? Description 2632 ================= ============== ========= ================================ 2633 "Name" string Required Source name of the kernel. 2634 "SymbolName" string Required Name of the kernel 2635 descriptor ELF symbol. 2636 "Language" string Source language of the kernel. 2637 Values include: 2638 2639 - "OpenCL C" 2640 - "OpenCL C++" 2641 - "HCC" 2642 - "OpenMP" 2643 2644 "LanguageVersion" sequence of - The first integer is the major 2645 2 integers version. 2646 - The second integer is the 2647 minor version. 2648 "Attrs" mapping Mapping of kernel attributes. 2649 See 2650 :ref:`amdgpu-amdhsa-code-object-kernel-attribute-metadata-map-v2-table` 2651 for the mapping definition. 2652 "Args" sequence of Sequence of mappings of the 2653 mapping kernel arguments. See 2654 :ref:`amdgpu-amdhsa-code-object-kernel-argument-metadata-map-v2-table` 2655 for the definition of the mapping. 2656 "CodeProps" mapping Mapping of properties related to 2657 the kernel code. See 2658 :ref:`amdgpu-amdhsa-code-object-kernel-code-properties-metadata-map-v2-table` 2659 for the mapping definition. 2660 ================= ============== ========= ================================ 2661 2662.. 2663 2664 .. table:: AMDHSA Code Object V2 Kernel Attribute Metadata Map 2665 :name: amdgpu-amdhsa-code-object-kernel-attribute-metadata-map-v2-table 2666 2667 =================== ============== ========= ============================== 2668 String Key Value Type Required? Description 2669 =================== ============== ========= ============================== 2670 "ReqdWorkGroupSize" sequence of If not 0, 0, 0 then all values 2671 3 integers must be >=1 and the dispatch 2672 work-group size X, Y, Z must 2673 correspond to the specified 2674 values. Defaults to 0, 0, 0. 2675 2676 Corresponds to the OpenCL 2677 ``reqd_work_group_size`` 2678 attribute. 2679 "WorkGroupSizeHint" sequence of The dispatch work-group size 2680 3 integers X, Y, Z is likely to be the 2681 specified values. 2682 2683 Corresponds to the OpenCL 2684 ``work_group_size_hint`` 2685 attribute. 2686 "VecTypeHint" string The name of a scalar or vector 2687 type. 2688 2689 Corresponds to the OpenCL 2690 ``vec_type_hint`` attribute. 2691 2692 "RuntimeHandle" string The external symbol name 2693 associated with a kernel. 2694 OpenCL runtime allocates a 2695 global buffer for the symbol 2696 and saves the kernel's address 2697 to it, which is used for 2698 device side enqueueing. Only 2699 available for device side 2700 enqueued kernels. 2701 =================== ============== ========= ============================== 2702 2703.. 2704 2705 .. table:: AMDHSA Code Object V2 Kernel Argument Metadata Map 2706 :name: amdgpu-amdhsa-code-object-kernel-argument-metadata-map-v2-table 2707 2708 ================= ============== ========= ================================ 2709 String Key Value Type Required? Description 2710 ================= ============== ========= ================================ 2711 "Name" string Kernel argument name. 2712 "TypeName" string Kernel argument type name. 2713 "Size" integer Required Kernel argument size in bytes. 2714 "Align" integer Required Kernel argument alignment in 2715 bytes. Must be a power of two. 2716 "ValueKind" string Required Kernel argument kind that 2717 specifies how to set up the 2718 corresponding argument. 2719 Values include: 2720 2721 "ByValue" 2722 The argument is copied 2723 directly into the kernarg. 2724 2725 "GlobalBuffer" 2726 A global address space pointer 2727 to the buffer data is passed 2728 in the kernarg. 2729 2730 "DynamicSharedPointer" 2731 A group address space pointer 2732 to dynamically allocated LDS 2733 is passed in the kernarg. 2734 2735 "Sampler" 2736 A global address space 2737 pointer to a S# is passed in 2738 the kernarg. 2739 2740 "Image" 2741 A global address space 2742 pointer to a T# is passed in 2743 the kernarg. 2744 2745 "Pipe" 2746 A global address space pointer 2747 to an OpenCL pipe is passed in 2748 the kernarg. 2749 2750 "Queue" 2751 A global address space pointer 2752 to an OpenCL device enqueue 2753 queue is passed in the 2754 kernarg. 2755 2756 "HiddenGlobalOffsetX" 2757 The OpenCL grid dispatch 2758 global offset for the X 2759 dimension is passed in the 2760 kernarg. 2761 2762 "HiddenGlobalOffsetY" 2763 The OpenCL grid dispatch 2764 global offset for the Y 2765 dimension is passed in the 2766 kernarg. 2767 2768 "HiddenGlobalOffsetZ" 2769 The OpenCL grid dispatch 2770 global offset for the Z 2771 dimension is passed in the 2772 kernarg. 2773 2774 "HiddenNone" 2775 An argument that is not used 2776 by the kernel. Space needs to 2777 be left for it, but it does 2778 not need to be set up. 2779 2780 "HiddenPrintfBuffer" 2781 A global address space pointer 2782 to the runtime printf buffer 2783 is passed in kernarg. 2784 2785 "HiddenHostcallBuffer" 2786 A global address space pointer 2787 to the runtime hostcall buffer 2788 is passed in kernarg. 2789 2790 "HiddenDefaultQueue" 2791 A global address space pointer 2792 to the OpenCL device enqueue 2793 queue that should be used by 2794 the kernel by default is 2795 passed in the kernarg. 2796 2797 "HiddenCompletionAction" 2798 A global address space pointer 2799 to help link enqueued kernels into 2800 the ancestor tree for determining 2801 when the parent kernel has finished. 2802 2803 "HiddenMultiGridSyncArg" 2804 A global address space pointer for 2805 multi-grid synchronization is 2806 passed in the kernarg. 2807 2808 "ValueType" string Unused and deprecated. This should no longer 2809 be emitted, but is accepted for compatibility. 2810 2811 2812 "PointeeAlign" integer Alignment in bytes of pointee 2813 type for pointer type kernel 2814 argument. Must be a power 2815 of 2. Only present if 2816 "ValueKind" is 2817 "DynamicSharedPointer". 2818 "AddrSpaceQual" string Kernel argument address space 2819 qualifier. Only present if 2820 "ValueKind" is "GlobalBuffer" or 2821 "DynamicSharedPointer". Values 2822 are: 2823 2824 - "Private" 2825 - "Global" 2826 - "Constant" 2827 - "Local" 2828 - "Generic" 2829 - "Region" 2830 2831 .. TODO:: 2832 2833 Is GlobalBuffer only Global 2834 or Constant? Is 2835 DynamicSharedPointer always 2836 Local? Can HCC allow Generic? 2837 How can Private or Region 2838 ever happen? 2839 2840 "AccQual" string Kernel argument access 2841 qualifier. Only present if 2842 "ValueKind" is "Image" or 2843 "Pipe". Values 2844 are: 2845 2846 - "ReadOnly" 2847 - "WriteOnly" 2848 - "ReadWrite" 2849 2850 .. TODO:: 2851 2852 Does this apply to 2853 GlobalBuffer? 2854 2855 "ActualAccQual" string The actual memory accesses 2856 performed by the kernel on the 2857 kernel argument. Only present if 2858 "ValueKind" is "GlobalBuffer", 2859 "Image", or "Pipe". This may be 2860 more restrictive than indicated 2861 by "AccQual" to reflect what the 2862 kernel actual does. If not 2863 present then the runtime must 2864 assume what is implied by 2865 "AccQual" and "IsConst". Values 2866 are: 2867 2868 - "ReadOnly" 2869 - "WriteOnly" 2870 - "ReadWrite" 2871 2872 "IsConst" boolean Indicates if the kernel argument 2873 is const qualified. Only present 2874 if "ValueKind" is 2875 "GlobalBuffer". 2876 2877 "IsRestrict" boolean Indicates if the kernel argument 2878 is restrict qualified. Only 2879 present if "ValueKind" is 2880 "GlobalBuffer". 2881 2882 "IsVolatile" boolean Indicates if the kernel argument 2883 is volatile qualified. Only 2884 present if "ValueKind" is 2885 "GlobalBuffer". 2886 2887 "IsPipe" boolean Indicates if the kernel argument 2888 is pipe qualified. Only present 2889 if "ValueKind" is "Pipe". 2890 2891 .. TODO:: 2892 2893 Can GlobalBuffer be pipe 2894 qualified? 2895 2896 ================= ============== ========= ================================ 2897 2898.. 2899 2900 .. table:: AMDHSA Code Object V2 Kernel Code Properties Metadata Map 2901 :name: amdgpu-amdhsa-code-object-kernel-code-properties-metadata-map-v2-table 2902 2903 ============================ ============== ========= ===================== 2904 String Key Value Type Required? Description 2905 ============================ ============== ========= ===================== 2906 "KernargSegmentSize" integer Required The size in bytes of 2907 the kernarg segment 2908 that holds the values 2909 of the arguments to 2910 the kernel. 2911 "GroupSegmentFixedSize" integer Required The amount of group 2912 segment memory 2913 required by a 2914 work-group in 2915 bytes. This does not 2916 include any 2917 dynamically allocated 2918 group segment memory 2919 that may be added 2920 when the kernel is 2921 dispatched. 2922 "PrivateSegmentFixedSize" integer Required The amount of fixed 2923 private address space 2924 memory required for a 2925 work-item in 2926 bytes. If the kernel 2927 uses a dynamic call 2928 stack then additional 2929 space must be added 2930 to this value for the 2931 call stack. 2932 "KernargSegmentAlign" integer Required The maximum byte 2933 alignment of 2934 arguments in the 2935 kernarg segment. Must 2936 be a power of 2. 2937 "WavefrontSize" integer Required Wavefront size. Must 2938 be a power of 2. 2939 "NumSGPRs" integer Required Number of scalar 2940 registers used by a 2941 wavefront for 2942 GFX6-GFX10. This 2943 includes the special 2944 SGPRs for VCC, Flat 2945 Scratch (GFX7-GFX10) 2946 and XNACK (for 2947 GFX8-GFX10). It does 2948 not include the 16 2949 SGPR added if a trap 2950 handler is 2951 enabled. It is not 2952 rounded up to the 2953 allocation 2954 granularity. 2955 "NumVGPRs" integer Required Number of vector 2956 registers used by 2957 each work-item for 2958 GFX6-GFX10 2959 "MaxFlatWorkGroupSize" integer Required Maximum flat 2960 work-group size 2961 supported by the 2962 kernel in work-items. 2963 Must be >=1 and 2964 consistent with 2965 ReqdWorkGroupSize if 2966 not 0, 0, 0. 2967 "NumSpilledSGPRs" integer Number of stores from 2968 a scalar register to 2969 a register allocator 2970 created spill 2971 location. 2972 "NumSpilledVGPRs" integer Number of stores from 2973 a vector register to 2974 a register allocator 2975 created spill 2976 location. 2977 ============================ ============== ========= ===================== 2978 2979.. _amdgpu-amdhsa-code-object-metadata-v3: 2980 2981Code Object V3 Metadata 2982+++++++++++++++++++++++ 2983 2984Code object V3 to V4 metadata is specified by the ``NT_AMDGPU_METADATA`` note 2985record (see :ref:`amdgpu-note-records-v3-v4`). 2986 2987The metadata is represented as Message Pack formatted binary data (see 2988[MsgPack]_). The top level is a Message Pack map that includes the 2989keys defined in table 2990:ref:`amdgpu-amdhsa-code-object-metadata-map-table-v3` and referenced 2991tables. 2992 2993Additional information can be added to the maps. To avoid conflicts, 2994any key names should be prefixed by "*vendor-name*." where 2995``vendor-name`` can be the name of the vendor and specific vendor 2996tool that generates the information. The prefix is abbreviated to 2997simply "." when it appears within a map that has been added by the 2998same *vendor-name*. 2999 3000 .. table:: AMDHSA Code Object V3 Metadata Map 3001 :name: amdgpu-amdhsa-code-object-metadata-map-table-v3 3002 3003 ================= ============== ========= ======================================= 3004 String Key Value Type Required? Description 3005 ================= ============== ========= ======================================= 3006 "amdhsa.version" sequence of Required - The first integer is the major 3007 2 integers version. Currently 1. 3008 - The second integer is the minor 3009 version. Currently 0. 3010 "amdhsa.printf" sequence of Each string is encoded information 3011 strings about a printf function call. The 3012 encoded information is organized as 3013 fields separated by colon (':'): 3014 3015 ``ID:N:S[0]:S[1]:...:S[N-1]:FormatString`` 3016 3017 where: 3018 3019 ``ID`` 3020 A 32-bit integer as a unique id for 3021 each printf function call 3022 3023 ``N`` 3024 A 32-bit integer equal to the number 3025 of arguments of printf function call 3026 minus 1 3027 3028 ``S[i]`` (where i = 0, 1, ... , N-1) 3029 32-bit integers for the size in bytes 3030 of the i-th FormatString argument of 3031 the printf function call 3032 3033 FormatString 3034 The format string passed to the 3035 printf function call. 3036 "amdhsa.kernels" sequence of Required Sequence of the maps for each 3037 map kernel in the code object. See 3038 :ref:`amdgpu-amdhsa-code-object-kernel-metadata-map-table-v3` 3039 for the definition of the keys included 3040 in that map. 3041 ================= ============== ========= ======================================= 3042 3043.. 3044 3045 .. table:: AMDHSA Code Object V3 Kernel Metadata Map 3046 :name: amdgpu-amdhsa-code-object-kernel-metadata-map-table-v3 3047 3048 =================================== ============== ========= ================================ 3049 String Key Value Type Required? Description 3050 =================================== ============== ========= ================================ 3051 ".name" string Required Source name of the kernel. 3052 ".symbol" string Required Name of the kernel 3053 descriptor ELF symbol. 3054 ".language" string Source language of the kernel. 3055 Values include: 3056 3057 - "OpenCL C" 3058 - "OpenCL C++" 3059 - "HCC" 3060 - "HIP" 3061 - "OpenMP" 3062 - "Assembler" 3063 3064 ".language_version" sequence of - The first integer is the major 3065 2 integers version. 3066 - The second integer is the 3067 minor version. 3068 ".args" sequence of Sequence of maps of the 3069 map kernel arguments. See 3070 :ref:`amdgpu-amdhsa-code-object-kernel-argument-metadata-map-table-v3` 3071 for the definition of the keys 3072 included in that map. 3073 ".reqd_workgroup_size" sequence of If not 0, 0, 0 then all values 3074 3 integers must be >=1 and the dispatch 3075 work-group size X, Y, Z must 3076 correspond to the specified 3077 values. Defaults to 0, 0, 0. 3078 3079 Corresponds to the OpenCL 3080 ``reqd_work_group_size`` 3081 attribute. 3082 ".workgroup_size_hint" sequence of The dispatch work-group size 3083 3 integers X, Y, Z is likely to be the 3084 specified values. 3085 3086 Corresponds to the OpenCL 3087 ``work_group_size_hint`` 3088 attribute. 3089 ".vec_type_hint" string The name of a scalar or vector 3090 type. 3091 3092 Corresponds to the OpenCL 3093 ``vec_type_hint`` attribute. 3094 3095 ".device_enqueue_symbol" string The external symbol name 3096 associated with a kernel. 3097 OpenCL runtime allocates a 3098 global buffer for the symbol 3099 and saves the kernel's address 3100 to it, which is used for 3101 device side enqueueing. Only 3102 available for device side 3103 enqueued kernels. 3104 ".kernarg_segment_size" integer Required The size in bytes of 3105 the kernarg segment 3106 that holds the values 3107 of the arguments to 3108 the kernel. 3109 ".group_segment_fixed_size" integer Required The amount of group 3110 segment memory 3111 required by a 3112 work-group in 3113 bytes. This does not 3114 include any 3115 dynamically allocated 3116 group segment memory 3117 that may be added 3118 when the kernel is 3119 dispatched. 3120 ".private_segment_fixed_size" integer Required The amount of fixed 3121 private address space 3122 memory required for a 3123 work-item in 3124 bytes. If the kernel 3125 uses a dynamic call 3126 stack then additional 3127 space must be added 3128 to this value for the 3129 call stack. 3130 ".kernarg_segment_align" integer Required The maximum byte 3131 alignment of 3132 arguments in the 3133 kernarg segment. Must 3134 be a power of 2. 3135 ".wavefront_size" integer Required Wavefront size. Must 3136 be a power of 2. 3137 ".sgpr_count" integer Required Number of scalar 3138 registers required by a 3139 wavefront for 3140 GFX6-GFX9. A register 3141 is required if it is 3142 used explicitly, or 3143 if a higher numbered 3144 register is used 3145 explicitly. This 3146 includes the special 3147 SGPRs for VCC, Flat 3148 Scratch (GFX7-GFX9) 3149 and XNACK (for 3150 GFX8-GFX9). It does 3151 not include the 16 3152 SGPR added if a trap 3153 handler is 3154 enabled. It is not 3155 rounded up to the 3156 allocation 3157 granularity. 3158 ".vgpr_count" integer Required Number of vector 3159 registers required by 3160 each work-item for 3161 GFX6-GFX9. A register 3162 is required if it is 3163 used explicitly, or 3164 if a higher numbered 3165 register is used 3166 explicitly. 3167 ".max_flat_workgroup_size" integer Required Maximum flat 3168 work-group size 3169 supported by the 3170 kernel in work-items. 3171 Must be >=1 and 3172 consistent with 3173 ReqdWorkGroupSize if 3174 not 0, 0, 0. 3175 ".sgpr_spill_count" integer Number of stores from 3176 a scalar register to 3177 a register allocator 3178 created spill 3179 location. 3180 ".vgpr_spill_count" integer Number of stores from 3181 a vector register to 3182 a register allocator 3183 created spill 3184 location. 3185 ".kind" string The kind of the kernel 3186 with the following 3187 values: 3188 3189 "normal" 3190 Regular kernels. 3191 3192 "init" 3193 These kernels must be 3194 invoked after loading 3195 the containing code 3196 object and must 3197 complete before any 3198 normal and fini 3199 kernels in the same 3200 code object are 3201 invoked. 3202 3203 "fini" 3204 These kernels must be 3205 invoked before 3206 unloading the 3207 containing code object 3208 and after all init and 3209 normal kernels in the 3210 same code object have 3211 been invoked and 3212 completed. 3213 3214 If omitted, "normal" is 3215 assumed. 3216 =================================== ============== ========= ================================ 3217 3218.. 3219 3220 .. table:: AMDHSA Code Object V3 Kernel Argument Metadata Map 3221 :name: amdgpu-amdhsa-code-object-kernel-argument-metadata-map-table-v3 3222 3223 ====================== ============== ========= ================================ 3224 String Key Value Type Required? Description 3225 ====================== ============== ========= ================================ 3226 ".name" string Kernel argument name. 3227 ".type_name" string Kernel argument type name. 3228 ".size" integer Required Kernel argument size in bytes. 3229 ".offset" integer Required Kernel argument offset in 3230 bytes. The offset must be a 3231 multiple of the alignment 3232 required by the argument. 3233 ".value_kind" string Required Kernel argument kind that 3234 specifies how to set up the 3235 corresponding argument. 3236 Values include: 3237 3238 "by_value" 3239 The argument is copied 3240 directly into the kernarg. 3241 3242 "global_buffer" 3243 A global address space pointer 3244 to the buffer data is passed 3245 in the kernarg. 3246 3247 "dynamic_shared_pointer" 3248 A group address space pointer 3249 to dynamically allocated LDS 3250 is passed in the kernarg. 3251 3252 "sampler" 3253 A global address space 3254 pointer to a S# is passed in 3255 the kernarg. 3256 3257 "image" 3258 A global address space 3259 pointer to a T# is passed in 3260 the kernarg. 3261 3262 "pipe" 3263 A global address space pointer 3264 to an OpenCL pipe is passed in 3265 the kernarg. 3266 3267 "queue" 3268 A global address space pointer 3269 to an OpenCL device enqueue 3270 queue is passed in the 3271 kernarg. 3272 3273 "hidden_global_offset_x" 3274 The OpenCL grid dispatch 3275 global offset for the X 3276 dimension is passed in the 3277 kernarg. 3278 3279 "hidden_global_offset_y" 3280 The OpenCL grid dispatch 3281 global offset for the Y 3282 dimension is passed in the 3283 kernarg. 3284 3285 "hidden_global_offset_z" 3286 The OpenCL grid dispatch 3287 global offset for the Z 3288 dimension is passed in the 3289 kernarg. 3290 3291 "hidden_none" 3292 An argument that is not used 3293 by the kernel. Space needs to 3294 be left for it, but it does 3295 not need to be set up. 3296 3297 "hidden_printf_buffer" 3298 A global address space pointer 3299 to the runtime printf buffer 3300 is passed in kernarg. 3301 3302 "hidden_hostcall_buffer" 3303 A global address space pointer 3304 to the runtime hostcall buffer 3305 is passed in kernarg. 3306 3307 "hidden_default_queue" 3308 A global address space pointer 3309 to the OpenCL device enqueue 3310 queue that should be used by 3311 the kernel by default is 3312 passed in the kernarg. 3313 3314 "hidden_completion_action" 3315 A global address space pointer 3316 to help link enqueued kernels into 3317 the ancestor tree for determining 3318 when the parent kernel has finished. 3319 3320 "hidden_multigrid_sync_arg" 3321 A global address space pointer for 3322 multi-grid synchronization is 3323 passed in the kernarg. 3324 3325 ".value_type" string Unused and deprecated. This should no longer 3326 be emitted, but is accepted for compatibility. 3327 3328 ".pointee_align" integer Alignment in bytes of pointee 3329 type for pointer type kernel 3330 argument. Must be a power 3331 of 2. Only present if 3332 ".value_kind" is 3333 "dynamic_shared_pointer". 3334 ".address_space" string Kernel argument address space 3335 qualifier. Only present if 3336 ".value_kind" is "global_buffer" or 3337 "dynamic_shared_pointer". Values 3338 are: 3339 3340 - "private" 3341 - "global" 3342 - "constant" 3343 - "local" 3344 - "generic" 3345 - "region" 3346 3347 .. TODO:: 3348 3349 Is "global_buffer" only "global" 3350 or "constant"? Is 3351 "dynamic_shared_pointer" always 3352 "local"? Can HCC allow "generic"? 3353 How can "private" or "region" 3354 ever happen? 3355 3356 ".access" string Kernel argument access 3357 qualifier. Only present if 3358 ".value_kind" is "image" or 3359 "pipe". Values 3360 are: 3361 3362 - "read_only" 3363 - "write_only" 3364 - "read_write" 3365 3366 .. TODO:: 3367 3368 Does this apply to 3369 "global_buffer"? 3370 3371 ".actual_access" string The actual memory accesses 3372 performed by the kernel on the 3373 kernel argument. Only present if 3374 ".value_kind" is "global_buffer", 3375 "image", or "pipe". This may be 3376 more restrictive than indicated 3377 by ".access" to reflect what the 3378 kernel actual does. If not 3379 present then the runtime must 3380 assume what is implied by 3381 ".access" and ".is_const" . Values 3382 are: 3383 3384 - "read_only" 3385 - "write_only" 3386 - "read_write" 3387 3388 ".is_const" boolean Indicates if the kernel argument 3389 is const qualified. Only present 3390 if ".value_kind" is 3391 "global_buffer". 3392 3393 ".is_restrict" boolean Indicates if the kernel argument 3394 is restrict qualified. Only 3395 present if ".value_kind" is 3396 "global_buffer". 3397 3398 ".is_volatile" boolean Indicates if the kernel argument 3399 is volatile qualified. Only 3400 present if ".value_kind" is 3401 "global_buffer". 3402 3403 ".is_pipe" boolean Indicates if the kernel argument 3404 is pipe qualified. Only present 3405 if ".value_kind" is "pipe". 3406 3407 .. TODO:: 3408 3409 Can "global_buffer" be pipe 3410 qualified? 3411 3412 ====================== ============== ========= ================================ 3413 3414.. _amdgpu-amdhsa-code-object-metadata-v4: 3415 3416Code Object V4 Metadata 3417+++++++++++++++++++++++ 3418 3419.. warning:: 3420 Code object V4 is not the default code object version emitted by this version 3421 of LLVM. 3422 3423Code object V4 metadata is the same as 3424:ref:`amdgpu-amdhsa-code-object-metadata-v3` with the changes and additions 3425defined in table :ref:`amdgpu-amdhsa-code-object-metadata-map-table-v3`. 3426 3427 .. table:: AMDHSA Code Object V4 Metadata Map Changes from :ref:`amdgpu-amdhsa-code-object-metadata-v3` 3428 :name: amdgpu-amdhsa-code-object-metadata-map-table-v4 3429 3430 ================= ============== ========= ======================================= 3431 String Key Value Type Required? Description 3432 ================= ============== ========= ======================================= 3433 "amdhsa.version" sequence of Required - The first integer is the major 3434 2 integers version. Currently 1. 3435 - The second integer is the minor 3436 version. Currently 1. 3437 "amdhsa.target" string Required The target name of the code using the syntax: 3438 3439 .. code:: 3440 3441 <target-triple> [ "-" <target-id> ] 3442 3443 A canonical target ID must be 3444 used. See :ref:`amdgpu-target-triples` 3445 and :ref:`amdgpu-target-id`. 3446 ================= ============== ========= ======================================= 3447 3448.. 3449 3450Kernel Dispatch 3451~~~~~~~~~~~~~~~ 3452 3453The HSA architected queuing language (AQL) defines a user space memory interface 3454that can be used to control the dispatch of kernels, in an agent independent 3455way. An agent can have zero or more AQL queues created for it using an HSA 3456compatible runtime (see :ref:`amdgpu-os`), in which AQL packets (all of which 3457are 64 bytes) can be placed. See the *HSA Platform System Architecture 3458Specification* [HSA]_ for the AQL queue mechanics and packet layouts. 3459 3460The packet processor of a kernel agent is responsible for detecting and 3461dispatching HSA kernels from the AQL queues associated with it. For AMD GPUs the 3462packet processor is implemented by the hardware command processor (CP), 3463asynchronous dispatch controller (ADC) and shader processor input controller 3464(SPI). 3465 3466An HSA compatible runtime can be used to allocate an AQL queue object. It uses 3467the kernel mode driver to initialize and register the AQL queue with CP. 3468 3469To dispatch a kernel the following actions are performed. This can occur in the 3470CPU host program, or from an HSA kernel executing on a GPU. 3471 34721. A pointer to an AQL queue for the kernel agent on which the kernel is to be 3473 executed is obtained. 34742. A pointer to the kernel descriptor (see 3475 :ref:`amdgpu-amdhsa-kernel-descriptor`) of the kernel to execute is obtained. 3476 It must be for a kernel that is contained in a code object that that was 3477 loaded by an HSA compatible runtime on the kernel agent with which the AQL 3478 queue is associated. 34793. Space is allocated for the kernel arguments using the HSA compatible runtime 3480 allocator for a memory region with the kernarg property for the kernel agent 3481 that will execute the kernel. It must be at least 16-byte aligned. 34824. Kernel argument values are assigned to the kernel argument memory 3483 allocation. The layout is defined in the *HSA Programmer's Language 3484 Reference* [HSA]_. For AMDGPU the kernel execution directly accesses the 3485 kernel argument memory in the same way constant memory is accessed. (Note 3486 that the HSA specification allows an implementation to copy the kernel 3487 argument contents to another location that is accessed by the kernel.) 34885. An AQL kernel dispatch packet is created on the AQL queue. The HSA compatible 3489 runtime api uses 64-bit atomic operations to reserve space in the AQL queue 3490 for the packet. The packet must be set up, and the final write must use an 3491 atomic store release to set the packet kind to ensure the packet contents are 3492 visible to the kernel agent. AQL defines a doorbell signal mechanism to 3493 notify the kernel agent that the AQL queue has been updated. These rules, and 3494 the layout of the AQL queue and kernel dispatch packet is defined in the *HSA 3495 System Architecture Specification* [HSA]_. 34966. A kernel dispatch packet includes information about the actual dispatch, 3497 such as grid and work-group size, together with information from the code 3498 object about the kernel, such as segment sizes. The HSA compatible runtime 3499 queries on the kernel symbol can be used to obtain the code object values 3500 which are recorded in the :ref:`amdgpu-amdhsa-code-object-metadata`. 35017. CP executes micro-code and is responsible for detecting and setting up the 3502 GPU to execute the wavefronts of a kernel dispatch. 35038. CP ensures that when the a wavefront starts executing the kernel machine 3504 code, the scalar general purpose registers (SGPR) and vector general purpose 3505 registers (VGPR) are set up as required by the machine code. The required 3506 setup is defined in the :ref:`amdgpu-amdhsa-kernel-descriptor`. The initial 3507 register state is defined in 3508 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`. 35099. The prolog of the kernel machine code (see 3510 :ref:`amdgpu-amdhsa-kernel-prolog`) sets up the machine state as necessary 3511 before continuing executing the machine code that corresponds to the kernel. 351210. When the kernel dispatch has completed execution, CP signals the completion 3513 signal specified in the kernel dispatch packet if not 0. 3514 3515.. _amdgpu-amdhsa-memory-spaces: 3516 3517Memory Spaces 3518~~~~~~~~~~~~~ 3519 3520The memory space properties are: 3521 3522 .. table:: AMDHSA Memory Spaces 3523 :name: amdgpu-amdhsa-memory-spaces-table 3524 3525 ================= =========== ======== ======= ================== 3526 Memory Space Name HSA Segment Hardware Address NULL Value 3527 Name Name Size 3528 ================= =========== ======== ======= ================== 3529 Private private scratch 32 0x00000000 3530 Local group LDS 32 0xFFFFFFFF 3531 Global global global 64 0x0000000000000000 3532 Constant constant *same as 64 0x0000000000000000 3533 global* 3534 Generic flat flat 64 0x0000000000000000 3535 Region N/A GDS 32 *not implemented 3536 for AMDHSA* 3537 ================= =========== ======== ======= ================== 3538 3539The global and constant memory spaces both use global virtual addresses, which 3540are the same virtual address space used by the CPU. However, some virtual 3541addresses may only be accessible to the CPU, some only accessible by the GPU, 3542and some by both. 3543 3544Using the constant memory space indicates that the data will not change during 3545the execution of the kernel. This allows scalar read instructions to be 3546used. The vector and scalar L1 caches are invalidated of volatile data before 3547each kernel dispatch execution to allow constant memory to change values between 3548kernel dispatches. 3549 3550The local memory space uses the hardware Local Data Store (LDS) which is 3551automatically allocated when the hardware creates work-groups of wavefronts, and 3552freed when all the wavefronts of a work-group have terminated. The data store 3553(DS) instructions can be used to access it. 3554 3555The private memory space uses the hardware scratch memory support. If the kernel 3556uses scratch, then the hardware allocates memory that is accessed using 3557wavefront lane dword (4 byte) interleaving. The mapping used from private 3558address to physical address is: 3559 3560 ``wavefront-scratch-base + 3561 (private-address * wavefront-size * 4) + 3562 (wavefront-lane-id * 4)`` 3563 3564There are different ways that the wavefront scratch base address is determined 3565by a wavefront (see :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). This 3566memory can be accessed in an interleaved manner using buffer instruction with 3567the scratch buffer descriptor and per wavefront scratch offset, by the scratch 3568instructions, or by flat instructions. If each lane of a wavefront accesses the 3569same private address, the interleaving results in adjacent dwords being accessed 3570and hence requires fewer cache lines to be fetched. Multi-dword access is not 3571supported except by flat and scratch instructions in GFX9-GFX10. 3572 3573The generic address space uses the hardware flat address support available in 3574GFX7-GFX10. This uses two fixed ranges of virtual addresses (the private and 3575local apertures), that are outside the range of addressible global memory, to 3576map from a flat address to a private or local address. 3577 3578FLAT instructions can take a flat address and access global, private (scratch) 3579and group (LDS) memory depending in if the address is within one of the 3580aperture ranges. Flat access to scratch requires hardware aperture setup and 3581setup in the kernel prologue (see 3582:ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`). Flat access to LDS requires 3583hardware aperture setup and M0 (GFX7-GFX8) register setup (see 3584:ref:`amdgpu-amdhsa-kernel-prolog-m0`). 3585 3586To convert between a segment address and a flat address the base address of the 3587apertures address can be used. For GFX7-GFX8 these are available in the 3588:ref:`amdgpu-amdhsa-hsa-aql-queue` the address of which can be obtained with 3589Queue Ptr SGPR (see :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). For 3590GFX9-GFX10 the aperture base addresses are directly available as inline constant 3591registers ``SRC_SHARED_BASE/LIMIT`` and ``SRC_PRIVATE_BASE/LIMIT``. In 64 bit 3592address mode the aperture sizes are 2^32 bytes and the base is aligned to 2^32 3593which makes it easier to convert from flat to segment or segment to flat. 3594 3595Image and Samplers 3596~~~~~~~~~~~~~~~~~~ 3597 3598Image and sample handles created by an HSA compatible runtime (see 3599:ref:`amdgpu-os`) are 64-bit addresses of a hardware 32-byte V# and 48 byte S# 3600object respectively. In order to support the HSA ``query_sampler`` operations 3601two extra dwords are used to store the HSA BRIG enumeration values for the 3602queries that are not trivially deducible from the S# representation. 3603 3604HSA Signals 3605~~~~~~~~~~~ 3606 3607HSA signal handles created by an HSA compatible runtime (see :ref:`amdgpu-os`) 3608are 64-bit addresses of a structure allocated in memory accessible from both the 3609CPU and GPU. The structure is defined by the runtime and subject to change 3610between releases. For example, see [AMD-ROCm-github]_. 3611 3612.. _amdgpu-amdhsa-hsa-aql-queue: 3613 3614HSA AQL Queue 3615~~~~~~~~~~~~~ 3616 3617The HSA AQL queue structure is defined by an HSA compatible runtime (see 3618:ref:`amdgpu-os`) and subject to change between releases. For example, see 3619[AMD-ROCm-github]_. For some processors it contains fields needed to implement 3620certain language features such as the flat address aperture bases. It also 3621contains fields used by CP such as managing the allocation of scratch memory. 3622 3623.. _amdgpu-amdhsa-kernel-descriptor: 3624 3625Kernel Descriptor 3626~~~~~~~~~~~~~~~~~ 3627 3628A kernel descriptor consists of the information needed by CP to initiate the 3629execution of a kernel, including the entry point address of the machine code 3630that implements the kernel. 3631 3632Code Object V3 Kernel Descriptor 3633++++++++++++++++++++++++++++++++ 3634 3635CP microcode requires the Kernel descriptor to be allocated on 64-byte 3636alignment. 3637 3638The fields used by CP for code objects before V3 also match those specified in 3639:ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 3640 3641 .. table:: Code Object V3 Kernel Descriptor 3642 :name: amdgpu-amdhsa-kernel-descriptor-v3-table 3643 3644 ======= ======= =============================== ============================ 3645 Bits Size Field Name Description 3646 ======= ======= =============================== ============================ 3647 31:0 4 bytes GROUP_SEGMENT_FIXED_SIZE The amount of fixed local 3648 address space memory 3649 required for a work-group 3650 in bytes. This does not 3651 include any dynamically 3652 allocated local address 3653 space memory that may be 3654 added when the kernel is 3655 dispatched. 3656 63:32 4 bytes PRIVATE_SEGMENT_FIXED_SIZE The amount of fixed 3657 private address space 3658 memory required for a 3659 work-item in bytes. 3660 Additional space may need to 3661 be added to this value if 3662 the call stack has 3663 non-inlined function calls. 3664 95:64 4 bytes KERNARG_SIZE The size of the kernarg 3665 memory pointed to by the 3666 AQL dispatch packet. The 3667 kernarg memory is used to 3668 pass arguments to the 3669 kernel. 3670 3671 * If the kernarg pointer in 3672 the dispatch packet is NULL 3673 then there are no kernel 3674 arguments. 3675 * If the kernarg pointer in 3676 the dispatch packet is 3677 not NULL and this value 3678 is 0 then the kernarg 3679 memory size is 3680 unspecified. 3681 * If the kernarg pointer in 3682 the dispatch packet is 3683 not NULL and this value 3684 is not 0 then the value 3685 specifies the kernarg 3686 memory size in bytes. It 3687 is recommended to provide 3688 a value as it may be used 3689 by CP to optimize making 3690 the kernarg memory 3691 visible to the kernel 3692 code. 3693 3694 127:96 4 bytes Reserved, must be 0. 3695 191:128 8 bytes KERNEL_CODE_ENTRY_BYTE_OFFSET Byte offset (possibly 3696 negative) from base 3697 address of kernel 3698 descriptor to kernel's 3699 entry point instruction 3700 which must be 256 byte 3701 aligned. 3702 351:272 20 Reserved, must be 0. 3703 bytes 3704 383:352 4 bytes COMPUTE_PGM_RSRC3 GFX6-GFX9 3705 Reserved, must be 0. 3706 GFX90A 3707 Compute Shader (CS) 3708 program settings used by 3709 CP to set up 3710 ``COMPUTE_PGM_RSRC3`` 3711 configuration 3712 register. See 3713 :ref:`amdgpu-amdhsa-compute_pgm_rsrc3-gfx90a-table`. 3714 GFX10 3715 Compute Shader (CS) 3716 program settings used by 3717 CP to set up 3718 ``COMPUTE_PGM_RSRC3`` 3719 configuration 3720 register. See 3721 :ref:`amdgpu-amdhsa-compute_pgm_rsrc3-gfx10-table`. 3722 415:384 4 bytes COMPUTE_PGM_RSRC1 Compute Shader (CS) 3723 program settings used by 3724 CP to set up 3725 ``COMPUTE_PGM_RSRC1`` 3726 configuration 3727 register. See 3728 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 3729 447:416 4 bytes COMPUTE_PGM_RSRC2 Compute Shader (CS) 3730 program settings used by 3731 CP to set up 3732 ``COMPUTE_PGM_RSRC2`` 3733 configuration 3734 register. See 3735 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 3736 458:448 7 bits *See separate bits below.* Enable the setup of the 3737 SGPR user data registers 3738 (see 3739 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 3740 3741 The total number of SGPR 3742 user data registers 3743 requested must not exceed 3744 16 and match value in 3745 ``compute_pgm_rsrc2.user_sgpr.user_sgpr_count``. 3746 Any requests beyond 16 3747 will be ignored. 3748 >448 1 bit ENABLE_SGPR_PRIVATE_SEGMENT If the *Target Properties* 3749 _BUFFER column of 3750 :ref:`amdgpu-processor-table` 3751 specifies *Architected flat 3752 scratch* then not supported 3753 and must be 0, 3754 >449 1 bit ENABLE_SGPR_DISPATCH_PTR 3755 >450 1 bit ENABLE_SGPR_QUEUE_PTR 3756 >451 1 bit ENABLE_SGPR_KERNARG_SEGMENT_PTR 3757 >452 1 bit ENABLE_SGPR_DISPATCH_ID 3758 >453 1 bit ENABLE_SGPR_FLAT_SCRATCH_INIT If the *Target Properties* 3759 column of 3760 :ref:`amdgpu-processor-table` 3761 specifies *Architected flat 3762 scratch* then not supported 3763 and must be 0, 3764 >454 1 bit ENABLE_SGPR_PRIVATE_SEGMENT 3765 _SIZE 3766 457:455 3 bits Reserved, must be 0. 3767 458 1 bit ENABLE_WAVEFRONT_SIZE32 GFX6-GFX9 3768 Reserved, must be 0. 3769 GFX10 3770 - If 0 execute in 3771 wavefront size 64 mode. 3772 - If 1 execute in 3773 native wavefront size 3774 32 mode. 3775 463:459 1 bit Reserved, must be 0. 3776 464 1 bit RESERVED_464 Deprecated, must be 0. 3777 467:465 3 bits Reserved, must be 0. 3778 468 1 bit RESERVED_468 Deprecated, must be 0. 3779 469:471 3 bits Reserved, must be 0. 3780 511:472 5 bytes Reserved, must be 0. 3781 512 **Total size 64 bytes.** 3782 ======= ==================================================================== 3783 3784.. 3785 3786 .. table:: compute_pgm_rsrc1 for GFX6-GFX10 3787 :name: amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table 3788 3789 ======= ======= =============================== =========================================================================== 3790 Bits Size Field Name Description 3791 ======= ======= =============================== =========================================================================== 3792 5:0 6 bits GRANULATED_WORKITEM_VGPR_COUNT Number of vector register 3793 blocks used by each work-item; 3794 granularity is device 3795 specific: 3796 3797 GFX6-GFX9 3798 - vgprs_used 0..256 3799 - max(0, ceil(vgprs_used / 4) - 1) 3800 GFX90A 3801 - vgprs_used 0..512 3802 - vgprs_used = align(arch_vgprs, 4) 3803 + acc_vgprs 3804 - max(0, ceil(vgprs_used / 8) - 1) 3805 GFX10 (wavefront size 64) 3806 - max_vgpr 1..256 3807 - max(0, ceil(vgprs_used / 4) - 1) 3808 GFX10 (wavefront size 32) 3809 - max_vgpr 1..256 3810 - max(0, ceil(vgprs_used / 8) - 1) 3811 3812 Where vgprs_used is defined 3813 as the highest VGPR number 3814 explicitly referenced plus 3815 one. 3816 3817 Used by CP to set up 3818 ``COMPUTE_PGM_RSRC1.VGPRS``. 3819 3820 The 3821 :ref:`amdgpu-assembler` 3822 calculates this 3823 automatically for the 3824 selected processor from 3825 values provided to the 3826 `.amdhsa_kernel` directive 3827 by the 3828 `.amdhsa_next_free_vgpr` 3829 nested directive (see 3830 :ref:`amdhsa-kernel-directives-table`). 3831 9:6 4 bits GRANULATED_WAVEFRONT_SGPR_COUNT Number of scalar register 3832 blocks used by a wavefront; 3833 granularity is device 3834 specific: 3835 3836 GFX6-GFX8 3837 - sgprs_used 0..112 3838 - max(0, ceil(sgprs_used / 8) - 1) 3839 GFX9 3840 - sgprs_used 0..112 3841 - 2 * max(0, ceil(sgprs_used / 16) - 1) 3842 GFX10 3843 Reserved, must be 0. 3844 (128 SGPRs always 3845 allocated.) 3846 3847 Where sgprs_used is 3848 defined as the highest 3849 SGPR number explicitly 3850 referenced plus one, plus 3851 a target specific number 3852 of additional special 3853 SGPRs for VCC, 3854 FLAT_SCRATCH (GFX7+) and 3855 XNACK_MASK (GFX8+), and 3856 any additional 3857 target specific 3858 limitations. It does not 3859 include the 16 SGPRs added 3860 if a trap handler is 3861 enabled. 3862 3863 The target specific 3864 limitations and special 3865 SGPR layout are defined in 3866 the hardware 3867 documentation, which can 3868 be found in the 3869 :ref:`amdgpu-processors` 3870 table. 3871 3872 Used by CP to set up 3873 ``COMPUTE_PGM_RSRC1.SGPRS``. 3874 3875 The 3876 :ref:`amdgpu-assembler` 3877 calculates this 3878 automatically for the 3879 selected processor from 3880 values provided to the 3881 `.amdhsa_kernel` directive 3882 by the 3883 `.amdhsa_next_free_sgpr` 3884 and `.amdhsa_reserve_*` 3885 nested directives (see 3886 :ref:`amdhsa-kernel-directives-table`). 3887 11:10 2 bits PRIORITY Must be 0. 3888 3889 Start executing wavefront 3890 at the specified priority. 3891 3892 CP is responsible for 3893 filling in 3894 ``COMPUTE_PGM_RSRC1.PRIORITY``. 3895 13:12 2 bits FLOAT_ROUND_MODE_32 Wavefront starts execution 3896 with specified rounding 3897 mode for single (32 3898 bit) floating point 3899 precision floating point 3900 operations. 3901 3902 Floating point rounding 3903 mode values are defined in 3904 :ref:`amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table`. 3905 3906 Used by CP to set up 3907 ``COMPUTE_PGM_RSRC1.FLOAT_MODE``. 3908 15:14 2 bits FLOAT_ROUND_MODE_16_64 Wavefront starts execution 3909 with specified rounding 3910 denorm mode for half/double (16 3911 and 64-bit) floating point 3912 precision floating point 3913 operations. 3914 3915 Floating point rounding 3916 mode values are defined in 3917 :ref:`amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table`. 3918 3919 Used by CP to set up 3920 ``COMPUTE_PGM_RSRC1.FLOAT_MODE``. 3921 17:16 2 bits FLOAT_DENORM_MODE_32 Wavefront starts execution 3922 with specified denorm mode 3923 for single (32 3924 bit) floating point 3925 precision floating point 3926 operations. 3927 3928 Floating point denorm mode 3929 values are defined in 3930 :ref:`amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table`. 3931 3932 Used by CP to set up 3933 ``COMPUTE_PGM_RSRC1.FLOAT_MODE``. 3934 19:18 2 bits FLOAT_DENORM_MODE_16_64 Wavefront starts execution 3935 with specified denorm mode 3936 for half/double (16 3937 and 64-bit) floating point 3938 precision floating point 3939 operations. 3940 3941 Floating point denorm mode 3942 values are defined in 3943 :ref:`amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table`. 3944 3945 Used by CP to set up 3946 ``COMPUTE_PGM_RSRC1.FLOAT_MODE``. 3947 20 1 bit PRIV Must be 0. 3948 3949 Start executing wavefront 3950 in privilege trap handler 3951 mode. 3952 3953 CP is responsible for 3954 filling in 3955 ``COMPUTE_PGM_RSRC1.PRIV``. 3956 21 1 bit ENABLE_DX10_CLAMP Wavefront starts execution 3957 with DX10 clamp mode 3958 enabled. Used by the vector 3959 ALU to force DX10 style 3960 treatment of NaN's (when 3961 set, clamp NaN to zero, 3962 otherwise pass NaN 3963 through). 3964 3965 Used by CP to set up 3966 ``COMPUTE_PGM_RSRC1.DX10_CLAMP``. 3967 22 1 bit DEBUG_MODE Must be 0. 3968 3969 Start executing wavefront 3970 in single step mode. 3971 3972 CP is responsible for 3973 filling in 3974 ``COMPUTE_PGM_RSRC1.DEBUG_MODE``. 3975 23 1 bit ENABLE_IEEE_MODE Wavefront starts execution 3976 with IEEE mode 3977 enabled. Floating point 3978 opcodes that support 3979 exception flag gathering 3980 will quiet and propagate 3981 signaling-NaN inputs per 3982 IEEE 754-2008. Min_dx10 and 3983 max_dx10 become IEEE 3984 754-2008 compliant due to 3985 signaling-NaN propagation 3986 and quieting. 3987 3988 Used by CP to set up 3989 ``COMPUTE_PGM_RSRC1.IEEE_MODE``. 3990 24 1 bit BULKY Must be 0. 3991 3992 Only one work-group allowed 3993 to execute on a compute 3994 unit. 3995 3996 CP is responsible for 3997 filling in 3998 ``COMPUTE_PGM_RSRC1.BULKY``. 3999 25 1 bit CDBG_USER Must be 0. 4000 4001 Flag that can be used to 4002 control debugging code. 4003 4004 CP is responsible for 4005 filling in 4006 ``COMPUTE_PGM_RSRC1.CDBG_USER``. 4007 26 1 bit FP16_OVFL GFX6-GFX8 4008 Reserved, must be 0. 4009 GFX9-GFX10 4010 Wavefront starts execution 4011 with specified fp16 overflow 4012 mode. 4013 4014 - If 0, fp16 overflow generates 4015 +/-INF values. 4016 - If 1, fp16 overflow that is the 4017 result of an +/-INF input value 4018 or divide by 0 produces a +/-INF, 4019 otherwise clamps computed 4020 overflow to +/-MAX_FP16 as 4021 appropriate. 4022 4023 Used by CP to set up 4024 ``COMPUTE_PGM_RSRC1.FP16_OVFL``. 4025 28:27 2 bits Reserved, must be 0. 4026 29 1 bit WGP_MODE GFX6-GFX9 4027 Reserved, must be 0. 4028 GFX10 4029 - If 0 execute work-groups in 4030 CU wavefront execution mode. 4031 - If 1 execute work-groups on 4032 in WGP wavefront execution mode. 4033 4034 See :ref:`amdgpu-amdhsa-memory-model`. 4035 4036 Used by CP to set up 4037 ``COMPUTE_PGM_RSRC1.WGP_MODE``. 4038 30 1 bit MEM_ORDERED GFX6-GFX9 4039 Reserved, must be 0. 4040 GFX10 4041 Controls the behavior of the 4042 s_waitcnt's vmcnt and vscnt 4043 counters. 4044 4045 - If 0 vmcnt reports completion 4046 of load and atomic with return 4047 out of order with sample 4048 instructions, and the vscnt 4049 reports the completion of 4050 store and atomic without 4051 return in order. 4052 - If 1 vmcnt reports completion 4053 of load, atomic with return 4054 and sample instructions in 4055 order, and the vscnt reports 4056 the completion of store and 4057 atomic without return in order. 4058 4059 Used by CP to set up 4060 ``COMPUTE_PGM_RSRC1.MEM_ORDERED``. 4061 31 1 bit FWD_PROGRESS GFX6-GFX9 4062 Reserved, must be 0. 4063 GFX10 4064 - If 0 execute SIMD wavefronts 4065 using oldest first policy. 4066 - If 1 execute SIMD wavefronts to 4067 ensure wavefronts will make some 4068 forward progress. 4069 4070 Used by CP to set up 4071 ``COMPUTE_PGM_RSRC1.FWD_PROGRESS``. 4072 32 **Total size 4 bytes** 4073 ======= =================================================================================================================== 4074 4075.. 4076 4077 .. table:: compute_pgm_rsrc2 for GFX6-GFX10 4078 :name: amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table 4079 4080 ======= ======= =============================== =========================================================================== 4081 Bits Size Field Name Description 4082 ======= ======= =============================== =========================================================================== 4083 0 1 bit ENABLE_PRIVATE_SEGMENT * Enable the setup of the 4084 private segment. 4085 * If the *Target Properties* 4086 column of 4087 :ref:`amdgpu-processor-table` 4088 does not specify 4089 *Architected flat 4090 scratch* then enable the 4091 setup of the SGPR 4092 wavefront scratch offset 4093 system register (see 4094 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4095 * If the *Target Properties* 4096 column of 4097 :ref:`amdgpu-processor-table` 4098 specifies *Architected 4099 flat scratch* then enable 4100 the setup of the 4101 FLAT_SCRATCH register 4102 pair (see 4103 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4104 4105 Used by CP to set up 4106 ``COMPUTE_PGM_RSRC2.SCRATCH_EN``. 4107 5:1 5 bits USER_SGPR_COUNT The total number of SGPR 4108 user data registers 4109 requested. This number must 4110 match the number of user 4111 data registers enabled. 4112 4113 Used by CP to set up 4114 ``COMPUTE_PGM_RSRC2.USER_SGPR``. 4115 6 1 bit ENABLE_TRAP_HANDLER Must be 0. 4116 4117 This bit represents 4118 ``COMPUTE_PGM_RSRC2.TRAP_PRESENT``, 4119 which is set by the CP if 4120 the runtime has installed a 4121 trap handler. 4122 7 1 bit ENABLE_SGPR_WORKGROUP_ID_X Enable the setup of the 4123 system SGPR register for 4124 the work-group id in the X 4125 dimension (see 4126 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4127 4128 Used by CP to set up 4129 ``COMPUTE_PGM_RSRC2.TGID_X_EN``. 4130 8 1 bit ENABLE_SGPR_WORKGROUP_ID_Y Enable the setup of the 4131 system SGPR register for 4132 the work-group id in the Y 4133 dimension (see 4134 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4135 4136 Used by CP to set up 4137 ``COMPUTE_PGM_RSRC2.TGID_Y_EN``. 4138 9 1 bit ENABLE_SGPR_WORKGROUP_ID_Z Enable the setup of the 4139 system SGPR register for 4140 the work-group id in the Z 4141 dimension (see 4142 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4143 4144 Used by CP to set up 4145 ``COMPUTE_PGM_RSRC2.TGID_Z_EN``. 4146 10 1 bit ENABLE_SGPR_WORKGROUP_INFO Enable the setup of the 4147 system SGPR register for 4148 work-group information (see 4149 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4150 4151 Used by CP to set up 4152 ``COMPUTE_PGM_RSRC2.TGID_SIZE_EN``. 4153 12:11 2 bits ENABLE_VGPR_WORKITEM_ID Enable the setup of the 4154 VGPR system registers used 4155 for the work-item ID. 4156 :ref:`amdgpu-amdhsa-system-vgpr-work-item-id-enumeration-values-table` 4157 defines the values. 4158 4159 Used by CP to set up 4160 ``COMPUTE_PGM_RSRC2.TIDIG_CMP_CNT``. 4161 13 1 bit ENABLE_EXCEPTION_ADDRESS_WATCH Must be 0. 4162 4163 Wavefront starts execution 4164 with address watch 4165 exceptions enabled which 4166 are generated when L1 has 4167 witnessed a thread access 4168 an *address of 4169 interest*. 4170 4171 CP is responsible for 4172 filling in the address 4173 watch bit in 4174 ``COMPUTE_PGM_RSRC2.EXCP_EN_MSB`` 4175 according to what the 4176 runtime requests. 4177 14 1 bit ENABLE_EXCEPTION_MEMORY Must be 0. 4178 4179 Wavefront starts execution 4180 with memory violation 4181 exceptions exceptions 4182 enabled which are generated 4183 when a memory violation has 4184 occurred for this wavefront from 4185 L1 or LDS 4186 (write-to-read-only-memory, 4187 mis-aligned atomic, LDS 4188 address out of range, 4189 illegal address, etc.). 4190 4191 CP sets the memory 4192 violation bit in 4193 ``COMPUTE_PGM_RSRC2.EXCP_EN_MSB`` 4194 according to what the 4195 runtime requests. 4196 23:15 9 bits GRANULATED_LDS_SIZE Must be 0. 4197 4198 CP uses the rounded value 4199 from the dispatch packet, 4200 not this value, as the 4201 dispatch may contain 4202 dynamically allocated group 4203 segment memory. CP writes 4204 directly to 4205 ``COMPUTE_PGM_RSRC2.LDS_SIZE``. 4206 4207 Amount of group segment 4208 (LDS) to allocate for each 4209 work-group. Granularity is 4210 device specific: 4211 4212 GFX6 4213 roundup(lds-size / (64 * 4)) 4214 GFX7-GFX10 4215 roundup(lds-size / (128 * 4)) 4216 4217 24 1 bit ENABLE_EXCEPTION_IEEE_754_FP Wavefront starts execution 4218 _INVALID_OPERATION with specified exceptions 4219 enabled. 4220 4221 Used by CP to set up 4222 ``COMPUTE_PGM_RSRC2.EXCP_EN`` 4223 (set from bits 0..6). 4224 4225 IEEE 754 FP Invalid 4226 Operation 4227 25 1 bit ENABLE_EXCEPTION_FP_DENORMAL FP Denormal one or more 4228 _SOURCE input operands is a 4229 denormal number 4230 26 1 bit ENABLE_EXCEPTION_IEEE_754_FP IEEE 754 FP Division by 4231 _DIVISION_BY_ZERO Zero 4232 27 1 bit ENABLE_EXCEPTION_IEEE_754_FP IEEE 754 FP FP Overflow 4233 _OVERFLOW 4234 28 1 bit ENABLE_EXCEPTION_IEEE_754_FP IEEE 754 FP Underflow 4235 _UNDERFLOW 4236 29 1 bit ENABLE_EXCEPTION_IEEE_754_FP IEEE 754 FP Inexact 4237 _INEXACT 4238 30 1 bit ENABLE_EXCEPTION_INT_DIVIDE_BY Integer Division by Zero 4239 _ZERO (rcp_iflag_f32 instruction 4240 only) 4241 31 1 bit Reserved, must be 0. 4242 32 **Total size 4 bytes.** 4243 ======= =================================================================================================================== 4244 4245.. 4246 4247 .. table:: compute_pgm_rsrc3 for GFX90A 4248 :name: amdgpu-amdhsa-compute_pgm_rsrc3-gfx90a-table 4249 4250 ======= ======= =============================== =========================================================================== 4251 Bits Size Field Name Description 4252 ======= ======= =============================== =========================================================================== 4253 5:0 6 bits ACCUM_OFFSET Offset of a first AccVGPR in the unified register file. Granularity 4. 4254 Value 0-63. 0 - accum-offset = 4, 1 - accum-offset = 8, ..., 4255 63 - accum-offset = 256. 4256 6:15 10 Reserved, must be 0. 4257 bits 4258 16 1 bit TG_SPLIT - If 0 the waves of a work-group are 4259 launched in the same CU. 4260 - If 1 the waves of a work-group can be 4261 launched in different CUs. The waves 4262 cannot use S_BARRIER or LDS. 4263 17:31 15 Reserved, must be 0. 4264 bits 4265 32 **Total size 4 bytes.** 4266 ======= =================================================================================================================== 4267 4268.. 4269 4270 .. table:: compute_pgm_rsrc3 for GFX10 4271 :name: amdgpu-amdhsa-compute_pgm_rsrc3-gfx10-table 4272 4273 ======= ======= =============================== =========================================================================== 4274 Bits Size Field Name Description 4275 ======= ======= =============================== =========================================================================== 4276 3:0 4 bits SHARED_VGPR_COUNT Number of shared VGPRs for wavefront size 64. Granularity 8. Value 0-120. 4277 compute_pgm_rsrc1.vgprs + shared_vgpr_cnt cannot exceed 64. 4278 31:4 28 Reserved, must be 0. 4279 bits 4280 32 **Total size 4 bytes.** 4281 ======= =================================================================================================================== 4282 4283.. 4284 4285 .. table:: Floating Point Rounding Mode Enumeration Values 4286 :name: amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table 4287 4288 ====================================== ===== ============================== 4289 Enumeration Name Value Description 4290 ====================================== ===== ============================== 4291 FLOAT_ROUND_MODE_NEAR_EVEN 0 Round Ties To Even 4292 FLOAT_ROUND_MODE_PLUS_INFINITY 1 Round Toward +infinity 4293 FLOAT_ROUND_MODE_MINUS_INFINITY 2 Round Toward -infinity 4294 FLOAT_ROUND_MODE_ZERO 3 Round Toward 0 4295 ====================================== ===== ============================== 4296 4297.. 4298 4299 .. table:: Floating Point Denorm Mode Enumeration Values 4300 :name: amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table 4301 4302 ====================================== ===== ============================== 4303 Enumeration Name Value Description 4304 ====================================== ===== ============================== 4305 FLOAT_DENORM_MODE_FLUSH_SRC_DST 0 Flush Source and Destination 4306 Denorms 4307 FLOAT_DENORM_MODE_FLUSH_DST 1 Flush Output Denorms 4308 FLOAT_DENORM_MODE_FLUSH_SRC 2 Flush Source Denorms 4309 FLOAT_DENORM_MODE_FLUSH_NONE 3 No Flush 4310 ====================================== ===== ============================== 4311 4312.. 4313 4314 .. table:: System VGPR Work-Item ID Enumeration Values 4315 :name: amdgpu-amdhsa-system-vgpr-work-item-id-enumeration-values-table 4316 4317 ======================================== ===== ============================ 4318 Enumeration Name Value Description 4319 ======================================== ===== ============================ 4320 SYSTEM_VGPR_WORKITEM_ID_X 0 Set work-item X dimension 4321 ID. 4322 SYSTEM_VGPR_WORKITEM_ID_X_Y 1 Set work-item X and Y 4323 dimensions ID. 4324 SYSTEM_VGPR_WORKITEM_ID_X_Y_Z 2 Set work-item X, Y and Z 4325 dimensions ID. 4326 SYSTEM_VGPR_WORKITEM_ID_UNDEFINED 3 Undefined. 4327 ======================================== ===== ============================ 4328 4329.. _amdgpu-amdhsa-initial-kernel-execution-state: 4330 4331Initial Kernel Execution State 4332~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 4333 4334This section defines the register state that will be set up by the packet 4335processor prior to the start of execution of every wavefront. This is limited by 4336the constraints of the hardware controllers of CP/ADC/SPI. 4337 4338The order of the SGPR registers is defined, but the compiler can specify which 4339ones are actually setup in the kernel descriptor using the ``enable_sgpr_*`` bit 4340fields (see :ref:`amdgpu-amdhsa-kernel-descriptor`). The register numbers used 4341for enabled registers are dense starting at SGPR0: the first enabled register is 4342SGPR0, the next enabled register is SGPR1 etc.; disabled registers do not have 4343an SGPR number. 4344 4345The initial SGPRs comprise up to 16 User SRGPs that are set by CP and apply to 4346all wavefronts of the grid. It is possible to specify more than 16 User SGPRs 4347using the ``enable_sgpr_*`` bit fields, in which case only the first 16 are 4348actually initialized. These are then immediately followed by the System SGPRs 4349that are set up by ADC/SPI and can have different values for each wavefront of 4350the grid dispatch. 4351 4352SGPR register initial state is defined in 4353:ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 4354 4355 .. table:: SGPR Register Set Up Order 4356 :name: amdgpu-amdhsa-sgpr-register-set-up-order-table 4357 4358 ========== ========================== ====== ============================== 4359 SGPR Order Name Number Description 4360 (kernel descriptor enable of 4361 field) SGPRs 4362 ========== ========================== ====== ============================== 4363 First Private Segment Buffer 4 See 4364 (enable_sgpr_private :ref:`amdgpu-amdhsa-kernel-prolog-private-segment-buffer`. 4365 _segment_buffer) 4366 then Dispatch Ptr 2 64-bit address of AQL dispatch 4367 (enable_sgpr_dispatch_ptr) packet for kernel dispatch 4368 actually executing. 4369 then Queue Ptr 2 64-bit address of amd_queue_t 4370 (enable_sgpr_queue_ptr) object for AQL queue on which 4371 the dispatch packet was 4372 queued. 4373 then Kernarg Segment Ptr 2 64-bit address of Kernarg 4374 (enable_sgpr_kernarg segment. This is directly 4375 _segment_ptr) copied from the 4376 kernarg_address in the kernel 4377 dispatch packet. 4378 4379 Having CP load it once avoids 4380 loading it at the beginning of 4381 every wavefront. 4382 then Dispatch Id 2 64-bit Dispatch ID of the 4383 (enable_sgpr_dispatch_id) dispatch packet being 4384 executed. 4385 then Flat Scratch Init 2 See 4386 (enable_sgpr_flat_scratch :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`. 4387 _init) 4388 then Private Segment Size 1 The 32-bit byte size of a 4389 (enable_sgpr_private single work-item's memory 4390 _segment_size) allocation. This is the 4391 value from the kernel 4392 dispatch packet Private 4393 Segment Byte Size rounded up 4394 by CP to a multiple of 4395 DWORD. 4396 4397 Having CP load it once avoids 4398 loading it at the beginning of 4399 every wavefront. 4400 4401 This is not used for 4402 GFX7-GFX8 since it is the same 4403 value as the second SGPR of 4404 Flat Scratch Init. However, it 4405 may be needed for GFX9-GFX10 which 4406 changes the meaning of the 4407 Flat Scratch Init value. 4408 then Work-Group Id X 1 32-bit work-group id in X 4409 (enable_sgpr_workgroup_id dimension of grid for 4410 _X) wavefront. 4411 then Work-Group Id Y 1 32-bit work-group id in Y 4412 (enable_sgpr_workgroup_id dimension of grid for 4413 _Y) wavefront. 4414 then Work-Group Id Z 1 32-bit work-group id in Z 4415 (enable_sgpr_workgroup_id dimension of grid for 4416 _Z) wavefront. 4417 then Work-Group Info 1 {first_wavefront, 14'b0000, 4418 (enable_sgpr_workgroup ordered_append_term[10:0], 4419 _info) threadgroup_size_in_wavefronts[5:0]} 4420 then Scratch Wavefront Offset 1 See 4421 (enable_sgpr_private :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`. 4422 _segment_wavefront_offset) and 4423 :ref:`amdgpu-amdhsa-kernel-prolog-private-segment-buffer`. 4424 ========== ========================== ====== ============================== 4425 4426The order of the VGPR registers is defined, but the compiler can specify which 4427ones are actually setup in the kernel descriptor using the ``enable_vgpr*`` bit 4428fields (see :ref:`amdgpu-amdhsa-kernel-descriptor`). The register numbers used 4429for enabled registers are dense starting at VGPR0: the first enabled register is 4430VGPR0, the next enabled register is VGPR1 etc.; disabled registers do not have a 4431VGPR number. 4432 4433There are different methods used for the VGPR initial state: 4434 4435* Unless the *Target Properties* column of :ref:`amdgpu-processor-table` 4436 specifies otherwise, a separate VGPR register is used per work-item ID. The 4437 VGPR register initial state for this method is defined in 4438 :ref:`amdgpu-amdhsa-vgpr-register-set-up-order-for-unpacked-work-item-id-method-table`. 4439* If *Target Properties* column of :ref:`amdgpu-processor-table` 4440 specifies *Packed work-item IDs*, the initial value of VGPR0 register is used 4441 for all work-item IDs. The register layout for this method is defined in 4442 :ref:`amdgpu-amdhsa-register-layout-for-packed-work-item-id-method-table`. 4443 4444 .. table:: VGPR Register Set Up Order for Unpacked Work-Item ID Method 4445 :name: amdgpu-amdhsa-vgpr-register-set-up-order-for-unpacked-work-item-id-method-table 4446 4447 ========== ========================== ====== ============================== 4448 VGPR Order Name Number Description 4449 (kernel descriptor enable of 4450 field) VGPRs 4451 ========== ========================== ====== ============================== 4452 First Work-Item Id X 1 32-bit work-item id in X 4453 (Always initialized) dimension of work-group for 4454 wavefront lane. 4455 then Work-Item Id Y 1 32-bit work-item id in Y 4456 (enable_vgpr_workitem_id dimension of work-group for 4457 > 0) wavefront lane. 4458 then Work-Item Id Z 1 32-bit work-item id in Z 4459 (enable_vgpr_workitem_id dimension of work-group for 4460 > 1) wavefront lane. 4461 ========== ========================== ====== ============================== 4462 4463.. 4464 4465 .. table:: Register Layout for Packed Work-Item ID Method 4466 :name: amdgpu-amdhsa-register-layout-for-packed-work-item-id-method-table 4467 4468 ======= ======= ================ ========================================= 4469 Bits Size Field Name Description 4470 ======= ======= ================ ========================================= 4471 0:9 10 bits Work-Item Id X Work-item id in X 4472 dimension of work-group for 4473 wavefront lane. 4474 4475 Always initialized. 4476 4477 10:19 10 bits Work-Item Id Y Work-item id in Y 4478 dimension of work-group for 4479 wavefront lane. 4480 4481 Initialized if enable_vgpr_workitem_id > 4482 0, otherwise set to 0. 4483 20:29 10 bits Work-Item Id Z Work-item id in Z 4484 dimension of work-group for 4485 wavefront lane. 4486 4487 Initialized if enable_vgpr_workitem_id > 4488 1, otherwise set to 0. 4489 30:31 2 bits Reserved, set to 0. 4490 ======= ======= ================ ========================================= 4491 4492The setting of registers is done by GPU CP/ADC/SPI hardware as follows: 4493 44941. SGPRs before the Work-Group Ids are set by CP using the 16 User Data 4495 registers. 44962. Work-group Id registers X, Y, Z are set by ADC which supports any 4497 combination including none. 44983. Scratch Wavefront Offset is set by SPI in a per wavefront basis which is why 4499 its value cannot be included with the flat scratch init value which is per 4500 queue (see :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`). 45014. The VGPRs are set by SPI which only supports specifying either (X), (X, Y) 4502 or (X, Y, Z). 45035. Flat Scratch register pair initialization is described in 4504 :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`. 4505 4506The global segment can be accessed either using buffer instructions (GFX6 which 4507has V# 64-bit address support), flat instructions (GFX7-GFX10), or global 4508instructions (GFX9-GFX10). 4509 4510If buffer operations are used, then the compiler can generate a V# with the 4511following properties: 4512 4513* base address of 0 4514* no swizzle 4515* ATC: 1 if IOMMU present (such as APU) 4516* ptr64: 1 4517* MTYPE set to support memory coherence that matches the runtime (such as CC for 4518 APU and NC for dGPU). 4519 4520.. _amdgpu-amdhsa-kernel-prolog: 4521 4522Kernel Prolog 4523~~~~~~~~~~~~~ 4524 4525The compiler performs initialization in the kernel prologue depending on the 4526target and information about things like stack usage in the kernel and called 4527functions. Some of this initialization requires the compiler to request certain 4528User and System SGPRs be present in the 4529:ref:`amdgpu-amdhsa-initial-kernel-execution-state` via the 4530:ref:`amdgpu-amdhsa-kernel-descriptor`. 4531 4532.. _amdgpu-amdhsa-kernel-prolog-cfi: 4533 4534CFI 4535+++ 4536 45371. The CFI return address is undefined. 4538 45392. The CFI CFA is defined using an expression which evaluates to a location 4540 description that comprises one memory location description for the 4541 ``DW_ASPACE_AMDGPU_private_lane`` address space address ``0``. 4542 4543.. _amdgpu-amdhsa-kernel-prolog-m0: 4544 4545M0 4546++ 4547 4548GFX6-GFX8 4549 The M0 register must be initialized with a value at least the total LDS size 4550 if the kernel may access LDS via DS or flat operations. Total LDS size is 4551 available in dispatch packet. For M0, it is also possible to use maximum 4552 possible value of LDS for given target (0x7FFF for GFX6 and 0xFFFF for 4553 GFX7-GFX8). 4554GFX9-GFX10 4555 The M0 register is not used for range checking LDS accesses and so does not 4556 need to be initialized in the prolog. 4557 4558.. _amdgpu-amdhsa-kernel-prolog-stack-pointer: 4559 4560Stack Pointer 4561+++++++++++++ 4562 4563If the kernel has function calls it must set up the ABI stack pointer described 4564in :ref:`amdgpu-amdhsa-function-call-convention-non-kernel-functions` by setting 4565SGPR32 to the unswizzled scratch offset of the address past the last local 4566allocation. 4567 4568.. _amdgpu-amdhsa-kernel-prolog-frame-pointer: 4569 4570Frame Pointer 4571+++++++++++++ 4572 4573If the kernel needs a frame pointer for the reasons defined in 4574``SIFrameLowering`` then SGPR33 is used and is always set to ``0`` in the 4575kernel prolog. If a frame pointer is not required then all uses of the frame 4576pointer are replaced with immediate ``0`` offsets. 4577 4578.. _amdgpu-amdhsa-kernel-prolog-flat-scratch: 4579 4580Flat Scratch 4581++++++++++++ 4582 4583There are different methods used for initializing flat scratch: 4584 4585* If the *Target Properties* column of :ref:`amdgpu-processor-table` 4586 specifies *Does not support generic address space*: 4587 4588 Flat scratch is not supported and there is no flat scratch register pair. 4589 4590* If the *Target Properties* column of :ref:`amdgpu-processor-table` 4591 specifies *Offset flat scratch*: 4592 4593 If the kernel or any function it calls may use flat operations to access 4594 scratch memory, the prolog code must set up the FLAT_SCRATCH register pair 4595 (FLAT_SCRATCH_LO/FLAT_SCRATCH_HI). Initialization uses Flat Scratch Init and 4596 Scratch Wavefront Offset SGPR registers (see 4597 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`): 4598 4599 1. The low word of Flat Scratch Init is the 32-bit byte offset from 4600 ``SH_HIDDEN_PRIVATE_BASE_VIMID`` to the base of scratch backing memory 4601 being managed by SPI for the queue executing the kernel dispatch. This is 4602 the same value used in the Scratch Segment Buffer V# base address. 4603 4604 CP obtains this from the runtime. (The Scratch Segment Buffer base address 4605 is ``SH_HIDDEN_PRIVATE_BASE_VIMID`` plus this offset.) 4606 4607 The prolog must add the value of Scratch Wavefront Offset to get the 4608 wavefront's byte scratch backing memory offset from 4609 ``SH_HIDDEN_PRIVATE_BASE_VIMID``. 4610 4611 The Scratch Wavefront Offset must also be used as an offset with Private 4612 segment address when using the Scratch Segment Buffer. 4613 4614 Since FLAT_SCRATCH_LO is in units of 256 bytes, the offset must be right 4615 shifted by 8 before moving into FLAT_SCRATCH_HI. 4616 4617 FLAT_SCRATCH_HI corresponds to SGPRn-4 on GFX7, and SGPRn-6 on GFX8 (where 4618 SGPRn is the highest numbered SGPR allocated to the wavefront). 4619 FLAT_SCRATCH_HI is multiplied by 256 (as it is in units of 256 bytes) and 4620 added to ``SH_HIDDEN_PRIVATE_BASE_VIMID`` to calculate the per wavefront 4621 FLAT SCRATCH BASE in flat memory instructions that access the scratch 4622 aperture. 4623 2. The second word of Flat Scratch Init is 32-bit byte size of a single 4624 work-items scratch memory usage. 4625 4626 CP obtains this from the runtime, and it is always a multiple of DWORD. CP 4627 checks that the value in the kernel dispatch packet Private Segment Byte 4628 Size is not larger and requests the runtime to increase the queue's scratch 4629 size if necessary. 4630 4631 CP directly loads from the kernel dispatch packet Private Segment Byte Size 4632 field and rounds up to a multiple of DWORD. Having CP load it once avoids 4633 loading it at the beginning of every wavefront. 4634 4635 The kernel prolog code must move it to FLAT_SCRATCH_LO which is SGPRn-3 on 4636 GFX7 and SGPRn-5 on GFX8. FLAT_SCRATCH_LO is used as the FLAT SCRATCH SIZE 4637 in flat memory instructions. 4638 4639* If the *Target Properties* column of :ref:`amdgpu-processor-table` 4640 specifies *Absolute flat scratch*: 4641 4642 If the kernel or any function it calls may use flat operations to access 4643 scratch memory, the prolog code must set up the FLAT_SCRATCH register pair 4644 (FLAT_SCRATCH_LO/FLAT_SCRATCH_HI which are in SGPRn-4/SGPRn-3). Initialization 4645 uses Flat Scratch Init and Scratch Wavefront Offset SGPR registers (see 4646 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`): 4647 4648 The Flat Scratch Init is the 64-bit address of the base of scratch backing 4649 memory being managed by SPI for the queue executing the kernel dispatch. 4650 4651 CP obtains this from the runtime. 4652 4653 The kernel prolog must add the value of the wave's Scratch Wavefront Offset 4654 and move the result as a 64-bit value to the FLAT_SCRATCH SGPR register pair 4655 which is SGPRn-6 and SGPRn-5. It is used as the FLAT SCRATCH BASE in flat 4656 memory instructions. 4657 4658 The Scratch Wavefront Offset must also be used as an offset with Private 4659 segment address when using the Scratch Segment Buffer (see 4660 :ref:`amdgpu-amdhsa-kernel-prolog-private-segment-buffer`). 4661 4662* If the *Target Properties* column of :ref:`amdgpu-processor-table` 4663 specifies *Architected flat scratch*: 4664 4665 If ENABLE_PRIVATE_SEGMENT is enabled in 4666 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table` then the FLAT_SCRATCH 4667 register pair will be initialized to the 64-bit address of the base of scratch 4668 backing memory being managed by SPI for the queue executing the kernel 4669 dispatch plus the value of the wave's Scratch Wavefront Offset for use as the 4670 flat scratch base in flat memory instructions. 4671 4672.. _amdgpu-amdhsa-kernel-prolog-private-segment-buffer: 4673 4674Private Segment Buffer 4675++++++++++++++++++++++ 4676 4677If the *Target Properties* column of :ref:`amdgpu-processor-table` specifies 4678*Architected flat scratch* then a Private Segment Buffer is not supported. 4679Instead the flat SCRATCH instructions are used. 4680 4681Otherwise, Private Segment Buffer SGPR register is used to initialize 4 SGPRs 4682that are used as a V# to access scratch. CP uses the value provided by the 4683runtime. It is used, together with Scratch Wavefront Offset as an offset, to 4684access the private memory space using a segment address. See 4685:ref:`amdgpu-amdhsa-initial-kernel-execution-state`. 4686 4687The scratch V# is a four-aligned SGPR and always selected for the kernel as 4688follows: 4689 4690 - If it is known during instruction selection that there is stack usage, 4691 SGPR0-3 is reserved for use as the scratch V#. Stack usage is assumed if 4692 optimizations are disabled (``-O0``), if stack objects already exist (for 4693 locals, etc.), or if there are any function calls. 4694 4695 - Otherwise, four high numbered SGPRs beginning at a four-aligned SGPR index 4696 are reserved for the tentative scratch V#. These will be used if it is 4697 determined that spilling is needed. 4698 4699 - If no use is made of the tentative scratch V#, then it is unreserved, 4700 and the register count is determined ignoring it. 4701 - If use is made of the tentative scratch V#, then its register numbers 4702 are shifted to the first four-aligned SGPR index after the highest one 4703 allocated by the register allocator, and all uses are updated. The 4704 register count includes them in the shifted location. 4705 - In either case, if the processor has the SGPR allocation bug, the 4706 tentative allocation is not shifted or unreserved in order to ensure 4707 the register count is higher to workaround the bug. 4708 4709 .. note:: 4710 4711 This approach of using a tentative scratch V# and shifting the register 4712 numbers if used avoids having to perform register allocation a second 4713 time if the tentative V# is eliminated. This is more efficient and 4714 avoids the problem that the second register allocation may perform 4715 spilling which will fail as there is no longer a scratch V#. 4716 4717When the kernel prolog code is being emitted it is known whether the scratch V# 4718described above is actually used. If it is, the prolog code must set it up by 4719copying the Private Segment Buffer to the scratch V# registers and then adding 4720the Private Segment Wavefront Offset to the queue base address in the V#. The 4721result is a V# with a base address pointing to the beginning of the wavefront 4722scratch backing memory. 4723 4724The Private Segment Buffer is always requested, but the Private Segment 4725Wavefront Offset is only requested if it is used (see 4726:ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4727 4728.. _amdgpu-amdhsa-memory-model: 4729 4730Memory Model 4731~~~~~~~~~~~~ 4732 4733This section describes the mapping of the LLVM memory model onto AMDGPU machine 4734code (see :ref:`memmodel`). 4735 4736The AMDGPU backend supports the memory synchronization scopes specified in 4737:ref:`amdgpu-memory-scopes`. 4738 4739The code sequences used to implement the memory model specify the order of 4740instructions that a single thread must execute. The ``s_waitcnt`` and cache 4741management instructions such as ``buffer_wbinvl1_vol`` are defined with respect 4742to other memory instructions executed by the same thread. This allows them to be 4743moved earlier or later which can allow them to be combined with other instances 4744of the same instruction, or hoisted/sunk out of loops to improve performance. 4745Only the instructions related to the memory model are given; additional 4746``s_waitcnt`` instructions are required to ensure registers are defined before 4747being used. These may be able to be combined with the memory model ``s_waitcnt`` 4748instructions as described above. 4749 4750The AMDGPU backend supports the following memory models: 4751 4752 HSA Memory Model [HSA]_ 4753 The HSA memory model uses a single happens-before relation for all address 4754 spaces (see :ref:`amdgpu-address-spaces`). 4755 OpenCL Memory Model [OpenCL]_ 4756 The OpenCL memory model which has separate happens-before relations for the 4757 global and local address spaces. Only a fence specifying both global and 4758 local address space, and seq_cst instructions join the relationships. Since 4759 the LLVM ``memfence`` instruction does not allow an address space to be 4760 specified the OpenCL fence has to conservatively assume both local and 4761 global address space was specified. However, optimizations can often be 4762 done to eliminate the additional ``s_waitcnt`` instructions when there are 4763 no intervening memory instructions which access the corresponding address 4764 space. The code sequences in the table indicate what can be omitted for the 4765 OpenCL memory. The target triple environment is used to determine if the 4766 source language is OpenCL (see :ref:`amdgpu-opencl`). 4767 4768``ds/flat_load/store/atomic`` instructions to local memory are termed LDS 4769operations. 4770 4771``buffer/global/flat_load/store/atomic`` instructions to global memory are 4772termed vector memory operations. 4773 4774Private address space uses ``buffer_load/store`` using the scratch V# 4775(GFX6-GFX8), or ``scratch_load/store`` (GFX9-GFX10). Since only a single thread 4776is accessing the memory, atomic memory orderings are not meaningful, and all 4777accesses are treated as non-atomic. 4778 4779Constant address space uses ``buffer/global_load`` instructions (or equivalent 4780scalar memory instructions). Since the constant address space contents do not 4781change during the execution of a kernel dispatch it is not legal to perform 4782stores, and atomic memory orderings are not meaningful, and all accesses are 4783treated as non-atomic. 4784 4785A memory synchronization scope wider than work-group is not meaningful for the 4786group (LDS) address space and is treated as work-group. 4787 4788The memory model does not support the region address space which is treated as 4789non-atomic. 4790 4791Acquire memory ordering is not meaningful on store atomic instructions and is 4792treated as non-atomic. 4793 4794Release memory ordering is not meaningful on load atomic instructions and is 4795treated a non-atomic. 4796 4797Acquire-release memory ordering is not meaningful on load or store atomic 4798instructions and is treated as acquire and release respectively. 4799 4800The memory order also adds the single thread optimization constraints defined in 4801table 4802:ref:`amdgpu-amdhsa-memory-model-single-thread-optimization-constraints-table`. 4803 4804 .. table:: AMDHSA Memory Model Single Thread Optimization Constraints 4805 :name: amdgpu-amdhsa-memory-model-single-thread-optimization-constraints-table 4806 4807 ============ ============================================================== 4808 LLVM Memory Optimization Constraints 4809 Ordering 4810 ============ ============================================================== 4811 unordered *none* 4812 monotonic *none* 4813 acquire - If a load atomic/atomicrmw then no following load/load 4814 atomic/store/store atomic/atomicrmw/fence instruction can be 4815 moved before the acquire. 4816 - If a fence then same as load atomic, plus no preceding 4817 associated fence-paired-atomic can be moved after the fence. 4818 release - If a store atomic/atomicrmw then no preceding load/load 4819 atomic/store/store atomic/atomicrmw/fence instruction can be 4820 moved after the release. 4821 - If a fence then same as store atomic, plus no following 4822 associated fence-paired-atomic can be moved before the 4823 fence. 4824 acq_rel Same constraints as both acquire and release. 4825 seq_cst - If a load atomic then same constraints as acquire, plus no 4826 preceding sequentially consistent load atomic/store 4827 atomic/atomicrmw/fence instruction can be moved after the 4828 seq_cst. 4829 - If a store atomic then the same constraints as release, plus 4830 no following sequentially consistent load atomic/store 4831 atomic/atomicrmw/fence instruction can be moved before the 4832 seq_cst. 4833 - If an atomicrmw/fence then same constraints as acq_rel. 4834 ============ ============================================================== 4835 4836The code sequences used to implement the memory model are defined in the 4837following sections: 4838 4839* :ref:`amdgpu-amdhsa-memory-model-gfx6-gfx9` 4840* :ref:`amdgpu-amdhsa-memory-model-gfx90a` 4841* :ref:`amdgpu-amdhsa-memory-model-gfx10` 4842 4843.. _amdgpu-amdhsa-memory-model-gfx6-gfx9: 4844 4845Memory Model GFX6-GFX9 4846++++++++++++++++++++++ 4847 4848For GFX6-GFX9: 4849 4850* Each agent has multiple shader arrays (SA). 4851* Each SA has multiple compute units (CU). 4852* Each CU has multiple SIMDs that execute wavefronts. 4853* The wavefronts for a single work-group are executed in the same CU but may be 4854 executed by different SIMDs. 4855* Each CU has a single LDS memory shared by the wavefronts of the work-groups 4856 executing on it. 4857* All LDS operations of a CU are performed as wavefront wide operations in a 4858 global order and involve no caching. Completion is reported to a wavefront in 4859 execution order. 4860* The LDS memory has multiple request queues shared by the SIMDs of a 4861 CU. Therefore, the LDS operations performed by different wavefronts of a 4862 work-group can be reordered relative to each other, which can result in 4863 reordering the visibility of vector memory operations with respect to LDS 4864 operations of other wavefronts in the same work-group. A ``s_waitcnt 4865 lgkmcnt(0)`` is required to ensure synchronization between LDS operations and 4866 vector memory operations between wavefronts of a work-group, but not between 4867 operations performed by the same wavefront. 4868* The vector memory operations are performed as wavefront wide operations and 4869 completion is reported to a wavefront in execution order. The exception is 4870 that for GFX7-GFX9 ``flat_load/store/atomic`` instructions can report out of 4871 vector memory order if they access LDS memory, and out of LDS operation order 4872 if they access global memory. 4873* The vector memory operations access a single vector L1 cache shared by all 4874 SIMDs a CU. Therefore, no special action is required for coherence between the 4875 lanes of a single wavefront, or for coherence between wavefronts in the same 4876 work-group. A ``buffer_wbinvl1_vol`` is required for coherence between 4877 wavefronts executing in different work-groups as they may be executing on 4878 different CUs. 4879* The scalar memory operations access a scalar L1 cache shared by all wavefronts 4880 on a group of CUs. The scalar and vector L1 caches are not coherent. However, 4881 scalar operations are used in a restricted way so do not impact the memory 4882 model. See :ref:`amdgpu-amdhsa-memory-spaces`. 4883* The vector and scalar memory operations use an L2 cache shared by all CUs on 4884 the same agent. 4885* The L2 cache has independent channels to service disjoint ranges of virtual 4886 addresses. 4887* Each CU has a separate request queue per channel. Therefore, the vector and 4888 scalar memory operations performed by wavefronts executing in different 4889 work-groups (which may be executing on different CUs) of an agent can be 4890 reordered relative to each other. A ``s_waitcnt vmcnt(0)`` is required to 4891 ensure synchronization between vector memory operations of different CUs. It 4892 ensures a previous vector memory operation has completed before executing a 4893 subsequent vector memory or LDS operation and so can be used to meet the 4894 requirements of acquire and release. 4895* The L2 cache can be kept coherent with other agents on some targets, or ranges 4896 of virtual addresses can be set up to bypass it to ensure system coherence. 4897 4898Scalar memory operations are only used to access memory that is proven to not 4899change during the execution of the kernel dispatch. This includes constant 4900address space and global address space for program scope ``const`` variables. 4901Therefore, the kernel machine code does not have to maintain the scalar cache to 4902ensure it is coherent with the vector caches. The scalar and vector caches are 4903invalidated between kernel dispatches by CP since constant address space data 4904may change between kernel dispatch executions. See 4905:ref:`amdgpu-amdhsa-memory-spaces`. 4906 4907The one exception is if scalar writes are used to spill SGPR registers. In this 4908case the AMDGPU backend ensures the memory location used to spill is never 4909accessed by vector memory operations at the same time. If scalar writes are used 4910then a ``s_dcache_wb`` is inserted before the ``s_endpgm`` and before a function 4911return since the locations may be used for vector memory instructions by a 4912future wavefront that uses the same scratch area, or a function call that 4913creates a frame at the same address, respectively. There is no need for a 4914``s_dcache_inv`` as all scalar writes are write-before-read in the same thread. 4915 4916For kernarg backing memory: 4917 4918* CP invalidates the L1 cache at the start of each kernel dispatch. 4919* On dGPU the kernarg backing memory is allocated in host memory accessed as 4920 MTYPE UC (uncached) to avoid needing to invalidate the L2 cache. This also 4921 causes it to be treated as non-volatile and so is not invalidated by 4922 ``*_vol``. 4923* On APU the kernarg backing memory it is accessed as MTYPE CC (cache coherent) 4924 and so the L2 cache will be coherent with the CPU and other agents. 4925 4926Scratch backing memory (which is used for the private address space) is accessed 4927with MTYPE NC_NV (non-coherent non-volatile). Since the private address space is 4928only accessed by a single thread, and is always write-before-read, there is 4929never a need to invalidate these entries from the L1 cache. Hence all cache 4930invalidates are done as ``*_vol`` to only invalidate the volatile cache lines. 4931 4932The code sequences used to implement the memory model for GFX6-GFX9 are defined 4933in table :ref:`amdgpu-amdhsa-memory-model-code-sequences-gfx6-gfx9-table`. 4934 4935 .. table:: AMDHSA Memory Model Code Sequences GFX6-GFX9 4936 :name: amdgpu-amdhsa-memory-model-code-sequences-gfx6-gfx9-table 4937 4938 ============ ============ ============== ========== ================================ 4939 LLVM Instr LLVM Memory LLVM Memory AMDGPU AMDGPU Machine Code 4940 Ordering Sync Scope Address GFX6-GFX9 4941 Space 4942 ============ ============ ============== ========== ================================ 4943 **Non-Atomic** 4944 ------------------------------------------------------------------------------------ 4945 load *none* *none* - global - !volatile & !nontemporal 4946 - generic 4947 - private 1. buffer/global/flat_load 4948 - constant 4949 - !volatile & nontemporal 4950 4951 1. buffer/global/flat_load 4952 glc=1 slc=1 4953 4954 - volatile 4955 4956 1. buffer/global/flat_load 4957 glc=1 4958 2. s_waitcnt vmcnt(0) 4959 4960 - Must happen before 4961 any following volatile 4962 global/generic 4963 load/store. 4964 - Ensures that 4965 volatile 4966 operations to 4967 different 4968 addresses will not 4969 be reordered by 4970 hardware. 4971 4972 load *none* *none* - local 1. ds_load 4973 store *none* *none* - global - !volatile & !nontemporal 4974 - generic 4975 - private 1. buffer/global/flat_store 4976 - constant 4977 - !volatile & nontemporal 4978 4979 1. buffer/global/flat_store 4980 glc=1 slc=1 4981 4982 - volatile 4983 4984 1. buffer/global/flat_store 4985 2. s_waitcnt vmcnt(0) 4986 4987 - Must happen before 4988 any following volatile 4989 global/generic 4990 load/store. 4991 - Ensures that 4992 volatile 4993 operations to 4994 different 4995 addresses will not 4996 be reordered by 4997 hardware. 4998 4999 store *none* *none* - local 1. ds_store 5000 **Unordered Atomic** 5001 ------------------------------------------------------------------------------------ 5002 load atomic unordered *any* *any* *Same as non-atomic*. 5003 store atomic unordered *any* *any* *Same as non-atomic*. 5004 atomicrmw unordered *any* *any* *Same as monotonic atomic*. 5005 **Monotonic Atomic** 5006 ------------------------------------------------------------------------------------ 5007 load atomic monotonic - singlethread - global 1. buffer/global/ds/flat_load 5008 - wavefront - local 5009 - workgroup - generic 5010 load atomic monotonic - agent - global 1. buffer/global/flat_load 5011 - system - generic glc=1 5012 store atomic monotonic - singlethread - global 1. buffer/global/flat_store 5013 - wavefront - generic 5014 - workgroup 5015 - agent 5016 - system 5017 store atomic monotonic - singlethread - local 1. ds_store 5018 - wavefront 5019 - workgroup 5020 atomicrmw monotonic - singlethread - global 1. buffer/global/flat_atomic 5021 - wavefront - generic 5022 - workgroup 5023 - agent 5024 - system 5025 atomicrmw monotonic - singlethread - local 1. ds_atomic 5026 - wavefront 5027 - workgroup 5028 **Acquire Atomic** 5029 ------------------------------------------------------------------------------------ 5030 load atomic acquire - singlethread - global 1. buffer/global/ds/flat_load 5031 - wavefront - local 5032 - generic 5033 load atomic acquire - workgroup - global 1. buffer/global_load 5034 load atomic acquire - workgroup - local 1. ds/flat_load 5035 - generic 2. s_waitcnt lgkmcnt(0) 5036 5037 - If OpenCL, omit. 5038 - Must happen before 5039 any following 5040 global/generic 5041 load/load 5042 atomic/store/store 5043 atomic/atomicrmw. 5044 - Ensures any 5045 following global 5046 data read is no 5047 older than a local load 5048 atomic value being 5049 acquired. 5050 5051 load atomic acquire - agent - global 1. buffer/global_load 5052 - system glc=1 5053 2. s_waitcnt vmcnt(0) 5054 5055 - Must happen before 5056 following 5057 buffer_wbinvl1_vol. 5058 - Ensures the load 5059 has completed 5060 before invalidating 5061 the cache. 5062 5063 3. buffer_wbinvl1_vol 5064 5065 - Must happen before 5066 any following 5067 global/generic 5068 load/load 5069 atomic/atomicrmw. 5070 - Ensures that 5071 following 5072 loads will not see 5073 stale global data. 5074 5075 load atomic acquire - agent - generic 1. flat_load glc=1 5076 - system 2. s_waitcnt vmcnt(0) & 5077 lgkmcnt(0) 5078 5079 - If OpenCL omit 5080 lgkmcnt(0). 5081 - Must happen before 5082 following 5083 buffer_wbinvl1_vol. 5084 - Ensures the flat_load 5085 has completed 5086 before invalidating 5087 the cache. 5088 5089 3. buffer_wbinvl1_vol 5090 5091 - Must happen before 5092 any following 5093 global/generic 5094 load/load 5095 atomic/atomicrmw. 5096 - Ensures that 5097 following loads 5098 will not see stale 5099 global data. 5100 5101 atomicrmw acquire - singlethread - global 1. buffer/global/ds/flat_atomic 5102 - wavefront - local 5103 - generic 5104 atomicrmw acquire - workgroup - global 1. buffer/global_atomic 5105 atomicrmw acquire - workgroup - local 1. ds/flat_atomic 5106 - generic 2. s_waitcnt lgkmcnt(0) 5107 5108 - If OpenCL, omit. 5109 - Must happen before 5110 any following 5111 global/generic 5112 load/load 5113 atomic/store/store 5114 atomic/atomicrmw. 5115 - Ensures any 5116 following global 5117 data read is no 5118 older than a local 5119 atomicrmw value 5120 being acquired. 5121 5122 atomicrmw acquire - agent - global 1. buffer/global_atomic 5123 - system 2. s_waitcnt vmcnt(0) 5124 5125 - Must happen before 5126 following 5127 buffer_wbinvl1_vol. 5128 - Ensures the 5129 atomicrmw has 5130 completed before 5131 invalidating the 5132 cache. 5133 5134 3. buffer_wbinvl1_vol 5135 5136 - Must happen before 5137 any following 5138 global/generic 5139 load/load 5140 atomic/atomicrmw. 5141 - Ensures that 5142 following loads 5143 will not see stale 5144 global data. 5145 5146 atomicrmw acquire - agent - generic 1. flat_atomic 5147 - system 2. s_waitcnt vmcnt(0) & 5148 lgkmcnt(0) 5149 5150 - If OpenCL, omit 5151 lgkmcnt(0). 5152 - Must happen before 5153 following 5154 buffer_wbinvl1_vol. 5155 - Ensures the 5156 atomicrmw has 5157 completed before 5158 invalidating the 5159 cache. 5160 5161 3. buffer_wbinvl1_vol 5162 5163 - Must happen before 5164 any following 5165 global/generic 5166 load/load 5167 atomic/atomicrmw. 5168 - Ensures that 5169 following loads 5170 will not see stale 5171 global data. 5172 5173 fence acquire - singlethread *none* *none* 5174 - wavefront 5175 fence acquire - workgroup *none* 1. s_waitcnt lgkmcnt(0) 5176 5177 - If OpenCL and 5178 address space is 5179 not generic, omit. 5180 - However, since LLVM 5181 currently has no 5182 address space on 5183 the fence need to 5184 conservatively 5185 always generate. If 5186 fence had an 5187 address space then 5188 set to address 5189 space of OpenCL 5190 fence flag, or to 5191 generic if both 5192 local and global 5193 flags are 5194 specified. 5195 - Must happen after 5196 any preceding 5197 local/generic load 5198 atomic/atomicrmw 5199 with an equal or 5200 wider sync scope 5201 and memory ordering 5202 stronger than 5203 unordered (this is 5204 termed the 5205 fence-paired-atomic). 5206 - Must happen before 5207 any following 5208 global/generic 5209 load/load 5210 atomic/store/store 5211 atomic/atomicrmw. 5212 - Ensures any 5213 following global 5214 data read is no 5215 older than the 5216 value read by the 5217 fence-paired-atomic. 5218 5219 fence acquire - agent *none* 1. s_waitcnt lgkmcnt(0) & 5220 - system vmcnt(0) 5221 5222 - If OpenCL and 5223 address space is 5224 not generic, omit 5225 lgkmcnt(0). 5226 - However, since LLVM 5227 currently has no 5228 address space on 5229 the fence need to 5230 conservatively 5231 always generate 5232 (see comment for 5233 previous fence). 5234 - Could be split into 5235 separate s_waitcnt 5236 vmcnt(0) and 5237 s_waitcnt 5238 lgkmcnt(0) to allow 5239 them to be 5240 independently moved 5241 according to the 5242 following rules. 5243 - s_waitcnt vmcnt(0) 5244 must happen after 5245 any preceding 5246 global/generic load 5247 atomic/atomicrmw 5248 with an equal or 5249 wider sync scope 5250 and memory ordering 5251 stronger than 5252 unordered (this is 5253 termed the 5254 fence-paired-atomic). 5255 - s_waitcnt lgkmcnt(0) 5256 must happen after 5257 any preceding 5258 local/generic load 5259 atomic/atomicrmw 5260 with an equal or 5261 wider sync scope 5262 and memory ordering 5263 stronger than 5264 unordered (this is 5265 termed the 5266 fence-paired-atomic). 5267 - Must happen before 5268 the following 5269 buffer_wbinvl1_vol. 5270 - Ensures that the 5271 fence-paired atomic 5272 has completed 5273 before invalidating 5274 the 5275 cache. Therefore 5276 any following 5277 locations read must 5278 be no older than 5279 the value read by 5280 the 5281 fence-paired-atomic. 5282 5283 2. buffer_wbinvl1_vol 5284 5285 - Must happen before any 5286 following global/generic 5287 load/load 5288 atomic/store/store 5289 atomic/atomicrmw. 5290 - Ensures that 5291 following loads 5292 will not see stale 5293 global data. 5294 5295 **Release Atomic** 5296 ------------------------------------------------------------------------------------ 5297 store atomic release - singlethread - global 1. buffer/global/ds/flat_store 5298 - wavefront - local 5299 - generic 5300 store atomic release - workgroup - global 1. s_waitcnt lgkmcnt(0) 5301 - generic 5302 - If OpenCL, omit. 5303 - Must happen after 5304 any preceding 5305 local/generic 5306 load/store/load 5307 atomic/store 5308 atomic/atomicrmw. 5309 - Must happen before 5310 the following 5311 store. 5312 - Ensures that all 5313 memory operations 5314 to local have 5315 completed before 5316 performing the 5317 store that is being 5318 released. 5319 5320 2. buffer/global/flat_store 5321 store atomic release - workgroup - local 1. ds_store 5322 store atomic release - agent - global 1. s_waitcnt lgkmcnt(0) & 5323 - system - generic vmcnt(0) 5324 5325 - If OpenCL and 5326 address space is 5327 not generic, omit 5328 lgkmcnt(0). 5329 - Could be split into 5330 separate s_waitcnt 5331 vmcnt(0) and 5332 s_waitcnt 5333 lgkmcnt(0) to allow 5334 them to be 5335 independently moved 5336 according to the 5337 following rules. 5338 - s_waitcnt vmcnt(0) 5339 must happen after 5340 any preceding 5341 global/generic 5342 load/store/load 5343 atomic/store 5344 atomic/atomicrmw. 5345 - s_waitcnt lgkmcnt(0) 5346 must happen after 5347 any preceding 5348 local/generic 5349 load/store/load 5350 atomic/store 5351 atomic/atomicrmw. 5352 - Must happen before 5353 the following 5354 store. 5355 - Ensures that all 5356 memory operations 5357 to memory have 5358 completed before 5359 performing the 5360 store that is being 5361 released. 5362 5363 2. buffer/global/flat_store 5364 atomicrmw release - singlethread - global 1. buffer/global/ds/flat_atomic 5365 - wavefront - local 5366 - generic 5367 atomicrmw release - workgroup - global 1. s_waitcnt lgkmcnt(0) 5368 - generic 5369 - If OpenCL, omit. 5370 - Must happen after 5371 any preceding 5372 local/generic 5373 load/store/load 5374 atomic/store 5375 atomic/atomicrmw. 5376 - Must happen before 5377 the following 5378 atomicrmw. 5379 - Ensures that all 5380 memory operations 5381 to local have 5382 completed before 5383 performing the 5384 atomicrmw that is 5385 being released. 5386 5387 2. buffer/global/flat_atomic 5388 atomicrmw release - workgroup - local 1. ds_atomic 5389 atomicrmw release - agent - global 1. s_waitcnt lgkmcnt(0) & 5390 - system - generic vmcnt(0) 5391 5392 - If OpenCL, omit 5393 lgkmcnt(0). 5394 - Could be split into 5395 separate s_waitcnt 5396 vmcnt(0) and 5397 s_waitcnt 5398 lgkmcnt(0) to allow 5399 them to be 5400 independently moved 5401 according to the 5402 following rules. 5403 - s_waitcnt vmcnt(0) 5404 must happen after 5405 any preceding 5406 global/generic 5407 load/store/load 5408 atomic/store 5409 atomic/atomicrmw. 5410 - s_waitcnt lgkmcnt(0) 5411 must happen after 5412 any preceding 5413 local/generic 5414 load/store/load 5415 atomic/store 5416 atomic/atomicrmw. 5417 - Must happen before 5418 the following 5419 atomicrmw. 5420 - Ensures that all 5421 memory operations 5422 to global and local 5423 have completed 5424 before performing 5425 the atomicrmw that 5426 is being released. 5427 5428 2. buffer/global/flat_atomic 5429 fence release - singlethread *none* *none* 5430 - wavefront 5431 fence release - workgroup *none* 1. s_waitcnt lgkmcnt(0) 5432 5433 - If OpenCL and 5434 address space is 5435 not generic, omit. 5436 - However, since LLVM 5437 currently has no 5438 address space on 5439 the fence need to 5440 conservatively 5441 always generate. If 5442 fence had an 5443 address space then 5444 set to address 5445 space of OpenCL 5446 fence flag, or to 5447 generic if both 5448 local and global 5449 flags are 5450 specified. 5451 - Must happen after 5452 any preceding 5453 local/generic 5454 load/load 5455 atomic/store/store 5456 atomic/atomicrmw. 5457 - Must happen before 5458 any following store 5459 atomic/atomicrmw 5460 with an equal or 5461 wider sync scope 5462 and memory ordering 5463 stronger than 5464 unordered (this is 5465 termed the 5466 fence-paired-atomic). 5467 - Ensures that all 5468 memory operations 5469 to local have 5470 completed before 5471 performing the 5472 following 5473 fence-paired-atomic. 5474 5475 fence release - agent *none* 1. s_waitcnt lgkmcnt(0) & 5476 - system vmcnt(0) 5477 5478 - If OpenCL and 5479 address space is 5480 not generic, omit 5481 lgkmcnt(0). 5482 - If OpenCL and 5483 address space is 5484 local, omit 5485 vmcnt(0). 5486 - However, since LLVM 5487 currently has no 5488 address space on 5489 the fence need to 5490 conservatively 5491 always generate. If 5492 fence had an 5493 address space then 5494 set to address 5495 space of OpenCL 5496 fence flag, or to 5497 generic if both 5498 local and global 5499 flags are 5500 specified. 5501 - Could be split into 5502 separate s_waitcnt 5503 vmcnt(0) and 5504 s_waitcnt 5505 lgkmcnt(0) to allow 5506 them to be 5507 independently moved 5508 according to the 5509 following rules. 5510 - s_waitcnt vmcnt(0) 5511 must happen after 5512 any preceding 5513 global/generic 5514 load/store/load 5515 atomic/store 5516 atomic/atomicrmw. 5517 - s_waitcnt lgkmcnt(0) 5518 must happen after 5519 any preceding 5520 local/generic 5521 load/store/load 5522 atomic/store 5523 atomic/atomicrmw. 5524 - Must happen before 5525 any following store 5526 atomic/atomicrmw 5527 with an equal or 5528 wider sync scope 5529 and memory ordering 5530 stronger than 5531 unordered (this is 5532 termed the 5533 fence-paired-atomic). 5534 - Ensures that all 5535 memory operations 5536 have 5537 completed before 5538 performing the 5539 following 5540 fence-paired-atomic. 5541 5542 **Acquire-Release Atomic** 5543 ------------------------------------------------------------------------------------ 5544 atomicrmw acq_rel - singlethread - global 1. buffer/global/ds/flat_atomic 5545 - wavefront - local 5546 - generic 5547 atomicrmw acq_rel - workgroup - global 1. s_waitcnt lgkmcnt(0) 5548 5549 - If OpenCL, omit. 5550 - Must happen after 5551 any preceding 5552 local/generic 5553 load/store/load 5554 atomic/store 5555 atomic/atomicrmw. 5556 - Must happen before 5557 the following 5558 atomicrmw. 5559 - Ensures that all 5560 memory operations 5561 to local have 5562 completed before 5563 performing the 5564 atomicrmw that is 5565 being released. 5566 5567 2. buffer/global_atomic 5568 5569 atomicrmw acq_rel - workgroup - local 1. ds_atomic 5570 2. s_waitcnt lgkmcnt(0) 5571 5572 - If OpenCL, omit. 5573 - Must happen before 5574 any following 5575 global/generic 5576 load/load 5577 atomic/store/store 5578 atomic/atomicrmw. 5579 - Ensures any 5580 following global 5581 data read is no 5582 older than the local load 5583 atomic value being 5584 acquired. 5585 5586 atomicrmw acq_rel - workgroup - generic 1. s_waitcnt lgkmcnt(0) 5587 5588 - If OpenCL, omit. 5589 - Must happen after 5590 any preceding 5591 local/generic 5592 load/store/load 5593 atomic/store 5594 atomic/atomicrmw. 5595 - Must happen before 5596 the following 5597 atomicrmw. 5598 - Ensures that all 5599 memory operations 5600 to local have 5601 completed before 5602 performing the 5603 atomicrmw that is 5604 being released. 5605 5606 2. flat_atomic 5607 3. s_waitcnt lgkmcnt(0) 5608 5609 - If OpenCL, omit. 5610 - Must happen before 5611 any following 5612 global/generic 5613 load/load 5614 atomic/store/store 5615 atomic/atomicrmw. 5616 - Ensures any 5617 following global 5618 data read is no 5619 older than a local load 5620 atomic value being 5621 acquired. 5622 5623 atomicrmw acq_rel - agent - global 1. s_waitcnt lgkmcnt(0) & 5624 - system vmcnt(0) 5625 5626 - If OpenCL, omit 5627 lgkmcnt(0). 5628 - Could be split into 5629 separate s_waitcnt 5630 vmcnt(0) and 5631 s_waitcnt 5632 lgkmcnt(0) to allow 5633 them to be 5634 independently moved 5635 according to the 5636 following rules. 5637 - s_waitcnt vmcnt(0) 5638 must happen after 5639 any preceding 5640 global/generic 5641 load/store/load 5642 atomic/store 5643 atomic/atomicrmw. 5644 - s_waitcnt lgkmcnt(0) 5645 must happen after 5646 any preceding 5647 local/generic 5648 load/store/load 5649 atomic/store 5650 atomic/atomicrmw. 5651 - Must happen before 5652 the following 5653 atomicrmw. 5654 - Ensures that all 5655 memory operations 5656 to global have 5657 completed before 5658 performing the 5659 atomicrmw that is 5660 being released. 5661 5662 2. buffer/global_atomic 5663 3. s_waitcnt vmcnt(0) 5664 5665 - Must happen before 5666 following 5667 buffer_wbinvl1_vol. 5668 - Ensures the 5669 atomicrmw has 5670 completed before 5671 invalidating the 5672 cache. 5673 5674 4. buffer_wbinvl1_vol 5675 5676 - Must happen before 5677 any following 5678 global/generic 5679 load/load 5680 atomic/atomicrmw. 5681 - Ensures that 5682 following loads 5683 will not see stale 5684 global data. 5685 5686 atomicrmw acq_rel - agent - generic 1. s_waitcnt lgkmcnt(0) & 5687 - system vmcnt(0) 5688 5689 - If OpenCL, omit 5690 lgkmcnt(0). 5691 - Could be split into 5692 separate s_waitcnt 5693 vmcnt(0) and 5694 s_waitcnt 5695 lgkmcnt(0) to allow 5696 them to be 5697 independently moved 5698 according to the 5699 following rules. 5700 - s_waitcnt vmcnt(0) 5701 must happen after 5702 any preceding 5703 global/generic 5704 load/store/load 5705 atomic/store 5706 atomic/atomicrmw. 5707 - s_waitcnt lgkmcnt(0) 5708 must happen after 5709 any preceding 5710 local/generic 5711 load/store/load 5712 atomic/store 5713 atomic/atomicrmw. 5714 - Must happen before 5715 the following 5716 atomicrmw. 5717 - Ensures that all 5718 memory operations 5719 to global have 5720 completed before 5721 performing the 5722 atomicrmw that is 5723 being released. 5724 5725 2. flat_atomic 5726 3. s_waitcnt vmcnt(0) & 5727 lgkmcnt(0) 5728 5729 - If OpenCL, omit 5730 lgkmcnt(0). 5731 - Must happen before 5732 following 5733 buffer_wbinvl1_vol. 5734 - Ensures the 5735 atomicrmw has 5736 completed before 5737 invalidating the 5738 cache. 5739 5740 4. buffer_wbinvl1_vol 5741 5742 - Must happen before 5743 any following 5744 global/generic 5745 load/load 5746 atomic/atomicrmw. 5747 - Ensures that 5748 following loads 5749 will not see stale 5750 global data. 5751 5752 fence acq_rel - singlethread *none* *none* 5753 - wavefront 5754 fence acq_rel - workgroup *none* 1. s_waitcnt lgkmcnt(0) 5755 5756 - If OpenCL and 5757 address space is 5758 not generic, omit. 5759 - However, 5760 since LLVM 5761 currently has no 5762 address space on 5763 the fence need to 5764 conservatively 5765 always generate 5766 (see comment for 5767 previous fence). 5768 - Must happen after 5769 any preceding 5770 local/generic 5771 load/load 5772 atomic/store/store 5773 atomic/atomicrmw. 5774 - Must happen before 5775 any following 5776 global/generic 5777 load/load 5778 atomic/store/store 5779 atomic/atomicrmw. 5780 - Ensures that all 5781 memory operations 5782 to local have 5783 completed before 5784 performing any 5785 following global 5786 memory operations. 5787 - Ensures that the 5788 preceding 5789 local/generic load 5790 atomic/atomicrmw 5791 with an equal or 5792 wider sync scope 5793 and memory ordering 5794 stronger than 5795 unordered (this is 5796 termed the 5797 acquire-fence-paired-atomic) 5798 has completed 5799 before following 5800 global memory 5801 operations. This 5802 satisfies the 5803 requirements of 5804 acquire. 5805 - Ensures that all 5806 previous memory 5807 operations have 5808 completed before a 5809 following 5810 local/generic store 5811 atomic/atomicrmw 5812 with an equal or 5813 wider sync scope 5814 and memory ordering 5815 stronger than 5816 unordered (this is 5817 termed the 5818 release-fence-paired-atomic). 5819 This satisfies the 5820 requirements of 5821 release. 5822 5823 fence acq_rel - agent *none* 1. s_waitcnt lgkmcnt(0) & 5824 - system vmcnt(0) 5825 5826 - If OpenCL and 5827 address space is 5828 not generic, omit 5829 lgkmcnt(0). 5830 - However, since LLVM 5831 currently has no 5832 address space on 5833 the fence need to 5834 conservatively 5835 always generate 5836 (see comment for 5837 previous fence). 5838 - Could be split into 5839 separate s_waitcnt 5840 vmcnt(0) and 5841 s_waitcnt 5842 lgkmcnt(0) to allow 5843 them to be 5844 independently moved 5845 according to the 5846 following rules. 5847 - s_waitcnt vmcnt(0) 5848 must happen after 5849 any preceding 5850 global/generic 5851 load/store/load 5852 atomic/store 5853 atomic/atomicrmw. 5854 - s_waitcnt lgkmcnt(0) 5855 must happen after 5856 any preceding 5857 local/generic 5858 load/store/load 5859 atomic/store 5860 atomic/atomicrmw. 5861 - Must happen before 5862 the following 5863 buffer_wbinvl1_vol. 5864 - Ensures that the 5865 preceding 5866 global/local/generic 5867 load 5868 atomic/atomicrmw 5869 with an equal or 5870 wider sync scope 5871 and memory ordering 5872 stronger than 5873 unordered (this is 5874 termed the 5875 acquire-fence-paired-atomic) 5876 has completed 5877 before invalidating 5878 the cache. This 5879 satisfies the 5880 requirements of 5881 acquire. 5882 - Ensures that all 5883 previous memory 5884 operations have 5885 completed before a 5886 following 5887 global/local/generic 5888 store 5889 atomic/atomicrmw 5890 with an equal or 5891 wider sync scope 5892 and memory ordering 5893 stronger than 5894 unordered (this is 5895 termed the 5896 release-fence-paired-atomic). 5897 This satisfies the 5898 requirements of 5899 release. 5900 5901 2. buffer_wbinvl1_vol 5902 5903 - Must happen before 5904 any following 5905 global/generic 5906 load/load 5907 atomic/store/store 5908 atomic/atomicrmw. 5909 - Ensures that 5910 following loads 5911 will not see stale 5912 global data. This 5913 satisfies the 5914 requirements of 5915 acquire. 5916 5917 **Sequential Consistent Atomic** 5918 ------------------------------------------------------------------------------------ 5919 load atomic seq_cst - singlethread - global *Same as corresponding 5920 - wavefront - local load atomic acquire, 5921 - generic except must generated 5922 all instructions even 5923 for OpenCL.* 5924 load atomic seq_cst - workgroup - global 1. s_waitcnt lgkmcnt(0) 5925 - generic 5926 5927 - Must 5928 happen after 5929 preceding 5930 local/generic load 5931 atomic/store 5932 atomic/atomicrmw 5933 with memory 5934 ordering of seq_cst 5935 and with equal or 5936 wider sync scope. 5937 (Note that seq_cst 5938 fences have their 5939 own s_waitcnt 5940 lgkmcnt(0) and so do 5941 not need to be 5942 considered.) 5943 - Ensures any 5944 preceding 5945 sequential 5946 consistent local 5947 memory instructions 5948 have completed 5949 before executing 5950 this sequentially 5951 consistent 5952 instruction. This 5953 prevents reordering 5954 a seq_cst store 5955 followed by a 5956 seq_cst load. (Note 5957 that seq_cst is 5958 stronger than 5959 acquire/release as 5960 the reordering of 5961 load acquire 5962 followed by a store 5963 release is 5964 prevented by the 5965 s_waitcnt of 5966 the release, but 5967 there is nothing 5968 preventing a store 5969 release followed by 5970 load acquire from 5971 completing out of 5972 order. The s_waitcnt 5973 could be placed after 5974 seq_store or before 5975 the seq_load. We 5976 choose the load to 5977 make the s_waitcnt be 5978 as late as possible 5979 so that the store 5980 may have already 5981 completed.) 5982 5983 2. *Following 5984 instructions same as 5985 corresponding load 5986 atomic acquire, 5987 except must generated 5988 all instructions even 5989 for OpenCL.* 5990 load atomic seq_cst - workgroup - local *Same as corresponding 5991 load atomic acquire, 5992 except must generated 5993 all instructions even 5994 for OpenCL.* 5995 5996 load atomic seq_cst - agent - global 1. s_waitcnt lgkmcnt(0) & 5997 - system - generic vmcnt(0) 5998 5999 - Could be split into 6000 separate s_waitcnt 6001 vmcnt(0) 6002 and s_waitcnt 6003 lgkmcnt(0) to allow 6004 them to be 6005 independently moved 6006 according to the 6007 following rules. 6008 - s_waitcnt lgkmcnt(0) 6009 must happen after 6010 preceding 6011 global/generic load 6012 atomic/store 6013 atomic/atomicrmw 6014 with memory 6015 ordering of seq_cst 6016 and with equal or 6017 wider sync scope. 6018 (Note that seq_cst 6019 fences have their 6020 own s_waitcnt 6021 lgkmcnt(0) and so do 6022 not need to be 6023 considered.) 6024 - s_waitcnt vmcnt(0) 6025 must happen after 6026 preceding 6027 global/generic load 6028 atomic/store 6029 atomic/atomicrmw 6030 with memory 6031 ordering of seq_cst 6032 and with equal or 6033 wider sync scope. 6034 (Note that seq_cst 6035 fences have their 6036 own s_waitcnt 6037 vmcnt(0) and so do 6038 not need to be 6039 considered.) 6040 - Ensures any 6041 preceding 6042 sequential 6043 consistent global 6044 memory instructions 6045 have completed 6046 before executing 6047 this sequentially 6048 consistent 6049 instruction. This 6050 prevents reordering 6051 a seq_cst store 6052 followed by a 6053 seq_cst load. (Note 6054 that seq_cst is 6055 stronger than 6056 acquire/release as 6057 the reordering of 6058 load acquire 6059 followed by a store 6060 release is 6061 prevented by the 6062 s_waitcnt of 6063 the release, but 6064 there is nothing 6065 preventing a store 6066 release followed by 6067 load acquire from 6068 completing out of 6069 order. The s_waitcnt 6070 could be placed after 6071 seq_store or before 6072 the seq_load. We 6073 choose the load to 6074 make the s_waitcnt be 6075 as late as possible 6076 so that the store 6077 may have already 6078 completed.) 6079 6080 2. *Following 6081 instructions same as 6082 corresponding load 6083 atomic acquire, 6084 except must generated 6085 all instructions even 6086 for OpenCL.* 6087 store atomic seq_cst - singlethread - global *Same as corresponding 6088 - wavefront - local store atomic release, 6089 - workgroup - generic except must generated 6090 - agent all instructions even 6091 - system for OpenCL.* 6092 atomicrmw seq_cst - singlethread - global *Same as corresponding 6093 - wavefront - local atomicrmw acq_rel, 6094 - workgroup - generic except must generated 6095 - agent all instructions even 6096 - system for OpenCL.* 6097 fence seq_cst - singlethread *none* *Same as corresponding 6098 - wavefront fence acq_rel, 6099 - workgroup except must generated 6100 - agent all instructions even 6101 - system for OpenCL.* 6102 ============ ============ ============== ========== ================================ 6103 6104.. _amdgpu-amdhsa-memory-model-gfx90a: 6105 6106Memory Model GFX90A 6107+++++++++++++++++++ 6108 6109For GFX90A: 6110 6111* Each agent has multiple shader arrays (SA). 6112* Each SA has multiple compute units (CU). 6113* Each CU has multiple SIMDs that execute wavefronts. 6114* The wavefronts for a single work-group are executed in the same CU but may be 6115 executed by different SIMDs. The exception is when in tgsplit execution mode 6116 when the wavefronts may be executed by different SIMDs in different CUs. 6117* Each CU has a single LDS memory shared by the wavefronts of the work-groups 6118 executing on it. The exception is when in tgsplit execution mode when no LDS 6119 is allocated as wavefronts of the same work-group can be in different CUs. 6120* All LDS operations of a CU are performed as wavefront wide operations in a 6121 global order and involve no caching. Completion is reported to a wavefront in 6122 execution order. 6123* The LDS memory has multiple request queues shared by the SIMDs of a 6124 CU. Therefore, the LDS operations performed by different wavefronts of a 6125 work-group can be reordered relative to each other, which can result in 6126 reordering the visibility of vector memory operations with respect to LDS 6127 operations of other wavefronts in the same work-group. A ``s_waitcnt 6128 lgkmcnt(0)`` is required to ensure synchronization between LDS operations and 6129 vector memory operations between wavefronts of a work-group, but not between 6130 operations performed by the same wavefront. 6131* The vector memory operations are performed as wavefront wide operations and 6132 completion is reported to a wavefront in execution order. The exception is 6133 that ``flat_load/store/atomic`` instructions can report out of vector memory 6134 order if they access LDS memory, and out of LDS operation order if they access 6135 global memory. 6136* The vector memory operations access a single vector L1 cache shared by all 6137 SIMDs a CU. Therefore: 6138 6139 * No special action is required for coherence between the lanes of a single 6140 wavefront. 6141 6142 * No special action is required for coherence between wavefronts in the same 6143 work-group since they execute on the same CU. The exception is when in 6144 tgsplit execution mode as wavefronts of the same work-group can be in 6145 different CUs and so a ``buffer_wbinvl1_vol`` is required as described in 6146 the following item. 6147 6148 * A ``buffer_wbinvl1_vol`` is required for coherence between wavefronts 6149 executing in different work-groups as they may be executing on different 6150 CUs. 6151 6152* The scalar memory operations access a scalar L1 cache shared by all wavefronts 6153 on a group of CUs. The scalar and vector L1 caches are not coherent. However, 6154 scalar operations are used in a restricted way so do not impact the memory 6155 model. See :ref:`amdgpu-amdhsa-memory-spaces`. 6156* The vector and scalar memory operations use an L2 cache shared by all CUs on 6157 the same agent. 6158 6159 * The L2 cache has independent channels to service disjoint ranges of virtual 6160 addresses. 6161 * Each CU has a separate request queue per channel. Therefore, the vector and 6162 scalar memory operations performed by wavefronts executing in different 6163 work-groups (which may be executing on different CUs), or the same 6164 work-group if executing in tgsplit mode, of an agent can be reordered 6165 relative to each other. A ``s_waitcnt vmcnt(0)`` is required to ensure 6166 synchronization between vector memory operations of different CUs. It 6167 ensures a previous vector memory operation has completed before executing a 6168 subsequent vector memory or LDS operation and so can be used to meet the 6169 requirements of acquire and release. 6170 * The L2 cache of one agent can be kept coherent with other agents by: 6171 using the MTYPE RW (read-write) or MTYPE CC (cache-coherent) with the PTE 6172 C-bit for memory local to the L2; and using the MTYPE NC (non-coherent) with 6173 the PTE C-bit set or MTYPE UC (uncached) for memory not local to the L2. 6174 6175 * Any local memory cache lines will be automatically invalidated by writes 6176 from CUs associated with other L2 caches, or writes from the CPU, due to 6177 the cache probe caused by coherent requests. Coherent requests are caused 6178 by GPU accesses to pages with the PTE C-bit set, by CPU accesses over 6179 XGMI, and by PCIe requests that are configured to be coherent requests. 6180 * XGMI accesses from the CPU to local memory may be cached on the CPU. 6181 Subsequent access from the GPU will automatically invalidate or writeback 6182 the CPU cache due to the L2 probe filter and and the PTE C-bit being set. 6183 * Since all work-groups on the same agent share the same L2, no L2 6184 invalidation or writeback is required for coherence. 6185 * To ensure coherence of local and remote memory writes of work-groups in 6186 different agents a ``buffer_wbl2`` is required. It will writeback dirty L2 6187 cache lines of MTYPE RW (used for local coarse grain memory) and MTYPE NC 6188 ()used for remote coarse grain memory). Note that MTYPE CC (used for local 6189 fine grain memory) causes write through to DRAM, and MTYPE UC (used for 6190 remote fine grain memory) bypasses the L2, so both will never result in 6191 dirty L2 cache lines. 6192 * To ensure coherence of local and remote memory reads of work-groups in 6193 different agents a ``buffer_invl2`` is required. It will invalidate L2 6194 cache lines with MTYPE NC (used for remote coarse grain memory). Note that 6195 MTYPE CC (used for local fine grain memory) and MTYPE RW (used for local 6196 coarse memory) cause local reads to be invalidated by remote writes with 6197 with the PTE C-bit so these cache lines are not invalidated. Note that 6198 MTYPE UC (used for remote fine grain memory) bypasses the L2, so will 6199 never result in L2 cache lines that need to be invalidated. 6200 6201 * PCIe access from the GPU to the CPU memory is kept coherent by using the 6202 MTYPE UC (uncached) which bypasses the L2. 6203 6204Scalar memory operations are only used to access memory that is proven to not 6205change during the execution of the kernel dispatch. This includes constant 6206address space and global address space for program scope ``const`` variables. 6207Therefore, the kernel machine code does not have to maintain the scalar cache to 6208ensure it is coherent with the vector caches. The scalar and vector caches are 6209invalidated between kernel dispatches by CP since constant address space data 6210may change between kernel dispatch executions. See 6211:ref:`amdgpu-amdhsa-memory-spaces`. 6212 6213The one exception is if scalar writes are used to spill SGPR registers. In this 6214case the AMDGPU backend ensures the memory location used to spill is never 6215accessed by vector memory operations at the same time. If scalar writes are used 6216then a ``s_dcache_wb`` is inserted before the ``s_endpgm`` and before a function 6217return since the locations may be used for vector memory instructions by a 6218future wavefront that uses the same scratch area, or a function call that 6219creates a frame at the same address, respectively. There is no need for a 6220``s_dcache_inv`` as all scalar writes are write-before-read in the same thread. 6221 6222For kernarg backing memory: 6223 6224* CP invalidates the L1 cache at the start of each kernel dispatch. 6225* On dGPU over XGMI or PCIe the kernarg backing memory is allocated in host 6226 memory accessed as MTYPE UC (uncached) to avoid needing to invalidate the L2 6227 cache. This also causes it to be treated as non-volatile and so is not 6228 invalidated by ``*_vol``. 6229* On APU the kernarg backing memory is accessed as MTYPE CC (cache coherent) and 6230 so the L2 cache will be coherent with the CPU and other agents. 6231 6232Scratch backing memory (which is used for the private address space) is accessed 6233with MTYPE NC_NV (non-coherent non-volatile). Since the private address space is 6234only accessed by a single thread, and is always write-before-read, there is 6235never a need to invalidate these entries from the L1 cache. Hence all cache 6236invalidates are done as ``*_vol`` to only invalidate the volatile cache lines. 6237 6238The code sequences used to implement the memory model for GFX90A are defined 6239in table :ref:`amdgpu-amdhsa-memory-model-code-sequences-gfx90a-table`. 6240 6241 .. table:: AMDHSA Memory Model Code Sequences GFX90A 6242 :name: amdgpu-amdhsa-memory-model-code-sequences-gfx90a-table 6243 6244 ============ ============ ============== ========== ================================ 6245 LLVM Instr LLVM Memory LLVM Memory AMDGPU AMDGPU Machine Code 6246 Ordering Sync Scope Address GFX90A 6247 Space 6248 ============ ============ ============== ========== ================================ 6249 **Non-Atomic** 6250 ------------------------------------------------------------------------------------ 6251 load *none* *none* - global - !volatile & !nontemporal 6252 - generic 6253 - private 1. buffer/global/flat_load 6254 - constant 6255 - !volatile & nontemporal 6256 6257 1. buffer/global/flat_load 6258 glc=1 slc=1 6259 6260 - volatile 6261 6262 1. buffer/global/flat_load 6263 glc=1 6264 2. s_waitcnt vmcnt(0) 6265 6266 - Must happen before 6267 any following volatile 6268 global/generic 6269 load/store. 6270 - Ensures that 6271 volatile 6272 operations to 6273 different 6274 addresses will not 6275 be reordered by 6276 hardware. 6277 6278 load *none* *none* - local 1. ds_load 6279 store *none* *none* - global - !volatile & !nontemporal 6280 - generic 6281 - private 1. buffer/global/flat_store 6282 - constant 6283 - !volatile & nontemporal 6284 6285 1. buffer/global/flat_store 6286 glc=1 slc=1 6287 6288 - volatile 6289 6290 1. buffer/global/flat_store 6291 2. s_waitcnt vmcnt(0) 6292 6293 - Must happen before 6294 any following volatile 6295 global/generic 6296 load/store. 6297 - Ensures that 6298 volatile 6299 operations to 6300 different 6301 addresses will not 6302 be reordered by 6303 hardware. 6304 6305 store *none* *none* - local 1. ds_store 6306 **Unordered Atomic** 6307 ------------------------------------------------------------------------------------ 6308 load atomic unordered *any* *any* *Same as non-atomic*. 6309 store atomic unordered *any* *any* *Same as non-atomic*. 6310 atomicrmw unordered *any* *any* *Same as monotonic atomic*. 6311 **Monotonic Atomic** 6312 ------------------------------------------------------------------------------------ 6313 load atomic monotonic - singlethread - global 1. buffer/global/flat_load 6314 - wavefront - generic 6315 load atomic monotonic - workgroup - global 1. buffer/global/flat_load 6316 - generic glc=1 6317 6318 - If not TgSplit execution 6319 mode, omit glc=1. 6320 6321 load atomic monotonic - singlethread - local *If TgSplit execution mode, 6322 - wavefront local address space cannot 6323 - workgroup be used.* 6324 6325 1. ds_load 6326 load atomic monotonic - agent - global 1. buffer/global/flat_load 6327 - generic glc=1 6328 load atomic monotonic - system - global 1. buffer/global/flat_load 6329 - generic glc=1 6330 store atomic monotonic - singlethread - global 1. buffer/global/flat_store 6331 - wavefront - generic 6332 - workgroup 6333 - agent 6334 store atomic monotonic - system - global 1. buffer/global/flat_store 6335 - generic 6336 store atomic monotonic - singlethread - local *If TgSplit execution mode, 6337 - wavefront local address space cannot 6338 - workgroup be used.* 6339 6340 1. ds_store 6341 atomicrmw monotonic - singlethread - global 1. buffer/global/flat_atomic 6342 - wavefront - generic 6343 - workgroup 6344 - agent 6345 atomicrmw monotonic - system - global 1. buffer/global/flat_atomic 6346 - generic 6347 atomicrmw monotonic - singlethread - local *If TgSplit execution mode, 6348 - wavefront local address space cannot 6349 - workgroup be used.* 6350 6351 1. ds_atomic 6352 **Acquire Atomic** 6353 ------------------------------------------------------------------------------------ 6354 load atomic acquire - singlethread - global 1. buffer/global/ds/flat_load 6355 - wavefront - local 6356 - generic 6357 load atomic acquire - workgroup - global 1. buffer/global_load glc=1 6358 6359 - If not TgSplit execution 6360 mode, omit glc=1. 6361 6362 2. s_waitcnt vmcnt(0) 6363 6364 - If not TgSplit execution 6365 mode, omit. 6366 - Must happen before the 6367 following buffer_wbinvl1_vol. 6368 6369 3. buffer_wbinvl1_vol 6370 6371 - If not TgSplit execution 6372 mode, omit. 6373 - Must happen before 6374 any following 6375 global/generic 6376 load/load 6377 atomic/store/store 6378 atomic/atomicrmw. 6379 - Ensures that 6380 following 6381 loads will not see 6382 stale data. 6383 6384 load atomic acquire - workgroup - local *If TgSplit execution mode, 6385 local address space cannot 6386 be used.* 6387 6388 1. ds_load 6389 2. s_waitcnt lgkmcnt(0) 6390 6391 - If OpenCL, omit. 6392 - Must happen before 6393 any following 6394 global/generic 6395 load/load 6396 atomic/store/store 6397 atomic/atomicrmw. 6398 - Ensures any 6399 following global 6400 data read is no 6401 older than the local load 6402 atomic value being 6403 acquired. 6404 6405 load atomic acquire - workgroup - generic 1. flat_load glc=1 6406 6407 - If not TgSplit execution 6408 mode, omit glc=1. 6409 6410 2. s_waitcnt lgkm/vmcnt(0) 6411 6412 - Use lgkmcnt(0) if not 6413 TgSplit execution mode 6414 and vmcnt(0) if TgSplit 6415 execution mode. 6416 - If OpenCL, omit lgkmcnt(0). 6417 - Must happen before 6418 the following 6419 buffer_wbinvl1_vol and any 6420 following global/generic 6421 load/load 6422 atomic/store/store 6423 atomic/atomicrmw. 6424 - Ensures any 6425 following global 6426 data read is no 6427 older than a local load 6428 atomic value being 6429 acquired. 6430 6431 3. buffer_wbinvl1_vol 6432 6433 - If not TgSplit execution 6434 mode, omit. 6435 - Ensures that 6436 following 6437 loads will not see 6438 stale data. 6439 6440 load atomic acquire - agent - global 1. buffer/global_load 6441 glc=1 6442 2. s_waitcnt vmcnt(0) 6443 6444 - Must happen before 6445 following 6446 buffer_wbinvl1_vol. 6447 - Ensures the load 6448 has completed 6449 before invalidating 6450 the cache. 6451 6452 3. buffer_wbinvl1_vol 6453 6454 - Must happen before 6455 any following 6456 global/generic 6457 load/load 6458 atomic/atomicrmw. 6459 - Ensures that 6460 following 6461 loads will not see 6462 stale global data. 6463 6464 load atomic acquire - system - global 1. buffer/global/flat_load 6465 glc=1 6466 2. s_waitcnt vmcnt(0) 6467 6468 - Must happen before 6469 following buffer_invl2 and 6470 buffer_wbinvl1_vol. 6471 - Ensures the load 6472 has completed 6473 before invalidating 6474 the cache. 6475 6476 3. buffer_invl2; 6477 buffer_wbinvl1_vol 6478 6479 - Must happen before 6480 any following 6481 global/generic 6482 load/load 6483 atomic/atomicrmw. 6484 - Ensures that 6485 following 6486 loads will not see 6487 stale L1 global data, 6488 nor see stale L2 MTYPE 6489 NC global data. 6490 MTYPE RW and CC memory will 6491 never be stale in L2 due to 6492 the memory probes. 6493 6494 load atomic acquire - agent - generic 1. flat_load glc=1 6495 2. s_waitcnt vmcnt(0) & 6496 lgkmcnt(0) 6497 6498 - If TgSplit execution mode, 6499 omit lgkmcnt(0). 6500 - If OpenCL omit 6501 lgkmcnt(0). 6502 - Must happen before 6503 following 6504 buffer_wbinvl1_vol. 6505 - Ensures the flat_load 6506 has completed 6507 before invalidating 6508 the cache. 6509 6510 3. buffer_wbinvl1_vol 6511 6512 - Must happen before 6513 any following 6514 global/generic 6515 load/load 6516 atomic/atomicrmw. 6517 - Ensures that 6518 following loads 6519 will not see stale 6520 global data. 6521 6522 load atomic acquire - system - generic 1. flat_load glc=1 6523 2. s_waitcnt vmcnt(0) & 6524 lgkmcnt(0) 6525 6526 - If TgSplit execution mode, 6527 omit lgkmcnt(0). 6528 - If OpenCL omit 6529 lgkmcnt(0). 6530 - Must happen before 6531 following 6532 buffer_invl2 and 6533 buffer_wbinvl1_vol. 6534 - Ensures the flat_load 6535 has completed 6536 before invalidating 6537 the caches. 6538 6539 3. buffer_invl2; 6540 buffer_wbinvl1_vol 6541 6542 - Must happen before 6543 any following 6544 global/generic 6545 load/load 6546 atomic/atomicrmw. 6547 - Ensures that 6548 following 6549 loads will not see 6550 stale L1 global data, 6551 nor see stale L2 MTYPE 6552 NC global data. 6553 MTYPE RW and CC memory will 6554 never be stale in L2 due to 6555 the memory probes. 6556 6557 atomicrmw acquire - singlethread - global 1. buffer/global/flat_atomic 6558 - wavefront - generic 6559 atomicrmw acquire - singlethread - local *If TgSplit execution mode, 6560 - wavefront local address space cannot 6561 be used.* 6562 6563 1. ds_atomic 6564 atomicrmw acquire - workgroup - global 1. buffer/global_atomic 6565 2. s_waitcnt vmcnt(0) 6566 6567 - If not TgSplit execution 6568 mode, omit. 6569 - Must happen before the 6570 following buffer_wbinvl1_vol. 6571 - Ensures the atomicrmw 6572 has completed 6573 before invalidating 6574 the cache. 6575 6576 3. buffer_wbinvl1_vol 6577 6578 - If not TgSplit execution 6579 mode, omit. 6580 - Must happen before 6581 any following 6582 global/generic 6583 load/load 6584 atomic/atomicrmw. 6585 - Ensures that 6586 following loads 6587 will not see stale 6588 global data. 6589 6590 atomicrmw acquire - workgroup - local *If TgSplit execution mode, 6591 local address space cannot 6592 be used.* 6593 6594 1. ds_atomic 6595 2. s_waitcnt lgkmcnt(0) 6596 6597 - If OpenCL, omit. 6598 - Must happen before 6599 any following 6600 global/generic 6601 load/load 6602 atomic/store/store 6603 atomic/atomicrmw. 6604 - Ensures any 6605 following global 6606 data read is no 6607 older than the local 6608 atomicrmw value 6609 being acquired. 6610 6611 atomicrmw acquire - workgroup - generic 1. flat_atomic 6612 2. s_waitcnt lgkm/vmcnt(0) 6613 6614 - Use lgkmcnt(0) if not 6615 TgSplit execution mode 6616 and vmcnt(0) if TgSplit 6617 execution mode. 6618 - If OpenCL, omit lgkmcnt(0). 6619 - Must happen before 6620 the following 6621 buffer_wbinvl1_vol and 6622 any following 6623 global/generic 6624 load/load 6625 atomic/store/store 6626 atomic/atomicrmw. 6627 - Ensures any 6628 following global 6629 data read is no 6630 older than a local 6631 atomicrmw value 6632 being acquired. 6633 6634 3. buffer_wbinvl1_vol 6635 6636 - If not TgSplit execution 6637 mode, omit. 6638 - Ensures that 6639 following 6640 loads will not see 6641 stale data. 6642 6643 atomicrmw acquire - agent - global 1. buffer/global_atomic 6644 2. s_waitcnt vmcnt(0) 6645 6646 - Must happen before 6647 following 6648 buffer_wbinvl1_vol. 6649 - Ensures the 6650 atomicrmw has 6651 completed before 6652 invalidating the 6653 cache. 6654 6655 3. buffer_wbinvl1_vol 6656 6657 - Must happen before 6658 any following 6659 global/generic 6660 load/load 6661 atomic/atomicrmw. 6662 - Ensures that 6663 following loads 6664 will not see stale 6665 global data. 6666 6667 atomicrmw acquire - system - global 1. buffer/global_atomic 6668 2. s_waitcnt vmcnt(0) 6669 6670 - Must happen before 6671 following buffer_invl2 and 6672 buffer_wbinvl1_vol. 6673 - Ensures the 6674 atomicrmw has 6675 completed before 6676 invalidating the 6677 caches. 6678 6679 3. buffer_invl2; 6680 buffer_wbinvl1_vol 6681 6682 - Must happen before 6683 any following 6684 global/generic 6685 load/load 6686 atomic/atomicrmw. 6687 - Ensures that 6688 following 6689 loads will not see 6690 stale L1 global data, 6691 nor see stale L2 MTYPE 6692 NC global data. 6693 MTYPE RW and CC memory will 6694 never be stale in L2 due to 6695 the memory probes. 6696 6697 atomicrmw acquire - agent - generic 1. flat_atomic 6698 2. s_waitcnt vmcnt(0) & 6699 lgkmcnt(0) 6700 6701 - If TgSplit execution mode, 6702 omit lgkmcnt(0). 6703 - If OpenCL, omit 6704 lgkmcnt(0). 6705 - Must happen before 6706 following 6707 buffer_wbinvl1_vol. 6708 - Ensures the 6709 atomicrmw has 6710 completed before 6711 invalidating the 6712 cache. 6713 6714 3. buffer_wbinvl1_vol 6715 6716 - Must happen before 6717 any following 6718 global/generic 6719 load/load 6720 atomic/atomicrmw. 6721 - Ensures that 6722 following loads 6723 will not see stale 6724 global data. 6725 6726 atomicrmw acquire - system - generic 1. flat_atomic 6727 2. s_waitcnt vmcnt(0) & 6728 lgkmcnt(0) 6729 6730 - If TgSplit execution mode, 6731 omit lgkmcnt(0). 6732 - If OpenCL, omit 6733 lgkmcnt(0). 6734 - Must happen before 6735 following 6736 buffer_invl2 and 6737 buffer_wbinvl1_vol. 6738 - Ensures the 6739 atomicrmw has 6740 completed before 6741 invalidating the 6742 caches. 6743 6744 3. buffer_invl2; 6745 buffer_wbinvl1_vol 6746 6747 - Must happen before 6748 any following 6749 global/generic 6750 load/load 6751 atomic/atomicrmw. 6752 - Ensures that 6753 following 6754 loads will not see 6755 stale L1 global data, 6756 nor see stale L2 MTYPE 6757 NC global data. 6758 MTYPE RW and CC memory will 6759 never be stale in L2 due to 6760 the memory probes. 6761 6762 fence acquire - singlethread *none* *none* 6763 - wavefront 6764 fence acquire - workgroup *none* 1. s_waitcnt lgkm/vmcnt(0) 6765 6766 - Use lgkmcnt(0) if not 6767 TgSplit execution mode 6768 and vmcnt(0) if TgSplit 6769 execution mode. 6770 - If OpenCL and 6771 address space is 6772 not generic, omit 6773 lgkmcnt(0). 6774 - If OpenCL and 6775 address space is 6776 local, omit 6777 vmcnt(0). 6778 - However, since LLVM 6779 currently has no 6780 address space on 6781 the fence need to 6782 conservatively 6783 always generate. If 6784 fence had an 6785 address space then 6786 set to address 6787 space of OpenCL 6788 fence flag, or to 6789 generic if both 6790 local and global 6791 flags are 6792 specified. 6793 - s_waitcnt vmcnt(0) 6794 must happen after 6795 any preceding 6796 global/generic load 6797 atomic/ 6798 atomicrmw 6799 with an equal or 6800 wider sync scope 6801 and memory ordering 6802 stronger than 6803 unordered (this is 6804 termed the 6805 fence-paired-atomic). 6806 - s_waitcnt lgkmcnt(0) 6807 must happen after 6808 any preceding 6809 local/generic load 6810 atomic/atomicrmw 6811 with an equal or 6812 wider sync scope 6813 and memory ordering 6814 stronger than 6815 unordered (this is 6816 termed the 6817 fence-paired-atomic). 6818 - Must happen before 6819 the following 6820 buffer_wbinvl1_vol and 6821 any following 6822 global/generic 6823 load/load 6824 atomic/store/store 6825 atomic/atomicrmw. 6826 - Ensures any 6827 following global 6828 data read is no 6829 older than the 6830 value read by the 6831 fence-paired-atomic. 6832 6833 2. buffer_wbinvl1_vol 6834 6835 - If not TgSplit execution 6836 mode, omit. 6837 - Ensures that 6838 following 6839 loads will not see 6840 stale data. 6841 6842 fence acquire - agent *none* 1. s_waitcnt lgkmcnt(0) & 6843 vmcnt(0) 6844 6845 - If TgSplit execution mode, 6846 omit lgkmcnt(0). 6847 - If OpenCL and 6848 address space is 6849 not generic, omit 6850 lgkmcnt(0). 6851 - However, since LLVM 6852 currently has no 6853 address space on 6854 the fence need to 6855 conservatively 6856 always generate 6857 (see comment for 6858 previous fence). 6859 - Could be split into 6860 separate s_waitcnt 6861 vmcnt(0) and 6862 s_waitcnt 6863 lgkmcnt(0) to allow 6864 them to be 6865 independently moved 6866 according to the 6867 following rules. 6868 - s_waitcnt vmcnt(0) 6869 must happen after 6870 any preceding 6871 global/generic load 6872 atomic/atomicrmw 6873 with an equal or 6874 wider sync scope 6875 and memory ordering 6876 stronger than 6877 unordered (this is 6878 termed the 6879 fence-paired-atomic). 6880 - s_waitcnt lgkmcnt(0) 6881 must happen after 6882 any preceding 6883 local/generic load 6884 atomic/atomicrmw 6885 with an equal or 6886 wider sync scope 6887 and memory ordering 6888 stronger than 6889 unordered (this is 6890 termed the 6891 fence-paired-atomic). 6892 - Must happen before 6893 the following 6894 buffer_wbinvl1_vol. 6895 - Ensures that the 6896 fence-paired atomic 6897 has completed 6898 before invalidating 6899 the 6900 cache. Therefore 6901 any following 6902 locations read must 6903 be no older than 6904 the value read by 6905 the 6906 fence-paired-atomic. 6907 6908 2. buffer_wbinvl1_vol 6909 6910 - Must happen before any 6911 following global/generic 6912 load/load 6913 atomic/store/store 6914 atomic/atomicrmw. 6915 - Ensures that 6916 following loads 6917 will not see stale 6918 global data. 6919 6920 fence acquire - system *none* 1. s_waitcnt lgkmcnt(0) & 6921 vmcnt(0) 6922 6923 - If TgSplit execution mode, 6924 omit lgkmcnt(0). 6925 - If OpenCL and 6926 address space is 6927 not generic, omit 6928 lgkmcnt(0). 6929 - However, since LLVM 6930 currently has no 6931 address space on 6932 the fence need to 6933 conservatively 6934 always generate 6935 (see comment for 6936 previous fence). 6937 - Could be split into 6938 separate s_waitcnt 6939 vmcnt(0) and 6940 s_waitcnt 6941 lgkmcnt(0) to allow 6942 them to be 6943 independently moved 6944 according to the 6945 following rules. 6946 - s_waitcnt vmcnt(0) 6947 must happen after 6948 any preceding 6949 global/generic load 6950 atomic/atomicrmw 6951 with an equal or 6952 wider sync scope 6953 and memory ordering 6954 stronger than 6955 unordered (this is 6956 termed the 6957 fence-paired-atomic). 6958 - s_waitcnt lgkmcnt(0) 6959 must happen after 6960 any preceding 6961 local/generic load 6962 atomic/atomicrmw 6963 with an equal or 6964 wider sync scope 6965 and memory ordering 6966 stronger than 6967 unordered (this is 6968 termed the 6969 fence-paired-atomic). 6970 - Must happen before 6971 the following buffer_invl2 and 6972 buffer_wbinvl1_vol. 6973 - Ensures that the 6974 fence-paired atomic 6975 has completed 6976 before invalidating 6977 the 6978 cache. Therefore 6979 any following 6980 locations read must 6981 be no older than 6982 the value read by 6983 the 6984 fence-paired-atomic. 6985 6986 2. buffer_invl2; 6987 buffer_wbinvl1_vol 6988 6989 - Must happen before any 6990 following global/generic 6991 load/load 6992 atomic/store/store 6993 atomic/atomicrmw. 6994 - Ensures that 6995 following 6996 loads will not see 6997 stale L1 global data, 6998 nor see stale L2 MTYPE 6999 NC global data. 7000 MTYPE RW and CC memory will 7001 never be stale in L2 due to 7002 the memory probes. 7003 **Release Atomic** 7004 ------------------------------------------------------------------------------------ 7005 store atomic release - singlethread - global 1. buffer/global/flat_store 7006 - wavefront - generic 7007 store atomic release - singlethread - local *If TgSplit execution mode, 7008 - wavefront local address space cannot 7009 be used.* 7010 7011 1. ds_store 7012 store atomic release - workgroup - global 1. s_waitcnt lgkm/vmcnt(0) 7013 - generic 7014 - Use lgkmcnt(0) if not 7015 TgSplit execution mode 7016 and vmcnt(0) if TgSplit 7017 execution mode. 7018 - If OpenCL, omit lgkmcnt(0). 7019 - s_waitcnt vmcnt(0) 7020 must happen after 7021 any preceding 7022 global/generic load/store/ 7023 load atomic/store atomic/ 7024 atomicrmw. 7025 - s_waitcnt lgkmcnt(0) 7026 must happen after 7027 any preceding 7028 local/generic 7029 load/store/load 7030 atomic/store 7031 atomic/atomicrmw. 7032 - Must happen before 7033 the following 7034 store. 7035 - Ensures that all 7036 memory operations 7037 have 7038 completed before 7039 performing the 7040 store that is being 7041 released. 7042 7043 2. buffer/global/flat_store 7044 store atomic release - workgroup - local *If TgSplit execution mode, 7045 local address space cannot 7046 be used.* 7047 7048 1. ds_store 7049 store atomic release - agent - global 1. s_waitcnt lgkmcnt(0) & 7050 - generic vmcnt(0) 7051 7052 - If TgSplit execution mode, 7053 omit lgkmcnt(0). 7054 - If OpenCL and 7055 address space is 7056 not generic, omit 7057 lgkmcnt(0). 7058 - Could be split into 7059 separate s_waitcnt 7060 vmcnt(0) and 7061 s_waitcnt 7062 lgkmcnt(0) to allow 7063 them to be 7064 independently moved 7065 according to the 7066 following rules. 7067 - s_waitcnt vmcnt(0) 7068 must happen after 7069 any preceding 7070 global/generic 7071 load/store/load 7072 atomic/store 7073 atomic/atomicrmw. 7074 - s_waitcnt lgkmcnt(0) 7075 must happen after 7076 any preceding 7077 local/generic 7078 load/store/load 7079 atomic/store 7080 atomic/atomicrmw. 7081 - Must happen before 7082 the following 7083 store. 7084 - Ensures that all 7085 memory operations 7086 to memory have 7087 completed before 7088 performing the 7089 store that is being 7090 released. 7091 7092 2. buffer/global/flat_store 7093 store atomic release - system - global 1. buffer_wbl2 7094 - generic 7095 - Must happen before 7096 following s_waitcnt. 7097 - Performs L2 writeback to 7098 ensure previous 7099 global/generic 7100 store/atomicrmw are 7101 visible at system scope. 7102 7103 2. s_waitcnt lgkmcnt(0) & 7104 vmcnt(0) 7105 7106 - If TgSplit execution mode, 7107 omit lgkmcnt(0). 7108 - If OpenCL and 7109 address space is 7110 not generic, omit 7111 lgkmcnt(0). 7112 - Could be split into 7113 separate s_waitcnt 7114 vmcnt(0) and 7115 s_waitcnt 7116 lgkmcnt(0) to allow 7117 them to be 7118 independently moved 7119 according to the 7120 following rules. 7121 - s_waitcnt vmcnt(0) 7122 must happen after any 7123 preceding 7124 global/generic 7125 load/store/load 7126 atomic/store 7127 atomic/atomicrmw. 7128 - s_waitcnt lgkmcnt(0) 7129 must happen after any 7130 preceding 7131 local/generic 7132 load/store/load 7133 atomic/store 7134 atomic/atomicrmw. 7135 - Must happen before 7136 the following 7137 store. 7138 - Ensures that all 7139 memory operations 7140 to memory and the L2 7141 writeback have 7142 completed before 7143 performing the 7144 store that is being 7145 released. 7146 7147 3. buffer/global/flat_store 7148 atomicrmw release - singlethread - global 1. buffer/global/flat_atomic 7149 - wavefront - generic 7150 atomicrmw release - singlethread - local *If TgSplit execution mode, 7151 - wavefront local address space cannot 7152 be used.* 7153 7154 1. ds_atomic 7155 atomicrmw release - workgroup - global 1. s_waitcnt lgkm/vmcnt(0) 7156 - generic 7157 - Use lgkmcnt(0) if not 7158 TgSplit execution mode 7159 and vmcnt(0) if TgSplit 7160 execution mode. 7161 - If OpenCL, omit 7162 lgkmcnt(0). 7163 - s_waitcnt vmcnt(0) 7164 must happen after 7165 any preceding 7166 global/generic load/store/ 7167 load atomic/store atomic/ 7168 atomicrmw. 7169 - s_waitcnt lgkmcnt(0) 7170 must happen after 7171 any preceding 7172 local/generic 7173 load/store/load 7174 atomic/store 7175 atomic/atomicrmw. 7176 - Must happen before 7177 the following 7178 atomicrmw. 7179 - Ensures that all 7180 memory operations 7181 have 7182 completed before 7183 performing the 7184 atomicrmw that is 7185 being released. 7186 7187 2. buffer/global/flat_atomic 7188 atomicrmw release - workgroup - local *If TgSplit execution mode, 7189 local address space cannot 7190 be used.* 7191 7192 1. ds_atomic 7193 atomicrmw release - agent - global 1. s_waitcnt lgkmcnt(0) & 7194 - generic vmcnt(0) 7195 7196 - If TgSplit execution mode, 7197 omit lgkmcnt(0). 7198 - If OpenCL, omit 7199 lgkmcnt(0). 7200 - Could be split into 7201 separate s_waitcnt 7202 vmcnt(0) and 7203 s_waitcnt 7204 lgkmcnt(0) to allow 7205 them to be 7206 independently moved 7207 according to the 7208 following rules. 7209 - s_waitcnt vmcnt(0) 7210 must happen after 7211 any preceding 7212 global/generic 7213 load/store/load 7214 atomic/store 7215 atomic/atomicrmw. 7216 - s_waitcnt lgkmcnt(0) 7217 must happen after 7218 any preceding 7219 local/generic 7220 load/store/load 7221 atomic/store 7222 atomic/atomicrmw. 7223 - Must happen before 7224 the following 7225 atomicrmw. 7226 - Ensures that all 7227 memory operations 7228 to global and local 7229 have completed 7230 before performing 7231 the atomicrmw that 7232 is being released. 7233 7234 2. buffer/global/flat_atomic 7235 atomicrmw release - system - global 1. buffer_wbl2 7236 - generic 7237 - Must happen before 7238 following s_waitcnt. 7239 - Performs L2 writeback to 7240 ensure previous 7241 global/generic 7242 store/atomicrmw are 7243 visible at system scope. 7244 7245 2. s_waitcnt lgkmcnt(0) & 7246 vmcnt(0) 7247 7248 - If TgSplit execution mode, 7249 omit lgkmcnt(0). 7250 - If OpenCL, omit 7251 lgkmcnt(0). 7252 - Could be split into 7253 separate s_waitcnt 7254 vmcnt(0) and 7255 s_waitcnt 7256 lgkmcnt(0) to allow 7257 them to be 7258 independently moved 7259 according to the 7260 following rules. 7261 - s_waitcnt vmcnt(0) 7262 must happen after 7263 any preceding 7264 global/generic 7265 load/store/load 7266 atomic/store 7267 atomic/atomicrmw. 7268 - s_waitcnt lgkmcnt(0) 7269 must happen after 7270 any preceding 7271 local/generic 7272 load/store/load 7273 atomic/store 7274 atomic/atomicrmw. 7275 - Must happen before 7276 the following 7277 atomicrmw. 7278 - Ensures that all 7279 memory operations 7280 to memory and the L2 7281 writeback have 7282 completed before 7283 performing the 7284 store that is being 7285 released. 7286 7287 3. buffer/global/flat_atomic 7288 fence release - singlethread *none* *none* 7289 - wavefront 7290 fence release - workgroup *none* 1. s_waitcnt lgkm/vmcnt(0) 7291 7292 - Use lgkmcnt(0) if not 7293 TgSplit execution mode 7294 and vmcnt(0) if TgSplit 7295 execution mode. 7296 - If OpenCL and 7297 address space is 7298 not generic, omit 7299 lgkmcnt(0). 7300 - If OpenCL and 7301 address space is 7302 local, omit 7303 vmcnt(0). 7304 - However, since LLVM 7305 currently has no 7306 address space on 7307 the fence need to 7308 conservatively 7309 always generate. If 7310 fence had an 7311 address space then 7312 set to address 7313 space of OpenCL 7314 fence flag, or to 7315 generic if both 7316 local and global 7317 flags are 7318 specified. 7319 - s_waitcnt vmcnt(0) 7320 must happen after 7321 any preceding 7322 global/generic 7323 load/store/ 7324 load atomic/store atomic/ 7325 atomicrmw. 7326 - s_waitcnt lgkmcnt(0) 7327 must happen after 7328 any preceding 7329 local/generic 7330 load/load 7331 atomic/store/store 7332 atomic/atomicrmw. 7333 - Must happen before 7334 any following store 7335 atomic/atomicrmw 7336 with an equal or 7337 wider sync scope 7338 and memory ordering 7339 stronger than 7340 unordered (this is 7341 termed the 7342 fence-paired-atomic). 7343 - Ensures that all 7344 memory operations 7345 have 7346 completed before 7347 performing the 7348 following 7349 fence-paired-atomic. 7350 7351 fence release - agent *none* 1. s_waitcnt lgkmcnt(0) & 7352 vmcnt(0) 7353 7354 - If TgSplit execution mode, 7355 omit lgkmcnt(0). 7356 - If OpenCL and 7357 address space is 7358 not generic, omit 7359 lgkmcnt(0). 7360 - If OpenCL and 7361 address space is 7362 local, omit 7363 vmcnt(0). 7364 - However, since LLVM 7365 currently has no 7366 address space on 7367 the fence need to 7368 conservatively 7369 always generate. If 7370 fence had an 7371 address space then 7372 set to address 7373 space of OpenCL 7374 fence flag, or to 7375 generic if both 7376 local and global 7377 flags are 7378 specified. 7379 - Could be split into 7380 separate s_waitcnt 7381 vmcnt(0) and 7382 s_waitcnt 7383 lgkmcnt(0) to allow 7384 them to be 7385 independently moved 7386 according to the 7387 following rules. 7388 - s_waitcnt vmcnt(0) 7389 must happen after 7390 any preceding 7391 global/generic 7392 load/store/load 7393 atomic/store 7394 atomic/atomicrmw. 7395 - s_waitcnt lgkmcnt(0) 7396 must happen after 7397 any preceding 7398 local/generic 7399 load/store/load 7400 atomic/store 7401 atomic/atomicrmw. 7402 - Must happen before 7403 any following store 7404 atomic/atomicrmw 7405 with an equal or 7406 wider sync scope 7407 and memory ordering 7408 stronger than 7409 unordered (this is 7410 termed the 7411 fence-paired-atomic). 7412 - Ensures that all 7413 memory operations 7414 have 7415 completed before 7416 performing the 7417 following 7418 fence-paired-atomic. 7419 7420 fence release - system *none* 1. buffer_wbl2 7421 7422 - If OpenCL and 7423 address space is 7424 local, omit. 7425 - Must happen before 7426 following s_waitcnt. 7427 - Performs L2 writeback to 7428 ensure previous 7429 global/generic 7430 store/atomicrmw are 7431 visible at system scope. 7432 7433 2. s_waitcnt lgkmcnt(0) & 7434 vmcnt(0) 7435 7436 - If TgSplit execution mode, 7437 omit lgkmcnt(0). 7438 - If OpenCL and 7439 address space is 7440 not generic, omit 7441 lgkmcnt(0). 7442 - If OpenCL and 7443 address space is 7444 local, omit 7445 vmcnt(0). 7446 - However, since LLVM 7447 currently has no 7448 address space on 7449 the fence need to 7450 conservatively 7451 always generate. If 7452 fence had an 7453 address space then 7454 set to address 7455 space of OpenCL 7456 fence flag, or to 7457 generic if both 7458 local and global 7459 flags are 7460 specified. 7461 - Could be split into 7462 separate s_waitcnt 7463 vmcnt(0) and 7464 s_waitcnt 7465 lgkmcnt(0) to allow 7466 them to be 7467 independently moved 7468 according to the 7469 following rules. 7470 - s_waitcnt vmcnt(0) 7471 must happen after 7472 any preceding 7473 global/generic 7474 load/store/load 7475 atomic/store 7476 atomic/atomicrmw. 7477 - s_waitcnt lgkmcnt(0) 7478 must happen after 7479 any preceding 7480 local/generic 7481 load/store/load 7482 atomic/store 7483 atomic/atomicrmw. 7484 - Must happen before 7485 any following store 7486 atomic/atomicrmw 7487 with an equal or 7488 wider sync scope 7489 and memory ordering 7490 stronger than 7491 unordered (this is 7492 termed the 7493 fence-paired-atomic). 7494 - Ensures that all 7495 memory operations 7496 have 7497 completed before 7498 performing the 7499 following 7500 fence-paired-atomic. 7501 7502 **Acquire-Release Atomic** 7503 ------------------------------------------------------------------------------------ 7504 atomicrmw acq_rel - singlethread - global 1. buffer/global/flat_atomic 7505 - wavefront - generic 7506 atomicrmw acq_rel - singlethread - local *If TgSplit execution mode, 7507 - wavefront local address space cannot 7508 be used.* 7509 7510 1. ds_atomic 7511 atomicrmw acq_rel - workgroup - global 1. s_waitcnt lgkm/vmcnt(0) 7512 7513 - Use lgkmcnt(0) if not 7514 TgSplit execution mode 7515 and vmcnt(0) if TgSplit 7516 execution mode. 7517 - If OpenCL, omit 7518 lgkmcnt(0). 7519 - Must happen after 7520 any preceding 7521 local/generic 7522 load/store/load 7523 atomic/store 7524 atomic/atomicrmw. 7525 - s_waitcnt vmcnt(0) 7526 must happen after 7527 any preceding 7528 global/generic load/store/ 7529 load atomic/store atomic/ 7530 atomicrmw. 7531 - s_waitcnt lgkmcnt(0) 7532 must happen after 7533 any preceding 7534 local/generic 7535 load/store/load 7536 atomic/store 7537 atomic/atomicrmw. 7538 - Must happen before 7539 the following 7540 atomicrmw. 7541 - Ensures that all 7542 memory operations 7543 have 7544 completed before 7545 performing the 7546 atomicrmw that is 7547 being released. 7548 7549 2. buffer/global_atomic 7550 3. s_waitcnt vmcnt(0) 7551 7552 - If not TgSplit execution 7553 mode, omit. 7554 - Must happen before 7555 the following 7556 buffer_wbinvl1_vol. 7557 - Ensures any 7558 following global 7559 data read is no 7560 older than the 7561 atomicrmw value 7562 being acquired. 7563 7564 4. buffer_wbinvl1_vol 7565 7566 - If not TgSplit execution 7567 mode, omit. 7568 - Ensures that 7569 following 7570 loads will not see 7571 stale data. 7572 7573 atomicrmw acq_rel - workgroup - local *If TgSplit execution mode, 7574 local address space cannot 7575 be used.* 7576 7577 1. ds_atomic 7578 2. s_waitcnt lgkmcnt(0) 7579 7580 - If OpenCL, omit. 7581 - Must happen before 7582 any following 7583 global/generic 7584 load/load 7585 atomic/store/store 7586 atomic/atomicrmw. 7587 - Ensures any 7588 following global 7589 data read is no 7590 older than the local load 7591 atomic value being 7592 acquired. 7593 7594 atomicrmw acq_rel - workgroup - generic 1. s_waitcnt lgkm/vmcnt(0) 7595 7596 - Use lgkmcnt(0) if not 7597 TgSplit execution mode 7598 and vmcnt(0) if TgSplit 7599 execution mode. 7600 - If OpenCL, omit 7601 lgkmcnt(0). 7602 - s_waitcnt vmcnt(0) 7603 must happen after 7604 any preceding 7605 global/generic load/store/ 7606 load atomic/store atomic/ 7607 atomicrmw. 7608 - s_waitcnt lgkmcnt(0) 7609 must happen after 7610 any preceding 7611 local/generic 7612 load/store/load 7613 atomic/store 7614 atomic/atomicrmw. 7615 - Must happen before 7616 the following 7617 atomicrmw. 7618 - Ensures that all 7619 memory operations 7620 have 7621 completed before 7622 performing the 7623 atomicrmw that is 7624 being released. 7625 7626 2. flat_atomic 7627 3. s_waitcnt lgkmcnt(0) & 7628 vmcnt(0) 7629 7630 - If not TgSplit execution 7631 mode, omit vmcnt(0). 7632 - If OpenCL, omit 7633 lgkmcnt(0). 7634 - Must happen before 7635 the following 7636 buffer_wbinvl1_vol and 7637 any following 7638 global/generic 7639 load/load 7640 atomic/store/store 7641 atomic/atomicrmw. 7642 - Ensures any 7643 following global 7644 data read is no 7645 older than a local load 7646 atomic value being 7647 acquired. 7648 7649 3. buffer_wbinvl1_vol 7650 7651 - If not TgSplit execution 7652 mode, omit. 7653 - Ensures that 7654 following 7655 loads will not see 7656 stale data. 7657 7658 atomicrmw acq_rel - agent - global 1. s_waitcnt lgkmcnt(0) & 7659 vmcnt(0) 7660 7661 - If TgSplit execution mode, 7662 omit lgkmcnt(0). 7663 - If OpenCL, omit 7664 lgkmcnt(0). 7665 - Could be split into 7666 separate s_waitcnt 7667 vmcnt(0) and 7668 s_waitcnt 7669 lgkmcnt(0) to allow 7670 them to be 7671 independently moved 7672 according to the 7673 following rules. 7674 - s_waitcnt vmcnt(0) 7675 must happen after 7676 any preceding 7677 global/generic 7678 load/store/load 7679 atomic/store 7680 atomic/atomicrmw. 7681 - s_waitcnt lgkmcnt(0) 7682 must happen after 7683 any preceding 7684 local/generic 7685 load/store/load 7686 atomic/store 7687 atomic/atomicrmw. 7688 - Must happen before 7689 the following 7690 atomicrmw. 7691 - Ensures that all 7692 memory operations 7693 to global have 7694 completed before 7695 performing the 7696 atomicrmw that is 7697 being released. 7698 7699 2. buffer/global_atomic 7700 3. s_waitcnt vmcnt(0) 7701 7702 - Must happen before 7703 following 7704 buffer_wbinvl1_vol. 7705 - Ensures the 7706 atomicrmw has 7707 completed before 7708 invalidating the 7709 cache. 7710 7711 4. buffer_wbinvl1_vol 7712 7713 - Must happen before 7714 any following 7715 global/generic 7716 load/load 7717 atomic/atomicrmw. 7718 - Ensures that 7719 following loads 7720 will not see stale 7721 global data. 7722 7723 atomicrmw acq_rel - system - global 1. buffer_wbl2 7724 7725 - Must happen before 7726 following s_waitcnt. 7727 - Performs L2 writeback to 7728 ensure previous 7729 global/generic 7730 store/atomicrmw are 7731 visible at system scope. 7732 7733 2. s_waitcnt lgkmcnt(0) & 7734 vmcnt(0) 7735 7736 - If TgSplit execution mode, 7737 omit lgkmcnt(0). 7738 - If OpenCL, omit 7739 lgkmcnt(0). 7740 - Could be split into 7741 separate s_waitcnt 7742 vmcnt(0) and 7743 s_waitcnt 7744 lgkmcnt(0) to allow 7745 them to be 7746 independently moved 7747 according to the 7748 following rules. 7749 - s_waitcnt vmcnt(0) 7750 must happen after 7751 any preceding 7752 global/generic 7753 load/store/load 7754 atomic/store 7755 atomic/atomicrmw. 7756 - s_waitcnt lgkmcnt(0) 7757 must happen after 7758 any preceding 7759 local/generic 7760 load/store/load 7761 atomic/store 7762 atomic/atomicrmw. 7763 - Must happen before 7764 the following 7765 atomicrmw. 7766 - Ensures that all 7767 memory operations 7768 to global and L2 writeback 7769 have completed before 7770 performing the 7771 atomicrmw that is 7772 being released. 7773 7774 3. buffer/global_atomic 7775 4. s_waitcnt vmcnt(0) 7776 7777 - Must happen before 7778 following buffer_invl2 and 7779 buffer_wbinvl1_vol. 7780 - Ensures the 7781 atomicrmw has 7782 completed before 7783 invalidating the 7784 caches. 7785 7786 5. buffer_invl2; 7787 buffer_wbinvl1_vol 7788 7789 - Must happen before 7790 any following 7791 global/generic 7792 load/load 7793 atomic/atomicrmw. 7794 - Ensures that 7795 following 7796 loads will not see 7797 stale L1 global data, 7798 nor see stale L2 MTYPE 7799 NC global data. 7800 MTYPE RW and CC memory will 7801 never be stale in L2 due to 7802 the memory probes. 7803 7804 atomicrmw acq_rel - agent - generic 1. s_waitcnt lgkmcnt(0) & 7805 vmcnt(0) 7806 7807 - If TgSplit execution mode, 7808 omit lgkmcnt(0). 7809 - If OpenCL, omit 7810 lgkmcnt(0). 7811 - Could be split into 7812 separate s_waitcnt 7813 vmcnt(0) and 7814 s_waitcnt 7815 lgkmcnt(0) to allow 7816 them to be 7817 independently moved 7818 according to the 7819 following rules. 7820 - s_waitcnt vmcnt(0) 7821 must happen after 7822 any preceding 7823 global/generic 7824 load/store/load 7825 atomic/store 7826 atomic/atomicrmw. 7827 - s_waitcnt lgkmcnt(0) 7828 must happen after 7829 any preceding 7830 local/generic 7831 load/store/load 7832 atomic/store 7833 atomic/atomicrmw. 7834 - Must happen before 7835 the following 7836 atomicrmw. 7837 - Ensures that all 7838 memory operations 7839 to global have 7840 completed before 7841 performing the 7842 atomicrmw that is 7843 being released. 7844 7845 2. flat_atomic 7846 3. s_waitcnt vmcnt(0) & 7847 lgkmcnt(0) 7848 7849 - If TgSplit execution mode, 7850 omit lgkmcnt(0). 7851 - If OpenCL, omit 7852 lgkmcnt(0). 7853 - Must happen before 7854 following 7855 buffer_wbinvl1_vol. 7856 - Ensures the 7857 atomicrmw has 7858 completed before 7859 invalidating the 7860 cache. 7861 7862 4. buffer_wbinvl1_vol 7863 7864 - Must happen before 7865 any following 7866 global/generic 7867 load/load 7868 atomic/atomicrmw. 7869 - Ensures that 7870 following loads 7871 will not see stale 7872 global data. 7873 7874 atomicrmw acq_rel - system - generic 1. buffer_wbl2 7875 7876 - Must happen before 7877 following s_waitcnt. 7878 - Performs L2 writeback to 7879 ensure previous 7880 global/generic 7881 store/atomicrmw are 7882 visible at system scope. 7883 7884 2. s_waitcnt lgkmcnt(0) & 7885 vmcnt(0) 7886 7887 - If TgSplit execution mode, 7888 omit lgkmcnt(0). 7889 - If OpenCL, omit 7890 lgkmcnt(0). 7891 - Could be split into 7892 separate s_waitcnt 7893 vmcnt(0) and 7894 s_waitcnt 7895 lgkmcnt(0) to allow 7896 them to be 7897 independently moved 7898 according to the 7899 following rules. 7900 - s_waitcnt vmcnt(0) 7901 must happen after 7902 any preceding 7903 global/generic 7904 load/store/load 7905 atomic/store 7906 atomic/atomicrmw. 7907 - s_waitcnt lgkmcnt(0) 7908 must happen after 7909 any preceding 7910 local/generic 7911 load/store/load 7912 atomic/store 7913 atomic/atomicrmw. 7914 - Must happen before 7915 the following 7916 atomicrmw. 7917 - Ensures that all 7918 memory operations 7919 to global and L2 writeback 7920 have completed before 7921 performing the 7922 atomicrmw that is 7923 being released. 7924 7925 3. flat_atomic 7926 4. s_waitcnt vmcnt(0) & 7927 lgkmcnt(0) 7928 7929 - If TgSplit execution mode, 7930 omit lgkmcnt(0). 7931 - If OpenCL, omit 7932 lgkmcnt(0). 7933 - Must happen before 7934 following buffer_invl2 and 7935 buffer_wbinvl1_vol. 7936 - Ensures the 7937 atomicrmw has 7938 completed before 7939 invalidating the 7940 caches. 7941 7942 5. buffer_invl2; 7943 buffer_wbinvl1_vol 7944 7945 - Must happen before 7946 any following 7947 global/generic 7948 load/load 7949 atomic/atomicrmw. 7950 - Ensures that 7951 following 7952 loads will not see 7953 stale L1 global data, 7954 nor see stale L2 MTYPE 7955 NC global data. 7956 MTYPE RW and CC memory will 7957 never be stale in L2 due to 7958 the memory probes. 7959 7960 fence acq_rel - singlethread *none* *none* 7961 - wavefront 7962 fence acq_rel - workgroup *none* 1. s_waitcnt lgkm/vmcnt(0) 7963 7964 - Use lgkmcnt(0) if not 7965 TgSplit execution mode 7966 and vmcnt(0) if TgSplit 7967 execution mode. 7968 - If OpenCL and 7969 address space is 7970 not generic, omit 7971 lgkmcnt(0). 7972 - If OpenCL and 7973 address space is 7974 local, omit 7975 vmcnt(0). 7976 - However, 7977 since LLVM 7978 currently has no 7979 address space on 7980 the fence need to 7981 conservatively 7982 always generate 7983 (see comment for 7984 previous fence). 7985 - s_waitcnt vmcnt(0) 7986 must happen after 7987 any preceding 7988 global/generic 7989 load/store/ 7990 load atomic/store atomic/ 7991 atomicrmw. 7992 - s_waitcnt lgkmcnt(0) 7993 must happen after 7994 any preceding 7995 local/generic 7996 load/load 7997 atomic/store/store 7998 atomic/atomicrmw. 7999 - Must happen before 8000 any following 8001 global/generic 8002 load/load 8003 atomic/store/store 8004 atomic/atomicrmw. 8005 - Ensures that all 8006 memory operations 8007 have 8008 completed before 8009 performing any 8010 following global 8011 memory operations. 8012 - Ensures that the 8013 preceding 8014 local/generic load 8015 atomic/atomicrmw 8016 with an equal or 8017 wider sync scope 8018 and memory ordering 8019 stronger than 8020 unordered (this is 8021 termed the 8022 acquire-fence-paired-atomic) 8023 has completed 8024 before following 8025 global memory 8026 operations. This 8027 satisfies the 8028 requirements of 8029 acquire. 8030 - Ensures that all 8031 previous memory 8032 operations have 8033 completed before a 8034 following 8035 local/generic store 8036 atomic/atomicrmw 8037 with an equal or 8038 wider sync scope 8039 and memory ordering 8040 stronger than 8041 unordered (this is 8042 termed the 8043 release-fence-paired-atomic). 8044 This satisfies the 8045 requirements of 8046 release. 8047 - Must happen before 8048 the following 8049 buffer_wbinvl1_vol. 8050 - Ensures that the 8051 acquire-fence-paired 8052 atomic has completed 8053 before invalidating 8054 the 8055 cache. Therefore 8056 any following 8057 locations read must 8058 be no older than 8059 the value read by 8060 the 8061 acquire-fence-paired-atomic. 8062 8063 2. buffer_wbinvl1_vol 8064 8065 - If not TgSplit execution 8066 mode, omit. 8067 - Ensures that 8068 following 8069 loads will not see 8070 stale data. 8071 8072 fence acq_rel - agent *none* 1. s_waitcnt lgkmcnt(0) & 8073 vmcnt(0) 8074 8075 - If TgSplit execution mode, 8076 omit lgkmcnt(0). 8077 - If OpenCL and 8078 address space is 8079 not generic, omit 8080 lgkmcnt(0). 8081 - However, since LLVM 8082 currently has no 8083 address space on 8084 the fence need to 8085 conservatively 8086 always generate 8087 (see comment for 8088 previous fence). 8089 - Could be split into 8090 separate s_waitcnt 8091 vmcnt(0) and 8092 s_waitcnt 8093 lgkmcnt(0) to allow 8094 them to be 8095 independently moved 8096 according to the 8097 following rules. 8098 - s_waitcnt vmcnt(0) 8099 must happen after 8100 any preceding 8101 global/generic 8102 load/store/load 8103 atomic/store 8104 atomic/atomicrmw. 8105 - s_waitcnt lgkmcnt(0) 8106 must happen after 8107 any preceding 8108 local/generic 8109 load/store/load 8110 atomic/store 8111 atomic/atomicrmw. 8112 - Must happen before 8113 the following 8114 buffer_wbinvl1_vol. 8115 - Ensures that the 8116 preceding 8117 global/local/generic 8118 load 8119 atomic/atomicrmw 8120 with an equal or 8121 wider sync scope 8122 and memory ordering 8123 stronger than 8124 unordered (this is 8125 termed the 8126 acquire-fence-paired-atomic) 8127 has completed 8128 before invalidating 8129 the cache. This 8130 satisfies the 8131 requirements of 8132 acquire. 8133 - Ensures that all 8134 previous memory 8135 operations have 8136 completed before a 8137 following 8138 global/local/generic 8139 store 8140 atomic/atomicrmw 8141 with an equal or 8142 wider sync scope 8143 and memory ordering 8144 stronger than 8145 unordered (this is 8146 termed the 8147 release-fence-paired-atomic). 8148 This satisfies the 8149 requirements of 8150 release. 8151 8152 2. buffer_wbinvl1_vol 8153 8154 - Must happen before 8155 any following 8156 global/generic 8157 load/load 8158 atomic/store/store 8159 atomic/atomicrmw. 8160 - Ensures that 8161 following loads 8162 will not see stale 8163 global data. This 8164 satisfies the 8165 requirements of 8166 acquire. 8167 8168 fence acq_rel - system *none* 1. buffer_wbl2 8169 8170 - If OpenCL and 8171 address space is 8172 local, omit. 8173 - Must happen before 8174 following s_waitcnt. 8175 - Performs L2 writeback to 8176 ensure previous 8177 global/generic 8178 store/atomicrmw are 8179 visible at system scope. 8180 8181 2. s_waitcnt lgkmcnt(0) & 8182 vmcnt(0) 8183 8184 - If TgSplit execution mode, 8185 omit lgkmcnt(0). 8186 - If OpenCL and 8187 address space is 8188 not generic, omit 8189 lgkmcnt(0). 8190 - However, since LLVM 8191 currently has no 8192 address space on 8193 the fence need to 8194 conservatively 8195 always generate 8196 (see comment for 8197 previous fence). 8198 - Could be split into 8199 separate s_waitcnt 8200 vmcnt(0) and 8201 s_waitcnt 8202 lgkmcnt(0) to allow 8203 them to be 8204 independently moved 8205 according to the 8206 following rules. 8207 - s_waitcnt vmcnt(0) 8208 must happen after 8209 any preceding 8210 global/generic 8211 load/store/load 8212 atomic/store 8213 atomic/atomicrmw. 8214 - s_waitcnt lgkmcnt(0) 8215 must happen after 8216 any preceding 8217 local/generic 8218 load/store/load 8219 atomic/store 8220 atomic/atomicrmw. 8221 - Must happen before 8222 the following buffer_invl2 and 8223 buffer_wbinvl1_vol. 8224 - Ensures that the 8225 preceding 8226 global/local/generic 8227 load 8228 atomic/atomicrmw 8229 with an equal or 8230 wider sync scope 8231 and memory ordering 8232 stronger than 8233 unordered (this is 8234 termed the 8235 acquire-fence-paired-atomic) 8236 has completed 8237 before invalidating 8238 the cache. This 8239 satisfies the 8240 requirements of 8241 acquire. 8242 - Ensures that all 8243 previous memory 8244 operations have 8245 completed before a 8246 following 8247 global/local/generic 8248 store 8249 atomic/atomicrmw 8250 with an equal or 8251 wider sync scope 8252 and memory ordering 8253 stronger than 8254 unordered (this is 8255 termed the 8256 release-fence-paired-atomic). 8257 This satisfies the 8258 requirements of 8259 release. 8260 8261 3. buffer_invl2; 8262 buffer_wbinvl1_vol 8263 8264 - Must happen before 8265 any following 8266 global/generic 8267 load/load 8268 atomic/store/store 8269 atomic/atomicrmw. 8270 - Ensures that 8271 following 8272 loads will not see 8273 stale L1 global data, 8274 nor see stale L2 MTYPE 8275 NC global data. 8276 MTYPE RW and CC memory will 8277 never be stale in L2 due to 8278 the memory probes. 8279 8280 **Sequential Consistent Atomic** 8281 ------------------------------------------------------------------------------------ 8282 load atomic seq_cst - singlethread - global *Same as corresponding 8283 - wavefront - local load atomic acquire, 8284 - generic except must generated 8285 all instructions even 8286 for OpenCL.* 8287 load atomic seq_cst - workgroup - global 1. s_waitcnt lgkm/vmcnt(0) 8288 - generic 8289 - Use lgkmcnt(0) if not 8290 TgSplit execution mode 8291 and vmcnt(0) if TgSplit 8292 execution mode. 8293 - s_waitcnt lgkmcnt(0) must 8294 happen after 8295 preceding 8296 local/generic load 8297 atomic/store 8298 atomic/atomicrmw 8299 with memory 8300 ordering of seq_cst 8301 and with equal or 8302 wider sync scope. 8303 (Note that seq_cst 8304 fences have their 8305 own s_waitcnt 8306 lgkmcnt(0) and so do 8307 not need to be 8308 considered.) 8309 - s_waitcnt vmcnt(0) 8310 must happen after 8311 preceding 8312 global/generic load 8313 atomic/store 8314 atomic/atomicrmw 8315 with memory 8316 ordering of seq_cst 8317 and with equal or 8318 wider sync scope. 8319 (Note that seq_cst 8320 fences have their 8321 own s_waitcnt 8322 vmcnt(0) and so do 8323 not need to be 8324 considered.) 8325 - Ensures any 8326 preceding 8327 sequential 8328 consistent global/local 8329 memory instructions 8330 have completed 8331 before executing 8332 this sequentially 8333 consistent 8334 instruction. This 8335 prevents reordering 8336 a seq_cst store 8337 followed by a 8338 seq_cst load. (Note 8339 that seq_cst is 8340 stronger than 8341 acquire/release as 8342 the reordering of 8343 load acquire 8344 followed by a store 8345 release is 8346 prevented by the 8347 s_waitcnt of 8348 the release, but 8349 there is nothing 8350 preventing a store 8351 release followed by 8352 load acquire from 8353 completing out of 8354 order. The s_waitcnt 8355 could be placed after 8356 seq_store or before 8357 the seq_load. We 8358 choose the load to 8359 make the s_waitcnt be 8360 as late as possible 8361 so that the store 8362 may have already 8363 completed.) 8364 8365 2. *Following 8366 instructions same as 8367 corresponding load 8368 atomic acquire, 8369 except must generated 8370 all instructions even 8371 for OpenCL.* 8372 load atomic seq_cst - workgroup - local *If TgSplit execution mode, 8373 local address space cannot 8374 be used.* 8375 8376 *Same as corresponding 8377 load atomic acquire, 8378 except must generated 8379 all instructions even 8380 for OpenCL.* 8381 8382 load atomic seq_cst - agent - global 1. s_waitcnt lgkmcnt(0) & 8383 - system - generic vmcnt(0) 8384 8385 - If TgSplit execution mode, 8386 omit lgkmcnt(0). 8387 - Could be split into 8388 separate s_waitcnt 8389 vmcnt(0) 8390 and s_waitcnt 8391 lgkmcnt(0) to allow 8392 them to be 8393 independently moved 8394 according to the 8395 following rules. 8396 - s_waitcnt lgkmcnt(0) 8397 must happen after 8398 preceding 8399 global/generic load 8400 atomic/store 8401 atomic/atomicrmw 8402 with memory 8403 ordering of seq_cst 8404 and with equal or 8405 wider sync scope. 8406 (Note that seq_cst 8407 fences have their 8408 own s_waitcnt 8409 lgkmcnt(0) and so do 8410 not need to be 8411 considered.) 8412 - s_waitcnt vmcnt(0) 8413 must happen after 8414 preceding 8415 global/generic load 8416 atomic/store 8417 atomic/atomicrmw 8418 with memory 8419 ordering of seq_cst 8420 and with equal or 8421 wider sync scope. 8422 (Note that seq_cst 8423 fences have their 8424 own s_waitcnt 8425 vmcnt(0) and so do 8426 not need to be 8427 considered.) 8428 - Ensures any 8429 preceding 8430 sequential 8431 consistent global 8432 memory instructions 8433 have completed 8434 before executing 8435 this sequentially 8436 consistent 8437 instruction. This 8438 prevents reordering 8439 a seq_cst store 8440 followed by a 8441 seq_cst load. (Note 8442 that seq_cst is 8443 stronger than 8444 acquire/release as 8445 the reordering of 8446 load acquire 8447 followed by a store 8448 release is 8449 prevented by the 8450 s_waitcnt of 8451 the release, but 8452 there is nothing 8453 preventing a store 8454 release followed by 8455 load acquire from 8456 completing out of 8457 order. The s_waitcnt 8458 could be placed after 8459 seq_store or before 8460 the seq_load. We 8461 choose the load to 8462 make the s_waitcnt be 8463 as late as possible 8464 so that the store 8465 may have already 8466 completed.) 8467 8468 2. *Following 8469 instructions same as 8470 corresponding load 8471 atomic acquire, 8472 except must generated 8473 all instructions even 8474 for OpenCL.* 8475 store atomic seq_cst - singlethread - global *Same as corresponding 8476 - wavefront - local store atomic release, 8477 - workgroup - generic except must generated 8478 - agent all instructions even 8479 - system for OpenCL.* 8480 atomicrmw seq_cst - singlethread - global *Same as corresponding 8481 - wavefront - local atomicrmw acq_rel, 8482 - workgroup - generic except must generated 8483 - agent all instructions even 8484 - system for OpenCL.* 8485 fence seq_cst - singlethread *none* *Same as corresponding 8486 - wavefront fence acq_rel, 8487 - workgroup except must generated 8488 - agent all instructions even 8489 - system for OpenCL.* 8490 ============ ============ ============== ========== ================================ 8491 8492.. _amdgpu-amdhsa-memory-model-gfx10: 8493 8494Memory Model GFX10 8495++++++++++++++++++ 8496 8497For GFX10: 8498 8499* Each agent has multiple shader arrays (SA). 8500* Each SA has multiple work-group processors (WGP). 8501* Each WGP has multiple compute units (CU). 8502* Each CU has multiple SIMDs that execute wavefronts. 8503* The wavefronts for a single work-group are executed in the same 8504 WGP. In CU wavefront execution mode the wavefronts may be executed by 8505 different SIMDs in the same CU. In WGP wavefront execution mode the 8506 wavefronts may be executed by different SIMDs in different CUs in the same 8507 WGP. 8508* Each WGP has a single LDS memory shared by the wavefronts of the work-groups 8509 executing on it. 8510* All LDS operations of a WGP are performed as wavefront wide operations in a 8511 global order and involve no caching. Completion is reported to a wavefront in 8512 execution order. 8513* The LDS memory has multiple request queues shared by the SIMDs of a 8514 WGP. Therefore, the LDS operations performed by different wavefronts of a 8515 work-group can be reordered relative to each other, which can result in 8516 reordering the visibility of vector memory operations with respect to LDS 8517 operations of other wavefronts in the same work-group. A ``s_waitcnt 8518 lgkmcnt(0)`` is required to ensure synchronization between LDS operations and 8519 vector memory operations between wavefronts of a work-group, but not between 8520 operations performed by the same wavefront. 8521* The vector memory operations are performed as wavefront wide operations. 8522 Completion of load/store/sample operations are reported to a wavefront in 8523 execution order of other load/store/sample operations performed by that 8524 wavefront. 8525* The vector memory operations access a vector L0 cache. There is a single L0 8526 cache per CU. Each SIMD of a CU accesses the same L0 cache. Therefore, no 8527 special action is required for coherence between the lanes of a single 8528 wavefront. However, a ``buffer_gl0_inv`` is required for coherence between 8529 wavefronts executing in the same work-group as they may be executing on SIMDs 8530 of different CUs that access different L0s. A ``buffer_gl0_inv`` is also 8531 required for coherence between wavefronts executing in different work-groups 8532 as they may be executing on different WGPs. 8533* The scalar memory operations access a scalar L0 cache shared by all wavefronts 8534 on a WGP. The scalar and vector L0 caches are not coherent. However, scalar 8535 operations are used in a restricted way so do not impact the memory model. See 8536 :ref:`amdgpu-amdhsa-memory-spaces`. 8537* The vector and scalar memory L0 caches use an L1 cache shared by all WGPs on 8538 the same SA. Therefore, no special action is required for coherence between 8539 the wavefronts of a single work-group. However, a ``buffer_gl1_inv`` is 8540 required for coherence between wavefronts executing in different work-groups 8541 as they may be executing on different SAs that access different L1s. 8542* The L1 caches have independent quadrants to service disjoint ranges of virtual 8543 addresses. 8544* Each L0 cache has a separate request queue per L1 quadrant. Therefore, the 8545 vector and scalar memory operations performed by different wavefronts, whether 8546 executing in the same or different work-groups (which may be executing on 8547 different CUs accessing different L0s), can be reordered relative to each 8548 other. A ``s_waitcnt vmcnt(0) & vscnt(0)`` is required to ensure 8549 synchronization between vector memory operations of different wavefronts. It 8550 ensures a previous vector memory operation has completed before executing a 8551 subsequent vector memory or LDS operation and so can be used to meet the 8552 requirements of acquire, release and sequential consistency. 8553* The L1 caches use an L2 cache shared by all SAs on the same agent. 8554* The L2 cache has independent channels to service disjoint ranges of virtual 8555 addresses. 8556* Each L1 quadrant of a single SA accesses a different L2 channel. Each L1 8557 quadrant has a separate request queue per L2 channel. Therefore, the vector 8558 and scalar memory operations performed by wavefronts executing in different 8559 work-groups (which may be executing on different SAs) of an agent can be 8560 reordered relative to each other. A ``s_waitcnt vmcnt(0) & vscnt(0)`` is 8561 required to ensure synchronization between vector memory operations of 8562 different SAs. It ensures a previous vector memory operation has completed 8563 before executing a subsequent vector memory and so can be used to meet the 8564 requirements of acquire, release and sequential consistency. 8565* The L2 cache can be kept coherent with other agents on some targets, or ranges 8566 of virtual addresses can be set up to bypass it to ensure system coherence. 8567 8568Scalar memory operations are only used to access memory that is proven to not 8569change during the execution of the kernel dispatch. This includes constant 8570address space and global address space for program scope ``const`` variables. 8571Therefore, the kernel machine code does not have to maintain the scalar cache to 8572ensure it is coherent with the vector caches. The scalar and vector caches are 8573invalidated between kernel dispatches by CP since constant address space data 8574may change between kernel dispatch executions. See 8575:ref:`amdgpu-amdhsa-memory-spaces`. 8576 8577The one exception is if scalar writes are used to spill SGPR registers. In this 8578case the AMDGPU backend ensures the memory location used to spill is never 8579accessed by vector memory operations at the same time. If scalar writes are used 8580then a ``s_dcache_wb`` is inserted before the ``s_endpgm`` and before a function 8581return since the locations may be used for vector memory instructions by a 8582future wavefront that uses the same scratch area, or a function call that 8583creates a frame at the same address, respectively. There is no need for a 8584``s_dcache_inv`` as all scalar writes are write-before-read in the same thread. 8585 8586For kernarg backing memory: 8587 8588* CP invalidates the L0 and L1 caches at the start of each kernel dispatch. 8589* On dGPU the kernarg backing memory is accessed as MTYPE UC (uncached) to avoid 8590 needing to invalidate the L2 cache. 8591* On APU the kernarg backing memory is accessed as MTYPE CC (cache coherent) and 8592 so the L2 cache will be coherent with the CPU and other agents. 8593 8594Scratch backing memory (which is used for the private address space) is accessed 8595with MTYPE NC (non-coherent). Since the private address space is only accessed 8596by a single thread, and is always write-before-read, there is never a need to 8597invalidate these entries from the L0 or L1 caches. 8598 8599Wavefronts are executed in native mode with in-order reporting of loads and 8600sample instructions. In this mode vmcnt reports completion of load, atomic with 8601return and sample instructions in order, and the vscnt reports the completion of 8602store and atomic without return in order. See ``MEM_ORDERED`` field in 8603:ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 8604 8605Wavefronts can be executed in WGP or CU wavefront execution mode: 8606 8607* In WGP wavefront execution mode the wavefronts of a work-group are executed 8608 on the SIMDs of both CUs of the WGP. Therefore, explicit management of the per 8609 CU L0 caches is required for work-group synchronization. Also accesses to L1 8610 at work-group scope need to be explicitly ordered as the accesses from 8611 different CUs are not ordered. 8612* In CU wavefront execution mode the wavefronts of a work-group are executed on 8613 the SIMDs of a single CU of the WGP. Therefore, all global memory access by 8614 the work-group access the same L0 which in turn ensures L1 accesses are 8615 ordered and so do not require explicit management of the caches for 8616 work-group synchronization. 8617 8618See ``WGP_MODE`` field in 8619:ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table` and 8620:ref:`amdgpu-target-features`. 8621 8622The code sequences used to implement the memory model for GFX10 are defined in 8623table :ref:`amdgpu-amdhsa-memory-model-code-sequences-gfx10-table`. 8624 8625 .. table:: AMDHSA Memory Model Code Sequences GFX10 8626 :name: amdgpu-amdhsa-memory-model-code-sequences-gfx10-table 8627 8628 ============ ============ ============== ========== ================================ 8629 LLVM Instr LLVM Memory LLVM Memory AMDGPU AMDGPU Machine Code 8630 Ordering Sync Scope Address GFX10 8631 Space 8632 ============ ============ ============== ========== ================================ 8633 **Non-Atomic** 8634 ------------------------------------------------------------------------------------ 8635 load *none* *none* - global - !volatile & !nontemporal 8636 - generic 8637 - private 1. buffer/global/flat_load 8638 - constant 8639 - !volatile & nontemporal 8640 8641 1. buffer/global/flat_load 8642 slc=1 8643 8644 - volatile 8645 8646 1. buffer/global/flat_load 8647 glc=1 dlc=1 8648 2. s_waitcnt vmcnt(0) 8649 8650 - Must happen before 8651 any following volatile 8652 global/generic 8653 load/store. 8654 - Ensures that 8655 volatile 8656 operations to 8657 different 8658 addresses will not 8659 be reordered by 8660 hardware. 8661 8662 load *none* *none* - local 1. ds_load 8663 store *none* *none* - global - !volatile & !nontemporal 8664 - generic 8665 - private 1. buffer/global/flat_store 8666 - constant 8667 - !volatile & nontemporal 8668 8669 1. buffer/global/flat_store 8670 slc=1 8671 8672 - volatile 8673 8674 1. buffer/global/flat_store 8675 2. s_waitcnt vscnt(0) 8676 8677 - Must happen before 8678 any following volatile 8679 global/generic 8680 load/store. 8681 - Ensures that 8682 volatile 8683 operations to 8684 different 8685 addresses will not 8686 be reordered by 8687 hardware. 8688 8689 store *none* *none* - local 1. ds_store 8690 **Unordered Atomic** 8691 ------------------------------------------------------------------------------------ 8692 load atomic unordered *any* *any* *Same as non-atomic*. 8693 store atomic unordered *any* *any* *Same as non-atomic*. 8694 atomicrmw unordered *any* *any* *Same as monotonic atomic*. 8695 **Monotonic Atomic** 8696 ------------------------------------------------------------------------------------ 8697 load atomic monotonic - singlethread - global 1. buffer/global/flat_load 8698 - wavefront - generic 8699 load atomic monotonic - workgroup - global 1. buffer/global/flat_load 8700 - generic glc=1 8701 8702 - If CU wavefront execution 8703 mode, omit glc=1. 8704 8705 load atomic monotonic - singlethread - local 1. ds_load 8706 - wavefront 8707 - workgroup 8708 load atomic monotonic - agent - global 1. buffer/global/flat_load 8709 - system - generic glc=1 dlc=1 8710 store atomic monotonic - singlethread - global 1. buffer/global/flat_store 8711 - wavefront - generic 8712 - workgroup 8713 - agent 8714 - system 8715 store atomic monotonic - singlethread - local 1. ds_store 8716 - wavefront 8717 - workgroup 8718 atomicrmw monotonic - singlethread - global 1. buffer/global/flat_atomic 8719 - wavefront - generic 8720 - workgroup 8721 - agent 8722 - system 8723 atomicrmw monotonic - singlethread - local 1. ds_atomic 8724 - wavefront 8725 - workgroup 8726 **Acquire Atomic** 8727 ------------------------------------------------------------------------------------ 8728 load atomic acquire - singlethread - global 1. buffer/global/ds/flat_load 8729 - wavefront - local 8730 - generic 8731 load atomic acquire - workgroup - global 1. buffer/global_load glc=1 8732 8733 - If CU wavefront execution 8734 mode, omit glc=1. 8735 8736 2. s_waitcnt vmcnt(0) 8737 8738 - If CU wavefront execution 8739 mode, omit. 8740 - Must happen before 8741 the following buffer_gl0_inv 8742 and before any following 8743 global/generic 8744 load/load 8745 atomic/store/store 8746 atomic/atomicrmw. 8747 8748 3. buffer_gl0_inv 8749 8750 - If CU wavefront execution 8751 mode, omit. 8752 - Ensures that 8753 following 8754 loads will not see 8755 stale data. 8756 8757 load atomic acquire - workgroup - local 1. ds_load 8758 2. s_waitcnt lgkmcnt(0) 8759 8760 - If OpenCL, omit. 8761 - Must happen before 8762 the following buffer_gl0_inv 8763 and before any following 8764 global/generic load/load 8765 atomic/store/store 8766 atomic/atomicrmw. 8767 - Ensures any 8768 following global 8769 data read is no 8770 older than the local load 8771 atomic value being 8772 acquired. 8773 8774 3. buffer_gl0_inv 8775 8776 - If CU wavefront execution 8777 mode, omit. 8778 - If OpenCL, omit. 8779 - Ensures that 8780 following 8781 loads will not see 8782 stale data. 8783 8784 load atomic acquire - workgroup - generic 1. flat_load glc=1 8785 8786 - If CU wavefront execution 8787 mode, omit glc=1. 8788 8789 2. s_waitcnt lgkmcnt(0) & 8790 vmcnt(0) 8791 8792 - If CU wavefront execution 8793 mode, omit vmcnt(0). 8794 - If OpenCL, omit 8795 lgkmcnt(0). 8796 - Must happen before 8797 the following 8798 buffer_gl0_inv and any 8799 following global/generic 8800 load/load 8801 atomic/store/store 8802 atomic/atomicrmw. 8803 - Ensures any 8804 following global 8805 data read is no 8806 older than a local load 8807 atomic value being 8808 acquired. 8809 8810 3. buffer_gl0_inv 8811 8812 - If CU wavefront execution 8813 mode, omit. 8814 - Ensures that 8815 following 8816 loads will not see 8817 stale data. 8818 8819 load atomic acquire - agent - global 1. buffer/global_load 8820 - system glc=1 dlc=1 8821 2. s_waitcnt vmcnt(0) 8822 8823 - Must happen before 8824 following 8825 buffer_gl*_inv. 8826 - Ensures the load 8827 has completed 8828 before invalidating 8829 the caches. 8830 8831 3. buffer_gl0_inv; 8832 buffer_gl1_inv 8833 8834 - Must happen before 8835 any following 8836 global/generic 8837 load/load 8838 atomic/atomicrmw. 8839 - Ensures that 8840 following 8841 loads will not see 8842 stale global data. 8843 8844 load atomic acquire - agent - generic 1. flat_load glc=1 dlc=1 8845 - system 2. s_waitcnt vmcnt(0) & 8846 lgkmcnt(0) 8847 8848 - If OpenCL omit 8849 lgkmcnt(0). 8850 - Must happen before 8851 following 8852 buffer_gl*_invl. 8853 - Ensures the flat_load 8854 has completed 8855 before invalidating 8856 the caches. 8857 8858 3. buffer_gl0_inv; 8859 buffer_gl1_inv 8860 8861 - Must happen before 8862 any following 8863 global/generic 8864 load/load 8865 atomic/atomicrmw. 8866 - Ensures that 8867 following loads 8868 will not see stale 8869 global data. 8870 8871 atomicrmw acquire - singlethread - global 1. buffer/global/ds/flat_atomic 8872 - wavefront - local 8873 - generic 8874 atomicrmw acquire - workgroup - global 1. buffer/global_atomic 8875 2. s_waitcnt vm/vscnt(0) 8876 8877 - If CU wavefront execution 8878 mode, omit. 8879 - Use vmcnt(0) if atomic with 8880 return and vscnt(0) if 8881 atomic with no-return. 8882 - Must happen before 8883 the following buffer_gl0_inv 8884 and before any following 8885 global/generic 8886 load/load 8887 atomic/store/store 8888 atomic/atomicrmw. 8889 8890 3. buffer_gl0_inv 8891 8892 - If CU wavefront execution 8893 mode, omit. 8894 - Ensures that 8895 following 8896 loads will not see 8897 stale data. 8898 8899 atomicrmw acquire - workgroup - local 1. ds_atomic 8900 2. s_waitcnt lgkmcnt(0) 8901 8902 - If OpenCL, omit. 8903 - Must happen before 8904 the following 8905 buffer_gl0_inv. 8906 - Ensures any 8907 following global 8908 data read is no 8909 older than the local 8910 atomicrmw value 8911 being acquired. 8912 8913 3. buffer_gl0_inv 8914 8915 - If OpenCL omit. 8916 - Ensures that 8917 following 8918 loads will not see 8919 stale data. 8920 8921 atomicrmw acquire - workgroup - generic 1. flat_atomic 8922 2. s_waitcnt lgkmcnt(0) & 8923 vm/vscnt(0) 8924 8925 - If CU wavefront execution 8926 mode, omit vm/vscnt(0). 8927 - If OpenCL, omit lgkmcnt(0). 8928 - Use vmcnt(0) if atomic with 8929 return and vscnt(0) if 8930 atomic with no-return. 8931 - Must happen before 8932 the following 8933 buffer_gl0_inv. 8934 - Ensures any 8935 following global 8936 data read is no 8937 older than a local 8938 atomicrmw value 8939 being acquired. 8940 8941 3. buffer_gl0_inv 8942 8943 - If CU wavefront execution 8944 mode, omit. 8945 - Ensures that 8946 following 8947 loads will not see 8948 stale data. 8949 8950 atomicrmw acquire - agent - global 1. buffer/global_atomic 8951 - system 2. s_waitcnt vm/vscnt(0) 8952 8953 - Use vmcnt(0) if atomic with 8954 return and vscnt(0) if 8955 atomic with no-return. 8956 - Must happen before 8957 following 8958 buffer_gl*_inv. 8959 - Ensures the 8960 atomicrmw has 8961 completed before 8962 invalidating the 8963 caches. 8964 8965 3. buffer_gl0_inv; 8966 buffer_gl1_inv 8967 8968 - Must happen before 8969 any following 8970 global/generic 8971 load/load 8972 atomic/atomicrmw. 8973 - Ensures that 8974 following loads 8975 will not see stale 8976 global data. 8977 8978 atomicrmw acquire - agent - generic 1. flat_atomic 8979 - system 2. s_waitcnt vm/vscnt(0) & 8980 lgkmcnt(0) 8981 8982 - If OpenCL, omit 8983 lgkmcnt(0). 8984 - Use vmcnt(0) if atomic with 8985 return and vscnt(0) if 8986 atomic with no-return. 8987 - Must happen before 8988 following 8989 buffer_gl*_inv. 8990 - Ensures the 8991 atomicrmw has 8992 completed before 8993 invalidating the 8994 caches. 8995 8996 3. buffer_gl0_inv; 8997 buffer_gl1_inv 8998 8999 - Must happen before 9000 any following 9001 global/generic 9002 load/load 9003 atomic/atomicrmw. 9004 - Ensures that 9005 following loads 9006 will not see stale 9007 global data. 9008 9009 fence acquire - singlethread *none* *none* 9010 - wavefront 9011 fence acquire - workgroup *none* 1. s_waitcnt lgkmcnt(0) & 9012 vmcnt(0) & vscnt(0) 9013 9014 - If CU wavefront execution 9015 mode, omit vmcnt(0) and 9016 vscnt(0). 9017 - If OpenCL and 9018 address space is 9019 not generic, omit 9020 lgkmcnt(0). 9021 - If OpenCL and 9022 address space is 9023 local, omit 9024 vmcnt(0) and vscnt(0). 9025 - However, since LLVM 9026 currently has no 9027 address space on 9028 the fence need to 9029 conservatively 9030 always generate. If 9031 fence had an 9032 address space then 9033 set to address 9034 space of OpenCL 9035 fence flag, or to 9036 generic if both 9037 local and global 9038 flags are 9039 specified. 9040 - Could be split into 9041 separate s_waitcnt 9042 vmcnt(0), s_waitcnt 9043 vscnt(0) and s_waitcnt 9044 lgkmcnt(0) to allow 9045 them to be 9046 independently moved 9047 according to the 9048 following rules. 9049 - s_waitcnt vmcnt(0) 9050 must happen after 9051 any preceding 9052 global/generic load 9053 atomic/ 9054 atomicrmw-with-return-value 9055 with an equal or 9056 wider sync scope 9057 and memory ordering 9058 stronger than 9059 unordered (this is 9060 termed the 9061 fence-paired-atomic). 9062 - s_waitcnt vscnt(0) 9063 must happen after 9064 any preceding 9065 global/generic 9066 atomicrmw-no-return-value 9067 with an equal or 9068 wider sync scope 9069 and memory ordering 9070 stronger than 9071 unordered (this is 9072 termed the 9073 fence-paired-atomic). 9074 - s_waitcnt lgkmcnt(0) 9075 must happen after 9076 any preceding 9077 local/generic load 9078 atomic/atomicrmw 9079 with an equal or 9080 wider sync scope 9081 and memory ordering 9082 stronger than 9083 unordered (this is 9084 termed the 9085 fence-paired-atomic). 9086 - Must happen before 9087 the following 9088 buffer_gl0_inv. 9089 - Ensures that the 9090 fence-paired atomic 9091 has completed 9092 before invalidating 9093 the 9094 cache. Therefore 9095 any following 9096 locations read must 9097 be no older than 9098 the value read by 9099 the 9100 fence-paired-atomic. 9101 9102 3. buffer_gl0_inv 9103 9104 - If CU wavefront execution 9105 mode, omit. 9106 - Ensures that 9107 following 9108 loads will not see 9109 stale data. 9110 9111 fence acquire - agent *none* 1. s_waitcnt lgkmcnt(0) & 9112 - system vmcnt(0) & vscnt(0) 9113 9114 - If OpenCL and 9115 address space is 9116 not generic, omit 9117 lgkmcnt(0). 9118 - If OpenCL and 9119 address space is 9120 local, omit 9121 vmcnt(0) and vscnt(0). 9122 - However, since LLVM 9123 currently has no 9124 address space on 9125 the fence need to 9126 conservatively 9127 always generate 9128 (see comment for 9129 previous fence). 9130 - Could be split into 9131 separate s_waitcnt 9132 vmcnt(0), s_waitcnt 9133 vscnt(0) and s_waitcnt 9134 lgkmcnt(0) to allow 9135 them to be 9136 independently moved 9137 according to the 9138 following rules. 9139 - s_waitcnt vmcnt(0) 9140 must happen after 9141 any preceding 9142 global/generic load 9143 atomic/ 9144 atomicrmw-with-return-value 9145 with an equal or 9146 wider sync scope 9147 and memory ordering 9148 stronger than 9149 unordered (this is 9150 termed the 9151 fence-paired-atomic). 9152 - s_waitcnt vscnt(0) 9153 must happen after 9154 any preceding 9155 global/generic 9156 atomicrmw-no-return-value 9157 with an equal or 9158 wider sync scope 9159 and memory ordering 9160 stronger than 9161 unordered (this is 9162 termed the 9163 fence-paired-atomic). 9164 - s_waitcnt lgkmcnt(0) 9165 must happen after 9166 any preceding 9167 local/generic load 9168 atomic/atomicrmw 9169 with an equal or 9170 wider sync scope 9171 and memory ordering 9172 stronger than 9173 unordered (this is 9174 termed the 9175 fence-paired-atomic). 9176 - Must happen before 9177 the following 9178 buffer_gl*_inv. 9179 - Ensures that the 9180 fence-paired atomic 9181 has completed 9182 before invalidating 9183 the 9184 caches. Therefore 9185 any following 9186 locations read must 9187 be no older than 9188 the value read by 9189 the 9190 fence-paired-atomic. 9191 9192 2. buffer_gl0_inv; 9193 buffer_gl1_inv 9194 9195 - Must happen before any 9196 following global/generic 9197 load/load 9198 atomic/store/store 9199 atomic/atomicrmw. 9200 - Ensures that 9201 following loads 9202 will not see stale 9203 global data. 9204 9205 **Release Atomic** 9206 ------------------------------------------------------------------------------------ 9207 store atomic release - singlethread - global 1. buffer/global/ds/flat_store 9208 - wavefront - local 9209 - generic 9210 store atomic release - workgroup - global 1. s_waitcnt lgkmcnt(0) & 9211 - generic vmcnt(0) & vscnt(0) 9212 9213 - If CU wavefront execution 9214 mode, omit vmcnt(0) and 9215 vscnt(0). 9216 - If OpenCL, omit 9217 lgkmcnt(0). 9218 - Could be split into 9219 separate s_waitcnt 9220 vmcnt(0), s_waitcnt 9221 vscnt(0) and s_waitcnt 9222 lgkmcnt(0) to allow 9223 them to be 9224 independently moved 9225 according to the 9226 following rules. 9227 - s_waitcnt vmcnt(0) 9228 must happen after 9229 any preceding 9230 global/generic load/load 9231 atomic/ 9232 atomicrmw-with-return-value. 9233 - s_waitcnt vscnt(0) 9234 must happen after 9235 any preceding 9236 global/generic 9237 store/store 9238 atomic/ 9239 atomicrmw-no-return-value. 9240 - s_waitcnt lgkmcnt(0) 9241 must happen after 9242 any preceding 9243 local/generic 9244 load/store/load 9245 atomic/store 9246 atomic/atomicrmw. 9247 - Must happen before 9248 the following 9249 store. 9250 - Ensures that all 9251 memory operations 9252 have 9253 completed before 9254 performing the 9255 store that is being 9256 released. 9257 9258 2. buffer/global/flat_store 9259 store atomic release - workgroup - local 1. s_waitcnt vmcnt(0) & vscnt(0) 9260 9261 - If CU wavefront execution 9262 mode, omit. 9263 - If OpenCL, omit. 9264 - Could be split into 9265 separate s_waitcnt 9266 vmcnt(0) and s_waitcnt 9267 vscnt(0) to allow 9268 them to be 9269 independently moved 9270 according to the 9271 following rules. 9272 - s_waitcnt vmcnt(0) 9273 must happen after 9274 any preceding 9275 global/generic load/load 9276 atomic/ 9277 atomicrmw-with-return-value. 9278 - s_waitcnt vscnt(0) 9279 must happen after 9280 any preceding 9281 global/generic 9282 store/store atomic/ 9283 atomicrmw-no-return-value. 9284 - Must happen before 9285 the following 9286 store. 9287 - Ensures that all 9288 global memory 9289 operations have 9290 completed before 9291 performing the 9292 store that is being 9293 released. 9294 9295 2. ds_store 9296 store atomic release - agent - global 1. s_waitcnt lgkmcnt(0) & 9297 - system - generic vmcnt(0) & vscnt(0) 9298 9299 - If OpenCL and 9300 address space is 9301 not generic, omit 9302 lgkmcnt(0). 9303 - Could be split into 9304 separate s_waitcnt 9305 vmcnt(0), s_waitcnt vscnt(0) 9306 and s_waitcnt 9307 lgkmcnt(0) to allow 9308 them to be 9309 independently moved 9310 according to the 9311 following rules. 9312 - s_waitcnt vmcnt(0) 9313 must happen after 9314 any preceding 9315 global/generic 9316 load/load 9317 atomic/ 9318 atomicrmw-with-return-value. 9319 - s_waitcnt vscnt(0) 9320 must happen after 9321 any preceding 9322 global/generic 9323 store/store atomic/ 9324 atomicrmw-no-return-value. 9325 - s_waitcnt lgkmcnt(0) 9326 must happen after 9327 any preceding 9328 local/generic 9329 load/store/load 9330 atomic/store 9331 atomic/atomicrmw. 9332 - Must happen before 9333 the following 9334 store. 9335 - Ensures that all 9336 memory operations 9337 have 9338 completed before 9339 performing the 9340 store that is being 9341 released. 9342 9343 2. buffer/global/flat_store 9344 atomicrmw release - singlethread - global 1. buffer/global/ds/flat_atomic 9345 - wavefront - local 9346 - generic 9347 atomicrmw release - workgroup - global 1. s_waitcnt lgkmcnt(0) & 9348 - generic vmcnt(0) & vscnt(0) 9349 9350 - If CU wavefront execution 9351 mode, omit vmcnt(0) and 9352 vscnt(0). 9353 - If OpenCL, omit lgkmcnt(0). 9354 - Could be split into 9355 separate s_waitcnt 9356 vmcnt(0), s_waitcnt 9357 vscnt(0) and s_waitcnt 9358 lgkmcnt(0) to allow 9359 them to be 9360 independently moved 9361 according to the 9362 following rules. 9363 - s_waitcnt vmcnt(0) 9364 must happen after 9365 any preceding 9366 global/generic load/load 9367 atomic/ 9368 atomicrmw-with-return-value. 9369 - s_waitcnt vscnt(0) 9370 must happen after 9371 any preceding 9372 global/generic 9373 store/store 9374 atomic/ 9375 atomicrmw-no-return-value. 9376 - s_waitcnt lgkmcnt(0) 9377 must happen after 9378 any preceding 9379 local/generic 9380 load/store/load 9381 atomic/store 9382 atomic/atomicrmw. 9383 - Must happen before 9384 the following 9385 atomicrmw. 9386 - Ensures that all 9387 memory operations 9388 have 9389 completed before 9390 performing the 9391 atomicrmw that is 9392 being released. 9393 9394 2. buffer/global/flat_atomic 9395 atomicrmw release - workgroup - local 1. s_waitcnt vmcnt(0) & vscnt(0) 9396 9397 - If CU wavefront execution 9398 mode, omit. 9399 - If OpenCL, omit. 9400 - Could be split into 9401 separate s_waitcnt 9402 vmcnt(0) and s_waitcnt 9403 vscnt(0) to allow 9404 them to be 9405 independently moved 9406 according to the 9407 following rules. 9408 - s_waitcnt vmcnt(0) 9409 must happen after 9410 any preceding 9411 global/generic load/load 9412 atomic/ 9413 atomicrmw-with-return-value. 9414 - s_waitcnt vscnt(0) 9415 must happen after 9416 any preceding 9417 global/generic 9418 store/store atomic/ 9419 atomicrmw-no-return-value. 9420 - Must happen before 9421 the following 9422 store. 9423 - Ensures that all 9424 global memory 9425 operations have 9426 completed before 9427 performing the 9428 store that is being 9429 released. 9430 9431 2. ds_atomic 9432 atomicrmw release - agent - global 1. s_waitcnt lgkmcnt(0) & 9433 - system - generic vmcnt(0) & vscnt(0) 9434 9435 - If OpenCL, omit 9436 lgkmcnt(0). 9437 - Could be split into 9438 separate s_waitcnt 9439 vmcnt(0), s_waitcnt 9440 vscnt(0) and s_waitcnt 9441 lgkmcnt(0) to allow 9442 them to be 9443 independently moved 9444 according to the 9445 following rules. 9446 - s_waitcnt vmcnt(0) 9447 must happen after 9448 any preceding 9449 global/generic 9450 load/load atomic/ 9451 atomicrmw-with-return-value. 9452 - s_waitcnt vscnt(0) 9453 must happen after 9454 any preceding 9455 global/generic 9456 store/store atomic/ 9457 atomicrmw-no-return-value. 9458 - s_waitcnt lgkmcnt(0) 9459 must happen after 9460 any preceding 9461 local/generic 9462 load/store/load 9463 atomic/store 9464 atomic/atomicrmw. 9465 - Must happen before 9466 the following 9467 atomicrmw. 9468 - Ensures that all 9469 memory operations 9470 to global and local 9471 have completed 9472 before performing 9473 the atomicrmw that 9474 is being released. 9475 9476 2. buffer/global/flat_atomic 9477 fence release - singlethread *none* *none* 9478 - wavefront 9479 fence release - workgroup *none* 1. s_waitcnt lgkmcnt(0) & 9480 vmcnt(0) & vscnt(0) 9481 9482 - If CU wavefront execution 9483 mode, omit vmcnt(0) and 9484 vscnt(0). 9485 - If OpenCL and 9486 address space is 9487 not generic, omit 9488 lgkmcnt(0). 9489 - If OpenCL and 9490 address space is 9491 local, omit 9492 vmcnt(0) and vscnt(0). 9493 - However, since LLVM 9494 currently has no 9495 address space on 9496 the fence need to 9497 conservatively 9498 always generate. If 9499 fence had an 9500 address space then 9501 set to address 9502 space of OpenCL 9503 fence flag, or to 9504 generic if both 9505 local and global 9506 flags are 9507 specified. 9508 - Could be split into 9509 separate s_waitcnt 9510 vmcnt(0), s_waitcnt 9511 vscnt(0) and s_waitcnt 9512 lgkmcnt(0) to allow 9513 them to be 9514 independently moved 9515 according to the 9516 following rules. 9517 - s_waitcnt vmcnt(0) 9518 must happen after 9519 any preceding 9520 global/generic 9521 load/load 9522 atomic/ 9523 atomicrmw-with-return-value. 9524 - s_waitcnt vscnt(0) 9525 must happen after 9526 any preceding 9527 global/generic 9528 store/store atomic/ 9529 atomicrmw-no-return-value. 9530 - s_waitcnt lgkmcnt(0) 9531 must happen after 9532 any preceding 9533 local/generic 9534 load/store/load 9535 atomic/store atomic/ 9536 atomicrmw. 9537 - Must happen before 9538 any following store 9539 atomic/atomicrmw 9540 with an equal or 9541 wider sync scope 9542 and memory ordering 9543 stronger than 9544 unordered (this is 9545 termed the 9546 fence-paired-atomic). 9547 - Ensures that all 9548 memory operations 9549 have 9550 completed before 9551 performing the 9552 following 9553 fence-paired-atomic. 9554 9555 fence release - agent *none* 1. s_waitcnt lgkmcnt(0) & 9556 - system vmcnt(0) & vscnt(0) 9557 9558 - If OpenCL and 9559 address space is 9560 not generic, omit 9561 lgkmcnt(0). 9562 - If OpenCL and 9563 address space is 9564 local, omit 9565 vmcnt(0) and vscnt(0). 9566 - However, since LLVM 9567 currently has no 9568 address space on 9569 the fence need to 9570 conservatively 9571 always generate. If 9572 fence had an 9573 address space then 9574 set to address 9575 space of OpenCL 9576 fence flag, or to 9577 generic if both 9578 local and global 9579 flags are 9580 specified. 9581 - Could be split into 9582 separate s_waitcnt 9583 vmcnt(0), s_waitcnt 9584 vscnt(0) and s_waitcnt 9585 lgkmcnt(0) to allow 9586 them to be 9587 independently moved 9588 according to the 9589 following rules. 9590 - s_waitcnt vmcnt(0) 9591 must happen after 9592 any preceding 9593 global/generic 9594 load/load atomic/ 9595 atomicrmw-with-return-value. 9596 - s_waitcnt vscnt(0) 9597 must happen after 9598 any preceding 9599 global/generic 9600 store/store atomic/ 9601 atomicrmw-no-return-value. 9602 - s_waitcnt lgkmcnt(0) 9603 must happen after 9604 any preceding 9605 local/generic 9606 load/store/load 9607 atomic/store 9608 atomic/atomicrmw. 9609 - Must happen before 9610 any following store 9611 atomic/atomicrmw 9612 with an equal or 9613 wider sync scope 9614 and memory ordering 9615 stronger than 9616 unordered (this is 9617 termed the 9618 fence-paired-atomic). 9619 - Ensures that all 9620 memory operations 9621 have 9622 completed before 9623 performing the 9624 following 9625 fence-paired-atomic. 9626 9627 **Acquire-Release Atomic** 9628 ------------------------------------------------------------------------------------ 9629 atomicrmw acq_rel - singlethread - global 1. buffer/global/ds/flat_atomic 9630 - wavefront - local 9631 - generic 9632 atomicrmw acq_rel - workgroup - global 1. s_waitcnt lgkmcnt(0) & 9633 vmcnt(0) & vscnt(0) 9634 9635 - If CU wavefront execution 9636 mode, omit vmcnt(0) and 9637 vscnt(0). 9638 - If OpenCL, omit 9639 lgkmcnt(0). 9640 - Must happen after 9641 any preceding 9642 local/generic 9643 load/store/load 9644 atomic/store 9645 atomic/atomicrmw. 9646 - Could be split into 9647 separate s_waitcnt 9648 vmcnt(0), s_waitcnt 9649 vscnt(0), and s_waitcnt 9650 lgkmcnt(0) to allow 9651 them to be 9652 independently moved 9653 according to the 9654 following rules. 9655 - s_waitcnt vmcnt(0) 9656 must happen after 9657 any preceding 9658 global/generic load/load 9659 atomic/ 9660 atomicrmw-with-return-value. 9661 - s_waitcnt vscnt(0) 9662 must happen after 9663 any preceding 9664 global/generic 9665 store/store 9666 atomic/ 9667 atomicrmw-no-return-value. 9668 - s_waitcnt lgkmcnt(0) 9669 must happen after 9670 any preceding 9671 local/generic 9672 load/store/load 9673 atomic/store 9674 atomic/atomicrmw. 9675 - Must happen before 9676 the following 9677 atomicrmw. 9678 - Ensures that all 9679 memory operations 9680 have 9681 completed before 9682 performing the 9683 atomicrmw that is 9684 being released. 9685 9686 2. buffer/global_atomic 9687 3. s_waitcnt vm/vscnt(0) 9688 9689 - If CU wavefront execution 9690 mode, omit. 9691 - Use vmcnt(0) if atomic with 9692 return and vscnt(0) if 9693 atomic with no-return. 9694 - Must happen before 9695 the following 9696 buffer_gl0_inv. 9697 - Ensures any 9698 following global 9699 data read is no 9700 older than the 9701 atomicrmw value 9702 being acquired. 9703 9704 4. buffer_gl0_inv 9705 9706 - If CU wavefront execution 9707 mode, omit. 9708 - Ensures that 9709 following 9710 loads will not see 9711 stale data. 9712 9713 atomicrmw acq_rel - workgroup - local 1. s_waitcnt vmcnt(0) & vscnt(0) 9714 9715 - If CU wavefront execution 9716 mode, omit. 9717 - If OpenCL, omit. 9718 - Could be split into 9719 separate s_waitcnt 9720 vmcnt(0) and s_waitcnt 9721 vscnt(0) to allow 9722 them to be 9723 independently moved 9724 according to the 9725 following rules. 9726 - s_waitcnt vmcnt(0) 9727 must happen after 9728 any preceding 9729 global/generic load/load 9730 atomic/ 9731 atomicrmw-with-return-value. 9732 - s_waitcnt vscnt(0) 9733 must happen after 9734 any preceding 9735 global/generic 9736 store/store atomic/ 9737 atomicrmw-no-return-value. 9738 - Must happen before 9739 the following 9740 store. 9741 - Ensures that all 9742 global memory 9743 operations have 9744 completed before 9745 performing the 9746 store that is being 9747 released. 9748 9749 2. ds_atomic 9750 3. s_waitcnt lgkmcnt(0) 9751 9752 - If OpenCL, omit. 9753 - Must happen before 9754 the following 9755 buffer_gl0_inv. 9756 - Ensures any 9757 following global 9758 data read is no 9759 older than the local load 9760 atomic value being 9761 acquired. 9762 9763 4. buffer_gl0_inv 9764 9765 - If CU wavefront execution 9766 mode, omit. 9767 - If OpenCL omit. 9768 - Ensures that 9769 following 9770 loads will not see 9771 stale data. 9772 9773 atomicrmw acq_rel - workgroup - generic 1. s_waitcnt lgkmcnt(0) & 9774 vmcnt(0) & vscnt(0) 9775 9776 - If CU wavefront execution 9777 mode, omit vmcnt(0) and 9778 vscnt(0). 9779 - If OpenCL, omit lgkmcnt(0). 9780 - Could be split into 9781 separate s_waitcnt 9782 vmcnt(0), s_waitcnt 9783 vscnt(0) and s_waitcnt 9784 lgkmcnt(0) to allow 9785 them to be 9786 independently moved 9787 according to the 9788 following rules. 9789 - s_waitcnt vmcnt(0) 9790 must happen after 9791 any preceding 9792 global/generic load/load 9793 atomic/ 9794 atomicrmw-with-return-value. 9795 - s_waitcnt vscnt(0) 9796 must happen after 9797 any preceding 9798 global/generic 9799 store/store 9800 atomic/ 9801 atomicrmw-no-return-value. 9802 - s_waitcnt lgkmcnt(0) 9803 must happen after 9804 any preceding 9805 local/generic 9806 load/store/load 9807 atomic/store 9808 atomic/atomicrmw. 9809 - Must happen before 9810 the following 9811 atomicrmw. 9812 - Ensures that all 9813 memory operations 9814 have 9815 completed before 9816 performing the 9817 atomicrmw that is 9818 being released. 9819 9820 2. flat_atomic 9821 3. s_waitcnt lgkmcnt(0) & 9822 vmcnt(0) & vscnt(0) 9823 9824 - If CU wavefront execution 9825 mode, omit vmcnt(0) and 9826 vscnt(0). 9827 - If OpenCL, omit lgkmcnt(0). 9828 - Must happen before 9829 the following 9830 buffer_gl0_inv. 9831 - Ensures any 9832 following global 9833 data read is no 9834 older than the load 9835 atomic value being 9836 acquired. 9837 9838 3. buffer_gl0_inv 9839 9840 - If CU wavefront execution 9841 mode, omit. 9842 - Ensures that 9843 following 9844 loads will not see 9845 stale data. 9846 9847 atomicrmw acq_rel - agent - global 1. s_waitcnt lgkmcnt(0) & 9848 - system vmcnt(0) & vscnt(0) 9849 9850 - If OpenCL, omit 9851 lgkmcnt(0). 9852 - Could be split into 9853 separate s_waitcnt 9854 vmcnt(0), s_waitcnt 9855 vscnt(0) and s_waitcnt 9856 lgkmcnt(0) to allow 9857 them to be 9858 independently moved 9859 according to the 9860 following rules. 9861 - s_waitcnt vmcnt(0) 9862 must happen after 9863 any preceding 9864 global/generic 9865 load/load atomic/ 9866 atomicrmw-with-return-value. 9867 - s_waitcnt vscnt(0) 9868 must happen after 9869 any preceding 9870 global/generic 9871 store/store atomic/ 9872 atomicrmw-no-return-value. 9873 - s_waitcnt lgkmcnt(0) 9874 must happen after 9875 any preceding 9876 local/generic 9877 load/store/load 9878 atomic/store 9879 atomic/atomicrmw. 9880 - Must happen before 9881 the following 9882 atomicrmw. 9883 - Ensures that all 9884 memory operations 9885 to global have 9886 completed before 9887 performing the 9888 atomicrmw that is 9889 being released. 9890 9891 2. buffer/global_atomic 9892 3. s_waitcnt vm/vscnt(0) 9893 9894 - Use vmcnt(0) if atomic with 9895 return and vscnt(0) if 9896 atomic with no-return. 9897 - Must happen before 9898 following 9899 buffer_gl*_inv. 9900 - Ensures the 9901 atomicrmw has 9902 completed before 9903 invalidating the 9904 caches. 9905 9906 4. buffer_gl0_inv; 9907 buffer_gl1_inv 9908 9909 - Must happen before 9910 any following 9911 global/generic 9912 load/load 9913 atomic/atomicrmw. 9914 - Ensures that 9915 following loads 9916 will not see stale 9917 global data. 9918 9919 atomicrmw acq_rel - agent - generic 1. s_waitcnt lgkmcnt(0) & 9920 - system vmcnt(0) & vscnt(0) 9921 9922 - If OpenCL, omit 9923 lgkmcnt(0). 9924 - Could be split into 9925 separate s_waitcnt 9926 vmcnt(0), s_waitcnt 9927 vscnt(0), and s_waitcnt 9928 lgkmcnt(0) to allow 9929 them to be 9930 independently moved 9931 according to the 9932 following rules. 9933 - s_waitcnt vmcnt(0) 9934 must happen after 9935 any preceding 9936 global/generic 9937 load/load atomic 9938 atomicrmw-with-return-value. 9939 - s_waitcnt vscnt(0) 9940 must happen after 9941 any preceding 9942 global/generic 9943 store/store atomic/ 9944 atomicrmw-no-return-value. 9945 - s_waitcnt lgkmcnt(0) 9946 must happen after 9947 any preceding 9948 local/generic 9949 load/store/load 9950 atomic/store 9951 atomic/atomicrmw. 9952 - Must happen before 9953 the following 9954 atomicrmw. 9955 - Ensures that all 9956 memory operations 9957 have 9958 completed before 9959 performing the 9960 atomicrmw that is 9961 being released. 9962 9963 2. flat_atomic 9964 3. s_waitcnt vm/vscnt(0) & 9965 lgkmcnt(0) 9966 9967 - If OpenCL, omit 9968 lgkmcnt(0). 9969 - Use vmcnt(0) if atomic with 9970 return and vscnt(0) if 9971 atomic with no-return. 9972 - Must happen before 9973 following 9974 buffer_gl*_inv. 9975 - Ensures the 9976 atomicrmw has 9977 completed before 9978 invalidating the 9979 caches. 9980 9981 4. buffer_gl0_inv; 9982 buffer_gl1_inv 9983 9984 - Must happen before 9985 any following 9986 global/generic 9987 load/load 9988 atomic/atomicrmw. 9989 - Ensures that 9990 following loads 9991 will not see stale 9992 global data. 9993 9994 fence acq_rel - singlethread *none* *none* 9995 - wavefront 9996 fence acq_rel - workgroup *none* 1. s_waitcnt lgkmcnt(0) & 9997 vmcnt(0) & vscnt(0) 9998 9999 - If CU wavefront execution 10000 mode, omit vmcnt(0) and 10001 vscnt(0). 10002 - If OpenCL and 10003 address space is 10004 not generic, omit 10005 lgkmcnt(0). 10006 - If OpenCL and 10007 address space is 10008 local, omit 10009 vmcnt(0) and vscnt(0). 10010 - However, 10011 since LLVM 10012 currently has no 10013 address space on 10014 the fence need to 10015 conservatively 10016 always generate 10017 (see comment for 10018 previous fence). 10019 - Could be split into 10020 separate s_waitcnt 10021 vmcnt(0), s_waitcnt 10022 vscnt(0) and s_waitcnt 10023 lgkmcnt(0) to allow 10024 them to be 10025 independently moved 10026 according to the 10027 following rules. 10028 - s_waitcnt vmcnt(0) 10029 must happen after 10030 any preceding 10031 global/generic 10032 load/load 10033 atomic/ 10034 atomicrmw-with-return-value. 10035 - s_waitcnt vscnt(0) 10036 must happen after 10037 any preceding 10038 global/generic 10039 store/store atomic/ 10040 atomicrmw-no-return-value. 10041 - s_waitcnt lgkmcnt(0) 10042 must happen after 10043 any preceding 10044 local/generic 10045 load/store/load 10046 atomic/store atomic/ 10047 atomicrmw. 10048 - Must happen before 10049 any following 10050 global/generic 10051 load/load 10052 atomic/store/store 10053 atomic/atomicrmw. 10054 - Ensures that all 10055 memory operations 10056 have 10057 completed before 10058 performing any 10059 following global 10060 memory operations. 10061 - Ensures that the 10062 preceding 10063 local/generic load 10064 atomic/atomicrmw 10065 with an equal or 10066 wider sync scope 10067 and memory ordering 10068 stronger than 10069 unordered (this is 10070 termed the 10071 acquire-fence-paired-atomic) 10072 has completed 10073 before following 10074 global memory 10075 operations. This 10076 satisfies the 10077 requirements of 10078 acquire. 10079 - Ensures that all 10080 previous memory 10081 operations have 10082 completed before a 10083 following 10084 local/generic store 10085 atomic/atomicrmw 10086 with an equal or 10087 wider sync scope 10088 and memory ordering 10089 stronger than 10090 unordered (this is 10091 termed the 10092 release-fence-paired-atomic). 10093 This satisfies the 10094 requirements of 10095 release. 10096 - Must happen before 10097 the following 10098 buffer_gl0_inv. 10099 - Ensures that the 10100 acquire-fence-paired 10101 atomic has completed 10102 before invalidating 10103 the 10104 cache. Therefore 10105 any following 10106 locations read must 10107 be no older than 10108 the value read by 10109 the 10110 acquire-fence-paired-atomic. 10111 10112 3. buffer_gl0_inv 10113 10114 - If CU wavefront execution 10115 mode, omit. 10116 - Ensures that 10117 following 10118 loads will not see 10119 stale data. 10120 10121 fence acq_rel - agent *none* 1. s_waitcnt lgkmcnt(0) & 10122 - system vmcnt(0) & vscnt(0) 10123 10124 - If OpenCL and 10125 address space is 10126 not generic, omit 10127 lgkmcnt(0). 10128 - If OpenCL and 10129 address space is 10130 local, omit 10131 vmcnt(0) and vscnt(0). 10132 - However, since LLVM 10133 currently has no 10134 address space on 10135 the fence need to 10136 conservatively 10137 always generate 10138 (see comment for 10139 previous fence). 10140 - Could be split into 10141 separate s_waitcnt 10142 vmcnt(0), s_waitcnt 10143 vscnt(0) and s_waitcnt 10144 lgkmcnt(0) to allow 10145 them to be 10146 independently moved 10147 according to the 10148 following rules. 10149 - s_waitcnt vmcnt(0) 10150 must happen after 10151 any preceding 10152 global/generic 10153 load/load 10154 atomic/ 10155 atomicrmw-with-return-value. 10156 - s_waitcnt vscnt(0) 10157 must happen after 10158 any preceding 10159 global/generic 10160 store/store atomic/ 10161 atomicrmw-no-return-value. 10162 - s_waitcnt lgkmcnt(0) 10163 must happen after 10164 any preceding 10165 local/generic 10166 load/store/load 10167 atomic/store 10168 atomic/atomicrmw. 10169 - Must happen before 10170 the following 10171 buffer_gl*_inv. 10172 - Ensures that the 10173 preceding 10174 global/local/generic 10175 load 10176 atomic/atomicrmw 10177 with an equal or 10178 wider sync scope 10179 and memory ordering 10180 stronger than 10181 unordered (this is 10182 termed the 10183 acquire-fence-paired-atomic) 10184 has completed 10185 before invalidating 10186 the caches. This 10187 satisfies the 10188 requirements of 10189 acquire. 10190 - Ensures that all 10191 previous memory 10192 operations have 10193 completed before a 10194 following 10195 global/local/generic 10196 store 10197 atomic/atomicrmw 10198 with an equal or 10199 wider sync scope 10200 and memory ordering 10201 stronger than 10202 unordered (this is 10203 termed the 10204 release-fence-paired-atomic). 10205 This satisfies the 10206 requirements of 10207 release. 10208 10209 2. buffer_gl0_inv; 10210 buffer_gl1_inv 10211 10212 - Must happen before 10213 any following 10214 global/generic 10215 load/load 10216 atomic/store/store 10217 atomic/atomicrmw. 10218 - Ensures that 10219 following loads 10220 will not see stale 10221 global data. This 10222 satisfies the 10223 requirements of 10224 acquire. 10225 10226 **Sequential Consistent Atomic** 10227 ------------------------------------------------------------------------------------ 10228 load atomic seq_cst - singlethread - global *Same as corresponding 10229 - wavefront - local load atomic acquire, 10230 - generic except must generated 10231 all instructions even 10232 for OpenCL.* 10233 load atomic seq_cst - workgroup - global 1. s_waitcnt lgkmcnt(0) & 10234 - generic vmcnt(0) & vscnt(0) 10235 10236 - If CU wavefront execution 10237 mode, omit vmcnt(0) and 10238 vscnt(0). 10239 - Could be split into 10240 separate s_waitcnt 10241 vmcnt(0), s_waitcnt 10242 vscnt(0), and s_waitcnt 10243 lgkmcnt(0) to allow 10244 them to be 10245 independently moved 10246 according to the 10247 following rules. 10248 - s_waitcnt lgkmcnt(0) must 10249 happen after 10250 preceding 10251 local/generic load 10252 atomic/store 10253 atomic/atomicrmw 10254 with memory 10255 ordering of seq_cst 10256 and with equal or 10257 wider sync scope. 10258 (Note that seq_cst 10259 fences have their 10260 own s_waitcnt 10261 lgkmcnt(0) and so do 10262 not need to be 10263 considered.) 10264 - s_waitcnt vmcnt(0) 10265 must happen after 10266 preceding 10267 global/generic load 10268 atomic/ 10269 atomicrmw-with-return-value 10270 with memory 10271 ordering of seq_cst 10272 and with equal or 10273 wider sync scope. 10274 (Note that seq_cst 10275 fences have their 10276 own s_waitcnt 10277 vmcnt(0) and so do 10278 not need to be 10279 considered.) 10280 - s_waitcnt vscnt(0) 10281 Must happen after 10282 preceding 10283 global/generic store 10284 atomic/ 10285 atomicrmw-no-return-value 10286 with memory 10287 ordering of seq_cst 10288 and with equal or 10289 wider sync scope. 10290 (Note that seq_cst 10291 fences have their 10292 own s_waitcnt 10293 vscnt(0) and so do 10294 not need to be 10295 considered.) 10296 - Ensures any 10297 preceding 10298 sequential 10299 consistent global/local 10300 memory instructions 10301 have completed 10302 before executing 10303 this sequentially 10304 consistent 10305 instruction. This 10306 prevents reordering 10307 a seq_cst store 10308 followed by a 10309 seq_cst load. (Note 10310 that seq_cst is 10311 stronger than 10312 acquire/release as 10313 the reordering of 10314 load acquire 10315 followed by a store 10316 release is 10317 prevented by the 10318 s_waitcnt of 10319 the release, but 10320 there is nothing 10321 preventing a store 10322 release followed by 10323 load acquire from 10324 completing out of 10325 order. The s_waitcnt 10326 could be placed after 10327 seq_store or before 10328 the seq_load. We 10329 choose the load to 10330 make the s_waitcnt be 10331 as late as possible 10332 so that the store 10333 may have already 10334 completed.) 10335 10336 2. *Following 10337 instructions same as 10338 corresponding load 10339 atomic acquire, 10340 except must generated 10341 all instructions even 10342 for OpenCL.* 10343 load atomic seq_cst - workgroup - local 10344 10345 1. s_waitcnt vmcnt(0) & vscnt(0) 10346 10347 - If CU wavefront execution 10348 mode, omit. 10349 - Could be split into 10350 separate s_waitcnt 10351 vmcnt(0) and s_waitcnt 10352 vscnt(0) to allow 10353 them to be 10354 independently moved 10355 according to the 10356 following rules. 10357 - s_waitcnt vmcnt(0) 10358 Must happen after 10359 preceding 10360 global/generic load 10361 atomic/ 10362 atomicrmw-with-return-value 10363 with memory 10364 ordering of seq_cst 10365 and with equal or 10366 wider sync scope. 10367 (Note that seq_cst 10368 fences have their 10369 own s_waitcnt 10370 vmcnt(0) and so do 10371 not need to be 10372 considered.) 10373 - s_waitcnt vscnt(0) 10374 Must happen after 10375 preceding 10376 global/generic store 10377 atomic/ 10378 atomicrmw-no-return-value 10379 with memory 10380 ordering of seq_cst 10381 and with equal or 10382 wider sync scope. 10383 (Note that seq_cst 10384 fences have their 10385 own s_waitcnt 10386 vscnt(0) and so do 10387 not need to be 10388 considered.) 10389 - Ensures any 10390 preceding 10391 sequential 10392 consistent global 10393 memory instructions 10394 have completed 10395 before executing 10396 this sequentially 10397 consistent 10398 instruction. This 10399 prevents reordering 10400 a seq_cst store 10401 followed by a 10402 seq_cst load. (Note 10403 that seq_cst is 10404 stronger than 10405 acquire/release as 10406 the reordering of 10407 load acquire 10408 followed by a store 10409 release is 10410 prevented by the 10411 s_waitcnt of 10412 the release, but 10413 there is nothing 10414 preventing a store 10415 release followed by 10416 load acquire from 10417 completing out of 10418 order. The s_waitcnt 10419 could be placed after 10420 seq_store or before 10421 the seq_load. We 10422 choose the load to 10423 make the s_waitcnt be 10424 as late as possible 10425 so that the store 10426 may have already 10427 completed.) 10428 10429 2. *Following 10430 instructions same as 10431 corresponding load 10432 atomic acquire, 10433 except must generated 10434 all instructions even 10435 for OpenCL.* 10436 10437 load atomic seq_cst - agent - global 1. s_waitcnt lgkmcnt(0) & 10438 - system - generic vmcnt(0) & vscnt(0) 10439 10440 - Could be split into 10441 separate s_waitcnt 10442 vmcnt(0), s_waitcnt 10443 vscnt(0) and s_waitcnt 10444 lgkmcnt(0) to allow 10445 them to be 10446 independently moved 10447 according to the 10448 following rules. 10449 - s_waitcnt lgkmcnt(0) 10450 must happen after 10451 preceding 10452 local load 10453 atomic/store 10454 atomic/atomicrmw 10455 with memory 10456 ordering of seq_cst 10457 and with equal or 10458 wider sync scope. 10459 (Note that seq_cst 10460 fences have their 10461 own s_waitcnt 10462 lgkmcnt(0) and so do 10463 not need to be 10464 considered.) 10465 - s_waitcnt vmcnt(0) 10466 must happen after 10467 preceding 10468 global/generic load 10469 atomic/ 10470 atomicrmw-with-return-value 10471 with memory 10472 ordering of seq_cst 10473 and with equal or 10474 wider sync scope. 10475 (Note that seq_cst 10476 fences have their 10477 own s_waitcnt 10478 vmcnt(0) and so do 10479 not need to be 10480 considered.) 10481 - s_waitcnt vscnt(0) 10482 Must happen after 10483 preceding 10484 global/generic store 10485 atomic/ 10486 atomicrmw-no-return-value 10487 with memory 10488 ordering of seq_cst 10489 and with equal or 10490 wider sync scope. 10491 (Note that seq_cst 10492 fences have their 10493 own s_waitcnt 10494 vscnt(0) and so do 10495 not need to be 10496 considered.) 10497 - Ensures any 10498 preceding 10499 sequential 10500 consistent global 10501 memory instructions 10502 have completed 10503 before executing 10504 this sequentially 10505 consistent 10506 instruction. This 10507 prevents reordering 10508 a seq_cst store 10509 followed by a 10510 seq_cst load. (Note 10511 that seq_cst is 10512 stronger than 10513 acquire/release as 10514 the reordering of 10515 load acquire 10516 followed by a store 10517 release is 10518 prevented by the 10519 s_waitcnt of 10520 the release, but 10521 there is nothing 10522 preventing a store 10523 release followed by 10524 load acquire from 10525 completing out of 10526 order. The s_waitcnt 10527 could be placed after 10528 seq_store or before 10529 the seq_load. We 10530 choose the load to 10531 make the s_waitcnt be 10532 as late as possible 10533 so that the store 10534 may have already 10535 completed.) 10536 10537 2. *Following 10538 instructions same as 10539 corresponding load 10540 atomic acquire, 10541 except must generated 10542 all instructions even 10543 for OpenCL.* 10544 store atomic seq_cst - singlethread - global *Same as corresponding 10545 - wavefront - local store atomic release, 10546 - workgroup - generic except must generated 10547 - agent all instructions even 10548 - system for OpenCL.* 10549 atomicrmw seq_cst - singlethread - global *Same as corresponding 10550 - wavefront - local atomicrmw acq_rel, 10551 - workgroup - generic except must generated 10552 - agent all instructions even 10553 - system for OpenCL.* 10554 fence seq_cst - singlethread *none* *Same as corresponding 10555 - wavefront fence acq_rel, 10556 - workgroup except must generated 10557 - agent all instructions even 10558 - system for OpenCL.* 10559 ============ ============ ============== ========== ================================ 10560 10561Trap Handler ABI 10562~~~~~~~~~~~~~~~~ 10563 10564For code objects generated by the AMDGPU backend for HSA [HSA]_ compatible 10565runtimes (see :ref:`amdgpu-os`), the runtime installs a trap handler that 10566supports the ``s_trap`` instruction. For usage see: 10567 10568- :ref:`amdgpu-trap-handler-for-amdhsa-os-v2-table` 10569- :ref:`amdgpu-trap-handler-for-amdhsa-os-v3-table` 10570- :ref:`amdgpu-trap-handler-for-amdhsa-os-v4-table` 10571 10572 .. table:: AMDGPU Trap Handler for AMDHSA OS Code Object V2 10573 :name: amdgpu-trap-handler-for-amdhsa-os-v2-table 10574 10575 =================== =============== =============== ======================================= 10576 Usage Code Sequence Trap Handler Description 10577 Inputs 10578 =================== =============== =============== ======================================= 10579 reserved ``s_trap 0x00`` Reserved by hardware. 10580 ``debugtrap(arg)`` ``s_trap 0x01`` ``SGPR0-1``: Reserved for Finalizer HSA ``debugtrap`` 10581 ``queue_ptr`` intrinsic (not implemented). 10582 ``VGPR0``: 10583 ``arg`` 10584 ``llvm.trap`` ``s_trap 0x02`` ``SGPR0-1``: Causes wave to be halted with the PC at 10585 ``queue_ptr`` the trap instruction. The associated 10586 queue is signalled to put it into the 10587 error state. When the queue is put in 10588 the error state, the waves executing 10589 dispatches on the queue will be 10590 terminated. 10591 ``llvm.debugtrap`` ``s_trap 0x03`` *none* - If debugger not enabled then behaves 10592 as a no-operation. The trap handler 10593 is entered and immediately returns to 10594 continue execution of the wavefront. 10595 - If the debugger is enabled, causes 10596 the debug trap to be reported by the 10597 debugger and the wavefront is put in 10598 the halt state with the PC at the 10599 instruction. The debugger must 10600 increment the PC and resume the wave. 10601 reserved ``s_trap 0x04`` Reserved. 10602 reserved ``s_trap 0x05`` Reserved. 10603 reserved ``s_trap 0x06`` Reserved. 10604 reserved ``s_trap 0x07`` Reserved. 10605 reserved ``s_trap 0x08`` Reserved. 10606 reserved ``s_trap 0xfe`` Reserved. 10607 reserved ``s_trap 0xff`` Reserved. 10608 =================== =============== =============== ======================================= 10609 10610.. 10611 10612 .. table:: AMDGPU Trap Handler for AMDHSA OS Code Object V3 10613 :name: amdgpu-trap-handler-for-amdhsa-os-v3-table 10614 10615 =================== =============== =============== ======================================= 10616 Usage Code Sequence Trap Handler Description 10617 Inputs 10618 =================== =============== =============== ======================================= 10619 reserved ``s_trap 0x00`` Reserved by hardware. 10620 debugger breakpoint ``s_trap 0x01`` *none* Reserved for debugger to use for 10621 breakpoints. Causes wave to be halted 10622 with the PC at the trap instruction. 10623 The debugger is responsible to resume 10624 the wave, including the instruction 10625 that the breakpoint overwrote. 10626 ``llvm.trap`` ``s_trap 0x02`` ``SGPR0-1``: Causes wave to be halted with the PC at 10627 ``queue_ptr`` the trap instruction. The associated 10628 queue is signalled to put it into the 10629 error state. When the queue is put in 10630 the error state, the waves executing 10631 dispatches on the queue will be 10632 terminated. 10633 ``llvm.debugtrap`` ``s_trap 0x03`` *none* - If debugger not enabled then behaves 10634 as a no-operation. The trap handler 10635 is entered and immediately returns to 10636 continue execution of the wavefront. 10637 - If the debugger is enabled, causes 10638 the debug trap to be reported by the 10639 debugger and the wavefront is put in 10640 the halt state with the PC at the 10641 instruction. The debugger must 10642 increment the PC and resume the wave. 10643 reserved ``s_trap 0x04`` Reserved. 10644 reserved ``s_trap 0x05`` Reserved. 10645 reserved ``s_trap 0x06`` Reserved. 10646 reserved ``s_trap 0x07`` Reserved. 10647 reserved ``s_trap 0x08`` Reserved. 10648 reserved ``s_trap 0xfe`` Reserved. 10649 reserved ``s_trap 0xff`` Reserved. 10650 =================== =============== =============== ======================================= 10651 10652.. 10653 10654 .. table:: AMDGPU Trap Handler for AMDHSA OS Code Object V4 10655 :name: amdgpu-trap-handler-for-amdhsa-os-v4-table 10656 10657 =================== =============== ================ ================= ======================================= 10658 Usage Code Sequence GFX6-GFX8 Inputs GFX9-GFX10 Inputs Description 10659 =================== =============== ================ ================= ======================================= 10660 reserved ``s_trap 0x00`` Reserved by hardware. 10661 debugger breakpoint ``s_trap 0x01`` *none* *none* Reserved for debugger to use for 10662 breakpoints. Causes wave to be halted 10663 with the PC at the trap instruction. 10664 The debugger is responsible to resume 10665 the wave, including the instruction 10666 that the breakpoint overwrote. 10667 ``llvm.trap`` ``s_trap 0x02`` ``SGPR0-1``: *none* Causes wave to be halted with the PC at 10668 ``queue_ptr`` the trap instruction. The associated 10669 queue is signalled to put it into the 10670 error state. When the queue is put in 10671 the error state, the waves executing 10672 dispatches on the queue will be 10673 terminated. 10674 ``llvm.debugtrap`` ``s_trap 0x03`` *none* *none* - If debugger not enabled then behaves 10675 as a no-operation. The trap handler 10676 is entered and immediately returns to 10677 continue execution of the wavefront. 10678 - If the debugger is enabled, causes 10679 the debug trap to be reported by the 10680 debugger and the wavefront is put in 10681 the halt state with the PC at the 10682 instruction. The debugger must 10683 increment the PC and resume the wave. 10684 reserved ``s_trap 0x04`` Reserved. 10685 reserved ``s_trap 0x05`` Reserved. 10686 reserved ``s_trap 0x06`` Reserved. 10687 reserved ``s_trap 0x07`` Reserved. 10688 reserved ``s_trap 0x08`` Reserved. 10689 reserved ``s_trap 0xfe`` Reserved. 10690 reserved ``s_trap 0xff`` Reserved. 10691 =================== =============== ================ ================= ======================================= 10692 10693.. _amdgpu-amdhsa-function-call-convention: 10694 10695Call Convention 10696~~~~~~~~~~~~~~~ 10697 10698.. note:: 10699 10700 This section is currently incomplete and has inaccuracies. It is WIP that will 10701 be updated as information is determined. 10702 10703See :ref:`amdgpu-dwarf-address-space-identifier` for information on swizzled 10704addresses. Unswizzled addresses are normal linear addresses. 10705 10706.. _amdgpu-amdhsa-function-call-convention-kernel-functions: 10707 10708Kernel Functions 10709++++++++++++++++ 10710 10711This section describes the call convention ABI for the outer kernel function. 10712 10713See :ref:`amdgpu-amdhsa-initial-kernel-execution-state` for the kernel call 10714convention. 10715 10716The following is not part of the AMDGPU kernel calling convention but describes 10717how the AMDGPU implements function calls: 10718 107191. Clang decides the kernarg layout to match the *HSA Programmer's Language 10720 Reference* [HSA]_. 10721 10722 - All structs are passed directly. 10723 - Lambda values are passed *TBA*. 10724 10725 .. TODO:: 10726 10727 - Does this really follow HSA rules? Or are structs >16 bytes passed 10728 by-value struct? 10729 - What is ABI for lambda values? 10730 107314. The kernel performs certain setup in its prolog, as described in 10732 :ref:`amdgpu-amdhsa-kernel-prolog`. 10733 10734.. _amdgpu-amdhsa-function-call-convention-non-kernel-functions: 10735 10736Non-Kernel Functions 10737++++++++++++++++++++ 10738 10739This section describes the call convention ABI for functions other than the 10740outer kernel function. 10741 10742If a kernel has function calls then scratch is always allocated and used for 10743the call stack which grows from low address to high address using the swizzled 10744scratch address space. 10745 10746On entry to a function: 10747 107481. SGPR0-3 contain a V# with the following properties (see 10749 :ref:`amdgpu-amdhsa-kernel-prolog-private-segment-buffer`): 10750 10751 * Base address pointing to the beginning of the wavefront scratch backing 10752 memory. 10753 * Swizzled with dword element size and stride of wavefront size elements. 10754 107552. The FLAT_SCRATCH register pair is setup. See 10756 :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`. 107573. GFX6-GFX8: M0 register set to the size of LDS in bytes. See 10758 :ref:`amdgpu-amdhsa-kernel-prolog-m0`. 107594. The EXEC register is set to the lanes active on entry to the function. 107605. MODE register: *TBD* 107616. VGPR0-31 and SGPR4-29 are used to pass function input arguments as described 10762 below. 107637. SGPR30-31 return address (RA). The code address that the function must 10764 return to when it completes. The value is undefined if the function is *no 10765 return*. 107668. SGPR32 is used for the stack pointer (SP). It is an unswizzled scratch 10767 offset relative to the beginning of the wavefront scratch backing memory. 10768 10769 The unswizzled SP can be used with buffer instructions as an unswizzled SGPR 10770 offset with the scratch V# in SGPR0-3 to access the stack in a swizzled 10771 manner. 10772 10773 The unswizzled SP value can be converted into the swizzled SP value by: 10774 10775 | swizzled SP = unswizzled SP / wavefront size 10776 10777 This may be used to obtain the private address space address of stack 10778 objects and to convert this address to a flat address by adding the flat 10779 scratch aperture base address. 10780 10781 The swizzled SP value is always 4 bytes aligned for the ``r600`` 10782 architecture and 16 byte aligned for the ``amdgcn`` architecture. 10783 10784 .. note:: 10785 10786 The ``amdgcn`` value is selected to avoid dynamic stack alignment for the 10787 OpenCL language which has the largest base type defined as 16 bytes. 10788 10789 On entry, the swizzled SP value is the address of the first function 10790 argument passed on the stack. Other stack passed arguments are positive 10791 offsets from the entry swizzled SP value. 10792 10793 The function may use positive offsets beyond the last stack passed argument 10794 for stack allocated local variables and register spill slots. If necessary, 10795 the function may align these to greater alignment than 16 bytes. After these 10796 the function may dynamically allocate space for such things as runtime sized 10797 ``alloca`` local allocations. 10798 10799 If the function calls another function, it will place any stack allocated 10800 arguments after the last local allocation and adjust SGPR32 to the address 10801 after the last local allocation. 10802 108039. All other registers are unspecified. 1080410. Any necessary ``s_waitcnt`` has been performed to ensure memory is available 10805 to the function. 10806 10807On exit from a function: 10808 108091. VGPR0-31 and SGPR4-29 are used to pass function result arguments as 10810 described below. Any registers used are considered clobbered registers. 108112. The following registers are preserved and have the same value as on entry: 10812 10813 * FLAT_SCRATCH 10814 * EXEC 10815 * GFX6-GFX8: M0 10816 * All SGPR registers except the clobbered registers of SGPR4-31. 10817 * VGPR40-47 10818 * VGPR56-63 10819 * VGPR72-79 10820 * VGPR88-95 10821 * VGPR104-111 10822 * VGPR120-127 10823 * VGPR136-143 10824 * VGPR152-159 10825 * VGPR168-175 10826 * VGPR184-191 10827 * VGPR200-207 10828 * VGPR216-223 10829 * VGPR232-239 10830 * VGPR248-255 10831 10832 .. note:: 10833 10834 Except the argument registers, the VGPRs clobbered and the preserved 10835 registers are intermixed at regular intervals in order to keep a 10836 similar ratio independent of the number of allocated VGPRs. 10837 10838 * GFX90A: All AGPR registers except the clobbered registers AGPR0-31. 10839 * Lanes of all VGPRs that are inactive at the call site. 10840 10841 For the AMDGPU backend, an inter-procedural register allocation (IPRA) 10842 optimization may mark some of clobbered SGPR and VGPR registers as 10843 preserved if it can be determined that the called function does not change 10844 their value. 10845 108462. The PC is set to the RA provided on entry. 108473. MODE register: *TBD*. 108484. All other registers are clobbered. 108495. Any necessary ``s_waitcnt`` has been performed to ensure memory accessed by 10850 function is available to the caller. 10851 10852.. TODO:: 10853 10854 - How are function results returned? The address of structured types is passed 10855 by reference, but what about other types? 10856 10857The function input arguments are made up of the formal arguments explicitly 10858declared by the source language function plus the implicit input arguments used 10859by the implementation. 10860 10861The source language input arguments are: 10862 108631. Any source language implicit ``this`` or ``self`` argument comes first as a 10864 pointer type. 108652. Followed by the function formal arguments in left to right source order. 10866 10867The source language result arguments are: 10868 108691. The function result argument. 10870 10871The source language input or result struct type arguments that are less than or 10872equal to 16 bytes, are decomposed recursively into their base type fields, and 10873each field is passed as if a separate argument. For input arguments, if the 10874called function requires the struct to be in memory, for example because its 10875address is taken, then the function body is responsible for allocating a stack 10876location and copying the field arguments into it. Clang terms this *direct 10877struct*. 10878 10879The source language input struct type arguments that are greater than 16 bytes, 10880are passed by reference. The caller is responsible for allocating a stack 10881location to make a copy of the struct value and pass the address as the input 10882argument. The called function is responsible to perform the dereference when 10883accessing the input argument. Clang terms this *by-value struct*. 10884 10885A source language result struct type argument that is greater than 16 bytes, is 10886returned by reference. The caller is responsible for allocating a stack location 10887to hold the result value and passes the address as the last input argument 10888(before the implicit input arguments). In this case there are no result 10889arguments. The called function is responsible to perform the dereference when 10890storing the result value. Clang terms this *structured return (sret)*. 10891 10892*TODO: correct the ``sret`` definition.* 10893 10894.. TODO:: 10895 10896 Is this definition correct? Or is ``sret`` only used if passing in registers, and 10897 pass as non-decomposed struct as stack argument? Or something else? Is the 10898 memory location in the caller stack frame, or a stack memory argument and so 10899 no address is passed as the caller can directly write to the argument stack 10900 location? But then the stack location is still live after return. If an 10901 argument stack location is it the first stack argument or the last one? 10902 10903Lambda argument types are treated as struct types with an implementation defined 10904set of fields. 10905 10906.. TODO:: 10907 10908 Need to specify the ABI for lambda types for AMDGPU. 10909 10910For AMDGPU backend all source language arguments (including the decomposed 10911struct type arguments) are passed in VGPRs unless marked ``inreg`` in which case 10912they are passed in SGPRs. 10913 10914The AMDGPU backend walks the function call graph from the leaves to determine 10915which implicit input arguments are used, propagating to each caller of the 10916function. The used implicit arguments are appended to the function arguments 10917after the source language arguments in the following order: 10918 10919.. TODO:: 10920 10921 Is recursion or external functions supported? 10922 109231. Work-Item ID (1 VGPR) 10924 10925 The X, Y and Z work-item ID are packed into a single VGRP with the following 10926 layout. Only fields actually used by the function are set. The other bits 10927 are undefined. 10928 10929 The values come from the initial kernel execution state. See 10930 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`. 10931 10932 .. table:: Work-item implicit argument layout 10933 :name: amdgpu-amdhsa-workitem-implicit-argument-layout-table 10934 10935 ======= ======= ============== 10936 Bits Size Field Name 10937 ======= ======= ============== 10938 9:0 10 bits X Work-Item ID 10939 19:10 10 bits Y Work-Item ID 10940 29:20 10 bits Z Work-Item ID 10941 31:30 2 bits Unused 10942 ======= ======= ============== 10943 109442. Dispatch Ptr (2 SGPRs) 10945 10946 The value comes from the initial kernel execution state. See 10947 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 10948 109493. Queue Ptr (2 SGPRs) 10950 10951 The value comes from the initial kernel execution state. See 10952 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 10953 109544. Kernarg Segment Ptr (2 SGPRs) 10955 10956 The value comes from the initial kernel execution state. See 10957 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 10958 109595. Dispatch id (2 SGPRs) 10960 10961 The value comes from the initial kernel execution state. See 10962 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 10963 109646. Work-Group ID X (1 SGPR) 10965 10966 The value comes from the initial kernel execution state. See 10967 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 10968 109697. Work-Group ID Y (1 SGPR) 10970 10971 The value comes from the initial kernel execution state. See 10972 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 10973 109748. Work-Group ID Z (1 SGPR) 10975 10976 The value comes from the initial kernel execution state. See 10977 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 10978 109799. Implicit Argument Ptr (2 SGPRs) 10980 10981 The value is computed by adding an offset to Kernarg Segment Ptr to get the 10982 global address space pointer to the first kernarg implicit argument. 10983 10984The input and result arguments are assigned in order in the following manner: 10985 10986.. note:: 10987 10988 There are likely some errors and omissions in the following description that 10989 need correction. 10990 10991 .. TODO:: 10992 10993 Check the Clang source code to decipher how function arguments and return 10994 results are handled. Also see the AMDGPU specific values used. 10995 10996* VGPR arguments are assigned to consecutive VGPRs starting at VGPR0 up to 10997 VGPR31. 10998 10999 If there are more arguments than will fit in these registers, the remaining 11000 arguments are allocated on the stack in order on naturally aligned 11001 addresses. 11002 11003 .. TODO:: 11004 11005 How are overly aligned structures allocated on the stack? 11006 11007* SGPR arguments are assigned to consecutive SGPRs starting at SGPR0 up to 11008 SGPR29. 11009 11010 If there are more arguments than will fit in these registers, the remaining 11011 arguments are allocated on the stack in order on naturally aligned 11012 addresses. 11013 11014Note that decomposed struct type arguments may have some fields passed in 11015registers and some in memory. 11016 11017.. TODO:: 11018 11019 So, a struct which can pass some fields as decomposed register arguments, will 11020 pass the rest as decomposed stack elements? But an argument that will not start 11021 in registers will not be decomposed and will be passed as a non-decomposed 11022 stack value? 11023 11024The following is not part of the AMDGPU function calling convention but 11025describes how the AMDGPU implements function calls: 11026 110271. SGPR33 is used as a frame pointer (FP) if necessary. Like the SP it is an 11028 unswizzled scratch address. It is only needed if runtime sized ``alloca`` 11029 are used, or for the reasons defined in ``SIFrameLowering``. 110302. Runtime stack alignment is supported. SGPR34 is used as a base pointer (BP) 11031 to access the incoming stack arguments in the function. The BP is needed 11032 only when the function requires the runtime stack alignment. 11033 110343. Allocating SGPR arguments on the stack are not supported. 11035 110364. No CFI is currently generated. See 11037 :ref:`amdgpu-dwarf-call-frame-information`. 11038 11039 .. note:: 11040 11041 CFI will be generated that defines the CFA as the unswizzled address 11042 relative to the wave scratch base in the unswizzled private address space 11043 of the lowest address stack allocated local variable. 11044 11045 ``DW_AT_frame_base`` will be defined as the swizzled address in the 11046 swizzled private address space by dividing the CFA by the wavefront size 11047 (since CFA is always at least dword aligned which matches the scratch 11048 swizzle element size). 11049 11050 If no dynamic stack alignment was performed, the stack allocated arguments 11051 are accessed as negative offsets relative to ``DW_AT_frame_base``, and the 11052 local variables and register spill slots are accessed as positive offsets 11053 relative to ``DW_AT_frame_base``. 11054 110555. Function argument passing is implemented by copying the input physical 11056 registers to virtual registers on entry. The register allocator can spill if 11057 necessary. These are copied back to physical registers at call sites. The 11058 net effect is that each function call can have these values in entirely 11059 distinct locations. The IPRA can help avoid shuffling argument registers. 110606. Call sites are implemented by setting up the arguments at positive offsets 11061 from SP. Then SP is incremented to account for the known frame size before 11062 the call and decremented after the call. 11063 11064 .. note:: 11065 11066 The CFI will reflect the changed calculation needed to compute the CFA 11067 from SP. 11068 110697. 4 byte spill slots are used in the stack frame. One slot is allocated for an 11070 emergency spill slot. Buffer instructions are used for stack accesses and 11071 not the ``flat_scratch`` instruction. 11072 11073 .. TODO:: 11074 11075 Explain when the emergency spill slot is used. 11076 11077.. TODO:: 11078 11079 Possible broken issues: 11080 11081 - Stack arguments must be aligned to required alignment. 11082 - Stack is aligned to max(16, max formal argument alignment) 11083 - Direct argument < 64 bits should check register budget. 11084 - Register budget calculation should respect ``inreg`` for SGPR. 11085 - SGPR overflow is not handled. 11086 - struct with 1 member unpeeling is not checking size of member. 11087 - ``sret`` is after ``this`` pointer. 11088 - Caller is not implementing stack realignment: need an extra pointer. 11089 - Should say AMDGPU passes FP rather than SP. 11090 - Should CFI define CFA as address of locals or arguments. Difference is 11091 apparent when have implemented dynamic alignment. 11092 - If ``SCRATCH`` instruction could allow negative offsets, then can make FP be 11093 highest address of stack frame and use negative offset for locals. Would 11094 allow SP to be the same as FP and could support signal-handler-like as now 11095 have a real SP for the top of the stack. 11096 - How is ``sret`` passed on the stack? In argument stack area? Can it overlay 11097 arguments? 11098 11099AMDPAL 11100------ 11101 11102This section provides code conventions used when the target triple OS is 11103``amdpal`` (see :ref:`amdgpu-target-triples`). 11104 11105.. _amdgpu-amdpal-code-object-metadata-section: 11106 11107Code Object Metadata 11108~~~~~~~~~~~~~~~~~~~~ 11109 11110.. note:: 11111 11112 The metadata is currently in development and is subject to major 11113 changes. Only the current version is supported. *When this document 11114 was generated the version was 2.6.* 11115 11116Code object metadata is specified by the ``NT_AMDGPU_METADATA`` note 11117record (see :ref:`amdgpu-note-records-v3-v4`). 11118 11119The metadata is represented as Message Pack formatted binary data (see 11120[MsgPack]_). The top level is a Message Pack map that includes the keys 11121defined in table :ref:`amdgpu-amdpal-code-object-metadata-map-table` 11122and referenced tables. 11123 11124Additional information can be added to the maps. To avoid conflicts, any 11125key names should be prefixed by "*vendor-name*." where ``vendor-name`` 11126can be the name of the vendor and specific vendor tool that generates the 11127information. The prefix is abbreviated to simply "." when it appears 11128within a map that has been added by the same *vendor-name*. 11129 11130 .. table:: AMDPAL Code Object Metadata Map 11131 :name: amdgpu-amdpal-code-object-metadata-map-table 11132 11133 =================== ============== ========= ====================================================================== 11134 String Key Value Type Required? Description 11135 =================== ============== ========= ====================================================================== 11136 "amdpal.version" sequence of Required PAL code object metadata (major, minor) version. The current values 11137 2 integers are defined by *Util::Abi::PipelineMetadata(Major|Minor)Version*. 11138 "amdpal.pipelines" sequence of Required Per-pipeline metadata. See 11139 map :ref:`amdgpu-amdpal-code-object-pipeline-metadata-map-table` for the 11140 definition of the keys included in that map. 11141 =================== ============== ========= ====================================================================== 11142 11143.. 11144 11145 .. table:: AMDPAL Code Object Pipeline Metadata Map 11146 :name: amdgpu-amdpal-code-object-pipeline-metadata-map-table 11147 11148 ====================================== ============== ========= =================================================== 11149 String Key Value Type Required? Description 11150 ====================================== ============== ========= =================================================== 11151 ".name" string Source name of the pipeline. 11152 ".type" string Pipeline type, e.g. VsPs. Values include: 11153 11154 - "VsPs" 11155 - "Gs" 11156 - "Cs" 11157 - "Ngg" 11158 - "Tess" 11159 - "GsTess" 11160 - "NggTess" 11161 11162 ".internal_pipeline_hash" sequence of Required Internal compiler hash for this pipeline. Lower 11163 2 integers 64 bits is the "stable" portion of the hash, used 11164 for e.g. shader replacement lookup. Upper 64 bits 11165 is the "unique" portion of the hash, used for 11166 e.g. pipeline cache lookup. The value is 11167 implementation defined, and can not be relied on 11168 between different builds of the compiler. 11169 ".shaders" map Per-API shader metadata. See 11170 :ref:`amdgpu-amdpal-code-object-shader-map-table` 11171 for the definition of the keys included in that 11172 map. 11173 ".hardware_stages" map Per-hardware stage metadata. See 11174 :ref:`amdgpu-amdpal-code-object-hardware-stage-map-table` 11175 for the definition of the keys included in that 11176 map. 11177 ".shader_functions" map Per-shader function metadata. See 11178 :ref:`amdgpu-amdpal-code-object-shader-function-map-table` 11179 for the definition of the keys included in that 11180 map. 11181 ".registers" map Required Hardware register configuration. See 11182 :ref:`amdgpu-amdpal-code-object-register-map-table` 11183 for the definition of the keys included in that 11184 map. 11185 ".user_data_limit" integer Number of user data entries accessed by this 11186 pipeline. 11187 ".spill_threshold" integer The user data spill threshold. 0xFFFF for 11188 NoUserDataSpilling. 11189 ".uses_viewport_array_index" boolean Indicates whether or not the pipeline uses the 11190 viewport array index feature. Pipelines which use 11191 this feature can render into all 16 viewports, 11192 whereas pipelines which do not use it are 11193 restricted to viewport #0. 11194 ".es_gs_lds_size" integer Size in bytes of LDS space used internally for 11195 handling data-passing between the ES and GS 11196 shader stages. This can be zero if the data is 11197 passed using off-chip buffers. This value should 11198 be used to program all user-SGPRs which have been 11199 marked with "UserDataMapping::EsGsLdsSize" 11200 (typically only the GS and VS HW stages will ever 11201 have a user-SGPR so marked). 11202 ".nggSubgroupSize" integer Explicit maximum subgroup size for NGG shaders 11203 (maximum number of threads in a subgroup). 11204 ".num_interpolants" integer Graphics only. Number of PS interpolants. 11205 ".mesh_scratch_memory_size" integer Max mesh shader scratch memory used. 11206 ".api" string Name of the client graphics API. 11207 ".api_create_info" binary Graphics API shader create info binary blob. Can 11208 be defined by the driver using the compiler if 11209 they want to be able to correlate API-specific 11210 information used during creation at a later time. 11211 ====================================== ============== ========= =================================================== 11212 11213.. 11214 11215 .. table:: AMDPAL Code Object Shader Map 11216 :name: amdgpu-amdpal-code-object-shader-map-table 11217 11218 11219 +-------------+--------------+-------------------------------------------------------------------+ 11220 |String Key |Value Type |Description | 11221 +=============+==============+===================================================================+ 11222 |- ".compute" |map |See :ref:`amdgpu-amdpal-code-object-api-shader-metadata-map-table` | 11223 |- ".vertex" | |for the definition of the keys included in that map. | 11224 |- ".hull" | | | 11225 |- ".domain" | | | 11226 |- ".geometry"| | | 11227 |- ".pixel" | | | 11228 +-------------+--------------+-------------------------------------------------------------------+ 11229 11230.. 11231 11232 .. table:: AMDPAL Code Object API Shader Metadata Map 11233 :name: amdgpu-amdpal-code-object-api-shader-metadata-map-table 11234 11235 ==================== ============== ========= ===================================================================== 11236 String Key Value Type Required? Description 11237 ==================== ============== ========= ===================================================================== 11238 ".api_shader_hash" sequence of Required Input shader hash, typically passed in from the client. The value 11239 2 integers is implementation defined, and can not be relied on between 11240 different builds of the compiler. 11241 ".hardware_mapping" sequence of Required Flags indicating the HW stages this API shader maps to. Values 11242 string include: 11243 11244 - ".ls" 11245 - ".hs" 11246 - ".es" 11247 - ".gs" 11248 - ".vs" 11249 - ".ps" 11250 - ".cs" 11251 11252 ==================== ============== ========= ===================================================================== 11253 11254.. 11255 11256 .. table:: AMDPAL Code Object Hardware Stage Map 11257 :name: amdgpu-amdpal-code-object-hardware-stage-map-table 11258 11259 +-------------+--------------+-----------------------------------------------------------------------+ 11260 |String Key |Value Type |Description | 11261 +=============+==============+=======================================================================+ 11262 |- ".ls" |map |See :ref:`amdgpu-amdpal-code-object-hardware-stage-metadata-map-table` | 11263 |- ".hs" | |for the definition of the keys included in that map. | 11264 |- ".es" | | | 11265 |- ".gs" | | | 11266 |- ".vs" | | | 11267 |- ".ps" | | | 11268 |- ".cs" | | | 11269 +-------------+--------------+-----------------------------------------------------------------------+ 11270 11271.. 11272 11273 .. table:: AMDPAL Code Object Hardware Stage Metadata Map 11274 :name: amdgpu-amdpal-code-object-hardware-stage-metadata-map-table 11275 11276 ========================== ============== ========= =============================================================== 11277 String Key Value Type Required? Description 11278 ========================== ============== ========= =============================================================== 11279 ".entry_point" string The ELF symbol pointing to this pipeline's stage entry point. 11280 ".scratch_memory_size" integer Scratch memory size in bytes. 11281 ".lds_size" integer Local Data Share size in bytes. 11282 ".perf_data_buffer_size" integer Performance data buffer size in bytes. 11283 ".vgpr_count" integer Number of VGPRs used. 11284 ".sgpr_count" integer Number of SGPRs used. 11285 ".vgpr_limit" integer If non-zero, indicates the shader was compiled with a 11286 directive to instruct the compiler to limit the VGPR usage to 11287 be less than or equal to the specified value (only set if 11288 different from HW default). 11289 ".sgpr_limit" integer SGPR count upper limit (only set if different from HW 11290 default). 11291 ".threadgroup_dimensions" sequence of Thread-group X/Y/Z dimensions (Compute only). 11292 3 integers 11293 ".wavefront_size" integer Wavefront size (only set if different from HW default). 11294 ".uses_uavs" boolean The shader reads or writes UAVs. 11295 ".uses_rovs" boolean The shader reads or writes ROVs. 11296 ".writes_uavs" boolean The shader writes to one or more UAVs. 11297 ".writes_depth" boolean The shader writes out a depth value. 11298 ".uses_append_consume" boolean The shader uses append and/or consume operations, either 11299 memory or GDS. 11300 ".uses_prim_id" boolean The shader uses PrimID. 11301 ========================== ============== ========= =============================================================== 11302 11303.. 11304 11305 .. table:: AMDPAL Code Object Shader Function Map 11306 :name: amdgpu-amdpal-code-object-shader-function-map-table 11307 11308 =============== ============== ==================================================================== 11309 String Key Value Type Description 11310 =============== ============== ==================================================================== 11311 *symbol name* map *symbol name* is the ELF symbol name of the shader function code 11312 entry address. The value is the function's metadata. See 11313 :ref:`amdgpu-amdpal-code-object-shader-function-metadata-map-table`. 11314 =============== ============== ==================================================================== 11315 11316.. 11317 11318 .. table:: AMDPAL Code Object Shader Function Metadata Map 11319 :name: amdgpu-amdpal-code-object-shader-function-metadata-map-table 11320 11321 ============================= ============== ================================================================= 11322 String Key Value Type Description 11323 ============================= ============== ================================================================= 11324 ".api_shader_hash" sequence of Input shader hash, typically passed in from the client. The value 11325 2 integers is implementation defined, and can not be relied on between 11326 different builds of the compiler. 11327 ".scratch_memory_size" integer Size in bytes of scratch memory used by the shader. 11328 ".lds_size" integer Size in bytes of LDS memory. 11329 ".vgpr_count" integer Number of VGPRs used by the shader. 11330 ".sgpr_count" integer Number of SGPRs used by the shader. 11331 ".stack_frame_size_in_bytes" integer Amount of stack size used by the shader. 11332 ".shader_subtype" string Shader subtype/kind. Values include: 11333 11334 - "Unknown" 11335 11336 ============================= ============== ================================================================= 11337 11338.. 11339 11340 .. table:: AMDPAL Code Object Register Map 11341 :name: amdgpu-amdpal-code-object-register-map-table 11342 11343 ========================== ============== ==================================================================== 11344 32-bit Integer Key Value Type Description 11345 ========================== ============== ==================================================================== 11346 ``reg offset`` 32-bit integer ``reg offset`` is the dword offset into the GFXIP register space of 11347 a GRBM register (i.e., driver accessible GPU register number, not 11348 shader GPR register number). The driver is required to program each 11349 specified register to the corresponding specified value when 11350 executing this pipeline. Typically, the ``reg offsets`` are the 11351 ``uint16_t`` offsets to each register as defined by the hardware 11352 chip headers. The register is set to the provided value. However, a 11353 ``reg offset`` that specifies a user data register (e.g., 11354 COMPUTE_USER_DATA_0) needs special treatment. See 11355 :ref:`amdgpu-amdpal-code-object-user-data-section` section for more 11356 information. 11357 ========================== ============== ==================================================================== 11358 11359.. _amdgpu-amdpal-code-object-user-data-section: 11360 11361User Data 11362+++++++++ 11363 11364Each hardware stage has a set of 32-bit physical SPI *user data registers* 11365(either 16 or 32 based on graphics IP and the stage) which can be 11366written from a command buffer and then loaded into SGPRs when waves are 11367launched via a subsequent dispatch or draw operation. This is the way 11368most arguments are passed from the application/runtime to a hardware 11369shader. 11370 11371PAL abstracts this functionality by exposing a set of 128 *user data 11372entries* per pipeline a client can use to pass arguments from a command 11373buffer to one or more shaders in that pipeline. The ELF code object must 11374specify a mapping from virtualized *user data entries* to physical *user 11375data registers*, and PAL is responsible for implementing that mapping, 11376including spilling overflow *user data entries* to memory if needed. 11377 11378Since the *user data registers* are GRBM-accessible SPI registers, this 11379mapping is actually embedded in the ``.registers`` metadata entry. For 11380most registers, the value in that map is a literal 32-bit value that 11381should be written to the register by the driver. However, when the 11382register is a *user data register* (any USER_DATA register e.g., 11383SPI_SHADER_USER_DATA_PS_5), the value is instead an encoding that tells 11384the driver to write either a *user data entry* value or one of several 11385driver-internal values to the register. This encoding is described in 11386the following table: 11387 11388.. note:: 11389 11390 Currently, *user data registers* 0 and 1 (e.g., SPI_SHADER_USER_DATA_PS_0, 11391 and SPI_SHADER_USER_DATA_PS_1) are reserved. *User data register* 0 must 11392 always be programmed to the address of the GlobalTable, and *user data 11393 register* 1 must always be programmed to the address of the PerShaderTable. 11394 11395.. 11396 11397 .. table:: AMDPAL User Data Mapping 11398 :name: amdgpu-amdpal-code-object-metadata-user-data-mapping-table 11399 11400 ========== ================= =============================================================================== 11401 Value Name Description 11402 ========== ================= =============================================================================== 11403 0..127 *User Data Entry* 32-bit value of user_data_entry[N] as specified via *CmdSetUserData()* 11404 0x10000000 GlobalTable 32-bit pointer to GPU memory containing the global internal table (should 11405 always point to *user data register* 0). 11406 0x10000001 PerShaderTable 32-bit pointer to GPU memory containing the per-shader internal table. See 11407 :ref:`amdgpu-amdpal-code-object-metadata-user-data-per-shader-table-section` 11408 for more detail (should always point to *user data register* 1). 11409 0x10000002 SpillTable 32-bit pointer to GPU memory containing the user data spill table. See 11410 :ref:`amdgpu-amdpal-code-object-metadata-user-data-spill-table-section` for 11411 more detail. 11412 0x10000003 BaseVertex Vertex offset (32-bit unsigned integer). Not needed if the pipeline doesn't 11413 reference the draw index in the vertex shader. Only supported by the first 11414 stage in a graphics pipeline. 11415 0x10000004 BaseInstance Instance offset (32-bit unsigned integer). Only supported by the first stage in 11416 a graphics pipeline. 11417 0x10000005 DrawIndex Draw index (32-bit unsigned integer). Only supported by the first stage in a 11418 graphics pipeline. 11419 0x10000006 Workgroup Thread group count (32-bit unsigned integer). Low half of a 64-bit address of 11420 a buffer containing the grid dimensions for a Compute dispatch operation. The 11421 high half of the address is stored in the next sequential user-SGPR. Only 11422 supported by compute pipelines. 11423 0x1000000A EsGsLdsSize Indicates that PAL will program this user-SGPR to contain the amount of LDS 11424 space used for the ES/GS pseudo-ring-buffer for passing data between shader 11425 stages. 11426 0x1000000B ViewId View id (32-bit unsigned integer) identifies a view of graphic 11427 pipeline instancing. 11428 0x1000000C StreamOutTable 32-bit pointer to GPU memory containing the stream out target SRD table. This 11429 can only appear for one shader stage per pipeline. 11430 0x1000000D PerShaderPerfData 32-bit pointer to GPU memory containing the per-shader performance data buffer. 11431 0x1000000F VertexBufferTable 32-bit pointer to GPU memory containing the vertex buffer SRD table. This can 11432 only appear for one shader stage per pipeline. 11433 0x10000010 UavExportTable 32-bit pointer to GPU memory containing the UAV export SRD table. This can 11434 only appear for one shader stage per pipeline (PS). These replace color targets 11435 and are completely separate from any UAVs used by the shader. This is optional, 11436 and only used by the PS when UAV exports are used to replace color-target 11437 exports to optimize specific shaders. 11438 0x10000011 NggCullingData 64-bit pointer to GPU memory containing the hardware register data needed by 11439 some NGG pipelines to perform culling. This value contains the address of the 11440 first of two consecutive registers which provide the full GPU address. 11441 0x10000015 FetchShaderPtr 64-bit pointer to GPU memory containing the fetch shader subroutine. 11442 ========== ================= =============================================================================== 11443 11444.. _amdgpu-amdpal-code-object-metadata-user-data-per-shader-table-section: 11445 11446Per-Shader Table 11447################ 11448 11449Low 32 bits of the GPU address for an optional buffer in the ``.data`` 11450section of the ELF. The high 32 bits of the address match the high 32 bits 11451of the shader's program counter. 11452 11453The buffer can be anything the shader compiler needs it for, and 11454allows each shader to have its own region of the ``.data`` section. 11455Typically, this could be a table of buffer SRD's and the data pointed to 11456by the buffer SRD's, but it could be a flat-address region of memory as 11457well. Its layout and usage are defined by the shader compiler. 11458 11459Each shader's table in the ``.data`` section is referenced by the symbol 11460``_amdgpu_``\ *xs*\ ``_shdr_intrl_data`` where *xs* corresponds with the 11461hardware shader stage the data is for. E.g., 11462``_amdgpu_cs_shdr_intrl_data`` for the compute shader hardware stage. 11463 11464.. _amdgpu-amdpal-code-object-metadata-user-data-spill-table-section: 11465 11466Spill Table 11467########### 11468 11469It is possible for a hardware shader to need access to more *user data 11470entries* than there are slots available in user data registers for one 11471or more hardware shader stages. In that case, the PAL runtime expects 11472the necessary *user data entries* to be spilled to GPU memory and use 11473one user data register to point to the spilled user data memory. The 11474value of the *user data entry* must then represent the location where 11475a shader expects to read the low 32-bits of the table's GPU virtual 11476address. The *spill table* itself represents a set of 32-bit values 11477managed by the PAL runtime in GPU-accessible memory that can be made 11478indirectly accessible to a hardware shader. 11479 11480Unspecified OS 11481-------------- 11482 11483This section provides code conventions used when the target triple OS is 11484empty (see :ref:`amdgpu-target-triples`). 11485 11486Trap Handler ABI 11487~~~~~~~~~~~~~~~~ 11488 11489For code objects generated by AMDGPU backend for non-amdhsa OS, the runtime does 11490not install a trap handler. The ``llvm.trap`` and ``llvm.debugtrap`` 11491instructions are handled as follows: 11492 11493 .. table:: AMDGPU Trap Handler for Non-AMDHSA OS 11494 :name: amdgpu-trap-handler-for-non-amdhsa-os-table 11495 11496 =============== =============== =========================================== 11497 Usage Code Sequence Description 11498 =============== =============== =========================================== 11499 llvm.trap s_endpgm Causes wavefront to be terminated. 11500 llvm.debugtrap *none* Compiler warning given that there is no 11501 trap handler installed. 11502 =============== =============== =========================================== 11503 11504Source Languages 11505================ 11506 11507.. _amdgpu-opencl: 11508 11509OpenCL 11510------ 11511 11512When the language is OpenCL the following differences occur: 11513 115141. The OpenCL memory model is used (see :ref:`amdgpu-amdhsa-memory-model`). 115152. The AMDGPU backend appends additional arguments to the kernel's explicit 11516 arguments for the AMDHSA OS (see 11517 :ref:`opencl-kernel-implicit-arguments-appended-for-amdhsa-os-table`). 115183. Additional metadata is generated 11519 (see :ref:`amdgpu-amdhsa-code-object-metadata`). 11520 11521 .. table:: OpenCL kernel implicit arguments appended for AMDHSA OS 11522 :name: opencl-kernel-implicit-arguments-appended-for-amdhsa-os-table 11523 11524 ======== ==== ========= =========================================== 11525 Position Byte Byte Description 11526 Size Alignment 11527 ======== ==== ========= =========================================== 11528 1 8 8 OpenCL Global Offset X 11529 2 8 8 OpenCL Global Offset Y 11530 3 8 8 OpenCL Global Offset Z 11531 4 8 8 OpenCL address of printf buffer 11532 5 8 8 OpenCL address of virtual queue used by 11533 enqueue_kernel. 11534 6 8 8 OpenCL address of AqlWrap struct used by 11535 enqueue_kernel. 11536 7 8 8 Pointer argument used for Multi-gird 11537 synchronization. 11538 ======== ==== ========= =========================================== 11539 11540.. _amdgpu-hcc: 11541 11542HCC 11543--- 11544 11545When the language is HCC the following differences occur: 11546 115471. The HSA memory model is used (see :ref:`amdgpu-amdhsa-memory-model`). 11548 11549.. _amdgpu-assembler: 11550 11551Assembler 11552--------- 11553 11554AMDGPU backend has LLVM-MC based assembler which is currently in development. 11555It supports AMDGCN GFX6-GFX10. 11556 11557This section describes general syntax for instructions and operands. 11558 11559Instructions 11560~~~~~~~~~~~~ 11561 11562An instruction has the following :doc:`syntax<AMDGPUInstructionSyntax>`: 11563 11564 | ``<``\ *opcode*\ ``> <``\ *operand0*\ ``>, <``\ *operand1*\ ``>,... 11565 <``\ *modifier0*\ ``> <``\ *modifier1*\ ``>...`` 11566 11567:doc:`Operands<AMDGPUOperandSyntax>` are comma-separated while 11568:doc:`modifiers<AMDGPUModifierSyntax>` are space-separated. 11569 11570The order of operands and modifiers is fixed. 11571Most modifiers are optional and may be omitted. 11572 11573Links to detailed instruction syntax description may be found in the following 11574table. Note that features under development are not included 11575in this description. 11576 11577 =================================== ======================================= 11578 Core ISA ISA Extensions 11579 =================================== ======================================= 11580 :doc:`GFX7<AMDGPU/AMDGPUAsmGFX7>` \- 11581 :doc:`GFX8<AMDGPU/AMDGPUAsmGFX8>` \- 11582 :doc:`GFX9<AMDGPU/AMDGPUAsmGFX9>` :doc:`gfx900<AMDGPU/AMDGPUAsmGFX900>` 11583 11584 :doc:`gfx902<AMDGPU/AMDGPUAsmGFX900>` 11585 11586 :doc:`gfx904<AMDGPU/AMDGPUAsmGFX904>` 11587 11588 :doc:`gfx906<AMDGPU/AMDGPUAsmGFX906>` 11589 11590 :doc:`gfx908<AMDGPU/AMDGPUAsmGFX908>` 11591 11592 :doc:`gfx909<AMDGPU/AMDGPUAsmGFX900>` 11593 11594 :doc:`gfx90a<AMDGPU/AMDGPUAsmGFX90a>` 11595 11596 :doc:`GFX10<AMDGPU/AMDGPUAsmGFX10>` :doc:`gfx1011<AMDGPU/AMDGPUAsmGFX1011>` 11597 11598 :doc:`gfx1012<AMDGPU/AMDGPUAsmGFX1011>` 11599 =================================== ======================================= 11600 11601For more information about instructions, their semantics and supported 11602combinations of operands, refer to one of instruction set architecture manuals 11603[AMD-GCN-GFX6]_, [AMD-GCN-GFX7]_, [AMD-GCN-GFX8]_, 11604[AMD-GCN-GFX900-GFX904-VEGA]_, [AMD-GCN-GFX906-VEGA7NM]_ 11605[AMD-GCN-GFX908-CDNA1]_, [AMD-GCN-GFX10-RDNA1]_ and [AMD-GCN-GFX10-RDNA2]_. 11606 11607Operands 11608~~~~~~~~ 11609 11610Detailed description of operands may be found :doc:`here<AMDGPUOperandSyntax>`. 11611 11612Modifiers 11613~~~~~~~~~ 11614 11615Detailed description of modifiers may be found 11616:doc:`here<AMDGPUModifierSyntax>`. 11617 11618Instruction Examples 11619~~~~~~~~~~~~~~~~~~~~ 11620 11621DS 11622++ 11623 11624.. code-block:: nasm 11625 11626 ds_add_u32 v2, v4 offset:16 11627 ds_write_src2_b64 v2 offset0:4 offset1:8 11628 ds_cmpst_f32 v2, v4, v6 11629 ds_min_rtn_f64 v[8:9], v2, v[4:5] 11630 11631For full list of supported instructions, refer to "LDS/GDS instructions" in ISA 11632Manual. 11633 11634FLAT 11635++++ 11636 11637.. code-block:: nasm 11638 11639 flat_load_dword v1, v[3:4] 11640 flat_store_dwordx3 v[3:4], v[5:7] 11641 flat_atomic_swap v1, v[3:4], v5 glc 11642 flat_atomic_cmpswap v1, v[3:4], v[5:6] glc slc 11643 flat_atomic_fmax_x2 v[1:2], v[3:4], v[5:6] glc 11644 11645For full list of supported instructions, refer to "FLAT instructions" in ISA 11646Manual. 11647 11648MUBUF 11649+++++ 11650 11651.. code-block:: nasm 11652 11653 buffer_load_dword v1, off, s[4:7], s1 11654 buffer_store_dwordx4 v[1:4], v2, ttmp[4:7], s1 offen offset:4 glc tfe 11655 buffer_store_format_xy v[1:2], off, s[4:7], s1 11656 buffer_wbinvl1 11657 buffer_atomic_inc v1, v2, s[8:11], s4 idxen offset:4 slc 11658 11659For full list of supported instructions, refer to "MUBUF Instructions" in ISA 11660Manual. 11661 11662SMRD/SMEM 11663+++++++++ 11664 11665.. code-block:: nasm 11666 11667 s_load_dword s1, s[2:3], 0xfc 11668 s_load_dwordx8 s[8:15], s[2:3], s4 11669 s_load_dwordx16 s[88:103], s[2:3], s4 11670 s_dcache_inv_vol 11671 s_memtime s[4:5] 11672 11673For full list of supported instructions, refer to "Scalar Memory Operations" in 11674ISA Manual. 11675 11676SOP1 11677++++ 11678 11679.. code-block:: nasm 11680 11681 s_mov_b32 s1, s2 11682 s_mov_b64 s[0:1], 0x80000000 11683 s_cmov_b32 s1, 200 11684 s_wqm_b64 s[2:3], s[4:5] 11685 s_bcnt0_i32_b64 s1, s[2:3] 11686 s_swappc_b64 s[2:3], s[4:5] 11687 s_cbranch_join s[4:5] 11688 11689For full list of supported instructions, refer to "SOP1 Instructions" in ISA 11690Manual. 11691 11692SOP2 11693++++ 11694 11695.. code-block:: nasm 11696 11697 s_add_u32 s1, s2, s3 11698 s_and_b64 s[2:3], s[4:5], s[6:7] 11699 s_cselect_b32 s1, s2, s3 11700 s_andn2_b32 s2, s4, s6 11701 s_lshr_b64 s[2:3], s[4:5], s6 11702 s_ashr_i32 s2, s4, s6 11703 s_bfm_b64 s[2:3], s4, s6 11704 s_bfe_i64 s[2:3], s[4:5], s6 11705 s_cbranch_g_fork s[4:5], s[6:7] 11706 11707For full list of supported instructions, refer to "SOP2 Instructions" in ISA 11708Manual. 11709 11710SOPC 11711++++ 11712 11713.. code-block:: nasm 11714 11715 s_cmp_eq_i32 s1, s2 11716 s_bitcmp1_b32 s1, s2 11717 s_bitcmp0_b64 s[2:3], s4 11718 s_setvskip s3, s5 11719 11720For full list of supported instructions, refer to "SOPC Instructions" in ISA 11721Manual. 11722 11723SOPP 11724++++ 11725 11726.. code-block:: nasm 11727 11728 s_barrier 11729 s_nop 2 11730 s_endpgm 11731 s_waitcnt 0 ; Wait for all counters to be 0 11732 s_waitcnt vmcnt(0) & expcnt(0) & lgkmcnt(0) ; Equivalent to above 11733 s_waitcnt vmcnt(1) ; Wait for vmcnt counter to be 1. 11734 s_sethalt 9 11735 s_sleep 10 11736 s_sendmsg 0x1 11737 s_sendmsg sendmsg(MSG_INTERRUPT) 11738 s_trap 1 11739 11740For full list of supported instructions, refer to "SOPP Instructions" in ISA 11741Manual. 11742 11743Unless otherwise mentioned, little verification is performed on the operands 11744of SOPP Instructions, so it is up to the programmer to be familiar with the 11745range or acceptable values. 11746 11747VALU 11748++++ 11749 11750For vector ALU instruction opcodes (VOP1, VOP2, VOP3, VOPC, VOP_DPP, VOP_SDWA), 11751the assembler will automatically use optimal encoding based on its operands. To 11752force specific encoding, one can add a suffix to the opcode of the instruction: 11753 11754* _e32 for 32-bit VOP1/VOP2/VOPC 11755* _e64 for 64-bit VOP3 11756* _dpp for VOP_DPP 11757* _sdwa for VOP_SDWA 11758 11759VOP1/VOP2/VOP3/VOPC examples: 11760 11761.. code-block:: nasm 11762 11763 v_mov_b32 v1, v2 11764 v_mov_b32_e32 v1, v2 11765 v_nop 11766 v_cvt_f64_i32_e32 v[1:2], v2 11767 v_floor_f32_e32 v1, v2 11768 v_bfrev_b32_e32 v1, v2 11769 v_add_f32_e32 v1, v2, v3 11770 v_mul_i32_i24_e64 v1, v2, 3 11771 v_mul_i32_i24_e32 v1, -3, v3 11772 v_mul_i32_i24_e32 v1, -100, v3 11773 v_addc_u32 v1, s[0:1], v2, v3, s[2:3] 11774 v_max_f16_e32 v1, v2, v3 11775 11776VOP_DPP examples: 11777 11778.. code-block:: nasm 11779 11780 v_mov_b32 v0, v0 quad_perm:[0,2,1,1] 11781 v_sin_f32 v0, v0 row_shl:1 row_mask:0xa bank_mask:0x1 bound_ctrl:0 11782 v_mov_b32 v0, v0 wave_shl:1 11783 v_mov_b32 v0, v0 row_mirror 11784 v_mov_b32 v0, v0 row_bcast:31 11785 v_mov_b32 v0, v0 quad_perm:[1,3,0,1] row_mask:0xa bank_mask:0x1 bound_ctrl:0 11786 v_add_f32 v0, v0, |v0| row_shl:1 row_mask:0xa bank_mask:0x1 bound_ctrl:0 11787 v_max_f16 v1, v2, v3 row_shl:1 row_mask:0xa bank_mask:0x1 bound_ctrl:0 11788 11789VOP_SDWA examples: 11790 11791.. code-block:: nasm 11792 11793 v_mov_b32 v1, v2 dst_sel:BYTE_0 dst_unused:UNUSED_PRESERVE src0_sel:DWORD 11794 v_min_u32 v200, v200, v1 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:BYTE_1 src1_sel:DWORD 11795 v_sin_f32 v0, v0 dst_unused:UNUSED_PAD src0_sel:WORD_1 11796 v_fract_f32 v0, |v0| dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 11797 v_cmpx_le_u32 vcc, v1, v2 src0_sel:BYTE_2 src1_sel:WORD_0 11798 11799For full list of supported instructions, refer to "Vector ALU instructions". 11800 11801.. _amdgpu-amdhsa-assembler-predefined-symbols-v2: 11802 11803Code Object V2 Predefined Symbols 11804~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 11805 11806.. warning:: 11807 Code object V2 is not the default code object version emitted by 11808 this version of LLVM. 11809 11810The AMDGPU assembler defines and updates some symbols automatically. These 11811symbols do not affect code generation. 11812 11813.option.machine_version_major 11814+++++++++++++++++++++++++++++ 11815 11816Set to the GFX major generation number of the target being assembled for. For 11817example, when assembling for a "GFX9" target this will be set to the integer 11818value "9". The possible GFX major generation numbers are presented in 11819:ref:`amdgpu-processors`. 11820 11821.option.machine_version_minor 11822+++++++++++++++++++++++++++++ 11823 11824Set to the GFX minor generation number of the target being assembled for. For 11825example, when assembling for a "GFX810" target this will be set to the integer 11826value "1". The possible GFX minor generation numbers are presented in 11827:ref:`amdgpu-processors`. 11828 11829.option.machine_version_stepping 11830++++++++++++++++++++++++++++++++ 11831 11832Set to the GFX stepping generation number of the target being assembled for. 11833For example, when assembling for a "GFX704" target this will be set to the 11834integer value "4". The possible GFX stepping generation numbers are presented 11835in :ref:`amdgpu-processors`. 11836 11837.kernel.vgpr_count 11838++++++++++++++++++ 11839 11840Set to zero each time a 11841:ref:`amdgpu-amdhsa-assembler-directive-amdgpu_hsa_kernel` directive is 11842encountered. At each instruction, if the current value of this symbol is less 11843than or equal to the maximum VGPR number explicitly referenced within that 11844instruction then the symbol value is updated to equal that VGPR number plus 11845one. 11846 11847.kernel.sgpr_count 11848++++++++++++++++++ 11849 11850Set to zero each time a 11851:ref:`amdgpu-amdhsa-assembler-directive-amdgpu_hsa_kernel` directive is 11852encountered. At each instruction, if the current value of this symbol is less 11853than or equal to the maximum VGPR number explicitly referenced within that 11854instruction then the symbol value is updated to equal that SGPR number plus 11855one. 11856 11857.. _amdgpu-amdhsa-assembler-directives-v2: 11858 11859Code Object V2 Directives 11860~~~~~~~~~~~~~~~~~~~~~~~~~ 11861 11862.. warning:: 11863 Code object V2 is not the default code object version emitted by 11864 this version of LLVM. 11865 11866AMDGPU ABI defines auxiliary data in output code object. In assembly source, 11867one can specify them with assembler directives. 11868 11869.hsa_code_object_version major, minor 11870+++++++++++++++++++++++++++++++++++++ 11871 11872*major* and *minor* are integers that specify the version of the HSA code 11873object that will be generated by the assembler. 11874 11875.hsa_code_object_isa [major, minor, stepping, vendor, arch] 11876+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 11877 11878 11879*major*, *minor*, and *stepping* are all integers that describe the instruction 11880set architecture (ISA) version of the assembly program. 11881 11882*vendor* and *arch* are quoted strings. *vendor* should always be equal to 11883"AMD" and *arch* should always be equal to "AMDGPU". 11884 11885By default, the assembler will derive the ISA version, *vendor*, and *arch* 11886from the value of the -mcpu option that is passed to the assembler. 11887 11888.. _amdgpu-amdhsa-assembler-directive-amdgpu_hsa_kernel: 11889 11890.amdgpu_hsa_kernel (name) 11891+++++++++++++++++++++++++ 11892 11893This directives specifies that the symbol with given name is a kernel entry 11894point (label) and the object should contain corresponding symbol of type 11895STT_AMDGPU_HSA_KERNEL. 11896 11897.amd_kernel_code_t 11898++++++++++++++++++ 11899 11900This directive marks the beginning of a list of key / value pairs that are used 11901to specify the amd_kernel_code_t object that will be emitted by the assembler. 11902The list must be terminated by the *.end_amd_kernel_code_t* directive. For any 11903amd_kernel_code_t values that are unspecified a default value will be used. The 11904default value for all keys is 0, with the following exceptions: 11905 11906- *amd_code_version_major* defaults to 1. 11907- *amd_kernel_code_version_minor* defaults to 2. 11908- *amd_machine_kind* defaults to 1. 11909- *amd_machine_version_major*, *machine_version_minor*, and 11910 *amd_machine_version_stepping* are derived from the value of the -mcpu option 11911 that is passed to the assembler. 11912- *kernel_code_entry_byte_offset* defaults to 256. 11913- *wavefront_size* defaults 6 for all targets before GFX10. For GFX10 onwards 11914 defaults to 6 if target feature ``wavefrontsize64`` is enabled, otherwise 5. 11915 Note that wavefront size is specified as a power of two, so a value of **n** 11916 means a size of 2^ **n**. 11917- *call_convention* defaults to -1. 11918- *kernarg_segment_alignment*, *group_segment_alignment*, and 11919 *private_segment_alignment* default to 4. Note that alignments are specified 11920 as a power of 2, so a value of **n** means an alignment of 2^ **n**. 11921- *enable_tg_split* defaults to 1 if target feature ``tgsplit`` is enabled for 11922 GFX90A onwards. 11923- *enable_wgp_mode* defaults to 1 if target feature ``cumode`` is disabled for 11924 GFX10 onwards. 11925- *enable_mem_ordered* defaults to 1 for GFX10 onwards. 11926 11927The *.amd_kernel_code_t* directive must be placed immediately after the 11928function label and before any instructions. 11929 11930For a full list of amd_kernel_code_t keys, refer to AMDGPU ABI document, 11931comments in lib/Target/AMDGPU/AmdKernelCodeT.h and test/CodeGen/AMDGPU/hsa.s. 11932 11933.. _amdgpu-amdhsa-assembler-example-v2: 11934 11935Code Object V2 Example Source Code 11936~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 11937 11938.. warning:: 11939 Code Object V2 is not the default code object version emitted by 11940 this version of LLVM. 11941 11942Here is an example of a minimal assembly source file, defining one HSA kernel: 11943 11944.. code:: 11945 :number-lines: 11946 11947 .hsa_code_object_version 1,0 11948 .hsa_code_object_isa 11949 11950 .hsatext 11951 .globl hello_world 11952 .p2align 8 11953 .amdgpu_hsa_kernel hello_world 11954 11955 hello_world: 11956 11957 .amd_kernel_code_t 11958 enable_sgpr_kernarg_segment_ptr = 1 11959 is_ptr64 = 1 11960 compute_pgm_rsrc1_vgprs = 0 11961 compute_pgm_rsrc1_sgprs = 0 11962 compute_pgm_rsrc2_user_sgpr = 2 11963 compute_pgm_rsrc1_wgp_mode = 0 11964 compute_pgm_rsrc1_mem_ordered = 0 11965 compute_pgm_rsrc1_fwd_progress = 1 11966 .end_amd_kernel_code_t 11967 11968 s_load_dwordx2 s[0:1], s[0:1] 0x0 11969 v_mov_b32 v0, 3.14159 11970 s_waitcnt lgkmcnt(0) 11971 v_mov_b32 v1, s0 11972 v_mov_b32 v2, s1 11973 flat_store_dword v[1:2], v0 11974 s_endpgm 11975 .Lfunc_end0: 11976 .size hello_world, .Lfunc_end0-hello_world 11977 11978.. _amdgpu-amdhsa-assembler-predefined-symbols-v3-v4: 11979 11980Code Object V3 to V4 Predefined Symbols 11981~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 11982 11983The AMDGPU assembler defines and updates some symbols automatically. These 11984symbols do not affect code generation. 11985 11986.amdgcn.gfx_generation_number 11987+++++++++++++++++++++++++++++ 11988 11989Set to the GFX major generation number of the target being assembled for. For 11990example, when assembling for a "GFX9" target this will be set to the integer 11991value "9". The possible GFX major generation numbers are presented in 11992:ref:`amdgpu-processors`. 11993 11994.amdgcn.gfx_generation_minor 11995++++++++++++++++++++++++++++ 11996 11997Set to the GFX minor generation number of the target being assembled for. For 11998example, when assembling for a "GFX810" target this will be set to the integer 11999value "1". The possible GFX minor generation numbers are presented in 12000:ref:`amdgpu-processors`. 12001 12002.amdgcn.gfx_generation_stepping 12003+++++++++++++++++++++++++++++++ 12004 12005Set to the GFX stepping generation number of the target being assembled for. 12006For example, when assembling for a "GFX704" target this will be set to the 12007integer value "4". The possible GFX stepping generation numbers are presented 12008in :ref:`amdgpu-processors`. 12009 12010.. _amdgpu-amdhsa-assembler-symbol-next_free_vgpr: 12011 12012.amdgcn.next_free_vgpr 12013++++++++++++++++++++++ 12014 12015Set to zero before assembly begins. At each instruction, if the current value 12016of this symbol is less than or equal to the maximum VGPR number explicitly 12017referenced within that instruction then the symbol value is updated to equal 12018that VGPR number plus one. 12019 12020May be used to set the `.amdhsa_next_free_vgpr` directive in 12021:ref:`amdhsa-kernel-directives-table`. 12022 12023May be set at any time, e.g. manually set to zero at the start of each kernel. 12024 12025.. _amdgpu-amdhsa-assembler-symbol-next_free_sgpr: 12026 12027.amdgcn.next_free_sgpr 12028++++++++++++++++++++++ 12029 12030Set to zero before assembly begins. At each instruction, if the current value 12031of this symbol is less than or equal the maximum SGPR number explicitly 12032referenced within that instruction then the symbol value is updated to equal 12033that SGPR number plus one. 12034 12035May be used to set the `.amdhsa_next_free_spgr` directive in 12036:ref:`amdhsa-kernel-directives-table`. 12037 12038May be set at any time, e.g. manually set to zero at the start of each kernel. 12039 12040.. _amdgpu-amdhsa-assembler-directives-v3-v4: 12041 12042Code Object V3 to V4 Directives 12043~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 12044 12045Directives which begin with ``.amdgcn`` are valid for all ``amdgcn`` 12046architecture processors, and are not OS-specific. Directives which begin with 12047``.amdhsa`` are specific to ``amdgcn`` architecture processors when the 12048``amdhsa`` OS is specified. See :ref:`amdgpu-target-triples` and 12049:ref:`amdgpu-processors`. 12050 12051.. _amdgpu-assembler-directive-amdgcn-target: 12052 12053.amdgcn_target <target-triple> "-" <target-id> 12054++++++++++++++++++++++++++++++++++++++++++++++ 12055 12056Optional directive which declares the ``<target-triple>-<target-id>`` supported 12057by the containing assembler source file. Used by the assembler to validate 12058command-line options such as ``-triple``, ``-mcpu``, and 12059``--offload-arch=<target-id>``. A non-canonical target ID is allowed. See 12060:ref:`amdgpu-target-triples` and :ref:`amdgpu-target-id`. 12061 12062.. note:: 12063 12064 The target ID syntax used for code object V2 to V3 for this directive differs 12065 from that used elsewhere. See :ref:`amdgpu-target-id-v2-v3`. 12066 12067.amdhsa_kernel <name> 12068+++++++++++++++++++++ 12069 12070Creates a correctly aligned AMDHSA kernel descriptor and a symbol, 12071``<name>.kd``, in the current location of the current section. Only valid when 12072the OS is ``amdhsa``. ``<name>`` must be a symbol that labels the first 12073instruction to execute, and does not need to be previously defined. 12074 12075Marks the beginning of a list of directives used to generate the bytes of a 12076kernel descriptor, as described in :ref:`amdgpu-amdhsa-kernel-descriptor`. 12077Directives which may appear in this list are described in 12078:ref:`amdhsa-kernel-directives-table`. Directives may appear in any order, must 12079be valid for the target being assembled for, and cannot be repeated. Directives 12080support the range of values specified by the field they reference in 12081:ref:`amdgpu-amdhsa-kernel-descriptor`. If a directive is not specified, it is 12082assumed to have its default value, unless it is marked as "Required", in which 12083case it is an error to omit the directive. This list of directives is 12084terminated by an ``.end_amdhsa_kernel`` directive. 12085 12086 .. table:: AMDHSA Kernel Assembler Directives 12087 :name: amdhsa-kernel-directives-table 12088 12089 ======================================================== =================== ============ =================== 12090 Directive Default Supported On Description 12091 ======================================================== =================== ============ =================== 12092 ``.amdhsa_group_segment_fixed_size`` 0 GFX6-GFX10 Controls GROUP_SEGMENT_FIXED_SIZE in 12093 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 12094 ``.amdhsa_private_segment_fixed_size`` 0 GFX6-GFX10 Controls PRIVATE_SEGMENT_FIXED_SIZE in 12095 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 12096 ``.amdhsa_kernarg_size`` 0 GFX6-GFX10 Controls KERNARG_SIZE in 12097 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 12098 ``.amdhsa_user_sgpr_private_segment_buffer`` 0 GFX6-GFX10 Controls ENABLE_SGPR_PRIVATE_SEGMENT_BUFFER in 12099 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 12100 ``.amdhsa_user_sgpr_dispatch_ptr`` 0 GFX6-GFX10 Controls ENABLE_SGPR_DISPATCH_PTR in 12101 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 12102 ``.amdhsa_user_sgpr_queue_ptr`` 0 GFX6-GFX10 Controls ENABLE_SGPR_QUEUE_PTR in 12103 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 12104 ``.amdhsa_user_sgpr_kernarg_segment_ptr`` 0 GFX6-GFX10 Controls ENABLE_SGPR_KERNARG_SEGMENT_PTR in 12105 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 12106 ``.amdhsa_user_sgpr_dispatch_id`` 0 GFX6-GFX10 Controls ENABLE_SGPR_DISPATCH_ID in 12107 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 12108 ``.amdhsa_user_sgpr_flat_scratch_init`` 0 GFX6-GFX10 Controls ENABLE_SGPR_FLAT_SCRATCH_INIT in 12109 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 12110 ``.amdhsa_user_sgpr_private_segment_size`` 0 GFX6-GFX10 Controls ENABLE_SGPR_PRIVATE_SEGMENT_SIZE in 12111 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 12112 ``.amdhsa_wavefront_size32`` Target GFX10 Controls ENABLE_WAVEFRONT_SIZE32 in 12113 Feature :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 12114 Specific 12115 (wavefrontsize64) 12116 ``.amdhsa_system_sgpr_private_segment_wavefront_offset`` 0 GFX6-GFX10 Controls ENABLE_PRIVATE_SEGMENT in 12117 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 12118 ``.amdhsa_system_sgpr_workgroup_id_x`` 1 GFX6-GFX10 Controls ENABLE_SGPR_WORKGROUP_ID_X in 12119 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 12120 ``.amdhsa_system_sgpr_workgroup_id_y`` 0 GFX6-GFX10 Controls ENABLE_SGPR_WORKGROUP_ID_Y in 12121 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 12122 ``.amdhsa_system_sgpr_workgroup_id_z`` 0 GFX6-GFX10 Controls ENABLE_SGPR_WORKGROUP_ID_Z in 12123 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 12124 ``.amdhsa_system_sgpr_workgroup_info`` 0 GFX6-GFX10 Controls ENABLE_SGPR_WORKGROUP_INFO in 12125 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 12126 ``.amdhsa_system_vgpr_workitem_id`` 0 GFX6-GFX10 Controls ENABLE_VGPR_WORKITEM_ID in 12127 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 12128 Possible values are defined in 12129 :ref:`amdgpu-amdhsa-system-vgpr-work-item-id-enumeration-values-table`. 12130 ``.amdhsa_next_free_vgpr`` Required GFX6-GFX10 Maximum VGPR number explicitly referenced, plus one. 12131 Used to calculate GRANULATED_WORKITEM_VGPR_COUNT in 12132 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 12133 ``.amdhsa_next_free_sgpr`` Required GFX6-GFX10 Maximum SGPR number explicitly referenced, plus one. 12134 Used to calculate GRANULATED_WAVEFRONT_SGPR_COUNT in 12135 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 12136 ``.amdhsa_accum_offset`` Required GFX90A Offset of a first AccVGPR in the unified register file. 12137 Used to calculate ACCUM_OFFSET in 12138 :ref:`amdgpu-amdhsa-compute_pgm_rsrc3-gfx90a-table`. 12139 ``.amdhsa_reserve_vcc`` 1 GFX6-GFX10 Whether the kernel may use the special VCC SGPR. 12140 Used to calculate GRANULATED_WAVEFRONT_SGPR_COUNT in 12141 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 12142 ``.amdhsa_reserve_flat_scratch`` 1 GFX7-GFX10 Whether the kernel may use flat instructions to access 12143 scratch memory. Used to calculate 12144 GRANULATED_WAVEFRONT_SGPR_COUNT in 12145 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 12146 ``.amdhsa_reserve_xnack_mask`` Target GFX8-GFX10 Whether the kernel may trigger XNACK replay. 12147 Feature Used to calculate GRANULATED_WAVEFRONT_SGPR_COUNT in 12148 Specific :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 12149 (xnack) 12150 ``.amdhsa_float_round_mode_32`` 0 GFX6-GFX10 Controls FLOAT_ROUND_MODE_32 in 12151 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 12152 Possible values are defined in 12153 :ref:`amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table`. 12154 ``.amdhsa_float_round_mode_16_64`` 0 GFX6-GFX10 Controls FLOAT_ROUND_MODE_16_64 in 12155 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 12156 Possible values are defined in 12157 :ref:`amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table`. 12158 ``.amdhsa_float_denorm_mode_32`` 0 GFX6-GFX10 Controls FLOAT_DENORM_MODE_32 in 12159 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 12160 Possible values are defined in 12161 :ref:`amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table`. 12162 ``.amdhsa_float_denorm_mode_16_64`` 3 GFX6-GFX10 Controls FLOAT_DENORM_MODE_16_64 in 12163 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 12164 Possible values are defined in 12165 :ref:`amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table`. 12166 ``.amdhsa_dx10_clamp`` 1 GFX6-GFX10 Controls ENABLE_DX10_CLAMP in 12167 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 12168 ``.amdhsa_ieee_mode`` 1 GFX6-GFX10 Controls ENABLE_IEEE_MODE in 12169 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 12170 ``.amdhsa_fp16_overflow`` 0 GFX9-GFX10 Controls FP16_OVFL in 12171 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 12172 ``.amdhsa_tg_split`` Target GFX90A Controls TG_SPLIT in 12173 Feature :ref:`amdgpu-amdhsa-compute_pgm_rsrc3-gfx90a-table`. 12174 Specific 12175 (tgsplit) 12176 ``.amdhsa_workgroup_processor_mode`` Target GFX10 Controls ENABLE_WGP_MODE in 12177 Feature :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 12178 Specific 12179 (cumode) 12180 ``.amdhsa_memory_ordered`` 1 GFX10 Controls MEM_ORDERED in 12181 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 12182 ``.amdhsa_forward_progress`` 0 GFX10 Controls FWD_PROGRESS in 12183 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 12184 ``.amdhsa_exception_fp_ieee_invalid_op`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_INVALID_OPERATION in 12185 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 12186 ``.amdhsa_exception_fp_denorm_src`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_FP_DENORMAL_SOURCE in 12187 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 12188 ``.amdhsa_exception_fp_ieee_div_zero`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_DIVISION_BY_ZERO in 12189 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 12190 ``.amdhsa_exception_fp_ieee_overflow`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_OVERFLOW in 12191 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 12192 ``.amdhsa_exception_fp_ieee_underflow`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_UNDERFLOW in 12193 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 12194 ``.amdhsa_exception_fp_ieee_inexact`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_INEXACT in 12195 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 12196 ``.amdhsa_exception_int_div_zero`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_INT_DIVIDE_BY_ZERO in 12197 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 12198 ======================================================== =================== ============ =================== 12199 12200.amdgpu_metadata 12201++++++++++++++++ 12202 12203Optional directive which declares the contents of the ``NT_AMDGPU_METADATA`` 12204note record (see :ref:`amdgpu-elf-note-records-table-v3-v4`). 12205 12206The contents must be in the [YAML]_ markup format, with the same structure and 12207semantics described in :ref:`amdgpu-amdhsa-code-object-metadata-v3` or 12208:ref:`amdgpu-amdhsa-code-object-metadata-v4`. 12209 12210This directive is terminated by an ``.end_amdgpu_metadata`` directive. 12211 12212.. _amdgpu-amdhsa-assembler-example-v3-v4: 12213 12214Code Object V3 to V4 Example Source Code 12215~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 12216 12217Here is an example of a minimal assembly source file, defining one HSA kernel: 12218 12219.. code:: 12220 :number-lines: 12221 12222 .amdgcn_target "amdgcn-amd-amdhsa--gfx900+xnack" // optional 12223 12224 .text 12225 .globl hello_world 12226 .p2align 8 12227 .type hello_world,@function 12228 hello_world: 12229 s_load_dwordx2 s[0:1], s[0:1] 0x0 12230 v_mov_b32 v0, 3.14159 12231 s_waitcnt lgkmcnt(0) 12232 v_mov_b32 v1, s0 12233 v_mov_b32 v2, s1 12234 flat_store_dword v[1:2], v0 12235 s_endpgm 12236 .Lfunc_end0: 12237 .size hello_world, .Lfunc_end0-hello_world 12238 12239 .rodata 12240 .p2align 6 12241 .amdhsa_kernel hello_world 12242 .amdhsa_user_sgpr_kernarg_segment_ptr 1 12243 .amdhsa_next_free_vgpr .amdgcn.next_free_vgpr 12244 .amdhsa_next_free_sgpr .amdgcn.next_free_sgpr 12245 .end_amdhsa_kernel 12246 12247 .amdgpu_metadata 12248 --- 12249 amdhsa.version: 12250 - 1 12251 - 0 12252 amdhsa.kernels: 12253 - .name: hello_world 12254 .symbol: hello_world.kd 12255 .kernarg_segment_size: 48 12256 .group_segment_fixed_size: 0 12257 .private_segment_fixed_size: 0 12258 .kernarg_segment_align: 4 12259 .wavefront_size: 64 12260 .sgpr_count: 2 12261 .vgpr_count: 3 12262 .max_flat_workgroup_size: 256 12263 .args: 12264 - .size: 8 12265 .offset: 0 12266 .value_kind: global_buffer 12267 .address_space: global 12268 .actual_access: write_only 12269 //... 12270 .end_amdgpu_metadata 12271 12272This kernel is equivalent to the following HIP program: 12273 12274.. code:: 12275 :number-lines: 12276 12277 __global__ void hello_world(float *p) { 12278 *p = 3.14159f; 12279 } 12280 12281If an assembly source file contains multiple kernels and/or functions, the 12282:ref:`amdgpu-amdhsa-assembler-symbol-next_free_vgpr` and 12283:ref:`amdgpu-amdhsa-assembler-symbol-next_free_sgpr` symbols may be reset using 12284the ``.set <symbol>, <expression>`` directive. For example, in the case of two 12285kernels, where ``function1`` is only called from ``kernel1`` it is sufficient 12286to group the function with the kernel that calls it and reset the symbols 12287between the two connected components: 12288 12289.. code:: 12290 :number-lines: 12291 12292 .amdgcn_target "amdgcn-amd-amdhsa--gfx900+xnack" // optional 12293 12294 // gpr tracking symbols are implicitly set to zero 12295 12296 .text 12297 .globl kern0 12298 .p2align 8 12299 .type kern0,@function 12300 kern0: 12301 // ... 12302 s_endpgm 12303 .Lkern0_end: 12304 .size kern0, .Lkern0_end-kern0 12305 12306 .rodata 12307 .p2align 6 12308 .amdhsa_kernel kern0 12309 // ... 12310 .amdhsa_next_free_vgpr .amdgcn.next_free_vgpr 12311 .amdhsa_next_free_sgpr .amdgcn.next_free_sgpr 12312 .end_amdhsa_kernel 12313 12314 // reset symbols to begin tracking usage in func1 and kern1 12315 .set .amdgcn.next_free_vgpr, 0 12316 .set .amdgcn.next_free_sgpr, 0 12317 12318 .text 12319 .hidden func1 12320 .global func1 12321 .p2align 2 12322 .type func1,@function 12323 func1: 12324 // ... 12325 s_setpc_b64 s[30:31] 12326 .Lfunc1_end: 12327 .size func1, .Lfunc1_end-func1 12328 12329 .globl kern1 12330 .p2align 8 12331 .type kern1,@function 12332 kern1: 12333 // ... 12334 s_getpc_b64 s[4:5] 12335 s_add_u32 s4, s4, func1@rel32@lo+4 12336 s_addc_u32 s5, s5, func1@rel32@lo+4 12337 s_swappc_b64 s[30:31], s[4:5] 12338 // ... 12339 s_endpgm 12340 .Lkern1_end: 12341 .size kern1, .Lkern1_end-kern1 12342 12343 .rodata 12344 .p2align 6 12345 .amdhsa_kernel kern1 12346 // ... 12347 .amdhsa_next_free_vgpr .amdgcn.next_free_vgpr 12348 .amdhsa_next_free_sgpr .amdgcn.next_free_sgpr 12349 .end_amdhsa_kernel 12350 12351These symbols cannot identify connected components in order to automatically 12352track the usage for each kernel. However, in some cases careful organization of 12353the kernels and functions in the source file means there is minimal additional 12354effort required to accurately calculate GPR usage. 12355 12356Additional Documentation 12357======================== 12358 12359.. [AMD-GCN-GFX6] `AMD Southern Islands Series ISA <http://developer.amd.com/wordpress/media/2012/12/AMD_Southern_Islands_Instruction_Set_Architecture.pdf>`__ 12360.. [AMD-GCN-GFX7] `AMD Sea Islands Series ISA <http://developer.amd.com/wordpress/media/2013/07/AMD_Sea_Islands_Instruction_Set_Architecture.pdf>`_ 12361.. [AMD-GCN-GFX8] `AMD GCN3 Instruction Set Architecture <http://amd-dev.wpengine.netdna-cdn.com/wordpress/media/2013/12/AMD_GCN3_Instruction_Set_Architecture_rev1.1.pdf>`__ 12362.. [AMD-GCN-GFX900-GFX904-VEGA] `AMD Vega Instruction Set Architecture <http://developer.amd.com/wordpress/media/2013/12/Vega_Shader_ISA_28July2017.pdf>`__ 12363.. [AMD-GCN-GFX906-VEGA7NM] `AMD Vega 7nm Instruction Set Architecture <https://gpuopen.com/wp-content/uploads/2019/11/Vega_7nm_Shader_ISA_26November2019.pdf>`__ 12364.. [AMD-GCN-GFX908-CDNA1] `AMD Instinct MI100 Instruction Set Architecture <https://developer.amd.com/wp-content/resources/CDNA1_Shader_ISA_14December2020.pdf>`__ 12365.. [AMD-GCN-GFX10-RDNA1] `AMD RDNA 1.0 Instruction Set Architecture <https://gpuopen.com/wp-content/uploads/2019/08/RDNA_Shader_ISA_5August2019.pdf>`__ 12366.. [AMD-GCN-GFX10-RDNA2] `AMD RDNA 2 Instruction Set Architecture <https://developer.amd.com/wp-content/resources/RDNA2_Shader_ISA_November2020.pdf>`__ 12367.. [AMD-RADEON-HD-2000-3000] `AMD R6xx shader ISA <http://developer.amd.com/wordpress/media/2012/10/R600_Instruction_Set_Architecture.pdf>`__ 12368.. [AMD-RADEON-HD-4000] `AMD R7xx shader ISA <http://developer.amd.com/wordpress/media/2012/10/R700-Family_Instruction_Set_Architecture.pdf>`__ 12369.. [AMD-RADEON-HD-5000] `AMD Evergreen shader ISA <http://developer.amd.com/wordpress/media/2012/10/AMD_Evergreen-Family_Instruction_Set_Architecture.pdf>`__ 12370.. [AMD-RADEON-HD-6000] `AMD Cayman/Trinity shader ISA <http://developer.amd.com/wordpress/media/2012/10/AMD_HD_6900_Series_Instruction_Set_Architecture.pdf>`__ 12371.. [AMD-ROCm] `AMD ROCm™ Platform <https://rocmdocs.amd.com/>`__ 12372.. [AMD-ROCm-github] `AMD ROCm™ github <http://github.com/RadeonOpenCompute>`__ 12373.. [AMD-ROCm-Release-Notes] `AMD ROCm Release Notes <https://github.com/RadeonOpenCompute/ROCm>`__ 12374.. [CLANG-ATTR] `Attributes in Clang <https://clang.llvm.org/docs/AttributeReference.html>`__ 12375.. [DWARF] `DWARF Debugging Information Format <http://dwarfstd.org/>`__ 12376.. [ELF] `Executable and Linkable Format (ELF) <http://www.sco.com/developers/gabi/>`__ 12377.. [HRF] `Heterogeneous-race-free Memory Models <http://benedictgaster.org/wp-content/uploads/2014/01/asplos269-FINAL.pdf>`__ 12378.. [HSA] `Heterogeneous System Architecture (HSA) Foundation <http://www.hsafoundation.com/>`__ 12379.. [MsgPack] `Message Pack <http://www.msgpack.org/>`__ 12380.. [OpenCL] `The OpenCL Specification Version 2.0 <http://www.khronos.org/registry/cl/specs/opencl-2.0.pdf>`__ 12381.. [SEMVER] `Semantic Versioning <https://semver.org/>`__ 12382.. [YAML] `YAML Ain't Markup Language (YAML™) Version 1.2 <http://www.yaml.org/spec/1.2/spec.html>`__ 12383