1============================= 2User Guide for AMDGPU Backend 3============================= 4 5.. contents:: 6 :local: 7 8.. toctree:: 9 :hidden: 10 11 AMDGPU/AMDGPUAsmGFX7 12 AMDGPU/AMDGPUAsmGFX8 13 AMDGPU/AMDGPUAsmGFX9 14 AMDGPU/AMDGPUAsmGFX900 15 AMDGPU/AMDGPUAsmGFX904 16 AMDGPU/AMDGPUAsmGFX906 17 AMDGPU/AMDGPUAsmGFX908 18 AMDGPU/AMDGPUAsmGFX10 19 AMDGPU/AMDGPUAsmGFX1011 20 AMDGPUModifierSyntax 21 AMDGPUOperandSyntax 22 AMDGPUInstructionSyntax 23 AMDGPUInstructionNotation 24 AMDGPUDwarfProposalForHeterogeneousDebugging 25 26Introduction 27============ 28 29The AMDGPU backend provides ISA code generation for AMD GPUs, starting with the 30R600 family up until the current GCN families. It lives in the 31``llvm/lib/Target/AMDGPU`` directory. 32 33LLVM 34==== 35 36.. _amdgpu-target-triples: 37 38Target Triples 39-------------- 40 41Use the ``clang -target <Architecture>-<Vendor>-<OS>-<Environment>`` option to 42specify the target triple: 43 44 .. table:: AMDGPU Architectures 45 :name: amdgpu-architecture-table 46 47 ============ ============================================================== 48 Architecture Description 49 ============ ============================================================== 50 ``r600`` AMD GPUs HD2XXX-HD6XXX for graphics and compute shaders. 51 ``amdgcn`` AMD GPUs GCN GFX6 onwards for graphics and compute shaders. 52 ============ ============================================================== 53 54 .. table:: AMDGPU Vendors 55 :name: amdgpu-vendor-table 56 57 ============ ============================================================== 58 Vendor Description 59 ============ ============================================================== 60 ``amd`` Can be used for all AMD GPU usage. 61 ``mesa3d`` Can be used if the OS is ``mesa3d``. 62 ============ ============================================================== 63 64 .. table:: AMDGPU Operating Systems 65 :name: amdgpu-os-table 66 67 ============== ============================================================ 68 OS Description 69 ============== ============================================================ 70 *<empty>* Defaults to the *unknown* OS. 71 ``amdhsa`` Compute kernels executed on HSA [HSA]_ compatible runtimes 72 such as AMD's ROCm [AMD-ROCm]_. 73 ``amdpal`` Graphic shaders and compute kernels executed on AMD PAL 74 runtime. 75 ``mesa3d`` Graphic shaders and compute kernels executed on Mesa 3D 76 runtime. 77 ============== ============================================================ 78 79 .. table:: AMDGPU Environments 80 :name: amdgpu-environment-table 81 82 ============ ============================================================== 83 Environment Description 84 ============ ============================================================== 85 *<empty>* Default. 86 ============ ============================================================== 87 88.. _amdgpu-processors: 89 90Processors 91---------- 92 93Use the ``clang -mcpu <Processor>`` option to specify the AMDGPU processor. The 94names from both the *Processor* and *Alternative Processor* can be used. 95 96 .. table:: AMDGPU Processors 97 :name: amdgpu-processor-table 98 99 =========== =============== ============ ===== ================= ======= ====================== 100 Processor Alternative Target dGPU/ Target ROCm Example 101 Processor Triple APU Features Support Products 102 Architecture Supported 103 [Default] 104 =========== =============== ============ ===== ================= ======= ====================== 105 **Radeon HD 2000/3000 Series (R600)** [AMD-RADEON-HD-2000-3000]_ 106 ----------------------------------------------------------------------------------------------- 107 ``r600`` ``r600`` dGPU 108 ``r630`` ``r600`` dGPU 109 ``rs880`` ``r600`` dGPU 110 ``rv670`` ``r600`` dGPU 111 **Radeon HD 4000 Series (R700)** [AMD-RADEON-HD-4000]_ 112 ----------------------------------------------------------------------------------------------- 113 ``rv710`` ``r600`` dGPU 114 ``rv730`` ``r600`` dGPU 115 ``rv770`` ``r600`` dGPU 116 **Radeon HD 5000 Series (Evergreen)** [AMD-RADEON-HD-5000]_ 117 ----------------------------------------------------------------------------------------------- 118 ``cedar`` ``r600`` dGPU 119 ``cypress`` ``r600`` dGPU 120 ``juniper`` ``r600`` dGPU 121 ``redwood`` ``r600`` dGPU 122 ``sumo`` ``r600`` dGPU 123 **Radeon HD 6000 Series (Northern Islands)** [AMD-RADEON-HD-6000]_ 124 ----------------------------------------------------------------------------------------------- 125 ``barts`` ``r600`` dGPU 126 ``caicos`` ``r600`` dGPU 127 ``cayman`` ``r600`` dGPU 128 ``turks`` ``r600`` dGPU 129 **GCN GFX6 (Southern Islands (SI))** [AMD-GCN-GFX6]_ 130 ----------------------------------------------------------------------------------------------- 131 ``gfx600`` - ``tahiti`` ``amdgcn`` dGPU 132 ``gfx601`` - ``hainan`` ``amdgcn`` dGPU 133 - ``oland`` 134 - ``pitcairn`` 135 - ``verde`` 136 **GCN GFX7 (Sea Islands (CI))** [AMD-GCN-GFX7]_ 137 ----------------------------------------------------------------------------------------------- 138 ``gfx700`` - ``kaveri`` ``amdgcn`` APU - A6-7000 139 - A6 Pro-7050B 140 - A8-7100 141 - A8 Pro-7150B 142 - A10-7300 143 - A10 Pro-7350B 144 - FX-7500 145 - A8-7200P 146 - A10-7400P 147 - FX-7600P 148 ``gfx701`` - ``hawaii`` ``amdgcn`` dGPU ROCm - FirePro W8100 149 - FirePro W9100 150 - FirePro S9150 151 - FirePro S9170 152 ``gfx702`` ``amdgcn`` dGPU ROCm - Radeon R9 290 153 - Radeon R9 290x 154 - Radeon R390 155 - Radeon R390x 156 ``gfx703`` - ``kabini`` ``amdgcn`` APU - E1-2100 157 - ``mullins`` - E1-2200 158 - E1-2500 159 - E2-3000 160 - E2-3800 161 - A4-5000 162 - A4-5100 163 - A6-5200 164 - A4 Pro-3340B 165 ``gfx704`` - ``bonaire`` ``amdgcn`` dGPU - Radeon HD 7790 166 - Radeon HD 8770 167 - R7 260 168 - R7 260X 169 **GCN GFX8 (Volcanic Islands (VI))** [AMD-GCN-GFX8]_ 170 ----------------------------------------------------------------------------------------------- 171 ``gfx801`` - ``carrizo`` ``amdgcn`` APU - xnack - A6-8500P 172 [on] - Pro A6-8500B 173 - A8-8600P 174 - Pro A8-8600B 175 - FX-8800P 176 - Pro A12-8800B 177 \ ``amdgcn`` APU - xnack ROCm - A10-8700P 178 [on] - Pro A10-8700B 179 - A10-8780P 180 \ ``amdgcn`` APU - xnack - A10-9600P 181 [on] - A10-9630P 182 - A12-9700P 183 - A12-9730P 184 - FX-9800P 185 - FX-9830P 186 \ ``amdgcn`` APU - xnack - E2-9010 187 [on] - A6-9210 188 - A9-9410 189 ``gfx802`` - ``iceland`` ``amdgcn`` dGPU - xnack ROCm - FirePro S7150 190 - ``tonga`` [off] - FirePro S7100 191 - FirePro W7100 192 - Radeon R285 193 - Radeon R9 380 194 - Radeon R9 385 195 - Mobile FirePro 196 M7170 197 ``gfx803`` - ``fiji`` ``amdgcn`` dGPU - xnack ROCm - Radeon R9 Nano 198 [off] - Radeon R9 Fury 199 - Radeon R9 FuryX 200 - Radeon Pro Duo 201 - FirePro S9300x2 202 - Radeon Instinct MI8 203 \ - ``polaris10`` ``amdgcn`` dGPU - xnack ROCm - Radeon RX 470 204 [off] - Radeon RX 480 205 - Radeon Instinct MI6 206 \ - ``polaris11`` ``amdgcn`` dGPU - xnack ROCm - Radeon RX 460 207 [off] 208 ``gfx810`` - ``stoney`` ``amdgcn`` APU - xnack 209 [on] 210 **GCN GFX9** [AMD-GCN-GFX9]_ 211 ----------------------------------------------------------------------------------------------- 212 ``gfx900`` ``amdgcn`` dGPU - xnack ROCm - Radeon Vega 213 [off] Frontier Edition 214 - Radeon RX Vega 56 215 - Radeon RX Vega 64 216 - Radeon RX Vega 64 217 Liquid 218 - Radeon Instinct MI25 219 ``gfx902`` ``amdgcn`` APU - xnack - Ryzen 3 2200G 220 [on] - Ryzen 5 2400G 221 ``gfx904`` ``amdgcn`` dGPU - xnack *TBA* 222 [off] 223 .. TODO:: 224 Add product 225 names. 226 ``gfx906`` ``amdgcn`` dGPU - xnack - Radeon Instinct MI50 227 [off] - Radeon Instinct MI60 228 ``gfx908`` ``amdgcn`` dGPU - xnack *TBA* 229 [off] 230 sram-ecc 231 [on] 232 ``gfx909`` ``amdgcn`` APU - xnack *TBA* (Raven Ridge 2) 233 [on] 234 .. TODO:: 235 Add product 236 names. 237 **GCN GFX10** [AMD-GCN-GFX10]_ 238 ----------------------------------------------------------------------------------------------- 239 ``gfx1010`` ``amdgcn`` dGPU - xnack *TBA* 240 [off] 241 - wavefrontsize64 242 [off] 243 - cumode 244 [off] 245 .. TODO:: 246 Add product 247 names. 248 ``gfx1011`` ``amdgcn`` dGPU - xnack *TBA* 249 [off] 250 - wavefrontsize64 251 [off] 252 - cumode 253 [off] 254 .. TODO:: 255 Add product 256 names. 257 ``gfx1012`` ``amdgcn`` dGPU - xnack *TBA* 258 [off] 259 - wavefrontsize64 260 [off] 261 - cumode 262 [off] 263 .. TODO:: 264 Add product 265 names. 266 =========== =============== ============ ===== ================= ======= ====================== 267 268.. _amdgpu-target-features: 269 270Target Features 271--------------- 272 273Target features control how code is generated to support certain 274processor specific features. Not all target features are supported by 275all processors. The runtime must ensure that the features supported by 276the device used to execute the code match the features enabled when 277generating the code. A mismatch of features may result in incorrect 278execution, or a reduction in performance. 279 280The target features supported by each processor, and the default value 281used if not specified explicitly, is listed in 282:ref:`amdgpu-processor-table`. 283 284Use the ``clang -m[no-]<TargetFeature>`` option to specify the AMDGPU 285target features. 286 287For example: 288 289``-mxnack`` 290 Enable the ``xnack`` feature. 291``-mno-xnack`` 292 Disable the ``xnack`` feature. 293 294 .. table:: AMDGPU Target Features 295 :name: amdgpu-target-feature-table 296 297 ====================== ================================================== 298 Target Feature Description 299 ====================== ================================================== 300 -m[no-]xnack Enable/disable generating code that has 301 memory clauses that are compatible with 302 having XNACK replay enabled. 303 304 This is used for demand paging and page 305 migration. If XNACK replay is enabled in 306 the device, then if a page fault occurs 307 the code may execute incorrectly if the 308 ``xnack`` feature is not enabled. Executing 309 code that has the feature enabled on a 310 device that does not have XNACK replay 311 enabled will execute correctly but may 312 be less performant than code with the 313 feature disabled. 314 315 -m[no-]sram-ecc Enable/disable generating code that assumes SRAM 316 ECC is enabled/disabled. 317 318 -m[no-]wavefrontsize64 Control the default wavefront size used when 319 generating code for kernels. When disabled 320 native wavefront size 32 is used, when enabled 321 wavefront size 64 is used. 322 323 -m[no-]cumode Control the default wavefront execution mode used 324 when generating code for kernels. When disabled 325 native WGP wavefront execution mode is used, 326 when enabled CU wavefront execution mode is used 327 (see :ref:`amdgpu-amdhsa-memory-model`). 328 ====================== ================================================== 329 330.. _amdgpu-address-spaces: 331 332Address Spaces 333-------------- 334 335The AMDGPU architecture supports a number of memory address spaces. The address 336space names use the OpenCL standard names, with some additions. 337 338The AMDGPU address spaces correspond to target architecture specific LLVM 339address space numbers used in LLVM IR. 340 341The AMDGPU address spaces are described in 342:ref:`amdgpu-address-spaces-table`. Only 64-bit process address spaces are 343supported for the ``amdgcn`` target. 344 345 .. table:: AMDGPU Address Spaces 346 :name: amdgpu-address-spaces-table 347 348 ================================= =============== =========== ================ ======= ============================ 349 .. 64-Bit Process Address Space 350 --------------------------------- --------------- ----------- ---------------- ------------------------------------ 351 Address Space Name LLVM IR Address HSA Segment Hardware Address NULL Value 352 Space Number Name Name Size 353 ================================= =============== =========== ================ ======= ============================ 354 Generic 0 flat flat 64 0x0000000000000000 355 Global 1 global global 64 0x0000000000000000 356 Region 2 N/A GDS 32 *not implemented for AMDHSA* 357 Local 3 group LDS 32 0xFFFFFFFF 358 Constant 4 constant *same as global* 64 0x0000000000000000 359 Private 5 private scratch 32 0x00000000 360 Constant 32-bit 6 *TODO* 361 Buffer Fat Pointer (experimental) 7 *TODO* 362 ================================= =============== =========== ================ ======= ============================ 363 364**Generic** 365 The generic address space uses the hardware flat address support available in 366 GFX7-GFX10. This uses two fixed ranges of virtual addresses (the private and 367 local apertures), that are outside the range of addressable global memory, to 368 map from a flat address to a private or local address. 369 370 FLAT instructions can take a flat address and access global, private 371 (scratch), and group (LDS) memory depending on if the address is within one 372 of the aperture ranges. Flat access to scratch requires hardware aperture 373 setup and setup in the kernel prologue (see 374 :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`). Flat access to LDS requires 375 hardware aperture setup and M0 (GFX7-GFX8) register setup (see 376 :ref:`amdgpu-amdhsa-kernel-prolog-m0`). 377 378 To convert between a private or group address space address (termed a segment 379 address) and a flat address the base address of the corresponding aperture 380 can be used. For GFX7-GFX8 these are available in the 381 :ref:`amdgpu-amdhsa-hsa-aql-queue` the address of which can be obtained with 382 Queue Ptr SGPR (see :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). For 383 GFX9-GFX10 the aperture base addresses are directly available as inline 384 constant registers ``SRC_SHARED_BASE/LIMIT`` and ``SRC_PRIVATE_BASE/LIMIT``. 385 In 64-bit address mode the aperture sizes are 2^32 bytes and the base is 386 aligned to 2^32 which makes it easier to convert from flat to segment or 387 segment to flat. 388 389 A global address space address has the same value when used as a flat address 390 so no conversion is needed. 391 392**Global and Constant** 393 The global and constant address spaces both use global virtual addresses, 394 which are the same virtual address space used by the CPU. However, some 395 virtual addresses may only be accessible to the CPU, some only accessible 396 by the GPU, and some by both. 397 398 Using the constant address space indicates that the data will not change 399 during the execution of the kernel. This allows scalar read instructions to 400 be used. The vector and scalar L1 caches are invalidated of volatile data 401 before each kernel dispatch execution to allow constant memory to change 402 values between kernel dispatches. 403 404**Region** 405 The region address space uses the hardware Global Data Store (GDS). All 406 wavefronts executing on the same device will access the same memory for any 407 given region address. However, the same region address accessed by wavefronts 408 executing on different devices will access different memory. It is higher 409 performance than global memory. It is allocated by the runtime. The data 410 store (DS) instructions can be used to access it. 411 412**Local** 413 The local address space uses the hardware Local Data Store (LDS) which is 414 automatically allocated when the hardware creates the wavefronts of a 415 work-group, and freed when all the wavefronts of a work-group have 416 terminated. All wavefronts belonging to the same work-group will access the 417 same memory for any given local address. However, the same local address 418 accessed by wavefronts belonging to different work-groups will access 419 different memory. It is higher performance than global memory. The data store 420 (DS) instructions can be used to access it. 421 422**Private** 423 The private address space uses the hardware scratch memory support which 424 automatically allocates memory when it creates a wavefront and frees it when 425 a wavefronts terminates. The memory accessed by a lane of a wavefront for any 426 given private address will be different to the memory accessed by another lane 427 of the same or different wavefront for the same private address. 428 429 If a kernel dispatch uses scratch, then the hardware allocates memory from a 430 pool of backing memory allocated by the runtime for each wavefront. The lanes 431 of the wavefront access this using dword (4 byte) interleaving. The mapping 432 used from private address to backing memory address is: 433 434 ``wavefront-scratch-base + 435 ((private-address / 4) * wavefront-size * 4) + 436 (wavefront-lane-id * 4) + (private-address % 4)`` 437 438 If each lane of a wavefront accesses the same private address, the 439 interleaving results in adjacent dwords being accessed and hence requires 440 fewer cache lines to be fetched. 441 442 There are different ways that the wavefront scratch base address is 443 determined by a wavefront (see 444 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 445 446 Scratch memory can be accessed in an interleaved manner using buffer 447 instructions with the scratch buffer descriptor and per wavefront scratch 448 offset, by the scratch instructions, or by flat instructions. Multi-dword 449 access is not supported except by flat and scratch instructions in 450 GFX9-GFX10. 451 452**Constant 32-bit** 453 *TODO* 454 455**Buffer Fat Pointer** 456 The buffer fat pointer is an experimental address space that is currently 457 unsupported in the backend. It exposes a non-integral pointer that is in 458 the future intended to support the modelling of 128-bit buffer descriptors 459 plus a 32-bit offset into the buffer (in total encapsulating a 160-bit 460 *pointer*), allowing normal LLVM load/store/atomic operations to be used to 461 model the buffer descriptors used heavily in graphics workloads targeting 462 the backend. 463 464.. _amdgpu-memory-scopes: 465 466Memory Scopes 467------------- 468 469This section provides LLVM memory synchronization scopes supported by the AMDGPU 470backend memory model when the target triple OS is ``amdhsa`` (see 471:ref:`amdgpu-amdhsa-memory-model` and :ref:`amdgpu-target-triples`). 472 473The memory model supported is based on the HSA memory model [HSA]_ which is 474based in turn on HRF-indirect with scope inclusion [HRF]_. The happens-before 475relation is transitive over the synchronizes-with relation independent of scope 476and synchronizes-with allows the memory scope instances to be inclusive (see 477table :ref:`amdgpu-amdhsa-llvm-sync-scopes-table`). 478 479This is different to the OpenCL [OpenCL]_ memory model which does not have scope 480inclusion and requires the memory scopes to exactly match. However, this 481is conservatively correct for OpenCL. 482 483 .. table:: AMDHSA LLVM Sync Scopes 484 :name: amdgpu-amdhsa-llvm-sync-scopes-table 485 486 ======================= =================================================== 487 LLVM Sync Scope Description 488 ======================= =================================================== 489 *none* The default: ``system``. 490 491 Synchronizes with, and participates in modification 492 and seq_cst total orderings with, other operations 493 (except image operations) for all address spaces 494 (except private, or generic that accesses private) 495 provided the other operation's sync scope is: 496 497 - ``system``. 498 - ``agent`` and executed by a thread on the same 499 agent. 500 - ``workgroup`` and executed by a thread in the 501 same work-group. 502 - ``wavefront`` and executed by a thread in the 503 same wavefront. 504 505 ``agent`` Synchronizes with, and participates in modification 506 and seq_cst total orderings with, other operations 507 (except image operations) for all address spaces 508 (except private, or generic that accesses private) 509 provided the other operation's sync scope is: 510 511 - ``system`` or ``agent`` and executed by a thread 512 on the same agent. 513 - ``workgroup`` and executed by a thread in the 514 same work-group. 515 - ``wavefront`` and executed by a thread in the 516 same wavefront. 517 518 ``workgroup`` Synchronizes with, and participates in modification 519 and seq_cst total orderings with, other operations 520 (except image operations) for all address spaces 521 (except private, or generic that accesses private) 522 provided the other operation's sync scope is: 523 524 - ``system``, ``agent`` or ``workgroup`` and 525 executed by a thread in the same work-group. 526 - ``wavefront`` and executed by a thread in the 527 same wavefront. 528 529 ``wavefront`` Synchronizes with, and participates in modification 530 and seq_cst total orderings with, other operations 531 (except image operations) for all address spaces 532 (except private, or generic that accesses private) 533 provided the other operation's sync scope is: 534 535 - ``system``, ``agent``, ``workgroup`` or 536 ``wavefront`` and executed by a thread in the 537 same wavefront. 538 539 ``singlethread`` Only synchronizes with and participates in 540 modification and seq_cst total orderings with, 541 other operations (except image operations) running 542 in the same thread for all address spaces (for 543 example, in signal handlers). 544 545 ``one-as`` Same as ``system`` but only synchronizes with other 546 operations within the same address space. 547 548 ``agent-one-as`` Same as ``agent`` but only synchronizes with other 549 operations within the same address space. 550 551 ``workgroup-one-as`` Same as ``workgroup`` but only synchronizes with 552 other operations within the same address space. 553 554 ``wavefront-one-as`` Same as ``wavefront`` but only synchronizes with 555 other operations within the same address space. 556 557 ``singlethread-one-as`` Same as ``singlethread`` but only synchronizes with 558 other operations within the same address space. 559 ======================= =================================================== 560 561LLVM IR Intrinsics 562------------------ 563 564The AMDGPU backend implements the following LLVM IR intrinsics. 565 566*This section is WIP.* 567 568.. TODO:: 569 570 List AMDGPU intrinsics. 571 572LLVM IR Attributes 573------------------ 574 575The AMDGPU backend supports the following LLVM IR attributes. 576 577 .. table:: AMDGPU LLVM IR Attributes 578 :name: amdgpu-llvm-ir-attributes-table 579 580 ======================================= ========================================================== 581 LLVM Attribute Description 582 ======================================= ========================================================== 583 "amdgpu-flat-work-group-size"="min,max" Specify the minimum and maximum flat work group sizes that 584 will be specified when the kernel is dispatched. Generated 585 by the ``amdgpu_flat_work_group_size`` CLANG attribute [CLANG-ATTR]_. 586 "amdgpu-implicitarg-num-bytes"="n" Number of kernel argument bytes to add to the kernel 587 argument block size for the implicit arguments. This 588 varies by OS and language (for OpenCL see 589 :ref:`opencl-kernel-implicit-arguments-appended-for-amdhsa-os-table`). 590 "amdgpu-num-sgpr"="n" Specifies the number of SGPRs to use. Generated by 591 the ``amdgpu_num_sgpr`` CLANG attribute [CLANG-ATTR]_. 592 "amdgpu-num-vgpr"="n" Specifies the number of VGPRs to use. Generated by the 593 ``amdgpu_num_vgpr`` CLANG attribute [CLANG-ATTR]_. 594 "amdgpu-waves-per-eu"="m,n" Specify the minimum and maximum number of waves per 595 execution unit. Generated by the ``amdgpu_waves_per_eu`` 596 CLANG attribute [CLANG-ATTR]_. 597 "amdgpu-ieee" true/false. Specify whether the function expects the IEEE field of the 598 mode register to be set on entry. Overrides the default for 599 the calling convention. 600 "amdgpu-dx10-clamp" true/false. Specify whether the function expects the DX10_CLAMP field of 601 the mode register to be set on entry. Overrides the default 602 for the calling convention. 603 ======================================= ========================================================== 604 605.. _amdgpu-elf-code-object: 606 607ELF Code Object 608=============== 609 610The AMDGPU backend generates a standard ELF [ELF]_ relocatable code object that 611can be linked by ``lld`` to produce a standard ELF shared code object which can 612be loaded and executed on an AMDGPU target. 613 614.. _amdgpu-elf-header: 615 616Header 617------ 618 619The AMDGPU backend uses the following ELF header: 620 621 .. table:: AMDGPU ELF Header 622 :name: amdgpu-elf-header-table 623 624 ========================== =============================== 625 Field Value 626 ========================== =============================== 627 ``e_ident[EI_CLASS]`` ``ELFCLASS64`` 628 ``e_ident[EI_DATA]`` ``ELFDATA2LSB`` 629 ``e_ident[EI_OSABI]`` - ``ELFOSABI_NONE`` 630 - ``ELFOSABI_AMDGPU_HSA`` 631 - ``ELFOSABI_AMDGPU_PAL`` 632 - ``ELFOSABI_AMDGPU_MESA3D`` 633 ``e_ident[EI_ABIVERSION]`` - ``ELFABIVERSION_AMDGPU_HSA`` 634 - ``ELFABIVERSION_AMDGPU_PAL`` 635 - ``ELFABIVERSION_AMDGPU_MESA3D`` 636 ``e_type`` - ``ET_REL`` 637 - ``ET_DYN`` 638 ``e_machine`` ``EM_AMDGPU`` 639 ``e_entry`` 0 640 ``e_flags`` See :ref:`amdgpu-elf-header-e_flags-table` 641 ========================== =============================== 642 643.. 644 645 .. table:: AMDGPU ELF Header Enumeration Values 646 :name: amdgpu-elf-header-enumeration-values-table 647 648 =============================== ===== 649 Name Value 650 =============================== ===== 651 ``EM_AMDGPU`` 224 652 ``ELFOSABI_NONE`` 0 653 ``ELFOSABI_AMDGPU_HSA`` 64 654 ``ELFOSABI_AMDGPU_PAL`` 65 655 ``ELFOSABI_AMDGPU_MESA3D`` 66 656 ``ELFABIVERSION_AMDGPU_HSA`` 1 657 ``ELFABIVERSION_AMDGPU_PAL`` 0 658 ``ELFABIVERSION_AMDGPU_MESA3D`` 0 659 =============================== ===== 660 661``e_ident[EI_CLASS]`` 662 The ELF class is: 663 664 * ``ELFCLASS32`` for ``r600`` architecture. 665 666 * ``ELFCLASS64`` for ``amdgcn`` architecture which only supports 64-bit 667 process address space applications. 668 669``e_ident[EI_DATA]`` 670 All AMDGPU targets use ``ELFDATA2LSB`` for little-endian byte ordering. 671 672``e_ident[EI_OSABI]`` 673 One of the following AMDGPU target architecture specific OS ABIs 674 (see :ref:`amdgpu-os-table`): 675 676 * ``ELFOSABI_NONE`` for *unknown* OS. 677 678 * ``ELFOSABI_AMDGPU_HSA`` for ``amdhsa`` OS. 679 680 * ``ELFOSABI_AMDGPU_PAL`` for ``amdpal`` OS. 681 682 * ``ELFOSABI_AMDGPU_MESA3D`` for ``mesa3D`` OS. 683 684``e_ident[EI_ABIVERSION]`` 685 The ABI version of the AMDGPU target architecture specific OS ABI to which the code 686 object conforms: 687 688 * ``ELFABIVERSION_AMDGPU_HSA`` is used to specify the version of AMD HSA 689 runtime ABI. 690 691 * ``ELFABIVERSION_AMDGPU_PAL`` is used to specify the version of AMD PAL 692 runtime ABI. 693 694 * ``ELFABIVERSION_AMDGPU_MESA3D`` is used to specify the version of AMD MESA 695 3D runtime ABI. 696 697``e_type`` 698 Can be one of the following values: 699 700 701 ``ET_REL`` 702 The type produced by the AMDGPU backend compiler as it is relocatable code 703 object. 704 705 ``ET_DYN`` 706 The type produced by the linker as it is a shared code object. 707 708 The AMD HSA runtime loader requires a ``ET_DYN`` code object. 709 710``e_machine`` 711 The value ``EM_AMDGPU`` is used for the machine for all processors supported 712 by the ``r600`` and ``amdgcn`` architectures (see 713 :ref:`amdgpu-processor-table`). The specific processor is specified in the 714 ``EF_AMDGPU_MACH`` bit field of the ``e_flags`` (see 715 :ref:`amdgpu-elf-header-e_flags-table`). 716 717``e_entry`` 718 The entry point is 0 as the entry points for individual kernels must be 719 selected in order to invoke them through AQL packets. 720 721``e_flags`` 722 The AMDGPU backend uses the following ELF header flags: 723 724 .. table:: AMDGPU ELF Header ``e_flags`` 725 :name: amdgpu-elf-header-e_flags-table 726 727 ================================= ========== ============================= 728 Name Value Description 729 ================================= ========== ============================= 730 **AMDGPU Processor Flag** See :ref:`amdgpu-processor-table`. 731 -------------------------------------------- ----------------------------- 732 ``EF_AMDGPU_MACH`` 0x000000ff AMDGPU processor selection 733 mask for 734 ``EF_AMDGPU_MACH_xxx`` values 735 defined in 736 :ref:`amdgpu-ef-amdgpu-mach-table`. 737 ``EF_AMDGPU_XNACK`` 0x00000100 Indicates if the ``xnack`` 738 target feature is 739 enabled for all code 740 contained in the code object. 741 If the processor 742 does not support the 743 ``xnack`` target 744 feature then must 745 be 0. 746 See 747 :ref:`amdgpu-target-features`. 748 ``EF_AMDGPU_SRAM_ECC`` 0x00000200 Indicates if the ``sram-ecc`` 749 target feature is 750 enabled for all code 751 contained in the code object. 752 If the processor 753 does not support the 754 ``sram-ecc`` target 755 feature then must 756 be 0. 757 See 758 :ref:`amdgpu-target-features`. 759 ================================= ========== ============================= 760 761 .. table:: AMDGPU ``EF_AMDGPU_MACH`` Values 762 :name: amdgpu-ef-amdgpu-mach-table 763 764 ================================= ========== ============================= 765 Name Value Description (see 766 :ref:`amdgpu-processor-table`) 767 ================================= ========== ============================= 768 ``EF_AMDGPU_MACH_NONE`` 0x000 *not specified* 769 ``EF_AMDGPU_MACH_R600_R600`` 0x001 ``r600`` 770 ``EF_AMDGPU_MACH_R600_R630`` 0x002 ``r630`` 771 ``EF_AMDGPU_MACH_R600_RS880`` 0x003 ``rs880`` 772 ``EF_AMDGPU_MACH_R600_RV670`` 0x004 ``rv670`` 773 ``EF_AMDGPU_MACH_R600_RV710`` 0x005 ``rv710`` 774 ``EF_AMDGPU_MACH_R600_RV730`` 0x006 ``rv730`` 775 ``EF_AMDGPU_MACH_R600_RV770`` 0x007 ``rv770`` 776 ``EF_AMDGPU_MACH_R600_CEDAR`` 0x008 ``cedar`` 777 ``EF_AMDGPU_MACH_R600_CYPRESS`` 0x009 ``cypress`` 778 ``EF_AMDGPU_MACH_R600_JUNIPER`` 0x00a ``juniper`` 779 ``EF_AMDGPU_MACH_R600_REDWOOD`` 0x00b ``redwood`` 780 ``EF_AMDGPU_MACH_R600_SUMO`` 0x00c ``sumo`` 781 ``EF_AMDGPU_MACH_R600_BARTS`` 0x00d ``barts`` 782 ``EF_AMDGPU_MACH_R600_CAICOS`` 0x00e ``caicos`` 783 ``EF_AMDGPU_MACH_R600_CAYMAN`` 0x00f ``cayman`` 784 ``EF_AMDGPU_MACH_R600_TURKS`` 0x010 ``turks`` 785 *reserved* 0x011 - Reserved for ``r600`` 786 0x01f architecture processors. 787 ``EF_AMDGPU_MACH_AMDGCN_GFX600`` 0x020 ``gfx600`` 788 ``EF_AMDGPU_MACH_AMDGCN_GFX601`` 0x021 ``gfx601`` 789 ``EF_AMDGPU_MACH_AMDGCN_GFX700`` 0x022 ``gfx700`` 790 ``EF_AMDGPU_MACH_AMDGCN_GFX701`` 0x023 ``gfx701`` 791 ``EF_AMDGPU_MACH_AMDGCN_GFX702`` 0x024 ``gfx702`` 792 ``EF_AMDGPU_MACH_AMDGCN_GFX703`` 0x025 ``gfx703`` 793 ``EF_AMDGPU_MACH_AMDGCN_GFX704`` 0x026 ``gfx704`` 794 *reserved* 0x027 Reserved. 795 ``EF_AMDGPU_MACH_AMDGCN_GFX801`` 0x028 ``gfx801`` 796 ``EF_AMDGPU_MACH_AMDGCN_GFX802`` 0x029 ``gfx802`` 797 ``EF_AMDGPU_MACH_AMDGCN_GFX803`` 0x02a ``gfx803`` 798 ``EF_AMDGPU_MACH_AMDGCN_GFX810`` 0x02b ``gfx810`` 799 ``EF_AMDGPU_MACH_AMDGCN_GFX900`` 0x02c ``gfx900`` 800 ``EF_AMDGPU_MACH_AMDGCN_GFX902`` 0x02d ``gfx902`` 801 ``EF_AMDGPU_MACH_AMDGCN_GFX904`` 0x02e ``gfx904`` 802 ``EF_AMDGPU_MACH_AMDGCN_GFX906`` 0x02f ``gfx906`` 803 ``EF_AMDGPU_MACH_AMDGCN_GFX908`` 0x030 ``gfx908`` 804 ``EF_AMDGPU_MACH_AMDGCN_GFX909`` 0x031 ``gfx909`` 805 *reserved* 0x032 Reserved. 806 ``EF_AMDGPU_MACH_AMDGCN_GFX1010`` 0x033 ``gfx1010`` 807 ``EF_AMDGPU_MACH_AMDGCN_GFX1011`` 0x034 ``gfx1011`` 808 ``EF_AMDGPU_MACH_AMDGCN_GFX1012`` 0x035 ``gfx1012`` 809 ================================= ========== ============================= 810 811Sections 812-------- 813 814An AMDGPU target ELF code object has the standard ELF sections which include: 815 816 .. table:: AMDGPU ELF Sections 817 :name: amdgpu-elf-sections-table 818 819 ================== ================ ================================= 820 Name Type Attributes 821 ================== ================ ================================= 822 ``.bss`` ``SHT_NOBITS`` ``SHF_ALLOC`` + ``SHF_WRITE`` 823 ``.data`` ``SHT_PROGBITS`` ``SHF_ALLOC`` + ``SHF_WRITE`` 824 ``.debug_``\ *\** ``SHT_PROGBITS`` *none* 825 ``.dynamic`` ``SHT_DYNAMIC`` ``SHF_ALLOC`` 826 ``.dynstr`` ``SHT_PROGBITS`` ``SHF_ALLOC`` 827 ``.dynsym`` ``SHT_PROGBITS`` ``SHF_ALLOC`` 828 ``.got`` ``SHT_PROGBITS`` ``SHF_ALLOC`` + ``SHF_WRITE`` 829 ``.hash`` ``SHT_HASH`` ``SHF_ALLOC`` 830 ``.note`` ``SHT_NOTE`` *none* 831 ``.rela``\ *name* ``SHT_RELA`` *none* 832 ``.rela.dyn`` ``SHT_RELA`` *none* 833 ``.rodata`` ``SHT_PROGBITS`` ``SHF_ALLOC`` 834 ``.shstrtab`` ``SHT_STRTAB`` *none* 835 ``.strtab`` ``SHT_STRTAB`` *none* 836 ``.symtab`` ``SHT_SYMTAB`` *none* 837 ``.text`` ``SHT_PROGBITS`` ``SHF_ALLOC`` + ``SHF_EXECINSTR`` 838 ================== ================ ================================= 839 840These sections have their standard meanings (see [ELF]_) and are only generated 841if needed. 842 843``.debug``\ *\** 844 The standard DWARF sections. See :ref:`amdgpu-dwarf-debug-information` for 845 information on the DWARF produced by the AMDGPU backend. 846 847``.dynamic``, ``.dynstr``, ``.dynsym``, ``.hash`` 848 The standard sections used by a dynamic loader. 849 850``.note`` 851 See :ref:`amdgpu-note-records` for the note records supported by the AMDGPU 852 backend. 853 854``.rela``\ *name*, ``.rela.dyn`` 855 For relocatable code objects, *name* is the name of the section that the 856 relocation records apply. For example, ``.rela.text`` is the section name for 857 relocation records associated with the ``.text`` section. 858 859 For linked shared code objects, ``.rela.dyn`` contains all the relocation 860 records from each of the relocatable code object's ``.rela``\ *name* sections. 861 862 See :ref:`amdgpu-relocation-records` for the relocation records supported by 863 the AMDGPU backend. 864 865``.text`` 866 The executable machine code for the kernels and functions they call. Generated 867 as position independent code. See :ref:`amdgpu-code-conventions` for 868 information on conventions used in the isa generation. 869 870.. _amdgpu-note-records: 871 872Note Records 873------------ 874 875The AMDGPU backend code object contains ELF note records in the ``.note`` 876section. The set of generated notes and their semantics depend on the code 877object version; see :ref:`amdgpu-note-records-v2` and 878:ref:`amdgpu-note-records-v3`. 879 880As required by ``ELFCLASS32`` and ``ELFCLASS64``, minimal zero-byte padding 881must be generated after the ``name`` field to ensure the ``desc`` field is 4 882byte aligned. In addition, minimal zero-byte padding must be generated to 883ensure the ``desc`` field size is a multiple of 4 bytes. The ``sh_addralign`` 884field of the ``.note`` section must be at least 4 to indicate at least 8 byte 885alignment. 886 887.. _amdgpu-note-records-v2: 888 889Code Object V2 Note Records (-mattr=-code-object-v3) 890~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 891 892.. warning:: Code Object V2 is not the default code object version emitted by 893 this version of LLVM. For a description of the notes generated with the 894 default configuration (Code Object V3) see :ref:`amdgpu-note-records-v3`. 895 896The AMDGPU backend code object uses the following ELF note record in the 897``.note`` section when compiling for Code Object V2 (-mattr=-code-object-v3). 898 899Additional note records may be present, but any which are not documented here 900are deprecated and should not be used. 901 902 .. table:: AMDGPU Code Object V2 ELF Note Records 903 :name: amdgpu-elf-note-records-table-v2 904 905 ===== ============================== ====================================== 906 Name Type Description 907 ===== ============================== ====================================== 908 "AMD" ``NT_AMD_AMDGPU_HSA_METADATA`` <metadata null terminated string> 909 ===== ============================== ====================================== 910 911.. 912 913 .. table:: AMDGPU Code Object V2 ELF Note Record Enumeration Values 914 :name: amdgpu-elf-note-record-enumeration-values-table-v2 915 916 ============================== ===== 917 Name Value 918 ============================== ===== 919 *reserved* 0-9 920 ``NT_AMD_AMDGPU_HSA_METADATA`` 10 921 *reserved* 11 922 ============================== ===== 923 924``NT_AMD_AMDGPU_HSA_METADATA`` 925 Specifies extensible metadata associated with the code objects executed on HSA 926 [HSA]_ compatible runtimes such as AMD's ROCm [AMD-ROCm]_. It is required when 927 the target triple OS is ``amdhsa`` (see :ref:`amdgpu-target-triples`). See 928 :ref:`amdgpu-amdhsa-code-object-metadata-v2` for the syntax of the code 929 object metadata string. 930 931.. _amdgpu-note-records-v3: 932 933Code Object V3 Note Records (-mattr=+code-object-v3) 934~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 935 936The AMDGPU backend code object uses the following ELF note record in the 937``.note`` section when compiling for Code Object V3 (-mattr=+code-object-v3). 938 939Additional note records may be present, but any which are not documented here 940are deprecated and should not be used. 941 942 .. table:: AMDGPU Code Object V3 ELF Note Records 943 :name: amdgpu-elf-note-records-table-v3 944 945 ======== ============================== ====================================== 946 Name Type Description 947 ======== ============================== ====================================== 948 "AMDGPU" ``NT_AMDGPU_METADATA`` Metadata in Message Pack [MsgPack]_ 949 binary format. 950 ======== ============================== ====================================== 951 952.. 953 954 .. table:: AMDGPU Code Object V3 ELF Note Record Enumeration Values 955 :name: amdgpu-elf-note-record-enumeration-values-table-v3 956 957 ============================== ===== 958 Name Value 959 ============================== ===== 960 *reserved* 0-31 961 ``NT_AMDGPU_METADATA`` 32 962 ============================== ===== 963 964``NT_AMDGPU_METADATA`` 965 Specifies extensible metadata associated with an AMDGPU code 966 object. It is encoded as a map in the Message Pack [MsgPack]_ binary 967 data format. See :ref:`amdgpu-amdhsa-code-object-metadata-v3` for the 968 map keys defined for the ``amdhsa`` OS. 969 970.. _amdgpu-symbols: 971 972Symbols 973------- 974 975Symbols include the following: 976 977 .. table:: AMDGPU ELF Symbols 978 :name: amdgpu-elf-symbols-table 979 980 ===================== ================== ================ ================== 981 Name Type Section Description 982 ===================== ================== ================ ================== 983 *link-name* ``STT_OBJECT`` - ``.data`` Global variable 984 - ``.rodata`` 985 - ``.bss`` 986 *link-name*\ ``.kd`` ``STT_OBJECT`` - ``.rodata`` Kernel descriptor 987 *link-name* ``STT_FUNC`` - ``.text`` Kernel entry point 988 *link-name* ``STT_OBJECT`` - SHN_AMDGPU_LDS Global variable in LDS 989 ===================== ================== ================ ================== 990 991Global variable 992 Global variables both used and defined by the compilation unit. 993 994 If the symbol is defined in the compilation unit then it is allocated in the 995 appropriate section according to if it has initialized data or is readonly. 996 997 If the symbol is external then its section is ``STN_UNDEF`` and the loader 998 will resolve relocations using the definition provided by another code object 999 or explicitly defined by the runtime. 1000 1001 If the symbol resides in local/group memory (LDS) then its section is the 1002 special processor specific section name ``SHN_AMDGPU_LDS``, and the 1003 ``st_value`` field describes alignment requirements as it does for common 1004 symbols. 1005 1006 .. TODO:: 1007 1008 Add description of linked shared object symbols. Seems undefined symbols 1009 are marked as STT_NOTYPE. 1010 1011Kernel descriptor 1012 Every HSA kernel has an associated kernel descriptor. It is the address of the 1013 kernel descriptor that is used in the AQL dispatch packet used to invoke the 1014 kernel, not the kernel entry point. The layout of the HSA kernel descriptor is 1015 defined in :ref:`amdgpu-amdhsa-kernel-descriptor`. 1016 1017Kernel entry point 1018 Every HSA kernel also has a symbol for its machine code entry point. 1019 1020.. _amdgpu-relocation-records: 1021 1022Relocation Records 1023------------------ 1024 1025AMDGPU backend generates ``Elf64_Rela`` relocation records. Supported 1026relocatable fields are: 1027 1028``word32`` 1029 This specifies a 32-bit field occupying 4 bytes with arbitrary byte 1030 alignment. These values use the same byte order as other word values in the 1031 AMDGPU architecture. 1032 1033``word64`` 1034 This specifies a 64-bit field occupying 8 bytes with arbitrary byte 1035 alignment. These values use the same byte order as other word values in the 1036 AMDGPU architecture. 1037 1038Following notations are used for specifying relocation calculations: 1039 1040**A** 1041 Represents the addend used to compute the value of the relocatable field. 1042 1043**G** 1044 Represents the offset into the global offset table at which the relocation 1045 entry's symbol will reside during execution. 1046 1047**GOT** 1048 Represents the address of the global offset table. 1049 1050**P** 1051 Represents the place (section offset for ``et_rel`` or address for ``et_dyn``) 1052 of the storage unit being relocated (computed using ``r_offset``). 1053 1054**S** 1055 Represents the value of the symbol whose index resides in the relocation 1056 entry. Relocations not using this must specify a symbol index of 1057 ``STN_UNDEF``. 1058 1059**B** 1060 Represents the base address of a loaded executable or shared object which is 1061 the difference between the ELF address and the actual load address. 1062 Relocations using this are only valid in executable or shared objects. 1063 1064The following relocation types are supported: 1065 1066 .. table:: AMDGPU ELF Relocation Records 1067 :name: amdgpu-elf-relocation-records-table 1068 1069 ========================== ======= ===== ========== ============================== 1070 Relocation Type Kind Value Field Calculation 1071 ========================== ======= ===== ========== ============================== 1072 ``R_AMDGPU_NONE`` 0 *none* *none* 1073 ``R_AMDGPU_ABS32_LO`` Static, 1 ``word32`` (S + A) & 0xFFFFFFFF 1074 Dynamic 1075 ``R_AMDGPU_ABS32_HI`` Static, 2 ``word32`` (S + A) >> 32 1076 Dynamic 1077 ``R_AMDGPU_ABS64`` Static, 3 ``word64`` S + A 1078 Dynamic 1079 ``R_AMDGPU_REL32`` Static 4 ``word32`` S + A - P 1080 ``R_AMDGPU_REL64`` Static 5 ``word64`` S + A - P 1081 ``R_AMDGPU_ABS32`` Static, 6 ``word32`` S + A 1082 Dynamic 1083 ``R_AMDGPU_GOTPCREL`` Static 7 ``word32`` G + GOT + A - P 1084 ``R_AMDGPU_GOTPCREL32_LO`` Static 8 ``word32`` (G + GOT + A - P) & 0xFFFFFFFF 1085 ``R_AMDGPU_GOTPCREL32_HI`` Static 9 ``word32`` (G + GOT + A - P) >> 32 1086 ``R_AMDGPU_REL32_LO`` Static 10 ``word32`` (S + A - P) & 0xFFFFFFFF 1087 ``R_AMDGPU_REL32_HI`` Static 11 ``word32`` (S + A - P) >> 32 1088 *reserved* 12 1089 ``R_AMDGPU_RELATIVE64`` Dynamic 13 ``word64`` B + A 1090 ========================== ======= ===== ========== ============================== 1091 1092``R_AMDGPU_ABS32_LO`` and ``R_AMDGPU_ABS32_HI`` are only supported by 1093the ``mesa3d`` OS, which does not support ``R_AMDGPU_ABS64``. 1094 1095There is no current OS loader support for 32-bit programs and so 1096``R_AMDGPU_ABS32`` is not used. 1097 1098.. _amdgpu-dwarf-debug-information: 1099 1100DWARF Debug Information 1101======================= 1102 1103.. warning:: 1104 1105 This section describes a **provisional proposal** for AMDGPU DWARF [DWARF]_ 1106 that is not currently fully implemented and is subject to change. 1107 1108AMDGPU generates DWARF [DWARF]_ debugging information ELF sections (see 1109:ref:`amdgpu-elf-code-object`) which contain information that maps the code 1110object executable code and data to the source language constructs. It can be 1111used by tools such as debuggers and profilers. It uses features defined in 1112:doc:`AMDGPUDwarfProposalForHeterogeneousDebugging` that are made available in 1113DWARF Version 4 and DWARF Version 5 as an LLVM vendor extension. 1114 1115This section defines the AMDGPU target architecture specific DWARF mappings. 1116 1117.. _amdgpu-dwarf-register-identifier: 1118 1119Register Identifier 1120------------------- 1121 1122This section defines the AMDGPU target architecture register numbers used in 1123DWARF operation expressions (see DWARF Version 5 section 2.5 and 1124:ref:`amdgpu-dwarf-operation-expressions`) and Call Frame Information 1125instructions (see DWARF Version 5 section 6.4 and 1126:ref:`amdgpu-dwarf-call-frame-information`). 1127 1128A single code object can contain code for kernels that have different wavefront 1129sizes. The vector registers and some scalar registers are based on the wavefront 1130size. AMDGPU defines distinct DWARF registers for each wavefront size. This 1131simplifies the consumer of the DWARF so that each register has a fixed size, 1132rather than being dynamic according to the wavefront size mode. Similarly, 1133distinct DWARF registers are defined for those registers that vary in size 1134according to the process address size. This allows a consumer to treat a 1135specific AMDGPU processor as a single architecture regardless of how it is 1136configured at run time. The compiler explicitly specifies the DWARF registers 1137that match the mode in which the code it is generating will be executed. 1138 1139DWARF registers are encoded as numbers, which are mapped to architecture 1140registers. The mapping for AMDGPU is defined in 1141:ref:`amdgpu-dwarf-register-mapping-table`. All AMDGPU targets use the same 1142mapping. 1143 1144.. table:: AMDGPU DWARF Register Mapping 1145 :name: amdgpu-dwarf-register-mapping-table 1146 1147 ============== ================= ======== ================================== 1148 DWARF Register AMDGPU Register Bit Size Description 1149 ============== ================= ======== ================================== 1150 0 PC_32 32 Program Counter (PC) when 1151 executing in a 32-bit process 1152 address space. Used in the CFI to 1153 describe the PC of the calling 1154 frame. 1155 1 EXEC_MASK_32 32 Execution Mask Register when 1156 executing in wavefront 32 mode. 1157 2-15 *Reserved* 1158 16 PC_64 64 Program Counter (PC) when 1159 executing in a 64-bit process 1160 address space. Used in the CFI to 1161 describe the PC of the calling 1162 frame. 1163 17 EXEC_MASK_64 64 Execution Mask Register when 1164 executing in wavefront 64 mode. 1165 18-31 *Reserved* 1166 32-95 SGPR0-SGPR63 32 Scalar General Purpose 1167 Registers. 1168 96-127 *Reserved* 1169 128-511 *Reserved* 1170 512-1023 *Reserved* 1171 1024-1087 *Reserved* 1172 1088-1129 SGPR64-SGPR105 32 Scalar General Purpose Registers 1173 1130-1535 *Reserved* 1174 1536-1791 VGPR0-VGPR255 32*32 Vector General Purpose Registers 1175 when executing in wavefront 32 1176 mode. 1177 1792-2047 *Reserved* 1178 2048-2303 AGPR0-AGPR255 32*32 Vector Accumulation Registers 1179 when executing in wavefront 32 1180 ode. 1181 2304-2559 *Reserved* 1182 2560-2815 VGPR0-VGPR255 64*32 Vector General Purpose Registers 1183 when executing in wavefront 64 1184 mode. 1185 2816-3071 *Reserved* 1186 3072-3327 AGPR0-AGPR255 64*32 Vector Accumulation Registers 1187 when executing in wavefront 64 1188 mode. 1189 3328-3583 *Reserved* 1190 ============== ================= ======== ================================== 1191 1192The vector registers are represented as the full size for the wavefront. They 1193are organized as consecutive dwords (32-bits), one per lane, with the dword at 1194the least significant bit position corresponding to lane 0 and so forth. DWARF 1195location expressions involving the ``DW_OP_LLVM_offset`` and 1196``DW_OP_LLVM_push_lane`` operations are used to select the part of the vector 1197register corresponding to the lane that is executing the current thread of 1198execution in languages that are implemented using a SIMD or SIMT execution 1199model. 1200 1201If the wavefront size is 32 lanes then the wavefront 32 mode register 1202definitions are used. If the wavefront size is 64 lanes then the wavefront 64 1203mode register definitions are used. Some AMDGPU targets support executing in 1204both wavefront 32 and wavefront 64 mode. The register definitions corresponding 1205to the wavefront mode of the generated code will be used. 1206 1207If code is generated to execute in a 32-bit process address space, then the 120832-bit process address space register definitions are used. If code is generated 1209to execute in a 64-bit process address space, then the 64-bit process address 1210space register definitions are used. The ``amdgcn`` target only supports the 121164-bit process address space. 1212 1213.. _amdgpu-dwarf-address-class-identifier: 1214 1215Address Class Identifier 1216------------------------ 1217 1218The DWARF address class represents the source language memory space. See DWARF 1219Version 5 section 2.12 which is updated by the propoal in 1220:ref:`amdgpu-dwarf-segment_addresses`. 1221 1222The DWARF address class mapping used for AMDGPU is defined in 1223:ref:`amdgpu-dwarf-address-class-mapping-table`. 1224 1225.. table:: AMDGPU DWARF Address Class Mapping 1226 :name: amdgpu-dwarf-address-class-mapping-table 1227 1228 ========================= ====== ================= 1229 DWARF AMDGPU 1230 -------------------------------- ----------------- 1231 Address Class Name Value Address Space 1232 ========================= ====== ================= 1233 ``DW_ADDR_none`` 0x0000 Generic (Flat) 1234 ``DW_ADDR_LLVM_global`` 0x0001 Global 1235 ``DW_ADDR_LLVM_constant`` 0x0002 Global 1236 ``DW_ADDR_LLVM_group`` 0x0003 Local (group/LDS) 1237 ``DW_ADDR_LLVM_private`` 0x0004 Private (Scratch) 1238 ``DW_ADDR_AMDGPU_region`` 0x8000 Region (GDS) 1239 ========================= ====== ================= 1240 1241The DWARF address class values defined in the proposal at 1242:ref:`amdgpu-dwarf-segment_addresses` are used. 1243 1244In addition, ``DW_ADDR_AMDGPU_region`` is encoded as a vendor extension. This is 1245available for use for the AMD extension for access to the hardware GDS memory 1246which is scratchpad memory allocated per device. 1247 1248For AMDGPU if no ``DW_AT_address_class`` attribute is present, then the default 1249address class of ``DW_ADDR_none`` is used. 1250 1251See :ref:`amdgpu-dwarf-address-space-identifier` for information on the AMDGPU 1252mapping of DWARF address classes to DWARF address spaces, including address size 1253and NULL value. 1254 1255.. _amdgpu-dwarf-address-space-identifier: 1256 1257Address Space Identifier 1258------------------------ 1259 1260DWARF address spaces correspond to target architecture specific linear 1261addressable memory areas. See DWARF Version 5 section 2.12 and 1262:ref:`amdgpu-dwarf-segment_addresses`. 1263 1264The DWARF address space mapping used for AMDGPU is defined in 1265:ref:`amdgpu-dwarf-address-space-mapping-table`. 1266 1267.. table:: AMDGPU DWARF Address Space Mapping 1268 :name: amdgpu-dwarf-address-space-mapping-table 1269 1270 ======================================= ===== ======= ======== ================= ======================= 1271 DWARF AMDGPU Notes 1272 --------------------------------------- ----- ---------------- ----------------- ----------------------- 1273 Address Space Name Value Address Bit Size Address Space 1274 --------------------------------------- ----- ------- -------- ----------------- ----------------------- 1275 .. 64-bit 32-bit 1276 process process 1277 address address 1278 space space 1279 ======================================= ===== ======= ======== ================= ======================= 1280 ``DW_ASPACE_none`` 0x00 8 4 Global *default address space* 1281 ``DW_ASPACE_AMDGPU_generic`` 0x01 8 4 Generic (Flat) 1282 ``DW_ASPACE_AMDGPU_region`` 0x02 4 4 Region (GDS) 1283 ``DW_ASPACE_AMDGPU_local`` 0x03 4 4 Local (group/LDS) 1284 *Reserved* 0x04 1285 ``DW_ASPACE_AMDGPU_private_lane`` 0x05 4 4 Private (Scratch) *focused lane* 1286 ``DW_ASPACE_AMDGPU_private_wave`` 0x06 4 4 Private (Scratch) *unswizzled wavefront* 1287 *Reserved* 0x07- 1288 0x1F 1289 ``DW_ASPACE_AMDGPU_private_lane<0-63>`` 0x20- 4 4 Private (Scratch) *specific lane* 1290 0x5F 1291 ======================================= ===== ======= ======== ================= ======================= 1292 1293See :ref:`amdgpu-address-spaces` for information on the AMDGPU address spaces 1294including address size and NULL value. 1295 1296The ``DW_ASPACE_none`` address space is the default target architecture address 1297space used in DWARF operations that do not specify an address space. It 1298therefore has to map to the global address space so that the ``DW_OP_addr*`` and 1299related operations can refer to addresses in the program code. 1300 1301The ``DW_ASPACE_AMDGPU_generic`` address space allows location expressions to 1302specify the flat address space. If the address corresponds to an address in the 1303local address space, then it corresponds to the wavefront that is executing the 1304focused thread of execution. If the address corresponds to an address in the 1305private address space, then it corresponds to the lane that is executing the 1306focused thread of execution for languages that are implemented using a SIMD or 1307SIMT execution model. 1308 1309.. note:: 1310 1311 CUDA-like languages such as HIP that do not have address spaces in the 1312 language type system, but do allow variables to be allocated in different 1313 address spaces, need to explicitly specify the ``DW_ASPACE_AMDGPU_generic`` 1314 address space in the DWARF expression operations as the default address space 1315 is the global address space. 1316 1317The ``DW_ASPACE_AMDGPU_local`` address space allows location expressions to 1318specify the local address space corresponding to the wavefront that is executing 1319the focused thread of execution. 1320 1321The ``DW_ASPACE_AMDGPU_private_lane`` address space allows location expressions 1322to specify the private address space corresponding to the lane that is executing 1323the focused thread of execution for languages that are implemented using a SIMD 1324or SIMT execution model. 1325 1326The ``DW_ASPACE_AMDGPU_private_wave`` address space allows location expressions 1327to specify the unswizzled private address space corresponding to the wavefront 1328that is executing the focused thread of execution. The wavefront view of private 1329memory is the per wavefront unswizzled backing memory layout defined in 1330:ref:`amdgpu-address-spaces`, such that address 0 corresponds to the first 1331location for the backing memory of the wavefront (namely the address is not 1332offset by ``wavefront-scratch-base``). The following formula can be used to 1333convert from a ``DW_ASPACE_AMDGPU_private_lane`` address to a 1334``DW_ASPACE_AMDGPU_private_wave`` address: 1335 1336:: 1337 1338 private-address-wavefront = 1339 ((private-address-lane / 4) * wavefront-size * 4) + 1340 (wavefront-lane-id * 4) + (private-address-lane % 4) 1341 1342If the ``DW_ASPACE_AMDGPU_private_lane`` address is dword aligned, and the start 1343of the dwords for each lane starting with lane 0 is required, then this 1344simplifies to: 1345 1346:: 1347 1348 private-address-wavefront = 1349 private-address-lane * wavefront-size 1350 1351A compiler can use the ``DW_ASPACE_AMDGPU_private_wave`` address space to read a 1352complete spilled vector register back into a complete vector register in the 1353CFI. The frame pointer can be a private lane address which is dword aligned, 1354which can be shifted to multiply by the wavefront size, and then used to form a 1355private wavefront address that gives a location for a contiguous set of dwords, 1356one per lane, where the vector register dwords are spilled. The compiler knows 1357the wavefront size since it generates the code. Note that the type of the 1358address may have to be converted as the size of a 1359``DW_ASPACE_AMDGPU_private_lane`` address may be smaller than the size of a 1360``DW_ASPACE_AMDGPU_private_wave`` address. 1361 1362The ``DW_ASPACE_AMDGPU_private_lane<N>`` address space allows location 1363expressions to specify the private address space corresponding to a specific 1364lane N. For example, this can be used when the compiler spills scalar registers 1365to scratch memory, with each scalar register being saved to a different lane's 1366scratch memory. 1367 1368.. _amdgpu-dwarf-lane-identifier: 1369 1370Lane identifier 1371--------------- 1372 1373DWARF lane identifies specify a target architecture lane position for hardware 1374that executes in a SIMD or SIMT manner, and on which a source language maps its 1375threads of execution onto those lanes. The DWARF lane identifier is pushed by 1376the ``DW_OP_LLVM_push_lane`` DWARF expression operation. See DWARF Version 5 1377section 2.5 which is updated by the proposal in 1378:ref:`amdgpu-dwarf-operation-expressions`. 1379 1380For AMDGPU, the lane identifier corresponds to the hardware lane ID of a 1381wavefront. It is numbered from 0 to the wavefront size minus 1. 1382 1383Operation Expressions 1384--------------------- 1385 1386DWARF expressions are used to compute program values and the locations of 1387program objects. See DWARF Version 5 section 2.5 and 1388:ref:`amdgpu-dwarf-operation-expressions`. 1389 1390DWARF location descriptions describe how to access storage which includes memory 1391and registers. When accessing storage on AMDGPU, bytes are ordered with least 1392significant bytes first, and bits are ordered within bytes with least 1393significant bits first. 1394 1395For AMDGPU CFI expressions, ``DW_OP_LLVM_select_bit_piece`` is used to describe 1396unwinding vector registers that are spilled under the execution mask to memory: 1397the zero-single location description is the vector register, and the one-single 1398location description is the spilled memory location description. The 1399``DW_OP_LLVM_form_aspace_address`` is used to specify the address space of the 1400memory location description. 1401 1402In AMDGPU expressions, ``DW_OP_LLVM_select_bit_piece`` is used by the 1403``DW_AT_LLVM_lane_pc`` attribute expression where divergent control flow is 1404controlled by the execution mask. An undefined location description together 1405with ``DW_OP_LLVM_extend`` is used to indicate the lane was not active on entry 1406to the subprogram. See :ref:`amdgpu-dwarf-dw-at-llvm-lane-pc` for an example. 1407 1408Debugger Information Entry Attributes 1409------------------------------------- 1410 1411This section describes how certain debugger information entry attributes are 1412used by AMDGPU. See the sections in DWARF Version 5 section 2 which are updated 1413by the proposal in :ref:`amdgpu-dwarf-debugging-information-entry-attributes`. 1414 1415.. _amdgpu-dwarf-dw-at-llvm-lane-pc: 1416 1417``DW_AT_LLVM_lane_pc`` 1418~~~~~~~~~~~~~~~~~~~~~~ 1419 1420For AMDGPU, the ``DW_AT_LLVM_lane_pc`` attribute is used to specify the program 1421location of the separate lanes of a SIMT thread. 1422 1423If the lane is an active lane then this will be the same as the current program 1424location. 1425 1426If the lane is inactive, but was active on entry to the subprogram, then this is 1427the program location in the subprogram at which execution of the lane is 1428conceptual positioned. 1429 1430If the lane was not active on entry to the subprogram, then this will be the 1431undefined location. A client debugger can check if the lane is part of a valid 1432work-group by checking that the lane is in the range of the associated 1433work-group within the grid, accounting for partial work-groups. If it is not, 1434then the debugger can omit any information for the lane. Otherwise, the debugger 1435may repeatedly unwind the stack and inspect the ``DW_AT_LLVM_lane_pc`` of the 1436calling subprogram until it finds a non-undefined location. Conceptually the 1437lane only has the call frames that it has a non-undefined 1438``DW_AT_LLVM_lane_pc``. 1439 1440The following example illustrates how the AMDGPU backend can generate a DWARF 1441location list expression for the nested ``IF/THEN/ELSE`` structures of the 1442following subprogram pseudo code for a target with 64 lanes per wavefront. 1443 1444.. code:: 1445 :number-lines: 1446 1447 SUBPROGRAM X 1448 BEGIN 1449 a; 1450 IF (c1) THEN 1451 b; 1452 IF (c2) THEN 1453 c; 1454 ELSE 1455 d; 1456 ENDIF 1457 e; 1458 ELSE 1459 f; 1460 ENDIF 1461 g; 1462 END 1463 1464The AMDGPU backend may generate the following pseudo LLVM MIR to manipulate the 1465execution mask (``EXEC``) to linearize the control flow. The condition is 1466evaluated to make a mask of the lanes for which the condition evaluates to true. 1467First the ``THEN`` region is executed by setting the ``EXEC`` mask to the 1468logical ``AND`` of the current ``EXEC`` mask with the condition mask. Then the 1469``ELSE`` region is executed by negating the ``EXEC`` mask and logical ``AND`` of 1470the saved ``EXEC`` mask at the start of the region. After the ``IF/THEN/ELSE`` 1471region the ``EXEC`` mask is restored to the value it had at the beginning of the 1472region. This is shown below. Other approaches are possible, but the basic 1473concept is the same. 1474 1475.. code:: 1476 :number-lines: 1477 1478 $lex_start: 1479 a; 1480 %1 = EXEC 1481 %2 = c1 1482 $lex_1_start: 1483 EXEC = %1 & %2 1484 $if_1_then: 1485 b; 1486 %3 = EXEC 1487 %4 = c2 1488 $lex_1_1_start: 1489 EXEC = %3 & %4 1490 $lex_1_1_then: 1491 c; 1492 EXEC = ~EXEC & %3 1493 $lex_1_1_else: 1494 d; 1495 EXEC = %3 1496 $lex_1_1_end: 1497 e; 1498 EXEC = ~EXEC & %1 1499 $lex_1_else: 1500 f; 1501 EXEC = %1 1502 $lex_1_end: 1503 g; 1504 $lex_end: 1505 1506To create the DWARF location list expression that defines the location 1507description of a vector of lane program locations, the LLVM MIR ``DBG_VALUE`` 1508pseudo instruction can be used to annotate the linearized control flow. This can 1509be done by defining an artificial variable for the lane PC. The DWARF location 1510list expression created for it is used as the value of the 1511``DW_AT_LLVM_lane_pc`` attribute on the subprogram's debugger information entry. 1512 1513A DWARF procedure is defined for each well nested structured control flow region 1514which provides the conceptual lane program location for a lane if it is not 1515active (namely it is divergent). The DWARF operation expression for each region 1516conceptually inherits the value of the immediately enclosing region and modifies 1517it according to the semantics of the region. 1518 1519For an ``IF/THEN/ELSE`` region the divergent program location is at the start of 1520the region for the ``THEN`` region since it is executed first. For the ``ELSE`` 1521region the divergent program location is at the end of the ``IF/THEN/ELSE`` 1522region since the ``THEN`` region has completed. 1523 1524The lane PC artificial variable is assigned at each region transition. It uses 1525the immediately enclosing region's DWARF procedure to compute the program 1526location for each lane assuming they are divergent, and then modifies the result 1527by inserting the current program location for each lane that the ``EXEC`` mask 1528indicates is active. 1529 1530By having separate DWARF procedures for each region, they can be reused to 1531define the value for any nested region. This reduces the total size of the DWARF 1532operation expressions. 1533 1534The following provides an example using pseudo LLVM MIR. 1535 1536.. code:: 1537 :number-lines: 1538 1539 $lex_start: 1540 DEFINE_DWARF %__uint_64 = DW_TAG_base_type[ 1541 DW_AT_name = "__uint64"; 1542 DW_AT_byte_size = 8; 1543 DW_AT_encoding = DW_ATE_unsigned; 1544 ]; 1545 DEFINE_DWARF %__active_lane_pc = DW_TAG_dwarf_procedure[ 1546 DW_AT_name = "__active_lane_pc"; 1547 DW_AT_location = [ 1548 DW_OP_regx PC; 1549 DW_OP_LLVM_extend 64, 64; 1550 DW_OP_regval_type EXEC, %uint_64; 1551 DW_OP_LLVM_select_bit_piece 64, 64; 1552 ]; 1553 ]; 1554 DEFINE_DWARF %__divergent_lane_pc = DW_TAG_dwarf_procedure[ 1555 DW_AT_name = "__divergent_lane_pc"; 1556 DW_AT_location = [ 1557 DW_OP_LLVM_undefined; 1558 DW_OP_LLVM_extend 64, 64; 1559 ]; 1560 ]; 1561 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 1562 DW_OP_call_ref %__divergent_lane_pc; 1563 DW_OP_call_ref %__active_lane_pc; 1564 ]; 1565 a; 1566 %1 = EXEC; 1567 DBG_VALUE %1, $noreg, %__lex_1_save_exec; 1568 %2 = c1; 1569 $lex_1_start: 1570 EXEC = %1 & %2; 1571 $lex_1_then: 1572 DEFINE_DWARF %__divergent_lane_pc_1_then = DW_TAG_dwarf_procedure[ 1573 DW_AT_name = "__divergent_lane_pc_1_then"; 1574 DW_AT_location = DIExpression[ 1575 DW_OP_call_ref %__divergent_lane_pc; 1576 DW_OP_xaddr &lex_1_start; 1577 DW_OP_stack_value; 1578 DW_OP_LLVM_extend 64, 64; 1579 DW_OP_call_ref %__lex_1_save_exec; 1580 DW_OP_deref_type 64, %__uint_64; 1581 DW_OP_LLVM_select_bit_piece 64, 64; 1582 ]; 1583 ]; 1584 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 1585 DW_OP_call_ref %__divergent_lane_pc_1_then; 1586 DW_OP_call_ref %__active_lane_pc; 1587 ]; 1588 b; 1589 %3 = EXEC; 1590 DBG_VALUE %3, %__lex_1_1_save_exec; 1591 %4 = c2; 1592 $lex_1_1_start: 1593 EXEC = %3 & %4; 1594 $lex_1_1_then: 1595 DEFINE_DWARF %__divergent_lane_pc_1_1_then = DW_TAG_dwarf_procedure[ 1596 DW_AT_name = "__divergent_lane_pc_1_1_then"; 1597 DW_AT_location = DIExpression[ 1598 DW_OP_call_ref %__divergent_lane_pc_1_then; 1599 DW_OP_xaddr &lex_1_1_start; 1600 DW_OP_stack_value; 1601 DW_OP_LLVM_extend 64, 64; 1602 DW_OP_call_ref %__lex_1_1_save_exec; 1603 DW_OP_deref_type 64, %__uint_64; 1604 DW_OP_LLVM_select_bit_piece 64, 64; 1605 ]; 1606 ]; 1607 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 1608 DW_OP_call_ref %__divergent_lane_pc_1_1_then; 1609 DW_OP_call_ref %__active_lane_pc; 1610 ]; 1611 c; 1612 EXEC = ~EXEC & %3; 1613 $lex_1_1_else: 1614 DEFINE_DWARF %__divergent_lane_pc_1_1_else = DW_TAG_dwarf_procedure[ 1615 DW_AT_name = "__divergent_lane_pc_1_1_else"; 1616 DW_AT_location = DIExpression[ 1617 DW_OP_call_ref %__divergent_lane_pc_1_then; 1618 DW_OP_xaddr &lex_1_1_end; 1619 DW_OP_stack_value; 1620 DW_OP_LLVM_extend 64, 64; 1621 DW_OP_call_ref %__lex_1_1_save_exec; 1622 DW_OP_deref_type 64, %__uint_64; 1623 DW_OP_LLVM_select_bit_piece 64, 64; 1624 ]; 1625 ]; 1626 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 1627 DW_OP_call_ref %__divergent_lane_pc_1_1_else; 1628 DW_OP_call_ref %__active_lane_pc; 1629 ]; 1630 d; 1631 EXEC = %3; 1632 $lex_1_1_end: 1633 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 1634 DW_OP_call_ref %__divergent_lane_pc; 1635 DW_OP_call_ref %__active_lane_pc; 1636 ]; 1637 e; 1638 EXEC = ~EXEC & %1; 1639 $lex_1_else: 1640 DEFINE_DWARF %__divergent_lane_pc_1_else = DW_TAG_dwarf_procedure[ 1641 DW_AT_name = "__divergent_lane_pc_1_else"; 1642 DW_AT_location = DIExpression[ 1643 DW_OP_call_ref %__divergent_lane_pc; 1644 DW_OP_xaddr &lex_1_end; 1645 DW_OP_stack_value; 1646 DW_OP_LLVM_extend 64, 64; 1647 DW_OP_call_ref %__lex_1_save_exec; 1648 DW_OP_deref_type 64, %__uint_64; 1649 DW_OP_LLVM_select_bit_piece 64, 64; 1650 ]; 1651 ]; 1652 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 1653 DW_OP_call_ref %__divergent_lane_pc_1_else; 1654 DW_OP_call_ref %__active_lane_pc; 1655 ]; 1656 f; 1657 EXEC = %1; 1658 $lex_1_end: 1659 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc DIExpression[ 1660 DW_OP_call_ref %__divergent_lane_pc; 1661 DW_OP_call_ref %__active_lane_pc; 1662 ]; 1663 g; 1664 $lex_end: 1665 1666The DWARF procedure ``%__active_lane_pc`` is used to update the lane pc elements 1667that are active, with the current program location. 1668 1669Artificial variables %__lex_1_save_exec and %__lex_1_1_save_exec are created for 1670the execution masks saved on entry to a region. Using the ``DBG_VALUE`` pseudo 1671instruction, location list entries will be created that describe where the 1672artificial variables are allocated at any given program location. The compiler 1673may allocate them to registers or spill them to memory. 1674 1675The DWARF procedures for each region use the values of the saved execution mask 1676artificial variables to only update the lanes that are active on entry to the 1677region. All other lanes retain the value of the enclosing region where they were 1678last active. If they were not active on entry to the subprogram, then will have 1679the undefined location description. 1680 1681Other structured control flow regions can be handled similarly. For example, 1682loops would set the divergent program location for the region at the end of the 1683loop. Any lanes active will be in the loop, and any lanes not active must have 1684exited the loop. 1685 1686An ``IF/THEN/ELSEIF/ELSEIF/...`` region can be treated as a nest of 1687``IF/THEN/ELSE`` regions. 1688 1689The DWARF procedures can use the active lane artificial variable described in 1690:ref:`amdgpu-dwarf-amdgpu-dw-at-llvm-active-lane` rather than the actual 1691``EXEC`` mask in order to support whole or quad wavefront mode. 1692 1693.. _amdgpu-dwarf-amdgpu-dw-at-llvm-active-lane: 1694 1695``DW_AT_LLVM_active_lane`` 1696~~~~~~~~~~~~~~~~~~~~~~~~~~ 1697 1698The ``DW_AT_LLVM_active_lane`` attribute on a subprogram debugger information 1699entry is used to specify the lanes that are conceptually active for a SIMT 1700thread. 1701 1702The execution mask may be modified to implement whole or quad wavefront mode 1703operations. For example, all lanes may need to temporarily be made active to 1704execute a whole wavefront operation. Such regions would save the ``EXEC`` mask, 1705update it to enable the necessary lanes, perform the operations, and then 1706restore the ``EXEC`` mask from the saved value. While executing the whole 1707wavefront region, the conceptual execution mask is the saved value, not the 1708``EXEC`` value. 1709 1710This is handled by defining an artificial variable for the active lane mask. The 1711active lane mask artificial variable would be the actual ``EXEC`` mask for 1712normal regions, and the saved execution mask for regions where the mask is 1713temporarily updated. The location list expression created for this artificial 1714variable is used to define the value of the ``DW_AT_LLVM_active_lane`` 1715attribute. 1716 1717``DW_AT_LLVM_augmentation`` 1718~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1719 1720For AMDGPU, the ``DW_AT_LLVM_augmentation`` attribute of a compilation unit 1721debugger information entry has the following value for the augmentation string: 1722 1723:: 1724 1725 [amdgpu:v0.0] 1726 1727The "vX.Y" specifies the major X and minor Y version number of the AMDGPU 1728extensions used in the DWARF of the compilation unit. The version number 1729conforms to [SEMVER]_. 1730 1731Call Frame Information 1732---------------------- 1733 1734DWARF Call Frame Information (CFI) describes how a consumer can virtually 1735*unwind* call frames in a running process or core dump. See DWARF Version 5 1736section 6.4 and :ref:`amdgpu-dwarf-call-frame-information`. 1737 1738For AMDGPU, the Common Information Entry (CIE) fields have the following values: 1739 17401. ``augmentation`` string contains the following null-terminated UTF-8 string: 1741 1742 :: 1743 1744 [amd:v0.0] 1745 1746 The ``vX.Y`` specifies the major X and minor Y version number of the AMDGPU 1747 extensions used in this CIE or to the FDEs that use it. The version number 1748 conforms to [SEMVER]_. 1749 17502. ``address_size`` for the ``Global`` address space is defined in 1751 :ref:`amdgpu-dwarf-address-space-identifier`. 1752 17533. ``segment_selector_size`` is 0 as AMDGPU does not use a segment selector. 1754 17554. ``code_alignment_factor`` is 4 bytes. 1756 1757 .. TODO:: 1758 1759 Add to :ref:`amdgpu-processor-table` table. 1760 17615. ``data_alignment_factor`` is 4 bytes. 1762 1763 .. TODO:: 1764 1765 Add to :ref:`amdgpu-processor-table` table. 1766 17676. ``return_address_register`` is ``PC_32`` for 32-bit processes and ``PC_64`` 1768 for 64-bit processes defined in :ref:`amdgpu-dwarf-register-identifier`. 1769 17707. ``initial_instructions`` Since a subprogram X with fewer registers can be 1771 called from subprogram Y that has more allocated, X will not change any of 1772 the extra registers as it cannot access them. Therefore, the default rule 1773 for all columns is ``same value``. 1774 1775For AMDGPU the register number follows the numbering defined in 1776:ref:`amdgpu-dwarf-register-identifier`. 1777 1778For AMDGPU the instructions are variable size. A consumer can subtract 1 from 1779the return address to get the address of a byte within the call site 1780instructions. See DWARF Version 5 section 6.4.4. 1781 1782Accelerated Access 1783------------------ 1784 1785See DWARF Version 5 section 6.1. 1786 1787Lookup By Name Section Header 1788~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1789 1790See DWARF Version 5 section 6.1.1.4.1 and :ref:`amdgpu-dwarf-lookup-by-name`. 1791 1792For AMDGPU the lookup by name section header table: 1793 1794``augmentation_string_size`` (uword) 1795 1796 Set to the length of the ``augmentation_string`` value which is always a 1797 multiple of 4. 1798 1799``augmentation_string`` (sequence of UTF-8 characters) 1800 1801 Contains the following UTF-8 string null padded to a multiple of 4 bytes: 1802 1803 :: 1804 1805 [amdgpu:v0.0] 1806 1807 The "vX.Y" specifies the major X and minor Y version number of the AMDGPU 1808 extensions used in the DWARF of this index. The version number conforms to 1809 [SEMVER]_. 1810 1811 .. note:: 1812 1813 This is different to the DWARF Version 5 definition that requires the first 1814 4 characters to be the vendor ID. But this is consistent with the other 1815 augmentation strings and does allow multiple vendor contributions. However, 1816 backwards compatibility may be more desirable. 1817 1818Lookup By Address Section Header 1819~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1820 1821See DWARF Version 5 section 6.1.2. 1822 1823For AMDGPU the lookup by address section header table: 1824 1825``address_size`` (ubyte) 1826 1827 Match the address size for the ``Global`` address space defined in 1828 :ref:`amdgpu-dwarf-address-space-identifier`. 1829 1830``segment_selector_size`` (ubyte) 1831 1832 AMDGPU does not use a segment selector so this is 0. The entries in the 1833 ``.debug_aranges`` do not have a segment selector. 1834 1835Line Number Information 1836----------------------- 1837 1838See DWARF Version 5 section 6.2 and :ref:`amdgpu-dwarf-line-number-information`. 1839 1840AMDGPU does not use the ``isa`` state machine registers and always sets it to 0. 1841The instruction set must be obtained from the ELF file header ``e_flags`` field 1842in the ``EF_AMDGPU_MACH`` bit position (see :ref:`ELF Header 1843<amdgpu-elf-header>`). See DWARF Version 5 section 6.2.2. 1844 1845.. TODO:: 1846 1847 Should the ``isa`` state machine register be used to indicate if the code is 1848 in wavefront32 or wavefront64 mode? Or used to specify the architecture ISA? 1849 1850For AMDGPU the line number program header fields have the following values (see 1851DWARF Version 5 section 6.2.4): 1852 1853``address_size`` (ubyte) 1854 Matches the address size for the ``Global`` address space defined in 1855 :ref:`amdgpu-dwarf-address-space-identifier`. 1856 1857``segment_selector_size`` (ubyte) 1858 AMDGPU does not use a segment selector so this is 0. 1859 1860``minimum_instruction_length`` (ubyte) 1861 For GFX9-GFX10 this is 4. 1862 1863``maximum_operations_per_instruction`` (ubyte) 1864 For GFX9-GFX10 this is 1. 1865 1866Source text for online-compiled programs (for example, those compiled by the 1867OpenCL language runtime) may be embedded into the DWARF Version 5 line table. 1868See DWARF Version 5 section 6.2.4.1 which is updated by the proposal in 1869:ref:`DW_LNCT_LLVM_source 1870<amdgpu-dwarf-line-number-information-dw-lnct-llvm-source>`. 1871 1872The Clang option used to control source embedding in AMDGPU is defined in 1873:ref:`amdgpu-clang-debug-options-table`. 1874 1875 .. table:: AMDGPU Clang Debug Options 1876 :name: amdgpu-clang-debug-options-table 1877 1878 ==================== ================================================== 1879 Debug Flag Description 1880 ==================== ================================================== 1881 -g[no-]embed-source Enable/disable embedding source text in DWARF 1882 debug sections. Useful for environments where 1883 source cannot be written to disk, such as 1884 when performing online compilation. 1885 ==================== ================================================== 1886 1887For example: 1888 1889``-gembed-source`` 1890 Enable the embedded source. 1891 1892``-gno-embed-source`` 1893 Disable the embedded source. 1894 189532-Bit and 64-Bit DWARF Formats 1896------------------------------- 1897 1898See DWARF Version 5 section 7.4 and 1899:ref:`amdgpu-dwarf-32-bit-and-64-bit-dwarf-formats`. 1900 1901For AMDGPU: 1902 1903* For the ``amdgcn`` target architecture only the 64-bit process address space 1904 is supported. 1905 1906* The producer can generate either 32-bit or 64-bit DWARF format. LLVM generates 1907 the 32-bit DWARF format. 1908 1909Unit Headers 1910------------ 1911 1912For AMDGPU the following values apply for each of the unit headers described in 1913DWARF Version 5 sections 7.5.1.1, 7.5.1.2, and 7.5.1.3: 1914 1915``address_size`` (ubyte) 1916 Matches the address size for the ``Global`` address space defined in 1917 :ref:`amdgpu-dwarf-address-space-identifier`. 1918 1919.. _amdgpu-code-conventions: 1920 1921Code Conventions 1922================ 1923 1924This section provides code conventions used for each supported target triple OS 1925(see :ref:`amdgpu-target-triples`). 1926 1927AMDHSA 1928------ 1929 1930This section provides code conventions used when the target triple OS is 1931``amdhsa`` (see :ref:`amdgpu-target-triples`). 1932 1933.. _amdgpu-amdhsa-code-object-target-identification: 1934 1935Code Object Target Identification 1936~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1937 1938The AMDHSA OS uses the following syntax to specify the code object 1939target as a single string: 1940 1941 ``<Architecture>-<Vendor>-<OS>-<Environment>-<Processor><Target Features>`` 1942 1943Where: 1944 1945 - ``<Architecture>``, ``<Vendor>``, ``<OS>`` and ``<Environment>`` 1946 are the same as the *Target Triple* (see 1947 :ref:`amdgpu-target-triples`). 1948 1949 - ``<Processor>`` is the same as the *Processor* (see 1950 :ref:`amdgpu-processors`). 1951 1952 - ``<Target Features>`` is a list of the enabled *Target Features* 1953 (see :ref:`amdgpu-target-features`), each prefixed by a plus, that 1954 apply to *Processor*. The list must be in the same order as listed 1955 in the table :ref:`amdgpu-target-feature-table`. Note that *Target 1956 Features* must be included in the list if they are enabled even if 1957 that is the default for *Processor*. 1958 1959For example: 1960 1961 ``"amdgcn-amd-amdhsa--gfx902+xnack"`` 1962 1963.. _amdgpu-amdhsa-code-object-metadata: 1964 1965Code Object Metadata 1966~~~~~~~~~~~~~~~~~~~~ 1967 1968The code object metadata specifies extensible metadata associated with the code 1969objects executed on HSA [HSA]_ compatible runtimes such as AMD's ROCm 1970[AMD-ROCm]_. The encoding and semantics of this metadata depends on the code 1971object version; see :ref:`amdgpu-amdhsa-code-object-metadata-v2` and 1972:ref:`amdgpu-amdhsa-code-object-metadata-v3`. 1973 1974Code object metadata is specified in a note record (see 1975:ref:`amdgpu-note-records`) and is required when the target triple OS is 1976``amdhsa`` (see :ref:`amdgpu-target-triples`). It must contain the minimum 1977information necessary to support the ROCM kernel queries. For example, the 1978segment sizes needed in a dispatch packet. In addition, a high-level language 1979runtime may require other information to be included. For example, the AMD 1980OpenCL runtime records kernel argument information. 1981 1982.. _amdgpu-amdhsa-code-object-metadata-v2: 1983 1984Code Object V2 Metadata (-mattr=-code-object-v3) 1985++++++++++++++++++++++++++++++++++++++++++++++++ 1986 1987.. warning:: Code Object V2 is not the default code object version emitted by 1988 this version of LLVM. For a description of the metadata generated with the 1989 default configuration (Code Object V3) see 1990 :ref:`amdgpu-amdhsa-code-object-metadata-v3`. 1991 1992Code object V2 metadata is specified by the ``NT_AMD_AMDGPU_METADATA`` note 1993record (see :ref:`amdgpu-note-records-v2`). 1994 1995The metadata is specified as a YAML formatted string (see [YAML]_ and 1996:doc:`YamlIO`). 1997 1998.. TODO:: 1999 2000 Is the string null terminated? It probably should not if YAML allows it to 2001 contain null characters, otherwise it should be. 2002 2003The metadata is represented as a single YAML document comprised of the mapping 2004defined in table :ref:`amdgpu-amdhsa-code-object-metadata-map-table-v2` and 2005referenced tables. 2006 2007For boolean values, the string values of ``false`` and ``true`` are used for 2008false and true respectively. 2009 2010Additional information can be added to the mappings. To avoid conflicts, any 2011non-AMD key names should be prefixed by "*vendor-name*.". 2012 2013 .. table:: AMDHSA Code Object V2 Metadata Map 2014 :name: amdgpu-amdhsa-code-object-metadata-map-table-v2 2015 2016 ========== ============== ========= ======================================= 2017 String Key Value Type Required? Description 2018 ========== ============== ========= ======================================= 2019 "Version" sequence of Required - The first integer is the major 2020 2 integers version. Currently 1. 2021 - The second integer is the minor 2022 version. Currently 0. 2023 "Printf" sequence of Each string is encoded information 2024 strings about a printf function call. The 2025 encoded information is organized as 2026 fields separated by colon (':'): 2027 2028 ``ID:N:S[0]:S[1]:...:S[N-1]:FormatString`` 2029 2030 where: 2031 2032 ``ID`` 2033 A 32-bit integer as a unique id for 2034 each printf function call 2035 2036 ``N`` 2037 A 32-bit integer equal to the number 2038 of arguments of printf function call 2039 minus 1 2040 2041 ``S[i]`` (where i = 0, 1, ... , N-1) 2042 32-bit integers for the size in bytes 2043 of the i-th FormatString argument of 2044 the printf function call 2045 2046 FormatString 2047 The format string passed to the 2048 printf function call. 2049 "Kernels" sequence of Required Sequence of the mappings for each 2050 mapping kernel in the code object. See 2051 :ref:`amdgpu-amdhsa-code-object-kernel-metadata-map-table-v2` 2052 for the definition of the mapping. 2053 ========== ============== ========= ======================================= 2054 2055.. 2056 2057 .. table:: AMDHSA Code Object V2 Kernel Metadata Map 2058 :name: amdgpu-amdhsa-code-object-kernel-metadata-map-table-v2 2059 2060 ================= ============== ========= ================================ 2061 String Key Value Type Required? Description 2062 ================= ============== ========= ================================ 2063 "Name" string Required Source name of the kernel. 2064 "SymbolName" string Required Name of the kernel 2065 descriptor ELF symbol. 2066 "Language" string Source language of the kernel. 2067 Values include: 2068 2069 - "OpenCL C" 2070 - "OpenCL C++" 2071 - "HCC" 2072 - "OpenMP" 2073 2074 "LanguageVersion" sequence of - The first integer is the major 2075 2 integers version. 2076 - The second integer is the 2077 minor version. 2078 "Attrs" mapping Mapping of kernel attributes. 2079 See 2080 :ref:`amdgpu-amdhsa-code-object-kernel-attribute-metadata-map-table-v2` 2081 for the mapping definition. 2082 "Args" sequence of Sequence of mappings of the 2083 mapping kernel arguments. See 2084 :ref:`amdgpu-amdhsa-code-object-kernel-argument-metadata-map-table-v2` 2085 for the definition of the mapping. 2086 "CodeProps" mapping Mapping of properties related to 2087 the kernel code. See 2088 :ref:`amdgpu-amdhsa-code-object-kernel-code-properties-metadata-map-table-v2` 2089 for the mapping definition. 2090 ================= ============== ========= ================================ 2091 2092.. 2093 2094 .. table:: AMDHSA Code Object V2 Kernel Attribute Metadata Map 2095 :name: amdgpu-amdhsa-code-object-kernel-attribute-metadata-map-table-v2 2096 2097 =================== ============== ========= ============================== 2098 String Key Value Type Required? Description 2099 =================== ============== ========= ============================== 2100 "ReqdWorkGroupSize" sequence of If not 0, 0, 0 then all values 2101 3 integers must be >=1 and the dispatch 2102 work-group size X, Y, Z must 2103 correspond to the specified 2104 values. Defaults to 0, 0, 0. 2105 2106 Corresponds to the OpenCL 2107 ``reqd_work_group_size`` 2108 attribute. 2109 "WorkGroupSizeHint" sequence of The dispatch work-group size 2110 3 integers X, Y, Z is likely to be the 2111 specified values. 2112 2113 Corresponds to the OpenCL 2114 ``work_group_size_hint`` 2115 attribute. 2116 "VecTypeHint" string The name of a scalar or vector 2117 type. 2118 2119 Corresponds to the OpenCL 2120 ``vec_type_hint`` attribute. 2121 2122 "RuntimeHandle" string The external symbol name 2123 associated with a kernel. 2124 OpenCL runtime allocates a 2125 global buffer for the symbol 2126 and saves the kernel's address 2127 to it, which is used for 2128 device side enqueueing. Only 2129 available for device side 2130 enqueued kernels. 2131 =================== ============== ========= ============================== 2132 2133.. 2134 2135 .. table:: AMDHSA Code Object V2 Kernel Argument Metadata Map 2136 :name: amdgpu-amdhsa-code-object-kernel-argument-metadata-map-table-v2 2137 2138 ================= ============== ========= ================================ 2139 String Key Value Type Required? Description 2140 ================= ============== ========= ================================ 2141 "Name" string Kernel argument name. 2142 "TypeName" string Kernel argument type name. 2143 "Size" integer Required Kernel argument size in bytes. 2144 "Align" integer Required Kernel argument alignment in 2145 bytes. Must be a power of two. 2146 "ValueKind" string Required Kernel argument kind that 2147 specifies how to set up the 2148 corresponding argument. 2149 Values include: 2150 2151 "ByValue" 2152 The argument is copied 2153 directly into the kernarg. 2154 2155 "GlobalBuffer" 2156 A global address space pointer 2157 to the buffer data is passed 2158 in the kernarg. 2159 2160 "DynamicSharedPointer" 2161 A group address space pointer 2162 to dynamically allocated LDS 2163 is passed in the kernarg. 2164 2165 "Sampler" 2166 A global address space 2167 pointer to a S# is passed in 2168 the kernarg. 2169 2170 "Image" 2171 A global address space 2172 pointer to a T# is passed in 2173 the kernarg. 2174 2175 "Pipe" 2176 A global address space pointer 2177 to an OpenCL pipe is passed in 2178 the kernarg. 2179 2180 "Queue" 2181 A global address space pointer 2182 to an OpenCL device enqueue 2183 queue is passed in the 2184 kernarg. 2185 2186 "HiddenGlobalOffsetX" 2187 The OpenCL grid dispatch 2188 global offset for the X 2189 dimension is passed in the 2190 kernarg. 2191 2192 "HiddenGlobalOffsetY" 2193 The OpenCL grid dispatch 2194 global offset for the Y 2195 dimension is passed in the 2196 kernarg. 2197 2198 "HiddenGlobalOffsetZ" 2199 The OpenCL grid dispatch 2200 global offset for the Z 2201 dimension is passed in the 2202 kernarg. 2203 2204 "HiddenNone" 2205 An argument that is not used 2206 by the kernel. Space needs to 2207 be left for it, but it does 2208 not need to be set up. 2209 2210 "HiddenPrintfBuffer" 2211 A global address space pointer 2212 to the runtime printf buffer 2213 is passed in kernarg. 2214 2215 "HiddenHostcallBuffer" 2216 A global address space pointer 2217 to the runtime hostcall buffer 2218 is passed in kernarg. 2219 2220 "HiddenDefaultQueue" 2221 A global address space pointer 2222 to the OpenCL device enqueue 2223 queue that should be used by 2224 the kernel by default is 2225 passed in the kernarg. 2226 2227 "HiddenCompletionAction" 2228 A global address space pointer 2229 to help link enqueued kernels into 2230 the ancestor tree for determining 2231 when the parent kernel has finished. 2232 2233 "HiddenMultiGridSyncArg" 2234 A global address space pointer for 2235 multi-grid synchronization is 2236 passed in the kernarg. 2237 2238 "ValueType" string Required Kernel argument value type. Only 2239 present if "ValueKind" is 2240 "ByValue". For vector data 2241 types, the value is for the 2242 element type. Values include: 2243 2244 - "Struct" 2245 - "I8" 2246 - "U8" 2247 - "I16" 2248 - "U16" 2249 - "F16" 2250 - "I32" 2251 - "U32" 2252 - "F32" 2253 - "I64" 2254 - "U64" 2255 - "F64" 2256 2257 .. TODO:: 2258 How can it be determined if a 2259 vector type, and what size 2260 vector? 2261 "PointeeAlign" integer Alignment in bytes of pointee 2262 type for pointer type kernel 2263 argument. Must be a power 2264 of 2. Only present if 2265 "ValueKind" is 2266 "DynamicSharedPointer". 2267 "AddrSpaceQual" string Kernel argument address space 2268 qualifier. Only present if 2269 "ValueKind" is "GlobalBuffer" or 2270 "DynamicSharedPointer". Values 2271 are: 2272 2273 - "Private" 2274 - "Global" 2275 - "Constant" 2276 - "Local" 2277 - "Generic" 2278 - "Region" 2279 2280 .. TODO:: 2281 Is GlobalBuffer only Global 2282 or Constant? Is 2283 DynamicSharedPointer always 2284 Local? Can HCC allow Generic? 2285 How can Private or Region 2286 ever happen? 2287 "AccQual" string Kernel argument access 2288 qualifier. Only present if 2289 "ValueKind" is "Image" or 2290 "Pipe". Values 2291 are: 2292 2293 - "ReadOnly" 2294 - "WriteOnly" 2295 - "ReadWrite" 2296 2297 .. TODO:: 2298 Does this apply to 2299 GlobalBuffer? 2300 "ActualAccQual" string The actual memory accesses 2301 performed by the kernel on the 2302 kernel argument. Only present if 2303 "ValueKind" is "GlobalBuffer", 2304 "Image", or "Pipe". This may be 2305 more restrictive than indicated 2306 by "AccQual" to reflect what the 2307 kernel actual does. If not 2308 present then the runtime must 2309 assume what is implied by 2310 "AccQual" and "IsConst". Values 2311 are: 2312 2313 - "ReadOnly" 2314 - "WriteOnly" 2315 - "ReadWrite" 2316 2317 "IsConst" boolean Indicates if the kernel argument 2318 is const qualified. Only present 2319 if "ValueKind" is 2320 "GlobalBuffer". 2321 2322 "IsRestrict" boolean Indicates if the kernel argument 2323 is restrict qualified. Only 2324 present if "ValueKind" is 2325 "GlobalBuffer". 2326 2327 "IsVolatile" boolean Indicates if the kernel argument 2328 is volatile qualified. Only 2329 present if "ValueKind" is 2330 "GlobalBuffer". 2331 2332 "IsPipe" boolean Indicates if the kernel argument 2333 is pipe qualified. Only present 2334 if "ValueKind" is "Pipe". 2335 2336 .. TODO:: 2337 Can GlobalBuffer be pipe 2338 qualified? 2339 ================= ============== ========= ================================ 2340 2341.. 2342 2343 .. table:: AMDHSA Code Object V2 Kernel Code Properties Metadata Map 2344 :name: amdgpu-amdhsa-code-object-kernel-code-properties-metadata-map-table-v2 2345 2346 ============================ ============== ========= ===================== 2347 String Key Value Type Required? Description 2348 ============================ ============== ========= ===================== 2349 "KernargSegmentSize" integer Required The size in bytes of 2350 the kernarg segment 2351 that holds the values 2352 of the arguments to 2353 the kernel. 2354 "GroupSegmentFixedSize" integer Required The amount of group 2355 segment memory 2356 required by a 2357 work-group in 2358 bytes. This does not 2359 include any 2360 dynamically allocated 2361 group segment memory 2362 that may be added 2363 when the kernel is 2364 dispatched. 2365 "PrivateSegmentFixedSize" integer Required The amount of fixed 2366 private address space 2367 memory required for a 2368 work-item in 2369 bytes. If the kernel 2370 uses a dynamic call 2371 stack then additional 2372 space must be added 2373 to this value for the 2374 call stack. 2375 "KernargSegmentAlign" integer Required The maximum byte 2376 alignment of 2377 arguments in the 2378 kernarg segment. Must 2379 be a power of 2. 2380 "WavefrontSize" integer Required Wavefront size. Must 2381 be a power of 2. 2382 "NumSGPRs" integer Required Number of scalar 2383 registers used by a 2384 wavefront for 2385 GFX6-GFX10. This 2386 includes the special 2387 SGPRs for VCC, Flat 2388 Scratch (GFX7-GFX10) 2389 and XNACK (for 2390 GFX8-GFX10). It does 2391 not include the 16 2392 SGPR added if a trap 2393 handler is 2394 enabled. It is not 2395 rounded up to the 2396 allocation 2397 granularity. 2398 "NumVGPRs" integer Required Number of vector 2399 registers used by 2400 each work-item for 2401 GFX6-GFX10 2402 "MaxFlatWorkGroupSize" integer Required Maximum flat 2403 work-group size 2404 supported by the 2405 kernel in work-items. 2406 Must be >=1 and 2407 consistent with 2408 ReqdWorkGroupSize if 2409 not 0, 0, 0. 2410 "NumSpilledSGPRs" integer Number of stores from 2411 a scalar register to 2412 a register allocator 2413 created spill 2414 location. 2415 "NumSpilledVGPRs" integer Number of stores from 2416 a vector register to 2417 a register allocator 2418 created spill 2419 location. 2420 ============================ ============== ========= ===================== 2421 2422.. _amdgpu-amdhsa-code-object-metadata-v3: 2423 2424Code Object V3 Metadata (-mattr=+code-object-v3) 2425++++++++++++++++++++++++++++++++++++++++++++++++ 2426 2427Code object V3 metadata is specified by the ``NT_AMDGPU_METADATA`` note record 2428(see :ref:`amdgpu-note-records-v3`). 2429 2430The metadata is represented as Message Pack formatted binary data (see 2431[MsgPack]_). The top level is a Message Pack map that includes the 2432keys defined in table 2433:ref:`amdgpu-amdhsa-code-object-metadata-map-table-v3` and referenced 2434tables. 2435 2436Additional information can be added to the maps. To avoid conflicts, 2437any key names should be prefixed by "*vendor-name*." where 2438``vendor-name`` can be the name of the vendor and specific vendor 2439tool that generates the information. The prefix is abbreviated to 2440simply "." when it appears within a map that has been added by the 2441same *vendor-name*. 2442 2443 .. table:: AMDHSA Code Object V3 Metadata Map 2444 :name: amdgpu-amdhsa-code-object-metadata-map-table-v3 2445 2446 ================= ============== ========= ======================================= 2447 String Key Value Type Required? Description 2448 ================= ============== ========= ======================================= 2449 "amdhsa.version" sequence of Required - The first integer is the major 2450 2 integers version. Currently 1. 2451 - The second integer is the minor 2452 version. Currently 0. 2453 "amdhsa.printf" sequence of Each string is encoded information 2454 strings about a printf function call. The 2455 encoded information is organized as 2456 fields separated by colon (':'): 2457 2458 ``ID:N:S[0]:S[1]:...:S[N-1]:FormatString`` 2459 2460 where: 2461 2462 ``ID`` 2463 A 32-bit integer as a unique id for 2464 each printf function call 2465 2466 ``N`` 2467 A 32-bit integer equal to the number 2468 of arguments of printf function call 2469 minus 1 2470 2471 ``S[i]`` (where i = 0, 1, ... , N-1) 2472 32-bit integers for the size in bytes 2473 of the i-th FormatString argument of 2474 the printf function call 2475 2476 FormatString 2477 The format string passed to the 2478 printf function call. 2479 "amdhsa.kernels" sequence of Required Sequence of the maps for each 2480 map kernel in the code object. See 2481 :ref:`amdgpu-amdhsa-code-object-kernel-metadata-map-table-v3` 2482 for the definition of the keys included 2483 in that map. 2484 ================= ============== ========= ======================================= 2485 2486.. 2487 2488 .. table:: AMDHSA Code Object V3 Kernel Metadata Map 2489 :name: amdgpu-amdhsa-code-object-kernel-metadata-map-table-v3 2490 2491 =================================== ============== ========= ================================ 2492 String Key Value Type Required? Description 2493 =================================== ============== ========= ================================ 2494 ".name" string Required Source name of the kernel. 2495 ".symbol" string Required Name of the kernel 2496 descriptor ELF symbol. 2497 ".language" string Source language of the kernel. 2498 Values include: 2499 2500 - "OpenCL C" 2501 - "OpenCL C++" 2502 - "HCC" 2503 - "HIP" 2504 - "OpenMP" 2505 - "Assembler" 2506 2507 ".language_version" sequence of - The first integer is the major 2508 2 integers version. 2509 - The second integer is the 2510 minor version. 2511 ".args" sequence of Sequence of maps of the 2512 map kernel arguments. See 2513 :ref:`amdgpu-amdhsa-code-object-kernel-argument-metadata-map-table-v3` 2514 for the definition of the keys 2515 included in that map. 2516 ".reqd_workgroup_size" sequence of If not 0, 0, 0 then all values 2517 3 integers must be >=1 and the dispatch 2518 work-group size X, Y, Z must 2519 correspond to the specified 2520 values. Defaults to 0, 0, 0. 2521 2522 Corresponds to the OpenCL 2523 ``reqd_work_group_size`` 2524 attribute. 2525 ".workgroup_size_hint" sequence of The dispatch work-group size 2526 3 integers X, Y, Z is likely to be the 2527 specified values. 2528 2529 Corresponds to the OpenCL 2530 ``work_group_size_hint`` 2531 attribute. 2532 ".vec_type_hint" string The name of a scalar or vector 2533 type. 2534 2535 Corresponds to the OpenCL 2536 ``vec_type_hint`` attribute. 2537 2538 ".device_enqueue_symbol" string The external symbol name 2539 associated with a kernel. 2540 OpenCL runtime allocates a 2541 global buffer for the symbol 2542 and saves the kernel's address 2543 to it, which is used for 2544 device side enqueueing. Only 2545 available for device side 2546 enqueued kernels. 2547 ".kernarg_segment_size" integer Required The size in bytes of 2548 the kernarg segment 2549 that holds the values 2550 of the arguments to 2551 the kernel. 2552 ".group_segment_fixed_size" integer Required The amount of group 2553 segment memory 2554 required by a 2555 work-group in 2556 bytes. This does not 2557 include any 2558 dynamically allocated 2559 group segment memory 2560 that may be added 2561 when the kernel is 2562 dispatched. 2563 ".private_segment_fixed_size" integer Required The amount of fixed 2564 private address space 2565 memory required for a 2566 work-item in 2567 bytes. If the kernel 2568 uses a dynamic call 2569 stack then additional 2570 space must be added 2571 to this value for the 2572 call stack. 2573 ".kernarg_segment_align" integer Required The maximum byte 2574 alignment of 2575 arguments in the 2576 kernarg segment. Must 2577 be a power of 2. 2578 ".wavefront_size" integer Required Wavefront size. Must 2579 be a power of 2. 2580 ".sgpr_count" integer Required Number of scalar 2581 registers required by a 2582 wavefront for 2583 GFX6-GFX9. A register 2584 is required if it is 2585 used explicitly, or 2586 if a higher numbered 2587 register is used 2588 explicitly. This 2589 includes the special 2590 SGPRs for VCC, Flat 2591 Scratch (GFX7-GFX9) 2592 and XNACK (for 2593 GFX8-GFX9). It does 2594 not include the 16 2595 SGPR added if a trap 2596 handler is 2597 enabled. It is not 2598 rounded up to the 2599 allocation 2600 granularity. 2601 ".vgpr_count" integer Required Number of vector 2602 registers required by 2603 each work-item for 2604 GFX6-GFX9. A register 2605 is required if it is 2606 used explicitly, or 2607 if a higher numbered 2608 register is used 2609 explicitly. 2610 ".max_flat_workgroup_size" integer Required Maximum flat 2611 work-group size 2612 supported by the 2613 kernel in work-items. 2614 Must be >=1 and 2615 consistent with 2616 ReqdWorkGroupSize if 2617 not 0, 0, 0. 2618 ".sgpr_spill_count" integer Number of stores from 2619 a scalar register to 2620 a register allocator 2621 created spill 2622 location. 2623 ".vgpr_spill_count" integer Number of stores from 2624 a vector register to 2625 a register allocator 2626 created spill 2627 location. 2628 =================================== ============== ========= ================================ 2629 2630.. 2631 2632 .. table:: AMDHSA Code Object V3 Kernel Argument Metadata Map 2633 :name: amdgpu-amdhsa-code-object-kernel-argument-metadata-map-table-v3 2634 2635 ====================== ============== ========= ================================ 2636 String Key Value Type Required? Description 2637 ====================== ============== ========= ================================ 2638 ".name" string Kernel argument name. 2639 ".type_name" string Kernel argument type name. 2640 ".size" integer Required Kernel argument size in bytes. 2641 ".offset" integer Required Kernel argument offset in 2642 bytes. The offset must be a 2643 multiple of the alignment 2644 required by the argument. 2645 ".value_kind" string Required Kernel argument kind that 2646 specifies how to set up the 2647 corresponding argument. 2648 Values include: 2649 2650 "by_value" 2651 The argument is copied 2652 directly into the kernarg. 2653 2654 "global_buffer" 2655 A global address space pointer 2656 to the buffer data is passed 2657 in the kernarg. 2658 2659 "dynamic_shared_pointer" 2660 A group address space pointer 2661 to dynamically allocated LDS 2662 is passed in the kernarg. 2663 2664 "sampler" 2665 A global address space 2666 pointer to a S# is passed in 2667 the kernarg. 2668 2669 "image" 2670 A global address space 2671 pointer to a T# is passed in 2672 the kernarg. 2673 2674 "pipe" 2675 A global address space pointer 2676 to an OpenCL pipe is passed in 2677 the kernarg. 2678 2679 "queue" 2680 A global address space pointer 2681 to an OpenCL device enqueue 2682 queue is passed in the 2683 kernarg. 2684 2685 "hidden_global_offset_x" 2686 The OpenCL grid dispatch 2687 global offset for the X 2688 dimension is passed in the 2689 kernarg. 2690 2691 "hidden_global_offset_y" 2692 The OpenCL grid dispatch 2693 global offset for the Y 2694 dimension is passed in the 2695 kernarg. 2696 2697 "hidden_global_offset_z" 2698 The OpenCL grid dispatch 2699 global offset for the Z 2700 dimension is passed in the 2701 kernarg. 2702 2703 "hidden_none" 2704 An argument that is not used 2705 by the kernel. Space needs to 2706 be left for it, but it does 2707 not need to be set up. 2708 2709 "hidden_printf_buffer" 2710 A global address space pointer 2711 to the runtime printf buffer 2712 is passed in kernarg. 2713 2714 "hidden_hostcall_buffer" 2715 A global address space pointer 2716 to the runtime hostcall buffer 2717 is passed in kernarg. 2718 2719 "hidden_default_queue" 2720 A global address space pointer 2721 to the OpenCL device enqueue 2722 queue that should be used by 2723 the kernel by default is 2724 passed in the kernarg. 2725 2726 "hidden_completion_action" 2727 A global address space pointer 2728 to help link enqueued kernels into 2729 the ancestor tree for determining 2730 when the parent kernel has finished. 2731 2732 "hidden_multigrid_sync_arg" 2733 A global address space pointer for 2734 multi-grid synchronization is 2735 passed in the kernarg. 2736 2737 ".value_type" string Required Kernel argument value type. Only 2738 present if ".value_kind" is 2739 "by_value". For vector data 2740 types, the value is for the 2741 element type. Values include: 2742 2743 - "struct" 2744 - "i8" 2745 - "u8" 2746 - "i16" 2747 - "u16" 2748 - "f16" 2749 - "i32" 2750 - "u32" 2751 - "f32" 2752 - "i64" 2753 - "u64" 2754 - "f64" 2755 2756 .. TODO:: 2757 How can it be determined if a 2758 vector type, and what size 2759 vector? 2760 ".pointee_align" integer Alignment in bytes of pointee 2761 type for pointer type kernel 2762 argument. Must be a power 2763 of 2. Only present if 2764 ".value_kind" is 2765 "dynamic_shared_pointer". 2766 ".address_space" string Kernel argument address space 2767 qualifier. Only present if 2768 ".value_kind" is "global_buffer" or 2769 "dynamic_shared_pointer". Values 2770 are: 2771 2772 - "private" 2773 - "global" 2774 - "constant" 2775 - "local" 2776 - "generic" 2777 - "region" 2778 2779 .. TODO:: 2780 Is "global_buffer" only "global" 2781 or "constant"? Is 2782 "dynamic_shared_pointer" always 2783 "local"? Can HCC allow "generic"? 2784 How can "private" or "region" 2785 ever happen? 2786 ".access" string Kernel argument access 2787 qualifier. Only present if 2788 ".value_kind" is "image" or 2789 "pipe". Values 2790 are: 2791 2792 - "read_only" 2793 - "write_only" 2794 - "read_write" 2795 2796 .. TODO:: 2797 Does this apply to 2798 "global_buffer"? 2799 ".actual_access" string The actual memory accesses 2800 performed by the kernel on the 2801 kernel argument. Only present if 2802 ".value_kind" is "global_buffer", 2803 "image", or "pipe". This may be 2804 more restrictive than indicated 2805 by ".access" to reflect what the 2806 kernel actual does. If not 2807 present then the runtime must 2808 assume what is implied by 2809 ".access" and ".is_const" . Values 2810 are: 2811 2812 - "read_only" 2813 - "write_only" 2814 - "read_write" 2815 2816 ".is_const" boolean Indicates if the kernel argument 2817 is const qualified. Only present 2818 if ".value_kind" is 2819 "global_buffer". 2820 2821 ".is_restrict" boolean Indicates if the kernel argument 2822 is restrict qualified. Only 2823 present if ".value_kind" is 2824 "global_buffer". 2825 2826 ".is_volatile" boolean Indicates if the kernel argument 2827 is volatile qualified. Only 2828 present if ".value_kind" is 2829 "global_buffer". 2830 2831 ".is_pipe" boolean Indicates if the kernel argument 2832 is pipe qualified. Only present 2833 if ".value_kind" is "pipe". 2834 2835 .. TODO:: 2836 Can "global_buffer" be pipe 2837 qualified? 2838 ====================== ============== ========= ================================ 2839 2840.. 2841 2842Kernel Dispatch 2843~~~~~~~~~~~~~~~ 2844 2845The HSA architected queuing language (AQL) defines a user space memory 2846interface that can be used to control the dispatch of kernels, in an agent 2847independent way. An agent can have zero or more AQL queues created for it using 2848the ROCm runtime, in which AQL packets (all of which are 64 bytes) can be 2849placed. See the *HSA Platform System Architecture Specification* [HSA]_ for the 2850AQL queue mechanics and packet layouts. 2851 2852The packet processor of a kernel agent is responsible for detecting and 2853dispatching HSA kernels from the AQL queues associated with it. For AMD GPUs the 2854packet processor is implemented by the hardware command processor (CP), 2855asynchronous dispatch controller (ADC) and shader processor input controller 2856(SPI). 2857 2858The ROCm runtime can be used to allocate an AQL queue object. It uses the kernel 2859mode driver to initialize and register the AQL queue with CP. 2860 2861To dispatch a kernel the following actions are performed. This can occur in the 2862CPU host program, or from an HSA kernel executing on a GPU. 2863 28641. A pointer to an AQL queue for the kernel agent on which the kernel is to be 2865 executed is obtained. 28662. A pointer to the kernel descriptor (see 2867 :ref:`amdgpu-amdhsa-kernel-descriptor`) of the kernel to execute is obtained. 2868 It must be for a kernel that is contained in a code object that that was 2869 loaded by the ROCm runtime on the kernel agent with which the AQL queue is 2870 associated. 28713. Space is allocated for the kernel arguments using the ROCm runtime allocator 2872 for a memory region with the kernarg property for the kernel agent that will 2873 execute the kernel. It must be at least 16-byte aligned. 28744. Kernel argument values are assigned to the kernel argument memory 2875 allocation. The layout is defined in the *HSA Programmer's Language 2876 Reference* [HSA]_. For AMDGPU the kernel execution directly accesses the 2877 kernel argument memory in the same way constant memory is accessed. (Note 2878 that the HSA specification allows an implementation to copy the kernel 2879 argument contents to another location that is accessed by the kernel.) 28805. An AQL kernel dispatch packet is created on the AQL queue. The ROCm runtime 2881 api uses 64-bit atomic operations to reserve space in the AQL queue for the 2882 packet. The packet must be set up, and the final write must use an atomic 2883 store release to set the packet kind to ensure the packet contents are 2884 visible to the kernel agent. AQL defines a doorbell signal mechanism to 2885 notify the kernel agent that the AQL queue has been updated. These rules, and 2886 the layout of the AQL queue and kernel dispatch packet is defined in the *HSA 2887 System Architecture Specification* [HSA]_. 28886. A kernel dispatch packet includes information about the actual dispatch, 2889 such as grid and work-group size, together with information from the code 2890 object about the kernel, such as segment sizes. The ROCm runtime queries on 2891 the kernel symbol can be used to obtain the code object values which are 2892 recorded in the :ref:`amdgpu-amdhsa-code-object-metadata`. 28937. CP executes micro-code and is responsible for detecting and setting up the 2894 GPU to execute the wavefronts of a kernel dispatch. 28958. CP ensures that when the a wavefront starts executing the kernel machine 2896 code, the scalar general purpose registers (SGPR) and vector general purpose 2897 registers (VGPR) are set up as required by the machine code. The required 2898 setup is defined in the :ref:`amdgpu-amdhsa-kernel-descriptor`. The initial 2899 register state is defined in 2900 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`. 29019. The prolog of the kernel machine code (see 2902 :ref:`amdgpu-amdhsa-kernel-prolog`) sets up the machine state as necessary 2903 before continuing executing the machine code that corresponds to the kernel. 290410. When the kernel dispatch has completed execution, CP signals the completion 2905 signal specified in the kernel dispatch packet if not 0. 2906 2907Image and Samplers 2908~~~~~~~~~~~~~~~~~~ 2909 2910Image and sample handles created by the ROCm runtime are 64-bit addresses of a 2911hardware 32-byte V# and 48 byte S# object respectively. In order to support the 2912HSA ``query_sampler`` operations two extra dwords are used to store the HSA BRIG 2913enumeration values for the queries that are not trivially deducible from the S# 2914representation. 2915 2916HSA Signals 2917~~~~~~~~~~~ 2918 2919HSA signal handles created by the ROCm runtime are 64-bit addresses of a 2920structure allocated in memory accessible from both the CPU and GPU. The 2921structure is defined by the ROCm runtime and subject to change between releases 2922(see [AMD-ROCm-github]_). 2923 2924.. _amdgpu-amdhsa-hsa-aql-queue: 2925 2926HSA AQL Queue 2927~~~~~~~~~~~~~ 2928 2929The HSA AQL queue structure is defined by the ROCm runtime and subject to change 2930between releases (see [AMD-ROCm-github]_). For some processors it contains 2931fields needed to implement certain language features such as the flat address 2932aperture bases. It also contains fields used by CP such as managing the 2933allocation of scratch memory. 2934 2935.. _amdgpu-amdhsa-kernel-descriptor: 2936 2937Kernel Descriptor 2938~~~~~~~~~~~~~~~~~ 2939 2940A kernel descriptor consists of the information needed by CP to initiate the 2941execution of a kernel, including the entry point address of the machine code 2942that implements the kernel. 2943 2944Kernel Descriptor for GFX6-GFX10 2945++++++++++++++++++++++++++++++++ 2946 2947CP microcode requires the Kernel descriptor to be allocated on 64-byte 2948alignment. 2949 2950 .. table:: Kernel Descriptor for GFX6-GFX10 2951 :name: amdgpu-amdhsa-kernel-descriptor-gfx6-gfx10-table 2952 2953 ======= ======= =============================== ============================ 2954 Bits Size Field Name Description 2955 ======= ======= =============================== ============================ 2956 31:0 4 bytes GROUP_SEGMENT_FIXED_SIZE The amount of fixed local 2957 address space memory 2958 required for a work-group 2959 in bytes. This does not 2960 include any dynamically 2961 allocated local address 2962 space memory that may be 2963 added when the kernel is 2964 dispatched. 2965 63:32 4 bytes PRIVATE_SEGMENT_FIXED_SIZE The amount of fixed 2966 private address space 2967 memory required for a 2968 work-item in bytes. If 2969 is_dynamic_callstack is 1 2970 then additional space must 2971 be added to this value for 2972 the call stack. 2973 127:64 8 bytes Reserved, must be 0. 2974 191:128 8 bytes KERNEL_CODE_ENTRY_BYTE_OFFSET Byte offset (possibly 2975 negative) from base 2976 address of kernel 2977 descriptor to kernel's 2978 entry point instruction 2979 which must be 256 byte 2980 aligned. 2981 351:272 20 Reserved, must be 0. 2982 bytes 2983 383:352 4 bytes COMPUTE_PGM_RSRC3 GFX6-9 2984 Reserved, must be 0. 2985 GFX10 2986 Compute Shader (CS) 2987 program settings used by 2988 CP to set up 2989 ``COMPUTE_PGM_RSRC3`` 2990 configuration 2991 register. See 2992 :ref:`amdgpu-amdhsa-compute_pgm_rsrc3-gfx10-table`. 2993 415:384 4 bytes COMPUTE_PGM_RSRC1 Compute Shader (CS) 2994 program settings used by 2995 CP to set up 2996 ``COMPUTE_PGM_RSRC1`` 2997 configuration 2998 register. See 2999 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 3000 447:416 4 bytes COMPUTE_PGM_RSRC2 Compute Shader (CS) 3001 program settings used by 3002 CP to set up 3003 ``COMPUTE_PGM_RSRC2`` 3004 configuration 3005 register. See 3006 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 3007 448 1 bit ENABLE_SGPR_PRIVATE_SEGMENT Enable the setup of the 3008 _BUFFER SGPR user data registers 3009 (see 3010 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 3011 3012 The total number of SGPR 3013 user data registers 3014 requested must not exceed 3015 16 and match value in 3016 ``compute_pgm_rsrc2.user_sgpr.user_sgpr_count``. 3017 Any requests beyond 16 3018 will be ignored. 3019 449 1 bit ENABLE_SGPR_DISPATCH_PTR *see above* 3020 450 1 bit ENABLE_SGPR_QUEUE_PTR *see above* 3021 451 1 bit ENABLE_SGPR_KERNARG_SEGMENT_PTR *see above* 3022 452 1 bit ENABLE_SGPR_DISPATCH_ID *see above* 3023 453 1 bit ENABLE_SGPR_FLAT_SCRATCH_INIT *see above* 3024 454 1 bit ENABLE_SGPR_PRIVATE_SEGMENT *see above* 3025 _SIZE 3026 457:455 3 bits Reserved, must be 0. 3027 458 1 bit ENABLE_WAVEFRONT_SIZE32 GFX6-9 3028 Reserved, must be 0. 3029 GFX10 3030 - If 0 execute in 3031 wavefront size 64 mode. 3032 - If 1 execute in 3033 native wavefront size 3034 32 mode. 3035 463:459 5 bits Reserved, must be 0. 3036 511:464 6 bytes Reserved, must be 0. 3037 512 **Total size 64 bytes.** 3038 ======= ==================================================================== 3039 3040.. 3041 3042 .. table:: compute_pgm_rsrc1 for GFX6-GFX10 3043 :name: amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table 3044 3045 ======= ======= =============================== =========================================================================== 3046 Bits Size Field Name Description 3047 ======= ======= =============================== =========================================================================== 3048 5:0 6 bits GRANULATED_WORKITEM_VGPR_COUNT Number of vector register 3049 blocks used by each work-item; 3050 granularity is device 3051 specific: 3052 3053 GFX6-GFX9 3054 - vgprs_used 0..256 3055 - max(0, ceil(vgprs_used / 4) - 1) 3056 GFX10 (wavefront size 64) 3057 - max_vgpr 1..256 3058 - max(0, ceil(vgprs_used / 4) - 1) 3059 GFX10 (wavefront size 32) 3060 - max_vgpr 1..256 3061 - max(0, ceil(vgprs_used / 8) - 1) 3062 3063 Where vgprs_used is defined 3064 as the highest VGPR number 3065 explicitly referenced plus 3066 one. 3067 3068 Used by CP to set up 3069 ``COMPUTE_PGM_RSRC1.VGPRS``. 3070 3071 The 3072 :ref:`amdgpu-assembler` 3073 calculates this 3074 automatically for the 3075 selected processor from 3076 values provided to the 3077 `.amdhsa_kernel` directive 3078 by the 3079 `.amdhsa_next_free_vgpr` 3080 nested directive (see 3081 :ref:`amdhsa-kernel-directives-table`). 3082 9:6 4 bits GRANULATED_WAVEFRONT_SGPR_COUNT Number of scalar register 3083 blocks used by a wavefront; 3084 granularity is device 3085 specific: 3086 3087 GFX6-GFX8 3088 - sgprs_used 0..112 3089 - max(0, ceil(sgprs_used / 8) - 1) 3090 GFX9 3091 - sgprs_used 0..112 3092 - 2 * max(0, ceil(sgprs_used / 16) - 1) 3093 GFX10 3094 Reserved, must be 0. 3095 (128 SGPRs always 3096 allocated.) 3097 3098 Where sgprs_used is 3099 defined as the highest 3100 SGPR number explicitly 3101 referenced plus one, plus 3102 a target specific number 3103 of additional special 3104 SGPRs for VCC, 3105 FLAT_SCRATCH (GFX7+) and 3106 XNACK_MASK (GFX8+), and 3107 any additional 3108 target specific 3109 limitations. It does not 3110 include the 16 SGPRs added 3111 if a trap handler is 3112 enabled. 3113 3114 The target specific 3115 limitations and special 3116 SGPR layout are defined in 3117 the hardware 3118 documentation, which can 3119 be found in the 3120 :ref:`amdgpu-processors` 3121 table. 3122 3123 Used by CP to set up 3124 ``COMPUTE_PGM_RSRC1.SGPRS``. 3125 3126 The 3127 :ref:`amdgpu-assembler` 3128 calculates this 3129 automatically for the 3130 selected processor from 3131 values provided to the 3132 `.amdhsa_kernel` directive 3133 by the 3134 `.amdhsa_next_free_sgpr` 3135 and `.amdhsa_reserve_*` 3136 nested directives (see 3137 :ref:`amdhsa-kernel-directives-table`). 3138 11:10 2 bits PRIORITY Must be 0. 3139 3140 Start executing wavefront 3141 at the specified priority. 3142 3143 CP is responsible for 3144 filling in 3145 ``COMPUTE_PGM_RSRC1.PRIORITY``. 3146 13:12 2 bits FLOAT_ROUND_MODE_32 Wavefront starts execution 3147 with specified rounding 3148 mode for single (32 3149 bit) floating point 3150 precision floating point 3151 operations. 3152 3153 Floating point rounding 3154 mode values are defined in 3155 :ref:`amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table`. 3156 3157 Used by CP to set up 3158 ``COMPUTE_PGM_RSRC1.FLOAT_MODE``. 3159 15:14 2 bits FLOAT_ROUND_MODE_16_64 Wavefront starts execution 3160 with specified rounding 3161 denorm mode for half/double (16 3162 and 64-bit) floating point 3163 precision floating point 3164 operations. 3165 3166 Floating point rounding 3167 mode values are defined in 3168 :ref:`amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table`. 3169 3170 Used by CP to set up 3171 ``COMPUTE_PGM_RSRC1.FLOAT_MODE``. 3172 17:16 2 bits FLOAT_DENORM_MODE_32 Wavefront starts execution 3173 with specified denorm mode 3174 for single (32 3175 bit) floating point 3176 precision floating point 3177 operations. 3178 3179 Floating point denorm mode 3180 values are defined in 3181 :ref:`amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table`. 3182 3183 Used by CP to set up 3184 ``COMPUTE_PGM_RSRC1.FLOAT_MODE``. 3185 19:18 2 bits FLOAT_DENORM_MODE_16_64 Wavefront starts execution 3186 with specified denorm mode 3187 for half/double (16 3188 and 64-bit) floating point 3189 precision floating point 3190 operations. 3191 3192 Floating point denorm mode 3193 values are defined in 3194 :ref:`amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table`. 3195 3196 Used by CP to set up 3197 ``COMPUTE_PGM_RSRC1.FLOAT_MODE``. 3198 20 1 bit PRIV Must be 0. 3199 3200 Start executing wavefront 3201 in privilege trap handler 3202 mode. 3203 3204 CP is responsible for 3205 filling in 3206 ``COMPUTE_PGM_RSRC1.PRIV``. 3207 21 1 bit ENABLE_DX10_CLAMP Wavefront starts execution 3208 with DX10 clamp mode 3209 enabled. Used by the vector 3210 ALU to force DX10 style 3211 treatment of NaN's (when 3212 set, clamp NaN to zero, 3213 otherwise pass NaN 3214 through). 3215 3216 Used by CP to set up 3217 ``COMPUTE_PGM_RSRC1.DX10_CLAMP``. 3218 22 1 bit DEBUG_MODE Must be 0. 3219 3220 Start executing wavefront 3221 in single step mode. 3222 3223 CP is responsible for 3224 filling in 3225 ``COMPUTE_PGM_RSRC1.DEBUG_MODE``. 3226 23 1 bit ENABLE_IEEE_MODE Wavefront starts execution 3227 with IEEE mode 3228 enabled. Floating point 3229 opcodes that support 3230 exception flag gathering 3231 will quiet and propagate 3232 signaling-NaN inputs per 3233 IEEE 754-2008. Min_dx10 and 3234 max_dx10 become IEEE 3235 754-2008 compliant due to 3236 signaling-NaN propagation 3237 and quieting. 3238 3239 Used by CP to set up 3240 ``COMPUTE_PGM_RSRC1.IEEE_MODE``. 3241 24 1 bit BULKY Must be 0. 3242 3243 Only one work-group allowed 3244 to execute on a compute 3245 unit. 3246 3247 CP is responsible for 3248 filling in 3249 ``COMPUTE_PGM_RSRC1.BULKY``. 3250 25 1 bit CDBG_USER Must be 0. 3251 3252 Flag that can be used to 3253 control debugging code. 3254 3255 CP is responsible for 3256 filling in 3257 ``COMPUTE_PGM_RSRC1.CDBG_USER``. 3258 26 1 bit FP16_OVFL GFX6-GFX8 3259 Reserved, must be 0. 3260 GFX9-GFX10 3261 Wavefront starts execution 3262 with specified fp16 overflow 3263 mode. 3264 3265 - If 0, fp16 overflow generates 3266 +/-INF values. 3267 - If 1, fp16 overflow that is the 3268 result of an +/-INF input value 3269 or divide by 0 produces a +/-INF, 3270 otherwise clamps computed 3271 overflow to +/-MAX_FP16 as 3272 appropriate. 3273 3274 Used by CP to set up 3275 ``COMPUTE_PGM_RSRC1.FP16_OVFL``. 3276 28:27 2 bits Reserved, must be 0. 3277 29 1 bit WGP_MODE GFX6-GFX9 3278 Reserved, must be 0. 3279 GFX10 3280 - If 0 execute work-groups in 3281 CU wavefront execution mode. 3282 - If 1 execute work-groups on 3283 in WGP wavefront execution mode. 3284 3285 See :ref:`amdgpu-amdhsa-memory-model`. 3286 3287 Used by CP to set up 3288 ``COMPUTE_PGM_RSRC1.WGP_MODE``. 3289 30 1 bit MEM_ORDERED GFX6-9 3290 Reserved, must be 0. 3291 GFX10 3292 Controls the behavior of the 3293 waitcnt's vmcnt and vscnt 3294 counters. 3295 3296 - If 0 vmcnt reports completion 3297 of load and atomic with return 3298 out of order with sample 3299 instructions, and the vscnt 3300 reports the completion of 3301 store and atomic without 3302 return in order. 3303 - If 1 vmcnt reports completion 3304 of load, atomic with return 3305 and sample instructions in 3306 order, and the vscnt reports 3307 the completion of store and 3308 atomic without return in order. 3309 3310 Used by CP to set up 3311 ``COMPUTE_PGM_RSRC1.MEM_ORDERED``. 3312 31 1 bit FWD_PROGRESS GFX6-9 3313 Reserved, must be 0. 3314 GFX10 3315 - If 0 execute SIMD wavefronts 3316 using oldest first policy. 3317 - If 1 execute SIMD wavefronts to 3318 ensure wavefronts will make some 3319 forward progress. 3320 3321 Used by CP to set up 3322 ``COMPUTE_PGM_RSRC1.FWD_PROGRESS``. 3323 32 **Total size 4 bytes** 3324 ======= =================================================================================================================== 3325 3326.. 3327 3328 .. table:: compute_pgm_rsrc2 for GFX6-GFX10 3329 :name: amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table 3330 3331 ======= ======= =============================== =========================================================================== 3332 Bits Size Field Name Description 3333 ======= ======= =============================== =========================================================================== 3334 0 1 bit ENABLE_SGPR_PRIVATE_SEGMENT Enable the setup of the 3335 _WAVEFRONT_OFFSET SGPR wavefront scratch offset 3336 system register (see 3337 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 3338 3339 Used by CP to set up 3340 ``COMPUTE_PGM_RSRC2.SCRATCH_EN``. 3341 5:1 5 bits USER_SGPR_COUNT The total number of SGPR 3342 user data registers 3343 requested. This number must 3344 match the number of user 3345 data registers enabled. 3346 3347 Used by CP to set up 3348 ``COMPUTE_PGM_RSRC2.USER_SGPR``. 3349 6 1 bit ENABLE_TRAP_HANDLER Must be 0. 3350 3351 This bit represents 3352 ``COMPUTE_PGM_RSRC2.TRAP_PRESENT``, 3353 which is set by the CP if 3354 the runtime has installed a 3355 trap handler. 3356 7 1 bit ENABLE_SGPR_WORKGROUP_ID_X Enable the setup of the 3357 system SGPR register for 3358 the work-group id in the X 3359 dimension (see 3360 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 3361 3362 Used by CP to set up 3363 ``COMPUTE_PGM_RSRC2.TGID_X_EN``. 3364 8 1 bit ENABLE_SGPR_WORKGROUP_ID_Y Enable the setup of the 3365 system SGPR register for 3366 the work-group id in the Y 3367 dimension (see 3368 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 3369 3370 Used by CP to set up 3371 ``COMPUTE_PGM_RSRC2.TGID_Y_EN``. 3372 9 1 bit ENABLE_SGPR_WORKGROUP_ID_Z Enable the setup of the 3373 system SGPR register for 3374 the work-group id in the Z 3375 dimension (see 3376 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 3377 3378 Used by CP to set up 3379 ``COMPUTE_PGM_RSRC2.TGID_Z_EN``. 3380 10 1 bit ENABLE_SGPR_WORKGROUP_INFO Enable the setup of the 3381 system SGPR register for 3382 work-group information (see 3383 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 3384 3385 Used by CP to set up 3386 ``COMPUTE_PGM_RSRC2.TGID_SIZE_EN``. 3387 12:11 2 bits ENABLE_VGPR_WORKITEM_ID Enable the setup of the 3388 VGPR system registers used 3389 for the work-item ID. 3390 :ref:`amdgpu-amdhsa-system-vgpr-work-item-id-enumeration-values-table` 3391 defines the values. 3392 3393 Used by CP to set up 3394 ``COMPUTE_PGM_RSRC2.TIDIG_CMP_CNT``. 3395 13 1 bit ENABLE_EXCEPTION_ADDRESS_WATCH Must be 0. 3396 3397 Wavefront starts execution 3398 with address watch 3399 exceptions enabled which 3400 are generated when L1 has 3401 witnessed a thread access 3402 an *address of 3403 interest*. 3404 3405 CP is responsible for 3406 filling in the address 3407 watch bit in 3408 ``COMPUTE_PGM_RSRC2.EXCP_EN_MSB`` 3409 according to what the 3410 runtime requests. 3411 14 1 bit ENABLE_EXCEPTION_MEMORY Must be 0. 3412 3413 Wavefront starts execution 3414 with memory violation 3415 exceptions exceptions 3416 enabled which are generated 3417 when a memory violation has 3418 occurred for this wavefront from 3419 L1 or LDS 3420 (write-to-read-only-memory, 3421 mis-aligned atomic, LDS 3422 address out of range, 3423 illegal address, etc.). 3424 3425 CP sets the memory 3426 violation bit in 3427 ``COMPUTE_PGM_RSRC2.EXCP_EN_MSB`` 3428 according to what the 3429 runtime requests. 3430 23:15 9 bits GRANULATED_LDS_SIZE Must be 0. 3431 3432 CP uses the rounded value 3433 from the dispatch packet, 3434 not this value, as the 3435 dispatch may contain 3436 dynamically allocated group 3437 segment memory. CP writes 3438 directly to 3439 ``COMPUTE_PGM_RSRC2.LDS_SIZE``. 3440 3441 Amount of group segment 3442 (LDS) to allocate for each 3443 work-group. Granularity is 3444 device specific: 3445 3446 GFX6: 3447 roundup(lds-size / (64 * 4)) 3448 GFX7-GFX10: 3449 roundup(lds-size / (128 * 4)) 3450 3451 24 1 bit ENABLE_EXCEPTION_IEEE_754_FP Wavefront starts execution 3452 _INVALID_OPERATION with specified exceptions 3453 enabled. 3454 3455 Used by CP to set up 3456 ``COMPUTE_PGM_RSRC2.EXCP_EN`` 3457 (set from bits 0..6). 3458 3459 IEEE 754 FP Invalid 3460 Operation 3461 25 1 bit ENABLE_EXCEPTION_FP_DENORMAL FP Denormal one or more 3462 _SOURCE input operands is a 3463 denormal number 3464 26 1 bit ENABLE_EXCEPTION_IEEE_754_FP IEEE 754 FP Division by 3465 _DIVISION_BY_ZERO Zero 3466 27 1 bit ENABLE_EXCEPTION_IEEE_754_FP IEEE 754 FP FP Overflow 3467 _OVERFLOW 3468 28 1 bit ENABLE_EXCEPTION_IEEE_754_FP IEEE 754 FP Underflow 3469 _UNDERFLOW 3470 29 1 bit ENABLE_EXCEPTION_IEEE_754_FP IEEE 754 FP Inexact 3471 _INEXACT 3472 30 1 bit ENABLE_EXCEPTION_INT_DIVIDE_BY Integer Division by Zero 3473 _ZERO (rcp_iflag_f32 instruction 3474 only) 3475 31 1 bit Reserved, must be 0. 3476 32 **Total size 4 bytes.** 3477 ======= =================================================================================================================== 3478 3479.. 3480 3481 .. table:: compute_pgm_rsrc3 for GFX10 3482 :name: amdgpu-amdhsa-compute_pgm_rsrc3-gfx10-table 3483 3484 ======= ======= =============================== =========================================================================== 3485 Bits Size Field Name Description 3486 ======= ======= =============================== =========================================================================== 3487 3:0 4 bits SHARED_VGPR_COUNT Number of shared VGPRs for wavefront size 64. Granularity 8. Value 0-120. 3488 compute_pgm_rsrc1.vgprs + shared_vgpr_cnt cannot exceed 64. 3489 31:4 28 Reserved, must be 0. 3490 bits 3491 32 **Total size 4 bytes.** 3492 ======= =================================================================================================================== 3493 3494.. 3495 3496 .. table:: Floating Point Rounding Mode Enumeration Values 3497 :name: amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table 3498 3499 ====================================== ===== ============================== 3500 Enumeration Name Value Description 3501 ====================================== ===== ============================== 3502 FLOAT_ROUND_MODE_NEAR_EVEN 0 Round Ties To Even 3503 FLOAT_ROUND_MODE_PLUS_INFINITY 1 Round Toward +infinity 3504 FLOAT_ROUND_MODE_MINUS_INFINITY 2 Round Toward -infinity 3505 FLOAT_ROUND_MODE_ZERO 3 Round Toward 0 3506 ====================================== ===== ============================== 3507 3508.. 3509 3510 .. table:: Floating Point Denorm Mode Enumeration Values 3511 :name: amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table 3512 3513 ====================================== ===== ============================== 3514 Enumeration Name Value Description 3515 ====================================== ===== ============================== 3516 FLOAT_DENORM_MODE_FLUSH_SRC_DST 0 Flush Source and Destination 3517 Denorms 3518 FLOAT_DENORM_MODE_FLUSH_DST 1 Flush Output Denorms 3519 FLOAT_DENORM_MODE_FLUSH_SRC 2 Flush Source Denorms 3520 FLOAT_DENORM_MODE_FLUSH_NONE 3 No Flush 3521 ====================================== ===== ============================== 3522 3523.. 3524 3525 .. table:: System VGPR Work-Item ID Enumeration Values 3526 :name: amdgpu-amdhsa-system-vgpr-work-item-id-enumeration-values-table 3527 3528 ======================================== ===== ============================ 3529 Enumeration Name Value Description 3530 ======================================== ===== ============================ 3531 SYSTEM_VGPR_WORKITEM_ID_X 0 Set work-item X dimension 3532 ID. 3533 SYSTEM_VGPR_WORKITEM_ID_X_Y 1 Set work-item X and Y 3534 dimensions ID. 3535 SYSTEM_VGPR_WORKITEM_ID_X_Y_Z 2 Set work-item X, Y and Z 3536 dimensions ID. 3537 SYSTEM_VGPR_WORKITEM_ID_UNDEFINED 3 Undefined. 3538 ======================================== ===== ============================ 3539 3540.. _amdgpu-amdhsa-initial-kernel-execution-state: 3541 3542Initial Kernel Execution State 3543~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 3544 3545This section defines the register state that will be set up by the packet 3546processor prior to the start of execution of every wavefront. This is limited by 3547the constraints of the hardware controllers of CP/ADC/SPI. 3548 3549The order of the SGPR registers is defined, but the compiler can specify which 3550ones are actually setup in the kernel descriptor using the ``enable_sgpr_*`` bit 3551fields (see :ref:`amdgpu-amdhsa-kernel-descriptor`). The register numbers used 3552for enabled registers are dense starting at SGPR0: the first enabled register is 3553SGPR0, the next enabled register is SGPR1 etc.; disabled registers do not have 3554an SGPR number. 3555 3556The initial SGPRs comprise up to 16 User SRGPs that are set by CP and apply to 3557all wavefronts of the grid. It is possible to specify more than 16 User SGPRs 3558using the ``enable_sgpr_*`` bit fields, in which case only the first 16 are 3559actually initialized. These are then immediately followed by the System SGPRs 3560that are set up by ADC/SPI and can have different values for each wavefront of 3561the grid dispatch. 3562 3563SGPR register initial state is defined in 3564:ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 3565 3566 .. table:: SGPR Register Set Up Order 3567 :name: amdgpu-amdhsa-sgpr-register-set-up-order-table 3568 3569 ========== ========================== ====== ============================== 3570 SGPR Order Name Number Description 3571 (kernel descriptor enable of 3572 field) SGPRs 3573 ========== ========================== ====== ============================== 3574 First Private Segment Buffer 4 V# that can be used, together 3575 (enable_sgpr_private with Scratch Wavefront Offset 3576 _segment_buffer) as an offset, to access the 3577 private address space using a 3578 segment address. 3579 3580 CP uses the value provided by 3581 the runtime. 3582 then Dispatch Ptr 2 64-bit address of AQL dispatch 3583 (enable_sgpr_dispatch_ptr) packet for kernel dispatch 3584 actually executing. 3585 then Queue Ptr 2 64-bit address of amd_queue_t 3586 (enable_sgpr_queue_ptr) object for AQL queue on which 3587 the dispatch packet was 3588 queued. 3589 then Kernarg Segment Ptr 2 64-bit address of Kernarg 3590 (enable_sgpr_kernarg segment. This is directly 3591 _segment_ptr) copied from the 3592 kernarg_address in the kernel 3593 dispatch packet. 3594 3595 Having CP load it once avoids 3596 loading it at the beginning of 3597 every wavefront. 3598 then Dispatch Id 2 64-bit Dispatch ID of the 3599 (enable_sgpr_dispatch_id) dispatch packet being 3600 executed. 3601 then Flat Scratch Init 2 This is 2 SGPRs: 3602 (enable_sgpr_flat_scratch 3603 _init) GFX6 3604 Not supported. 3605 GFX7-GFX8 3606 The first SGPR is a 32-bit 3607 byte offset from 3608 ``SH_HIDDEN_PRIVATE_BASE_VIMID`` 3609 to per SPI base of memory 3610 for scratch for the queue 3611 executing the kernel 3612 dispatch. CP obtains this 3613 from the runtime. (The 3614 Scratch Segment Buffer base 3615 address is 3616 ``SH_HIDDEN_PRIVATE_BASE_VIMID`` 3617 plus this offset.) The value 3618 of Scratch Wavefront Offset must 3619 be added to this offset by 3620 the kernel machine code, 3621 right shifted by 8, and 3622 moved to the FLAT_SCRATCH_HI 3623 SGPR register. 3624 FLAT_SCRATCH_HI corresponds 3625 to SGPRn-4 on GFX7, and 3626 SGPRn-6 on GFX8 (where SGPRn 3627 is the highest numbered SGPR 3628 allocated to the wavefront). 3629 FLAT_SCRATCH_HI is 3630 multiplied by 256 (as it is 3631 in units of 256 bytes) and 3632 added to 3633 ``SH_HIDDEN_PRIVATE_BASE_VIMID`` 3634 to calculate the per wavefront 3635 FLAT SCRATCH BASE in flat 3636 memory instructions that 3637 access the scratch 3638 aperture. 3639 3640 The second SGPR is 32-bit 3641 byte size of a single 3642 work-item's scratch memory 3643 usage. CP obtains this from 3644 the runtime, and it is 3645 always a multiple of DWORD. 3646 CP checks that the value in 3647 the kernel dispatch packet 3648 Private Segment Byte Size is 3649 not larger and requests the 3650 runtime to increase the 3651 queue's scratch size if 3652 necessary. The kernel code 3653 must move it to 3654 FLAT_SCRATCH_LO which is 3655 SGPRn-3 on GFX7 and SGPRn-5 3656 on GFX8. FLAT_SCRATCH_LO is 3657 used as the FLAT SCRATCH 3658 SIZE in flat memory 3659 instructions. Having CP load 3660 it once avoids loading it at 3661 the beginning of every 3662 wavefront. 3663 GFX9-GFX10 3664 This is the 3665 64-bit base address of the 3666 per SPI scratch backing 3667 memory managed by SPI for 3668 the queue executing the 3669 kernel dispatch. CP obtains 3670 this from the runtime (and 3671 divides it if there are 3672 multiple Shader Arrays each 3673 with its own SPI). The value 3674 of Scratch Wavefront Offset must 3675 be added by the kernel 3676 machine code and the result 3677 moved to the FLAT_SCRATCH 3678 SGPR which is SGPRn-6 and 3679 SGPRn-5. It is used as the 3680 FLAT SCRATCH BASE in flat 3681 memory instructions. 3682 then Private Segment Size 1 The 32-bit byte size of a 3683 (enable_sgpr_private single 3684 work-item's 3685 scratch_segment_size) memory 3686 allocation. This is the 3687 value from the kernel 3688 dispatch packet Private 3689 Segment Byte Size rounded up 3690 by CP to a multiple of 3691 DWORD. 3692 3693 Having CP load it once avoids 3694 loading it at the beginning of 3695 every wavefront. 3696 3697 This is not used for 3698 GFX7-GFX8 since it is the same 3699 value as the second SGPR of 3700 Flat Scratch Init. However, it 3701 may be needed for GFX9-GFX10 which 3702 changes the meaning of the 3703 Flat Scratch Init value. 3704 then Grid Work-Group Count X 1 32-bit count of the number of 3705 (enable_sgpr_grid work-groups in the X dimension 3706 _workgroup_count_X) for the grid being 3707 executed. Computed from the 3708 fields in the kernel dispatch 3709 packet as ((grid_size.x + 3710 workgroup_size.x - 1) / 3711 workgroup_size.x). 3712 then Grid Work-Group Count Y 1 32-bit count of the number of 3713 (enable_sgpr_grid work-groups in the Y dimension 3714 _workgroup_count_Y && for the grid being 3715 less than 16 previous executed. Computed from the 3716 SGPRs) fields in the kernel dispatch 3717 packet as ((grid_size.y + 3718 workgroup_size.y - 1) / 3719 workgroupSize.y). 3720 3721 Only initialized if <16 3722 previous SGPRs initialized. 3723 then Grid Work-Group Count Z 1 32-bit count of the number of 3724 (enable_sgpr_grid work-groups in the Z dimension 3725 _workgroup_count_Z && for the grid being 3726 less than 16 previous executed. Computed from the 3727 SGPRs) fields in the kernel dispatch 3728 packet as ((grid_size.z + 3729 workgroup_size.z - 1) / 3730 workgroupSize.z). 3731 3732 Only initialized if <16 3733 previous SGPRs initialized. 3734 then Work-Group Id X 1 32-bit work-group id in X 3735 (enable_sgpr_workgroup_id dimension of grid for 3736 _X) wavefront. 3737 then Work-Group Id Y 1 32-bit work-group id in Y 3738 (enable_sgpr_workgroup_id dimension of grid for 3739 _Y) wavefront. 3740 then Work-Group Id Z 1 32-bit work-group id in Z 3741 (enable_sgpr_workgroup_id dimension of grid for 3742 _Z) wavefront. 3743 then Work-Group Info 1 {first_wavefront, 14'b0000, 3744 (enable_sgpr_workgroup ordered_append_term[10:0], 3745 _info) threadgroup_size_in_wavefronts[5:0]} 3746 then Scratch Wavefront Offset 1 32-bit byte offset from base 3747 (enable_sgpr_private of scratch base of queue 3748 _segment_wavefront_offset) executing the kernel 3749 dispatch. Must be used as an 3750 offset with Private 3751 segment address when using 3752 Scratch Segment Buffer. It 3753 must be used to set up FLAT 3754 SCRATCH for flat addressing 3755 (see 3756 :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`). 3757 ========== ========================== ====== ============================== 3758 3759The order of the VGPR registers is defined, but the compiler can specify which 3760ones are actually setup in the kernel descriptor using the ``enable_vgpr*`` bit 3761fields (see :ref:`amdgpu-amdhsa-kernel-descriptor`). The register numbers used 3762for enabled registers are dense starting at VGPR0: the first enabled register is 3763VGPR0, the next enabled register is VGPR1 etc.; disabled registers do not have a 3764VGPR number. 3765 3766VGPR register initial state is defined in 3767:ref:`amdgpu-amdhsa-vgpr-register-set-up-order-table`. 3768 3769 .. table:: VGPR Register Set Up Order 3770 :name: amdgpu-amdhsa-vgpr-register-set-up-order-table 3771 3772 ========== ========================== ====== ============================== 3773 VGPR Order Name Number Description 3774 (kernel descriptor enable of 3775 field) VGPRs 3776 ========== ========================== ====== ============================== 3777 First Work-Item Id X 1 32-bit work item id in X 3778 (Always initialized) dimension of work-group for 3779 wavefront lane. 3780 then Work-Item Id Y 1 32-bit work item id in Y 3781 (enable_vgpr_workitem_id dimension of work-group for 3782 > 0) wavefront lane. 3783 then Work-Item Id Z 1 32-bit work item id in Z 3784 (enable_vgpr_workitem_id dimension of work-group for 3785 > 1) wavefront lane. 3786 ========== ========================== ====== ============================== 3787 3788The setting of registers is done by GPU CP/ADC/SPI hardware as follows: 3789 37901. SGPRs before the Work-Group Ids are set by CP using the 16 User Data 3791 registers. 37922. Work-group Id registers X, Y, Z are set by ADC which supports any 3793 combination including none. 37943. Scratch Wavefront Offset is set by SPI in a per wavefront basis which is why 3795 its value cannot be included with the flat scratch init value which is per 3796 queue. 37974. The VGPRs are set by SPI which only supports specifying either (X), (X, Y) 3798 or (X, Y, Z). 3799 3800Flat Scratch register pair are adjacent SGRRs so they can be moved as a 64-bit 3801value to the hardware required SGPRn-3 and SGPRn-4 respectively. 3802 3803The global segment can be accessed either using buffer instructions (GFX6 which 3804has V# 64-bit address support), flat instructions (GFX7-GFX10), or global 3805instructions (GFX9-GFX10). 3806 3807If buffer operations are used, then the compiler can generate a V# with the 3808following properties: 3809 3810* base address of 0 3811* no swizzle 3812* ATC: 1 if IOMMU present (such as APU) 3813* ptr64: 1 3814* MTYPE set to support memory coherence that matches the runtime (such as CC for 3815 APU and NC for dGPU). 3816 3817.. _amdgpu-amdhsa-kernel-prolog: 3818 3819Kernel Prolog 3820~~~~~~~~~~~~~ 3821 3822The compiler performs initialization in the kernel prologue depending on the 3823target and information about things like stack usage in the kernel and called 3824functions. Some of this initialization requires the compiler to request certain 3825User and System SGPRs be present in the 3826:ref:`amdgpu-amdhsa-initial-kernel-execution-state` via the 3827:ref:`amdgpu-amdhsa-kernel-descriptor`. 3828 3829.. _amdgpu-amdhsa-kernel-prolog-cfi: 3830 3831CFI 3832+++ 3833 38341. The CFI return address is undefined. 3835 38362. The CFI CFA is defined using an expression which evaluates to a location 3837 description that comprises one memory location description for the 3838 ``DW_ASPACE_AMDGPU_private_lane`` address space address ``0``. 3839 3840.. _amdgpu-amdhsa-kernel-prolog-m0: 3841 3842M0 3843++ 3844 3845GFX6-GFX8 3846 The M0 register must be initialized with a value at least the total LDS size 3847 if the kernel may access LDS via DS or flat operations. Total LDS size is 3848 available in dispatch packet. For M0, it is also possible to use maximum 3849 possible value of LDS for given target (0x7FFF for GFX6 and 0xFFFF for 3850 GFX7-GFX8). 3851GFX9-GFX10 3852 The M0 register is not used for range checking LDS accesses and so does not 3853 need to be initialized in the prolog. 3854 3855.. _amdgpu-amdhsa-kernel-prolog-stack-pointer: 3856 3857Stack Pointer 3858+++++++++++++ 3859 3860If the kernel has function calls it must set up the ABI stack pointer described 3861in :ref:`amdgpu-amdhsa-function-call-convention-non-kernel-functions` by setting 3862SGPR32 to the unswizzled scratch offset of the address past the last local 3863allocation. 3864 3865.. _amdgpu-amdhsa-kernel-prolog-frame-pointer: 3866 3867Frame Pointer 3868+++++++++++++ 3869 3870If the kernel needs a frame pointer for the reasons defined in 3871``SIFrameLowering`` then SGPR33 is used and is always set to ``0`` in the 3872kernel prolog. If a frame pointer is not required then all uses of the frame 3873pointer are replaced with immediate ``0`` offsets. 3874 3875.. _amdgpu-amdhsa-kernel-prolog-flat-scratch: 3876 3877Flat Scratch 3878++++++++++++ 3879 3880If the kernel or any function it calls may use flat operations to access 3881scratch memory, the prolog code must set up the FLAT_SCRATCH register pair 3882(FLAT_SCRATCH_LO/FLAT_SCRATCH_HI which are in SGPRn-4/SGPRn-3). Initialization 3883uses Flat Scratch Init and Scratch Wavefront Offset SGPR registers (see 3884:ref:`amdgpu-amdhsa-initial-kernel-execution-state`): 3885 3886GFX6 3887 Flat scratch is not supported. 3888 3889GFX7-GFX8 3890 3891 1. The low word of Flat Scratch Init is 32-bit byte offset from 3892 ``SH_HIDDEN_PRIVATE_BASE_VIMID`` to the base of scratch backing memory 3893 being managed by SPI for the queue executing the kernel dispatch. This is 3894 the same value used in the Scratch Segment Buffer V# base address. The 3895 prolog must add the value of Scratch Wavefront Offset to get the 3896 wavefront's byte scratch backing memory offset from 3897 ``SH_HIDDEN_PRIVATE_BASE_VIMID``. Since FLAT_SCRATCH_LO is in units of 256 3898 bytes, the offset must be right shifted by 8 before moving into 3899 FLAT_SCRATCH_LO. 3900 2. The second word of Flat Scratch Init is 32-bit byte size of a single 3901 work-items scratch memory usage. This is directly loaded from the kernel 3902 dispatch packet Private Segment Byte Size and rounded up to a multiple of 3903 DWORD. Having CP load it once avoids loading it at the beginning of every 3904 wavefront. The prolog must move it to FLAT_SCRATCH_LO for use as FLAT 3905 SCRATCH SIZE. 3906 3907GFX9-GFX10 3908 The Flat Scratch Init is the 64-bit address of the base of scratch backing 3909 memory being managed by SPI for the queue executing the kernel dispatch. The 3910 prolog must add the value of Scratch Wavefront Offset and moved to the 3911 FLAT_SCRATCH pair for use as the flat scratch base in flat memory 3912 instructions. 3913 3914.. _amdgpu-amdhsa-kernel-prolog-private-segment-buffer: 3915 3916Private Segment Buffer 3917++++++++++++++++++++++ 3918 3919A set of four SGPRs beginning at a four-aligned SGPR index are always selected 3920to serve as the scratch V# for the kernel as follows: 3921 3922 - If it is known during instruction selection that there is stack usage, 3923 SGPR0-3 is reserved for use as the scratch V#. Stack usage is assumed if 3924 optimizations are disabled (``-O0``), if stack objects already exist (for 3925 locals, etc.), or if there are any function calls. 3926 3927 - Otherwise, four high numbered SGPRs beginning at a four-aligned SGPR index 3928 are reserved for the tentative scratch V#. These will be used if it is 3929 determined that spilling is needed. 3930 3931 - If no use is made of the tentative scratch V#, then it is unreserved, 3932 and the register count is determined ignoring it. 3933 - If use is made of the tentative scratch V#, then its register numbers 3934 are shifted to the first four-aligned SGPR index after the highest one 3935 allocated by the register allocator, and all uses are updated. The 3936 register count includes them in the shifted location. 3937 - In either case, if the processor has the SGPR allocation bug, the 3938 tentative allocation is not shifted or unreserved in order to ensure 3939 the register count is higher to workaround the bug. 3940 3941 .. note:: 3942 3943 This approach of using a tentative scratch V# and shifting the register 3944 numbers if used avoids having to perform register allocation a second 3945 time if the tentative V# is eliminated. This is more efficient and 3946 avoids the problem that the second register allocation may perform 3947 spilling which will fail as there is no longer a scratch V#. 3948 3949When the kernel prolog code is being emitted it is known whether the scratch V# 3950described above is actually used. If it is, the prolog code must set it up by 3951copying the Private Segment Buffer to the scratch V# registers and then adding 3952the Private Segment Wavefront Offset to the queue base address in the V#. The 3953result is a V# with a base address pointing to the beginning of the wavefront 3954scratch backing memory. 3955 3956The Private Segment Buffer is always requested, but the Private Segment 3957Wavefront Offset is only requested if it is used (see 3958:ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 3959 3960.. _amdgpu-amdhsa-memory-model: 3961 3962Memory Model 3963~~~~~~~~~~~~ 3964 3965This section describes the mapping of LLVM memory model onto AMDGPU machine code 3966(see :ref:`memmodel`). 3967 3968The AMDGPU backend supports the memory synchronization scopes specified in 3969:ref:`amdgpu-memory-scopes`. 3970 3971The code sequences used to implement the memory model are defined in table 3972:ref:`amdgpu-amdhsa-memory-model-code-sequences-gfx6-gfx10-table`. 3973 3974The sequences specify the order of instructions that a single thread must 3975execute. The ``s_waitcnt`` and ``buffer_wbinvl1_vol`` are defined with respect 3976to other memory instructions executed by the same thread. This allows them to be 3977moved earlier or later which can allow them to be combined with other instances 3978of the same instruction, or hoisted/sunk out of loops to improve 3979performance. Only the instructions related to the memory model are given; 3980additional ``s_waitcnt`` instructions are required to ensure registers are 3981defined before being used. These may be able to be combined with the memory 3982model ``s_waitcnt`` instructions as described above. 3983 3984The AMDGPU backend supports the following memory models: 3985 3986 HSA Memory Model [HSA]_ 3987 The HSA memory model uses a single happens-before relation for all address 3988 spaces (see :ref:`amdgpu-address-spaces`). 3989 OpenCL Memory Model [OpenCL]_ 3990 The OpenCL memory model which has separate happens-before relations for the 3991 global and local address spaces. Only a fence specifying both global and 3992 local address space, and seq_cst instructions join the relationships. Since 3993 the LLVM ``memfence`` instruction does not allow an address space to be 3994 specified the OpenCL fence has to conservatively assume both local and 3995 global address space was specified. However, optimizations can often be 3996 done to eliminate the additional ``s_waitcnt`` instructions when there are 3997 no intervening memory instructions which access the corresponding address 3998 space. The code sequences in the table indicate what can be omitted for the 3999 OpenCL memory. The target triple environment is used to determine if the 4000 source language is OpenCL (see :ref:`amdgpu-opencl`). 4001 4002``ds/flat_load/store/atomic`` instructions to local memory are termed LDS 4003operations. 4004 4005``buffer/global/flat_load/store/atomic`` instructions to global memory are 4006termed vector memory operations. 4007 4008For GFX6-GFX9: 4009 4010* Each agent has multiple shader arrays (SA). 4011* Each SA has multiple compute units (CU). 4012* Each CU has multiple SIMDs that execute wavefronts. 4013* The wavefronts for a single work-group are executed in the same CU but may be 4014 executed by different SIMDs. 4015* Each CU has a single LDS memory shared by the wavefronts of the work-groups 4016 executing on it. 4017* All LDS operations of a CU are performed as wavefront wide operations in a 4018 global order and involve no caching. Completion is reported to a wavefront in 4019 execution order. 4020* The LDS memory has multiple request queues shared by the SIMDs of a 4021 CU. Therefore, the LDS operations performed by different wavefronts of a 4022 work-group can be reordered relative to each other, which can result in 4023 reordering the visibility of vector memory operations with respect to LDS 4024 operations of other wavefronts in the same work-group. A ``s_waitcnt 4025 lgkmcnt(0)`` is required to ensure synchronization between LDS operations and 4026 vector memory operations between wavefronts of a work-group, but not between 4027 operations performed by the same wavefront. 4028* The vector memory operations are performed as wavefront wide operations and 4029 completion is reported to a wavefront in execution order. The exception is 4030 that for GFX7-GFX9 ``flat_load/store/atomic`` instructions can report out of 4031 vector memory order if they access LDS memory, and out of LDS operation order 4032 if they access global memory. 4033* The vector memory operations access a single vector L1 cache shared by all 4034 SIMDs a CU. Therefore, no special action is required for coherence between the 4035 lanes of a single wavefront, or for coherence between wavefronts in the same 4036 work-group. A ``buffer_wbinvl1_vol`` is required for coherence between 4037 wavefronts executing in different work-groups as they may be executing on 4038 different CUs. 4039* The scalar memory operations access a scalar L1 cache shared by all wavefronts 4040 on a group of CUs. The scalar and vector L1 caches are not coherent. However, 4041 scalar operations are used in a restricted way so do not impact the memory 4042 model. See :ref:`amdgpu-address-spaces`. 4043* The vector and scalar memory operations use an L2 cache shared by all CUs on 4044 the same agent. 4045* The L2 cache has independent channels to service disjoint ranges of virtual 4046 addresses. 4047* Each CU has a separate request queue per channel. Therefore, the vector and 4048 scalar memory operations performed by wavefronts executing in different 4049 work-groups (which may be executing on different CUs) of an agent can be 4050 reordered relative to each other. A ``s_waitcnt vmcnt(0)`` is required to 4051 ensure synchronization between vector memory operations of different CUs. It 4052 ensures a previous vector memory operation has completed before executing a 4053 subsequent vector memory or LDS operation and so can be used to meet the 4054 requirements of acquire and release. 4055* The L2 cache can be kept coherent with other agents on some targets, or ranges 4056 of virtual addresses can be set up to bypass it to ensure system coherence. 4057 4058For GFX10: 4059 4060* Each agent has multiple shader arrays (SA). 4061* Each SA has multiple work-group processors (WGP). 4062* Each WGP has multiple compute units (CU). 4063* Each CU has multiple SIMDs that execute wavefronts. 4064* The wavefronts for a single work-group are executed in the same 4065 WGP. In CU wavefront execution mode the wavefronts may be executed by 4066 different SIMDs in the same CU. In WGP wavefront execution mode the 4067 wavefronts may be executed by different SIMDs in different CUs in the same 4068 WGP. 4069* Each WGP has a single LDS memory shared by the wavefronts of the work-groups 4070 executing on it. 4071* All LDS operations of a WGP are performed as wavefront wide operations in a 4072 global order and involve no caching. Completion is reported to a wavefront in 4073 execution order. 4074* The LDS memory has multiple request queues shared by the SIMDs of a 4075 WGP. Therefore, the LDS operations performed by different wavefronts of a 4076 work-group can be reordered relative to each other, which can result in 4077 reordering the visibility of vector memory operations with respect to LDS 4078 operations of other wavefronts in the same work-group. A ``s_waitcnt 4079 lgkmcnt(0)`` is required to ensure synchronization between LDS operations and 4080 vector memory operations between wavefronts of a work-group, but not between 4081 operations performed by the same wavefront. 4082* The vector memory operations are performed as wavefront wide operations. 4083 Completion of load/store/sample operations are reported to a wavefront in 4084 execution order of other load/store/sample operations performed by that 4085 wavefront. 4086* The vector memory operations access a vector L0 cache. There is a single L0 4087 cache per CU. Each SIMD of a CU accesses the same L0 cache. Therefore, no 4088 special action is required for coherence between the lanes of a single 4089 wavefront. However, a ``BUFFER_GL0_INV`` is required for coherence between 4090 wavefronts executing in the same work-group as they may be executing on SIMDs 4091 of different CUs that access different L0s. A ``BUFFER_GL0_INV`` is also 4092 required for coherence between wavefronts executing in different work-groups 4093 as they may be executing on different WGPs. 4094* The scalar memory operations access a scalar L0 cache shared by all wavefronts 4095 on a WGP. The scalar and vector L0 caches are not coherent. However, scalar 4096 operations are used in a restricted way so do not impact the memory model. See 4097 :ref:`amdgpu-address-spaces`. 4098* The vector and scalar memory L0 caches use an L1 cache shared by all WGPs on 4099 the same SA. Therefore, no special action is required for coherence between 4100 the wavefronts of a single work-group. However, a ``BUFFER_GL1_INV`` is 4101 required for coherence between wavefronts executing in different work-groups 4102 as they may be executing on different SAs that access different L1s. 4103* The L1 caches have independent quadrants to service disjoint ranges of virtual 4104 addresses. 4105* Each L0 cache has a separate request queue per L1 quadrant. Therefore, the 4106 vector and scalar memory operations performed by different wavefronts, whether 4107 executing in the same or different work-groups (which may be executing on 4108 different CUs accessing different L0s), can be reordered relative to each 4109 other. A ``s_waitcnt vmcnt(0) & vscnt(0)`` is required to ensure 4110 synchronization between vector memory operations of different wavefronts. It 4111 ensures a previous vector memory operation has completed before executing a 4112 subsequent vector memory or LDS operation and so can be used to meet the 4113 requirements of acquire, release and sequential consistency. 4114* The L1 caches use an L2 cache shared by all SAs on the same agent. 4115* The L2 cache has independent channels to service disjoint ranges of virtual 4116 addresses. 4117* Each L1 quadrant of a single SA accesses a different L2 channel. Each L1 4118 quadrant has a separate request queue per L2 channel. Therefore, the vector 4119 and scalar memory operations performed by wavefronts executing in different 4120 work-groups (which may be executing on different SAs) of an agent can be 4121 reordered relative to each other. A ``s_waitcnt vmcnt(0) & vscnt(0)`` is 4122 required to ensure synchronization between vector memory operations of 4123 different SAs. It ensures a previous vector memory operation has completed 4124 before executing a subsequent vector memory and so can be used to meet the 4125 requirements of acquire, release and sequential consistency. 4126* The L2 cache can be kept coherent with other agents on some targets, or ranges 4127 of virtual addresses can be set up to bypass it to ensure system coherence. 4128 4129Private address space uses ``buffer_load/store`` using the scratch V# 4130(GFX6-GFX8), or ``scratch_load/store`` (GFX9-GFX10). Since only a single thread 4131is accessing the memory, atomic memory orderings are not meaningful, and all 4132accesses are treated as non-atomic. 4133 4134Constant address space uses ``buffer/global_load`` instructions (or equivalent 4135scalar memory instructions). Since the constant address space contents do not 4136change during the execution of a kernel dispatch it is not legal to perform 4137stores, and atomic memory orderings are not meaningful, and all access are 4138treated as non-atomic. 4139 4140A memory synchronization scope wider than work-group is not meaningful for the 4141group (LDS) address space and is treated as work-group. 4142 4143The memory model does not support the region address space which is treated as 4144non-atomic. 4145 4146Acquire memory ordering is not meaningful on store atomic instructions and is 4147treated as non-atomic. 4148 4149Release memory ordering is not meaningful on load atomic instructions and is 4150treated a non-atomic. 4151 4152Acquire-release memory ordering is not meaningful on load or store atomic 4153instructions and is treated as acquire and release respectively. 4154 4155AMDGPU backend only uses scalar memory operations to access memory that is 4156proven to not change during the execution of the kernel dispatch. This includes 4157constant address space and global address space for program scope const 4158variables. Therefore, the kernel machine code does not have to maintain the 4159scalar L1 cache to ensure it is coherent with the vector L1 cache. The scalar 4160and vector L1 caches are invalidated between kernel dispatches by CP since 4161constant address space data may change between kernel dispatch executions. See 4162:ref:`amdgpu-address-spaces`. 4163 4164The one exception is if scalar writes are used to spill SGPR registers. In this 4165case the AMDGPU backend ensures the memory location used to spill is never 4166accessed by vector memory operations at the same time. If scalar writes are used 4167then a ``s_dcache_wb`` is inserted before the ``s_endpgm`` and before a function 4168return since the locations may be used for vector memory instructions by a 4169future wavefront that uses the same scratch area, or a function call that 4170creates a frame at the same address, respectively. There is no need for a 4171``s_dcache_inv`` as all scalar writes are write-before-read in the same thread. 4172 4173For GFX6-GFX9, scratch backing memory (which is used for the private address 4174space) is accessed with MTYPE NC_NV (non-coherent non-volatile). Since the 4175private address space is only accessed by a single thread, and is always 4176write-before-read, there is never a need to invalidate these entries from the L1 4177cache. Hence all cache invalidates are done as ``*_vol`` to only invalidate the 4178volatile cache lines. 4179 4180For GFX10, scratch backing memory (which is used for the private address space) 4181is accessed with MTYPE NC (non-coherent). Since the private address space is 4182only accessed by a single thread, and is always write-before-read, there is 4183never a need to invalidate these entries from the L0 or L1 caches. 4184 4185For GFX10, wavefronts are executed in native mode with in-order reporting of 4186loads and sample instructions. In this mode vmcnt reports completion of load, 4187atomic with return and sample instructions in order, and the vscnt reports the 4188completion of store and atomic without return in order. See ``MEM_ORDERED`` 4189field in :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 4190 4191In GFX10, wavefronts can be executed in WGP or CU wavefront execution mode: 4192 4193* In WGP wavefront execution mode the wavefronts of a work-group are executed 4194 on the SIMDs of both CUs of the WGP. Therefore, explicit management of the per 4195 CU L0 caches is required for work-group synchronization. Also accesses to L1 4196 at work-group scope need to be explicitly ordered as the accesses from 4197 different CUs are not ordered. 4198* In CU wavefront execution mode the wavefronts of a work-group are executed on 4199 the SIMDs of a single CU of the WGP. Therefore, all global memory access by 4200 the work-group access the same L0 which in turn ensures L1 accesses are 4201 ordered and so do not require explicit management of the caches for 4202 work-group synchronization. 4203 4204See ``WGP_MODE`` field in 4205:ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table` and 4206:ref:`amdgpu-target-features`. 4207 4208On dGPU the kernarg backing memory is accessed as UC (uncached) to avoid needing 4209to invalidate the L2 cache. For GFX6-GFX9, this also causes it to be treated as 4210non-volatile and so is not invalidated by ``*_vol``. On APU it is accessed as CC 4211(cache coherent) and so the L2 cache will be coherent with the CPU and other 4212agents. 4213 4214 .. table:: AMDHSA Memory Model Code Sequences GFX6-GFX10 4215 :name: amdgpu-amdhsa-memory-model-code-sequences-gfx6-gfx10-table 4216 4217 ============ ============ ============== ========== =============================== ================================== 4218 LLVM Instr LLVM Memory LLVM Memory AMDGPU AMDGPU Machine Code AMDGPU Machine Code 4219 Ordering Sync Scope Address GFX6-9 GFX10 4220 Space 4221 ============ ============ ============== ========== =============================== ================================== 4222 **Non-Atomic** 4223 ---------------------------------------------------------------------------------------------------------------------- 4224 load *none* *none* - global - !volatile & !nontemporal - !volatile & !nontemporal 4225 - generic 4226 - private 1. buffer/global/flat_load 1. buffer/global/flat_load 4227 - constant 4228 - volatile & !nontemporal - volatile & !nontemporal 4229 4230 1. buffer/global/flat_load 1. buffer/global/flat_load 4231 glc=1 glc=1 dlc=1 4232 4233 - nontemporal - nontemporal 4234 4235 1. buffer/global/flat_load 1. buffer/global/flat_load 4236 glc=1 slc=1 slc=1 4237 4238 load *none* *none* - local 1. ds_load 1. ds_load 4239 store *none* *none* - global - !nontemporal - !nontemporal 4240 - generic 4241 - private 1. buffer/global/flat_store 1. buffer/global/flat_store 4242 - constant 4243 - nontemporal - nontemporal 4244 4245 1. buffer/global/flat_store 1. buffer/global/flat_store 4246 glc=1 slc=1 slc=1 4247 4248 store *none* *none* - local 1. ds_store 1. ds_store 4249 **Unordered Atomic** 4250 ---------------------------------------------------------------------------------------------------------------------- 4251 load atomic unordered *any* *any* *Same as non-atomic*. *Same as non-atomic*. 4252 store atomic unordered *any* *any* *Same as non-atomic*. *Same as non-atomic*. 4253 atomicrmw unordered *any* *any* *Same as monotonic *Same as monotonic 4254 atomic*. atomic*. 4255 **Monotonic Atomic** 4256 ---------------------------------------------------------------------------------------------------------------------- 4257 load atomic monotonic - singlethread - global 1. buffer/global/flat_load 1. buffer/global/flat_load 4258 - wavefront - generic 4259 load atomic monotonic - workgroup - global 1. buffer/global/flat_load 1. buffer/global/flat_load 4260 - generic glc=1 4261 4262 - If CU wavefront execution mode, omit glc=1. 4263 4264 load atomic monotonic - singlethread - local 1. ds_load 1. ds_load 4265 - wavefront 4266 - workgroup 4267 load atomic monotonic - agent - global 1. buffer/global/flat_load 1. buffer/global/flat_load 4268 - system - generic glc=1 glc=1 dlc=1 4269 store atomic monotonic - singlethread - global 1. buffer/global/flat_store 1. buffer/global/flat_store 4270 - wavefront - generic 4271 - workgroup 4272 - agent 4273 - system 4274 store atomic monotonic - singlethread - local 1. ds_store 1. ds_store 4275 - wavefront 4276 - workgroup 4277 atomicrmw monotonic - singlethread - global 1. buffer/global/flat_atomic 1. buffer/global/flat_atomic 4278 - wavefront - generic 4279 - workgroup 4280 - agent 4281 - system 4282 atomicrmw monotonic - singlethread - local 1. ds_atomic 1. ds_atomic 4283 - wavefront 4284 - workgroup 4285 **Acquire Atomic** 4286 ---------------------------------------------------------------------------------------------------------------------- 4287 load atomic acquire - singlethread - global 1. buffer/global/ds/flat_load 1. buffer/global/ds/flat_load 4288 - wavefront - local 4289 - generic 4290 load atomic acquire - workgroup - global 1. buffer/global/flat_load 1. buffer/global_load glc=1 4291 4292 - If CU wavefront execution mode, omit glc=1. 4293 4294 2. s_waitcnt vmcnt(0) 4295 4296 - If CU wavefront execution mode, omit. 4297 - Must happen before 4298 the following buffer_gl0_inv 4299 and before any following 4300 global/generic 4301 load/load 4302 atomic/store/store 4303 atomic/atomicrmw. 4304 4305 3. buffer_gl0_inv 4306 4307 - If CU wavefront execution mode, omit. 4308 - Ensures that 4309 following 4310 loads will not see 4311 stale data. 4312 4313 load atomic acquire - workgroup - local 1. ds_load 1. ds_load 4314 2. s_waitcnt lgkmcnt(0) 2. s_waitcnt lgkmcnt(0) 4315 4316 - If OpenCL, omit. - If OpenCL, omit. 4317 - Must happen before - Must happen before 4318 any following the following buffer_gl0_inv 4319 global/generic and before any following 4320 load/load global/generic load/load 4321 atomic/store/store atomic/store/store 4322 atomic/atomicrmw. atomic/atomicrmw. 4323 - Ensures any - Ensures any 4324 following global following global 4325 data read is no data read is no 4326 older than the load older than the load 4327 atomic value being atomic value being 4328 acquired. acquired. 4329 4330 3. buffer_gl0_inv 4331 4332 - If CU wavefront execution mode, omit. 4333 - If OpenCL, omit. 4334 - Ensures that 4335 following 4336 loads will not see 4337 stale data. 4338 4339 load atomic acquire - workgroup - generic 1. flat_load 1. flat_load glc=1 4340 4341 - If CU wavefront execution mode, omit glc=1. 4342 4343 2. s_waitcnt lgkmcnt(0) 2. s_waitcnt lgkmcnt(0) & 4344 vmcnt(0) 4345 4346 - If CU wavefront execution mode, omit vmcnt. 4347 - If OpenCL, omit. - If OpenCL, omit 4348 lgkmcnt(0). 4349 - Must happen before - Must happen before 4350 any following the following 4351 global/generic buffer_gl0_inv and any 4352 load/load following global/generic 4353 atomic/store/store load/load 4354 atomic/atomicrmw. atomic/store/store 4355 atomic/atomicrmw. 4356 - Ensures any - Ensures any 4357 following global following global 4358 data read is no data read is no 4359 older than the load older than the load 4360 atomic value being atomic value being 4361 acquired. acquired. 4362 4363 3. buffer_gl0_inv 4364 4365 - If CU wavefront execution mode, omit. 4366 - Ensures that 4367 following 4368 loads will not see 4369 stale data. 4370 4371 load atomic acquire - agent - global 1. buffer/global/flat_load 1. buffer/global_load 4372 - system glc=1 glc=1 dlc=1 4373 2. s_waitcnt vmcnt(0) 2. s_waitcnt vmcnt(0) 4374 4375 - Must happen before - Must happen before 4376 following following 4377 buffer_wbinvl1_vol. buffer_gl*_inv. 4378 - Ensures the load - Ensures the load 4379 has completed has completed 4380 before invalidating before invalidating 4381 the cache. the caches. 4382 4383 3. buffer_wbinvl1_vol 3. buffer_gl0_inv; 4384 buffer_gl1_inv 4385 4386 - Must happen before - Must happen before 4387 any following any following 4388 global/generic global/generic 4389 load/load load/load 4390 atomic/atomicrmw. atomic/atomicrmw. 4391 - Ensures that - Ensures that 4392 following following 4393 loads will not see loads will not see 4394 stale global data. stale global data. 4395 4396 load atomic acquire - agent - generic 1. flat_load glc=1 1. flat_load glc=1 dlc=1 4397 - system 2. s_waitcnt vmcnt(0) & 2. s_waitcnt vmcnt(0) & 4398 lgkmcnt(0) lgkmcnt(0) 4399 4400 - If OpenCL omit - If OpenCL omit 4401 lgkmcnt(0). lgkmcnt(0). 4402 - Must happen before - Must happen before 4403 following following 4404 buffer_wbinvl1_vol. buffer_gl*_invl. 4405 - Ensures the flat_load - Ensures the flat_load 4406 has completed has completed 4407 before invalidating before invalidating 4408 the cache. the caches. 4409 4410 3. buffer_wbinvl1_vol 3. buffer_gl0_inv; 4411 buffer_gl1_inv 4412 4413 - Must happen before - Must happen before 4414 any following any following 4415 global/generic global/generic 4416 load/load load/load 4417 atomic/atomicrmw. atomic/atomicrmw. 4418 - Ensures that - Ensures that 4419 following loads following loads 4420 will not see stale will not see stale 4421 global data. global data. 4422 4423 atomicrmw acquire - singlethread - global 1. buffer/global/ds/flat_atomic 1. buffer/global/ds/flat_atomic 4424 - wavefront - local 4425 - generic 4426 atomicrmw acquire - workgroup - global 1. buffer/global/flat_atomic 1. buffer/global_atomic 4427 2. s_waitcnt vm/vscnt(0) 4428 4429 - If CU wavefront execution mode, omit. 4430 - Use vmcnt if atomic with 4431 return and vscnt if atomic 4432 with no-return. 4433 - Must happen before 4434 the following buffer_gl0_inv 4435 and before any following 4436 global/generic 4437 load/load 4438 atomic/store/store 4439 atomic/atomicrmw. 4440 4441 3. buffer_gl0_inv 4442 4443 - If CU wavefront execution mode, omit. 4444 - Ensures that 4445 following 4446 loads will not see 4447 stale data. 4448 4449 atomicrmw acquire - workgroup - local 1. ds_atomic 1. ds_atomic 4450 2. waitcnt lgkmcnt(0) 2. waitcnt lgkmcnt(0) 4451 4452 - If OpenCL, omit. - If OpenCL, omit. 4453 - Must happen before - Must happen before 4454 any following the following 4455 global/generic buffer_gl0_inv. 4456 load/load 4457 atomic/store/store 4458 atomic/atomicrmw. 4459 - Ensures any - Ensures any 4460 following global following global 4461 data read is no data read is no 4462 older than the older than the 4463 atomicrmw value atomicrmw value 4464 being acquired. being acquired. 4465 4466 3. buffer_gl0_inv 4467 4468 - If OpenCL omit. 4469 - Ensures that 4470 following 4471 loads will not see 4472 stale data. 4473 4474 atomicrmw acquire - workgroup - generic 1. flat_atomic 1. flat_atomic 4475 2. waitcnt lgkmcnt(0) 2. waitcnt lgkmcnt(0) & 4476 vm/vscnt(0) 4477 4478 - If CU wavefront execution mode, omit vm/vscnt. 4479 - If OpenCL, omit. - If OpenCL, omit 4480 waitcnt lgkmcnt(0).. 4481 - Use vmcnt if atomic with 4482 return and vscnt if atomic 4483 with no-return. 4484 waitcnt lgkmcnt(0). 4485 - Must happen before - Must happen before 4486 any following the following 4487 global/generic buffer_gl0_inv. 4488 load/load 4489 atomic/store/store 4490 atomic/atomicrmw. 4491 - Ensures any - Ensures any 4492 following global following global 4493 data read is no data read is no 4494 older than the older than the 4495 atomicrmw value atomicrmw value 4496 being acquired. being acquired. 4497 4498 3. buffer_gl0_inv 4499 4500 - If CU wavefront execution mode, omit. 4501 - Ensures that 4502 following 4503 loads will not see 4504 stale data. 4505 4506 atomicrmw acquire - agent - global 1. buffer/global/flat_atomic 1. buffer/global_atomic 4507 - system 2. s_waitcnt vmcnt(0) 2. s_waitcnt vm/vscnt(0) 4508 4509 - Use vmcnt if atomic with 4510 return and vscnt if atomic 4511 with no-return. 4512 waitcnt lgkmcnt(0). 4513 - Must happen before - Must happen before 4514 following following 4515 buffer_wbinvl1_vol. buffer_gl*_inv. 4516 - Ensures the - Ensures the 4517 atomicrmw has atomicrmw has 4518 completed before completed before 4519 invalidating the invalidating the 4520 cache. caches. 4521 4522 3. buffer_wbinvl1_vol 3. buffer_gl0_inv; 4523 buffer_gl1_inv 4524 4525 - Must happen before - Must happen before 4526 any following any following 4527 global/generic global/generic 4528 load/load load/load 4529 atomic/atomicrmw. atomic/atomicrmw. 4530 - Ensures that - Ensures that 4531 following loads following loads 4532 will not see stale will not see stale 4533 global data. global data. 4534 4535 atomicrmw acquire - agent - generic 1. flat_atomic 1. flat_atomic 4536 - system 2. s_waitcnt vmcnt(0) & 2. s_waitcnt vm/vscnt(0) & 4537 lgkmcnt(0) lgkmcnt(0) 4538 4539 - If OpenCL, omit - If OpenCL, omit 4540 lgkmcnt(0). lgkmcnt(0). 4541 - Use vmcnt if atomic with 4542 return and vscnt if atomic 4543 with no-return. 4544 - Must happen before - Must happen before 4545 following following 4546 buffer_wbinvl1_vol. buffer_gl*_inv. 4547 - Ensures the - Ensures the 4548 atomicrmw has atomicrmw has 4549 completed before completed before 4550 invalidating the invalidating the 4551 cache. caches. 4552 4553 3. buffer_wbinvl1_vol 3. buffer_gl0_inv; 4554 buffer_gl1_inv 4555 4556 - Must happen before - Must happen before 4557 any following any following 4558 global/generic global/generic 4559 load/load load/load 4560 atomic/atomicrmw. atomic/atomicrmw. 4561 - Ensures that - Ensures that 4562 following loads following loads 4563 will not see stale will not see stale 4564 global data. global data. 4565 4566 fence acquire - singlethread *none* *none* *none* 4567 - wavefront 4568 fence acquire - workgroup *none* 1. s_waitcnt lgkmcnt(0) 1. s_waitcnt lgkmcnt(0) & 4569 vmcnt(0) & vscnt(0) 4570 4571 - If CU wavefront execution mode, omit vmcnt and 4572 vscnt. 4573 - If OpenCL and - If OpenCL and 4574 address space is address space is 4575 not generic, omit. not generic, omit 4576 lgkmcnt(0). 4577 - If OpenCL and 4578 address space is 4579 local, omit 4580 vmcnt(0) and vscnt(0). 4581 - However, since LLVM - However, since LLVM 4582 currently has no currently has no 4583 address space on address space on 4584 the fence need to the fence need to 4585 conservatively conservatively 4586 always generate. If always generate. If 4587 fence had an fence had an 4588 address space then address space then 4589 set to address set to address 4590 space of OpenCL space of OpenCL 4591 fence flag, or to fence flag, or to 4592 generic if both generic if both 4593 local and global local and global 4594 flags are flags are 4595 specified. specified. 4596 - Must happen after 4597 any preceding 4598 local/generic load 4599 atomic/atomicrmw 4600 with an equal or 4601 wider sync scope 4602 and memory ordering 4603 stronger than 4604 unordered (this is 4605 termed the 4606 fence-paired-atomic). 4607 - Must happen before 4608 any following 4609 global/generic 4610 load/load 4611 atomic/store/store 4612 atomic/atomicrmw. 4613 - Ensures any 4614 following global 4615 data read is no 4616 older than the 4617 value read by the 4618 fence-paired-atomic. 4619 - Could be split into 4620 separate s_waitcnt 4621 vmcnt(0), s_waitcnt 4622 vscnt(0) and s_waitcnt 4623 lgkmcnt(0) to allow 4624 them to be 4625 independently moved 4626 according to the 4627 following rules. 4628 - s_waitcnt vmcnt(0) 4629 must happen after 4630 any preceding 4631 global/generic load 4632 atomic/ 4633 atomicrmw-with-return-value 4634 with an equal or 4635 wider sync scope 4636 and memory ordering 4637 stronger than 4638 unordered (this is 4639 termed the 4640 fence-paired-atomic). 4641 - s_waitcnt vscnt(0) 4642 must happen after 4643 any preceding 4644 global/generic 4645 atomicrmw-no-return-value 4646 with an equal or 4647 wider sync scope 4648 and memory ordering 4649 stronger than 4650 unordered (this is 4651 termed the 4652 fence-paired-atomic). 4653 - s_waitcnt lgkmcnt(0) 4654 must happen after 4655 any preceding 4656 local/generic load 4657 atomic/atomicrmw 4658 with an equal or 4659 wider sync scope 4660 and memory ordering 4661 stronger than 4662 unordered (this is 4663 termed the 4664 fence-paired-atomic). 4665 - Must happen before 4666 the following 4667 buffer_gl0_inv. 4668 - Ensures that the 4669 fence-paired atomic 4670 has completed 4671 before invalidating 4672 the 4673 cache. Therefore 4674 any following 4675 locations read must 4676 be no older than 4677 the value read by 4678 the 4679 fence-paired-atomic. 4680 4681 3. buffer_gl0_inv 4682 4683 - If CU wavefront execution mode, omit. 4684 - Ensures that 4685 following 4686 loads will not see 4687 stale data. 4688 4689 fence acquire - agent *none* 1. s_waitcnt lgkmcnt(0) & 1. s_waitcnt lgkmcnt(0) & 4690 - system vmcnt(0) vmcnt(0) & vscnt(0) 4691 4692 - If OpenCL and - If OpenCL and 4693 address space is address space is 4694 not generic, omit not generic, omit 4695 lgkmcnt(0). lgkmcnt(0). 4696 - If OpenCL and 4697 address space is 4698 local, omit 4699 vmcnt(0) and vscnt(0). 4700 - However, since LLVM - However, since LLVM 4701 currently has no currently has no 4702 address space on address space on 4703 the fence need to the fence need to 4704 conservatively conservatively 4705 always generate always generate 4706 (see comment for (see comment for 4707 previous fence). previous fence). 4708 - Could be split into 4709 separate s_waitcnt 4710 vmcnt(0) and 4711 s_waitcnt 4712 lgkmcnt(0) to allow 4713 them to be 4714 independently moved 4715 according to the 4716 following rules. 4717 - s_waitcnt vmcnt(0) 4718 must happen after 4719 any preceding 4720 global/generic load 4721 atomic/atomicrmw 4722 with an equal or 4723 wider sync scope 4724 and memory ordering 4725 stronger than 4726 unordered (this is 4727 termed the 4728 fence-paired-atomic). 4729 - s_waitcnt lgkmcnt(0) 4730 must happen after 4731 any preceding 4732 local/generic load 4733 atomic/atomicrmw 4734 with an equal or 4735 wider sync scope 4736 and memory ordering 4737 stronger than 4738 unordered (this is 4739 termed the 4740 fence-paired-atomic). 4741 - Must happen before 4742 the following 4743 buffer_wbinvl1_vol. 4744 - Ensures that the 4745 fence-paired atomic 4746 has completed 4747 before invalidating 4748 the 4749 cache. Therefore 4750 any following 4751 locations read must 4752 be no older than 4753 the value read by 4754 the 4755 fence-paired-atomic. 4756 - Could be split into 4757 separate s_waitcnt 4758 vmcnt(0), s_waitcnt 4759 vscnt(0) and s_waitcnt 4760 lgkmcnt(0) to allow 4761 them to be 4762 independently moved 4763 according to the 4764 following rules. 4765 - s_waitcnt vmcnt(0) 4766 must happen after 4767 any preceding 4768 global/generic load 4769 atomic/ 4770 atomicrmw-with-return-value 4771 with an equal or 4772 wider sync scope 4773 and memory ordering 4774 stronger than 4775 unordered (this is 4776 termed the 4777 fence-paired-atomic). 4778 - s_waitcnt vscnt(0) 4779 must happen after 4780 any preceding 4781 global/generic 4782 atomicrmw-no-return-value 4783 with an equal or 4784 wider sync scope 4785 and memory ordering 4786 stronger than 4787 unordered (this is 4788 termed the 4789 fence-paired-atomic). 4790 - s_waitcnt lgkmcnt(0) 4791 must happen after 4792 any preceding 4793 local/generic load 4794 atomic/atomicrmw 4795 with an equal or 4796 wider sync scope 4797 and memory ordering 4798 stronger than 4799 unordered (this is 4800 termed the 4801 fence-paired-atomic). 4802 - Must happen before 4803 the following 4804 buffer_gl*_inv. 4805 - Ensures that the 4806 fence-paired atomic 4807 has completed 4808 before invalidating 4809 the 4810 caches. Therefore 4811 any following 4812 locations read must 4813 be no older than 4814 the value read by 4815 the 4816 fence-paired-atomic. 4817 4818 2. buffer_wbinvl1_vol 2. buffer_gl0_inv; 4819 buffer_gl1_inv 4820 4821 - Must happen before any - Must happen before any 4822 following global/generic following global/generic 4823 load/load load/load 4824 atomic/store/store atomic/store/store 4825 atomic/atomicrmw. atomic/atomicrmw. 4826 - Ensures that - Ensures that 4827 following loads following loads 4828 will not see stale will not see stale 4829 global data. global data. 4830 4831 **Release Atomic** 4832 ---------------------------------------------------------------------------------------------------------------------- 4833 store atomic release - singlethread - global 1. buffer/global/ds/flat_store 1. buffer/global/ds/flat_store 4834 - wavefront - local 4835 - generic 4836 store atomic release - workgroup - global 1. s_waitcnt lgkmcnt(0) 1. s_waitcnt lgkmcnt(0) & 4837 vmcnt(0) & vscnt(0) 4838 4839 - If CU wavefront execution mode, omit vmcnt and 4840 vscnt. 4841 - If OpenCL, omit. - If OpenCL, omit 4842 lgkmcnt(0). 4843 - Must happen after 4844 any preceding 4845 local/generic 4846 load/store/load 4847 atomic/store 4848 atomic/atomicrmw. 4849 - Could be split into 4850 separate s_waitcnt 4851 vmcnt(0), s_waitcnt 4852 vscnt(0) and s_waitcnt 4853 lgkmcnt(0) to allow 4854 them to be 4855 independently moved 4856 according to the 4857 following rules. 4858 - s_waitcnt vmcnt(0) 4859 must happen after 4860 any preceding 4861 global/generic load/load 4862 atomic/ 4863 atomicrmw-with-return-value. 4864 - s_waitcnt vscnt(0) 4865 must happen after 4866 any preceding 4867 global/generic 4868 store/store 4869 atomic/ 4870 atomicrmw-no-return-value. 4871 - s_waitcnt lgkmcnt(0) 4872 must happen after 4873 any preceding 4874 local/generic 4875 load/store/load 4876 atomic/store 4877 atomic/atomicrmw. 4878 - Must happen before - Must happen before 4879 the following the following 4880 store. store. 4881 - Ensures that all - Ensures that all 4882 memory operations memory operations 4883 to local have have 4884 completed before completed before 4885 performing the performing the 4886 store that is being store that is being 4887 released. released. 4888 4889 2. buffer/global/flat_store 2. buffer/global_store 4890 store atomic release - workgroup - local 1. waitcnt vmcnt(0) & vscnt(0) 4891 4892 - If CU wavefront execution mode, omit. 4893 - If OpenCL, omit. 4894 - Could be split into 4895 separate s_waitcnt 4896 vmcnt(0) and s_waitcnt 4897 vscnt(0) to allow 4898 them to be 4899 independently moved 4900 according to the 4901 following rules. 4902 - s_waitcnt vmcnt(0) 4903 must happen after 4904 any preceding 4905 global/generic load/load 4906 atomic/ 4907 atomicrmw-with-return-value. 4908 - s_waitcnt vscnt(0) 4909 must happen after 4910 any preceding 4911 global/generic 4912 store/store atomic/ 4913 atomicrmw-no-return-value. 4914 - Must happen before 4915 the following 4916 store. 4917 - Ensures that all 4918 global memory 4919 operations have 4920 completed before 4921 performing the 4922 store that is being 4923 released. 4924 4925 1. ds_store 2. ds_store 4926 store atomic release - workgroup - generic 1. s_waitcnt lgkmcnt(0) 1. s_waitcnt lgkmcnt(0) & 4927 vmcnt(0) & vscnt(0) 4928 4929 - If CU wavefront execution mode, omit vmcnt and 4930 vscnt. 4931 - If OpenCL, omit. - If OpenCL, omit 4932 lgkmcnt(0). 4933 - Must happen after 4934 any preceding 4935 local/generic 4936 load/store/load 4937 atomic/store 4938 atomic/atomicrmw. 4939 - Could be split into 4940 separate s_waitcnt 4941 vmcnt(0), s_waitcnt 4942 vscnt(0) and s_waitcnt 4943 lgkmcnt(0) to allow 4944 them to be 4945 independently moved 4946 according to the 4947 following rules. 4948 - s_waitcnt vmcnt(0) 4949 must happen after 4950 any preceding 4951 global/generic load/load 4952 atomic/ 4953 atomicrmw-with-return-value. 4954 - s_waitcnt vscnt(0) 4955 must happen after 4956 any preceding 4957 global/generic 4958 store/store 4959 atomic/ 4960 atomicrmw-no-return-value. 4961 - s_waitcnt lgkmcnt(0) 4962 must happen after 4963 any preceding 4964 local/generic load/store/load 4965 atomic/store atomic/atomicrmw. 4966 - Must happen before - Must happen before 4967 the following the following 4968 store. store. 4969 - Ensures that all - Ensures that all 4970 memory operations memory operations 4971 to local have have 4972 completed before completed before 4973 performing the performing the 4974 store that is being store that is being 4975 released. released. 4976 4977 2. flat_store 2. flat_store 4978 store atomic release - agent - global 1. s_waitcnt lgkmcnt(0) & 1. s_waitcnt lgkmcnt(0) & 4979 - system - generic vmcnt(0) vmcnt(0) & vscnt(0) 4980 4981 - If OpenCL, omit - If OpenCL, omit 4982 lgkmcnt(0). lgkmcnt(0). 4983 - Could be split into - Could be split into 4984 separate s_waitcnt separate s_waitcnt 4985 vmcnt(0) and vmcnt(0), s_waitcnt vscnt(0) 4986 s_waitcnt and s_waitcnt 4987 lgkmcnt(0) to allow lgkmcnt(0) to allow 4988 them to be them to be 4989 independently moved independently moved 4990 according to the according to the 4991 following rules. following rules. 4992 - s_waitcnt vmcnt(0) - s_waitcnt vmcnt(0) 4993 must happen after must happen after 4994 any preceding any preceding 4995 global/generic global/generic 4996 load/store/load load/load 4997 atomic/store atomic/ 4998 atomic/atomicrmw. atomicrmw-with-return-value. 4999 - s_waitcnt vscnt(0) 5000 must happen after 5001 any preceding 5002 global/generic 5003 store/store atomic/ 5004 atomicrmw-no-return-value. 5005 - s_waitcnt lgkmcnt(0) - s_waitcnt lgkmcnt(0) 5006 must happen after must happen after 5007 any preceding any preceding 5008 local/generic local/generic 5009 load/store/load load/store/load 5010 atomic/store atomic/store 5011 atomic/atomicrmw. atomic/atomicrmw. 5012 - Must happen before - Must happen before 5013 the following the following 5014 store. store. 5015 - Ensures that all - Ensures that all 5016 memory operations memory operations 5017 to memory have to memory have 5018 completed before completed before 5019 performing the performing the 5020 store that is being store that is being 5021 released. released. 5022 5023 2. buffer/global/ds/flat_store 2. buffer/global/ds/flat_store 5024 atomicrmw release - singlethread - global 1. buffer/global/ds/flat_atomic 1. buffer/global/ds/flat_atomic 5025 - wavefront - local 5026 - generic 5027 atomicrmw release - workgroup - global 1. s_waitcnt lgkmcnt(0) 1. s_waitcnt lgkmcnt(0) & 5028 vmcnt(0) & vscnt(0) 5029 5030 - If CU wavefront execution mode, omit vmcnt and 5031 vscnt. 5032 - If OpenCL, omit. 5033 5034 - Must happen after 5035 any preceding 5036 local/generic 5037 load/store/load 5038 atomic/store 5039 atomic/atomicrmw. 5040 - Could be split into 5041 separate s_waitcnt 5042 vmcnt(0), s_waitcnt 5043 vscnt(0) and s_waitcnt 5044 lgkmcnt(0) to allow 5045 them to be 5046 independently moved 5047 according to the 5048 following rules. 5049 - s_waitcnt vmcnt(0) 5050 must happen after 5051 any preceding 5052 global/generic load/load 5053 atomic/ 5054 atomicrmw-with-return-value. 5055 - s_waitcnt vscnt(0) 5056 must happen after 5057 any preceding 5058 global/generic 5059 store/store 5060 atomic/ 5061 atomicrmw-no-return-value. 5062 - s_waitcnt lgkmcnt(0) 5063 must happen after 5064 any preceding 5065 local/generic 5066 load/store/load 5067 atomic/store 5068 atomic/atomicrmw. 5069 - Must happen before - Must happen before 5070 the following the following 5071 atomicrmw. atomicrmw. 5072 - Ensures that all - Ensures that all 5073 memory operations memory operations 5074 to local have have 5075 completed before completed before 5076 performing the performing the 5077 atomicrmw that is atomicrmw that is 5078 being released. being released. 5079 5080 2. buffer/global/flat_atomic 2. buffer/global_atomic 5081 atomicrmw release - workgroup - local 1. waitcnt vmcnt(0) & vscnt(0) 5082 5083 - If CU wavefront execution mode, omit. 5084 - If OpenCL, omit. 5085 - Could be split into 5086 separate s_waitcnt 5087 vmcnt(0) and s_waitcnt 5088 vscnt(0) to allow 5089 them to be 5090 independently moved 5091 according to the 5092 following rules. 5093 - s_waitcnt vmcnt(0) 5094 must happen after 5095 any preceding 5096 global/generic load/load 5097 atomic/ 5098 atomicrmw-with-return-value. 5099 - s_waitcnt vscnt(0) 5100 must happen after 5101 any preceding 5102 global/generic 5103 store/store atomic/ 5104 atomicrmw-no-return-value. 5105 - Must happen before 5106 the following 5107 store. 5108 - Ensures that all 5109 global memory 5110 operations have 5111 completed before 5112 performing the 5113 store that is being 5114 released. 5115 5116 1. ds_atomic 2. ds_atomic 5117 atomicrmw release - workgroup - generic 1. s_waitcnt lgkmcnt(0) 1. s_waitcnt lgkmcnt(0) & 5118 vmcnt(0) & vscnt(0) 5119 5120 - If CU wavefront execution mode, omit vmcnt and 5121 vscnt. 5122 - If OpenCL, omit. - If OpenCL, omit 5123 waitcnt lgkmcnt(0). 5124 - Must happen after 5125 any preceding 5126 local/generic 5127 load/store/load 5128 atomic/store 5129 atomic/atomicrmw. 5130 - Could be split into 5131 separate s_waitcnt 5132 vmcnt(0), s_waitcnt 5133 vscnt(0) and s_waitcnt 5134 lgkmcnt(0) to allow 5135 them to be 5136 independently moved 5137 according to the 5138 following rules. 5139 - s_waitcnt vmcnt(0) 5140 must happen after 5141 any preceding 5142 global/generic load/load 5143 atomic/ 5144 atomicrmw-with-return-value. 5145 - s_waitcnt vscnt(0) 5146 must happen after 5147 any preceding 5148 global/generic 5149 store/store 5150 atomic/ 5151 atomicrmw-no-return-value. 5152 - s_waitcnt lgkmcnt(0) 5153 must happen after 5154 any preceding 5155 local/generic load/store/load 5156 atomic/store atomic/atomicrmw. 5157 - Must happen before - Must happen before 5158 the following the following 5159 atomicrmw. atomicrmw. 5160 - Ensures that all - Ensures that all 5161 memory operations memory operations 5162 to local have have 5163 completed before completed before 5164 performing the performing the 5165 atomicrmw that is atomicrmw that is 5166 being released. being released. 5167 5168 2. flat_atomic 2. flat_atomic 5169 atomicrmw release - agent - global 1. s_waitcnt lgkmcnt(0) & 1. s_waitcnt lkkmcnt(0) & 5170 - system - generic vmcnt(0) vmcnt(0) & vscnt(0) 5171 5172 - If OpenCL, omit - If OpenCL, omit 5173 lgkmcnt(0). lgkmcnt(0). 5174 - Could be split into - Could be split into 5175 separate s_waitcnt separate s_waitcnt 5176 vmcnt(0) and vmcnt(0), s_waitcnt 5177 s_waitcnt vscnt(0) and s_waitcnt 5178 lgkmcnt(0) to allow lgkmcnt(0) to allow 5179 them to be them to be 5180 independently moved independently moved 5181 according to the according to the 5182 following rules. following rules. 5183 - s_waitcnt vmcnt(0) - s_waitcnt vmcnt(0) 5184 must happen after must happen after 5185 any preceding any preceding 5186 global/generic global/generic 5187 load/store/load load/load atomic/ 5188 atomic/store atomicrmw-with-return-value. 5189 atomic/atomicrmw. 5190 - s_waitcnt vscnt(0) 5191 must happen after 5192 any preceding 5193 global/generic 5194 store/store atomic/ 5195 atomicrmw-no-return-value. 5196 - s_waitcnt lgkmcnt(0) - s_waitcnt lgkmcnt(0) 5197 must happen after must happen after 5198 any preceding any preceding 5199 local/generic local/generic 5200 load/store/load load/store/load 5201 atomic/store atomic/store 5202 atomic/atomicrmw. atomic/atomicrmw. 5203 - Must happen before - Must happen before 5204 the following the following 5205 atomicrmw. atomicrmw. 5206 - Ensures that all - Ensures that all 5207 memory operations memory operations 5208 to global and local to global and local 5209 have completed have completed 5210 before performing before performing 5211 the atomicrmw that the atomicrmw that 5212 is being released. is being released. 5213 5214 2. buffer/global/ds/flat_atomic 2. buffer/global/ds/flat_atomic 5215 fence release - singlethread *none* *none* *none* 5216 - wavefront 5217 fence release - workgroup *none* 1. s_waitcnt lgkmcnt(0) 1. s_waitcnt lgkmcnt(0) & 5218 vmcnt(0) & vscnt(0) 5219 5220 - If CU wavefront execution mode, omit vmcnt and 5221 vscnt. 5222 - If OpenCL and - If OpenCL and 5223 address space is address space is 5224 not generic, omit. not generic, omit 5225 lgkmcnt(0). 5226 - If OpenCL and 5227 address space is 5228 local, omit 5229 vmcnt(0) and vscnt(0). 5230 - However, since LLVM - However, since LLVM 5231 currently has no currently has no 5232 address space on address space on 5233 the fence need to the fence need to 5234 conservatively conservatively 5235 always generate. If always generate. If 5236 fence had an fence had an 5237 address space then address space then 5238 set to address set to address 5239 space of OpenCL space of OpenCL 5240 fence flag, or to fence flag, or to 5241 generic if both generic if both 5242 local and global local and global 5243 flags are flags are 5244 specified. specified. 5245 - Must happen after 5246 any preceding 5247 local/generic 5248 load/load 5249 atomic/store/store 5250 atomic/atomicrmw. 5251 - Could be split into 5252 separate s_waitcnt 5253 vmcnt(0), s_waitcnt 5254 vscnt(0) and s_waitcnt 5255 lgkmcnt(0) to allow 5256 them to be 5257 independently moved 5258 according to the 5259 following rules. 5260 - s_waitcnt vmcnt(0) 5261 must happen after 5262 any preceding 5263 global/generic 5264 load/load 5265 atomic/ 5266 atomicrmw-with-return-value. 5267 - s_waitcnt vscnt(0) 5268 must happen after 5269 any preceding 5270 global/generic 5271 store/store atomic/ 5272 atomicrmw-no-return-value. 5273 - s_waitcnt lgkmcnt(0) 5274 must happen after 5275 any preceding 5276 local/generic 5277 load/store/load 5278 atomic/store atomic/ 5279 atomicrmw. 5280 - Must happen before - Must happen before 5281 any following store any following store 5282 atomic/atomicrmw atomic/atomicrmw 5283 with an equal or with an equal or 5284 wider sync scope wider sync scope 5285 and memory ordering and memory ordering 5286 stronger than stronger than 5287 unordered (this is unordered (this is 5288 termed the termed the 5289 fence-paired-atomic). fence-paired-atomic). 5290 - Ensures that all - Ensures that all 5291 memory operations memory operations 5292 to local have have 5293 completed before completed before 5294 performing the performing the 5295 following following 5296 fence-paired-atomic. fence-paired-atomic. 5297 5298 fence release - agent *none* 1. s_waitcnt lgkmcnt(0) & 1. s_waitcnt lgkmcnt(0) & 5299 - system vmcnt(0) vmcnt(0) & vscnt(0) 5300 5301 - If OpenCL and - If OpenCL and 5302 address space is address space is 5303 not generic, omit not generic, omit 5304 lgkmcnt(0). lgkmcnt(0). 5305 - If OpenCL and - If OpenCL and 5306 address space is address space is 5307 local, omit local, omit 5308 vmcnt(0). vmcnt(0) and vscnt(0). 5309 - However, since LLVM - However, since LLVM 5310 currently has no currently has no 5311 address space on address space on 5312 the fence need to the fence need to 5313 conservatively conservatively 5314 always generate. If always generate. If 5315 fence had an fence had an 5316 address space then address space then 5317 set to address set to address 5318 space of OpenCL space of OpenCL 5319 fence flag, or to fence flag, or to 5320 generic if both generic if both 5321 local and global local and global 5322 flags are flags are 5323 specified. specified. 5324 - Could be split into - Could be split into 5325 separate s_waitcnt separate s_waitcnt 5326 vmcnt(0) and vmcnt(0), s_waitcnt 5327 s_waitcnt vscnt(0) and s_waitcnt 5328 lgkmcnt(0) to allow lgkmcnt(0) to allow 5329 them to be them to be 5330 independently moved independently moved 5331 according to the according to the 5332 following rules. following rules. 5333 - s_waitcnt vmcnt(0) - s_waitcnt vmcnt(0) 5334 must happen after must happen after 5335 any preceding any preceding 5336 global/generic global/generic 5337 load/store/load load/load atomic/ 5338 atomic/store atomicrmw-with-return-value. 5339 atomic/atomicrmw. 5340 - s_waitcnt vscnt(0) 5341 must happen after 5342 any preceding 5343 global/generic 5344 store/store atomic/ 5345 atomicrmw-no-return-value. 5346 - s_waitcnt lgkmcnt(0) - s_waitcnt lgkmcnt(0) 5347 must happen after must happen after 5348 any preceding any preceding 5349 local/generic local/generic 5350 load/store/load load/store/load 5351 atomic/store atomic/store 5352 atomic/atomicrmw. atomic/atomicrmw. 5353 - Must happen before - Must happen before 5354 any following store any following store 5355 atomic/atomicrmw atomic/atomicrmw 5356 with an equal or with an equal or 5357 wider sync scope wider sync scope 5358 and memory ordering and memory ordering 5359 stronger than stronger than 5360 unordered (this is unordered (this is 5361 termed the termed the 5362 fence-paired-atomic). fence-paired-atomic). 5363 - Ensures that all - Ensures that all 5364 memory operations memory operations 5365 have have 5366 completed before completed before 5367 performing the performing the 5368 following following 5369 fence-paired-atomic. fence-paired-atomic. 5370 5371 **Acquire-Release Atomic** 5372 ---------------------------------------------------------------------------------------------------------------------- 5373 atomicrmw acq_rel - singlethread - global 1. buffer/global/ds/flat_atomic 1. buffer/global/ds/flat_atomic 5374 - wavefront - local 5375 - generic 5376 atomicrmw acq_rel - workgroup - global 1. s_waitcnt lgkmcnt(0) 1. s_waitcnt lgkmcnt(0) & 5377 vmcnt(0) & vscnt(0) 5378 5379 - If CU wavefront execution mode, omit vmcnt and 5380 vscnt. 5381 - If OpenCL, omit. - If OpenCL, omit 5382 s_waitcnt lgkmcnt(0). 5383 - Must happen after - Must happen after 5384 any preceding any preceding 5385 local/generic local/generic 5386 load/store/load load/store/load 5387 atomic/store atomic/store 5388 atomic/atomicrmw. atomic/atomicrmw. 5389 - Could be split into 5390 separate s_waitcnt 5391 vmcnt(0), s_waitcnt 5392 vscnt(0) and s_waitcnt 5393 lgkmcnt(0) to allow 5394 them to be 5395 independently moved 5396 according to the 5397 following rules. 5398 - s_waitcnt vmcnt(0) 5399 must happen after 5400 any preceding 5401 global/generic load/load 5402 atomic/ 5403 atomicrmw-with-return-value. 5404 - s_waitcnt vscnt(0) 5405 must happen after 5406 any preceding 5407 global/generic 5408 store/store 5409 atomic/ 5410 atomicrmw-no-return-value. 5411 - s_waitcnt lgkmcnt(0) 5412 must happen after 5413 any preceding 5414 local/generic load/store/load 5415 atomic/store atomic/atomicrmw. 5416 - Must happen before - Must happen before 5417 the following the following 5418 atomicrmw. atomicrmw. 5419 - Ensures that all - Ensures that all 5420 memory operations memory operations 5421 to local have have 5422 completed before completed before 5423 performing the performing the 5424 atomicrmw that is atomicrmw that is 5425 being released. being released. 5426 5427 2. buffer/global/flat_atomic 2. buffer/global_atomic 5428 3. s_waitcnt vm/vscnt(0) 5429 5430 - If CU wavefront execution mode, omit vm/vscnt. 5431 - Use vmcnt if atomic with 5432 return and vscnt if atomic 5433 with no-return. 5434 waitcnt lgkmcnt(0). 5435 - Must happen before 5436 the following 5437 buffer_gl0_inv. 5438 - Ensures any 5439 following global 5440 data read is no 5441 older than the 5442 atomicrmw value 5443 being acquired. 5444 5445 4. buffer_gl0_inv 5446 5447 - If CU wavefront execution mode, omit. 5448 - Ensures that 5449 following 5450 loads will not see 5451 stale data. 5452 5453 atomicrmw acq_rel - workgroup - local 1. waitcnt vmcnt(0) & vscnt(0) 5454 5455 - If CU wavefront execution mode, omit. 5456 - If OpenCL, omit. 5457 - Could be split into 5458 separate s_waitcnt 5459 vmcnt(0) and s_waitcnt 5460 vscnt(0) to allow 5461 them to be 5462 independently moved 5463 according to the 5464 following rules. 5465 - s_waitcnt vmcnt(0) 5466 must happen after 5467 any preceding 5468 global/generic load/load 5469 atomic/ 5470 atomicrmw-with-return-value. 5471 - s_waitcnt vscnt(0) 5472 must happen after 5473 any preceding 5474 global/generic 5475 store/store atomic/ 5476 atomicrmw-no-return-value. 5477 - Must happen before 5478 the following 5479 store. 5480 - Ensures that all 5481 global memory 5482 operations have 5483 completed before 5484 performing the 5485 store that is being 5486 released. 5487 5488 1. ds_atomic 2. ds_atomic 5489 2. s_waitcnt lgkmcnt(0) 3. s_waitcnt lgkmcnt(0) 5490 5491 - If OpenCL, omit. - If OpenCL, omit. 5492 - Must happen before - Must happen before 5493 any following the following 5494 global/generic buffer_gl0_inv. 5495 load/load 5496 atomic/store/store 5497 atomic/atomicrmw. 5498 - Ensures any - Ensures any 5499 following global following global 5500 data read is no data read is no 5501 older than the load older than the load 5502 atomic value being atomic value being 5503 acquired. acquired. 5504 5505 4. buffer_gl0_inv 5506 5507 - If CU wavefront execution mode, omit. 5508 - If OpenCL omit. 5509 - Ensures that 5510 following 5511 loads will not see 5512 stale data. 5513 5514 atomicrmw acq_rel - workgroup - generic 1. s_waitcnt lgkmcnt(0) 1. s_waitcnt lgkmcnt(0) & 5515 vmcnt(0) & vscnt(0) 5516 5517 - If CU wavefront execution mode, omit vmcnt and 5518 vscnt. 5519 - If OpenCL, omit. - If OpenCL, omit 5520 waitcnt lgkmcnt(0). 5521 - Must happen after 5522 any preceding 5523 local/generic 5524 load/store/load 5525 atomic/store 5526 atomic/atomicrmw. 5527 - Could be split into 5528 separate s_waitcnt 5529 vmcnt(0), s_waitcnt 5530 vscnt(0) and s_waitcnt 5531 lgkmcnt(0) to allow 5532 them to be 5533 independently moved 5534 according to the 5535 following rules. 5536 - s_waitcnt vmcnt(0) 5537 must happen after 5538 any preceding 5539 global/generic load/load 5540 atomic/ 5541 atomicrmw-with-return-value. 5542 - s_waitcnt vscnt(0) 5543 must happen after 5544 any preceding 5545 global/generic 5546 store/store 5547 atomic/ 5548 atomicrmw-no-return-value. 5549 - s_waitcnt lgkmcnt(0) 5550 must happen after 5551 any preceding 5552 local/generic load/store/load 5553 atomic/store atomic/atomicrmw. 5554 - Must happen before - Must happen before 5555 the following the following 5556 atomicrmw. atomicrmw. 5557 - Ensures that all - Ensures that all 5558 memory operations memory operations 5559 to local have have 5560 completed before completed before 5561 performing the performing the 5562 atomicrmw that is atomicrmw that is 5563 being released. being released. 5564 5565 2. flat_atomic 2. flat_atomic 5566 3. s_waitcnt lgkmcnt(0) 3. s_waitcnt lgkmcnt(0) & 5567 vm/vscnt(0) 5568 5569 - If CU wavefront execution mode, omit vm/vscnt. 5570 - If OpenCL, omit. - If OpenCL, omit 5571 waitcnt lgkmcnt(0). 5572 - Must happen before - Must happen before 5573 any following the following 5574 global/generic buffer_gl0_inv. 5575 load/load 5576 atomic/store/store 5577 atomic/atomicrmw. 5578 - Ensures any - Ensures any 5579 following global following global 5580 data read is no data read is no 5581 older than the load older than the load 5582 atomic value being atomic value being 5583 acquired. acquired. 5584 5585 3. buffer_gl0_inv 5586 5587 - If CU wavefront execution mode, omit. 5588 - Ensures that 5589 following 5590 loads will not see 5591 stale data. 5592 5593 atomicrmw acq_rel - agent - global 1. s_waitcnt lgkmcnt(0) & 1. s_waitcnt lgkmcnt(0) & 5594 - system vmcnt(0) vmcnt(0) & vscnt(0) 5595 5596 - If OpenCL, omit - If OpenCL, omit 5597 lgkmcnt(0). lgkmcnt(0). 5598 - Could be split into - Could be split into 5599 separate s_waitcnt separate s_waitcnt 5600 vmcnt(0) and vmcnt(0), s_waitcnt 5601 s_waitcnt vscnt(0) and s_waitcnt 5602 lgkmcnt(0) to allow lgkmcnt(0) to allow 5603 them to be them to be 5604 independently moved independently moved 5605 according to the according to the 5606 following rules. following rules. 5607 - s_waitcnt vmcnt(0) - s_waitcnt vmcnt(0) 5608 must happen after must happen after 5609 any preceding any preceding 5610 global/generic global/generic 5611 load/store/load load/load atomic/ 5612 atomic/store atomicrmw-with-return-value. 5613 atomic/atomicrmw. 5614 - s_waitcnt vscnt(0) 5615 must happen after 5616 any preceding 5617 global/generic 5618 store/store atomic/ 5619 atomicrmw-no-return-value. 5620 - s_waitcnt lgkmcnt(0) - s_waitcnt lgkmcnt(0) 5621 must happen after must happen after 5622 any preceding any preceding 5623 local/generic local/generic 5624 load/store/load load/store/load 5625 atomic/store atomic/store 5626 atomic/atomicrmw. atomic/atomicrmw. 5627 - Must happen before - Must happen before 5628 the following the following 5629 atomicrmw. atomicrmw. 5630 - Ensures that all - Ensures that all 5631 memory operations memory operations 5632 to global have to global have 5633 completed before completed before 5634 performing the performing the 5635 atomicrmw that is atomicrmw that is 5636 being released. being released. 5637 5638 2. buffer/global/flat_atomic 2. buffer/global_atomic 5639 3. s_waitcnt vmcnt(0) 3. s_waitcnt vm/vscnt(0) 5640 5641 - Use vmcnt if atomic with 5642 return and vscnt if atomic 5643 with no-return. 5644 waitcnt lgkmcnt(0). 5645 - Must happen before - Must happen before 5646 following following 5647 buffer_wbinvl1_vol. buffer_gl*_inv. 5648 - Ensures the - Ensures the 5649 atomicrmw has atomicrmw has 5650 completed before completed before 5651 invalidating the invalidating the 5652 cache. caches. 5653 5654 4. buffer_wbinvl1_vol 4. buffer_gl0_inv; 5655 buffer_gl1_inv 5656 5657 - Must happen before - Must happen before 5658 any following any following 5659 global/generic global/generic 5660 load/load load/load 5661 atomic/atomicrmw. atomic/atomicrmw. 5662 - Ensures that - Ensures that 5663 following loads following loads 5664 will not see stale will not see stale 5665 global data. global data. 5666 5667 atomicrmw acq_rel - agent - generic 1. s_waitcnt lgkmcnt(0) & 1. s_waitcnt lgkmcnt(0) & 5668 - system vmcnt(0) vmcnt(0) & vscnt(0) 5669 5670 - If OpenCL, omit - If OpenCL, omit 5671 lgkmcnt(0). lgkmcnt(0). 5672 - Could be split into - Could be split into 5673 separate s_waitcnt separate s_waitcnt 5674 vmcnt(0) and vmcnt(0), s_waitcnt 5675 s_waitcnt vscnt(0) and s_waitcnt 5676 lgkmcnt(0) to allow lgkmcnt(0) to allow 5677 them to be them to be 5678 independently moved independently moved 5679 according to the according to the 5680 following rules. following rules. 5681 - s_waitcnt vmcnt(0) - s_waitcnt vmcnt(0) 5682 must happen after must happen after 5683 any preceding any preceding 5684 global/generic global/generic 5685 load/store/load load/load atomic 5686 atomic/store atomicrmw-with-return-value. 5687 atomic/atomicrmw. 5688 - s_waitcnt vscnt(0) 5689 must happen after 5690 any preceding 5691 global/generic 5692 store/store atomic/ 5693 atomicrmw-no-return-value. 5694 - s_waitcnt lgkmcnt(0) - s_waitcnt lgkmcnt(0) 5695 must happen after must happen after 5696 any preceding any preceding 5697 local/generic local/generic 5698 load/store/load load/store/load 5699 atomic/store atomic/store 5700 atomic/atomicrmw. atomic/atomicrmw. 5701 - Must happen before - Must happen before 5702 the following the following 5703 atomicrmw. atomicrmw. 5704 - Ensures that all - Ensures that all 5705 memory operations memory operations 5706 to global have have 5707 completed before completed before 5708 performing the performing the 5709 atomicrmw that is atomicrmw that is 5710 being released. being released. 5711 5712 2. flat_atomic 2. flat_atomic 5713 3. s_waitcnt vmcnt(0) & 3. s_waitcnt vm/vscnt(0) & 5714 lgkmcnt(0) lgkmcnt(0) 5715 5716 - If OpenCL, omit - If OpenCL, omit 5717 lgkmcnt(0). lgkmcnt(0). 5718 - Use vmcnt if atomic with 5719 return and vscnt if atomic 5720 with no-return. 5721 - Must happen before - Must happen before 5722 following following 5723 buffer_wbinvl1_vol. buffer_gl*_inv. 5724 - Ensures the - Ensures the 5725 atomicrmw has atomicrmw has 5726 completed before completed before 5727 invalidating the invalidating the 5728 cache. caches. 5729 5730 4. buffer_wbinvl1_vol 4. buffer_gl0_inv; 5731 buffer_gl1_inv 5732 5733 - Must happen before - Must happen before 5734 any following any following 5735 global/generic global/generic 5736 load/load load/load 5737 atomic/atomicrmw. atomic/atomicrmw. 5738 - Ensures that - Ensures that 5739 following loads following loads 5740 will not see stale will not see stale 5741 global data. global data. 5742 5743 fence acq_rel - singlethread *none* *none* *none* 5744 - wavefront 5745 fence acq_rel - workgroup *none* 1. s_waitcnt lgkmcnt(0) 1. s_waitcnt lgkmcnt(0) & 5746 vmcnt(0) & vscnt(0) 5747 5748 - If CU wavefront execution mode, omit vmcnt and 5749 vscnt. 5750 - If OpenCL and - If OpenCL and 5751 address space is address space is 5752 not generic, omit. not generic, omit 5753 lgkmcnt(0). 5754 - If OpenCL and 5755 address space is 5756 local, omit 5757 vmcnt(0) and vscnt(0). 5758 - However, - However, 5759 since LLVM since LLVM 5760 currently has no currently has no 5761 address space on address space on 5762 the fence need to the fence need to 5763 conservatively conservatively 5764 always generate always generate 5765 (see comment for (see comment for 5766 previous fence). previous fence). 5767 - Must happen after 5768 any preceding 5769 local/generic 5770 load/load 5771 atomic/store/store 5772 atomic/atomicrmw. 5773 - Could be split into 5774 separate s_waitcnt 5775 vmcnt(0), s_waitcnt 5776 vscnt(0) and s_waitcnt 5777 lgkmcnt(0) to allow 5778 them to be 5779 independently moved 5780 according to the 5781 following rules. 5782 - s_waitcnt vmcnt(0) 5783 must happen after 5784 any preceding 5785 global/generic 5786 load/load 5787 atomic/ 5788 atomicrmw-with-return-value. 5789 - s_waitcnt vscnt(0) 5790 must happen after 5791 any preceding 5792 global/generic 5793 store/store atomic/ 5794 atomicrmw-no-return-value. 5795 - s_waitcnt lgkmcnt(0) 5796 must happen after 5797 any preceding 5798 local/generic 5799 load/store/load 5800 atomic/store atomic/ 5801 atomicrmw. 5802 - Must happen before - Must happen before 5803 any following any following 5804 global/generic global/generic 5805 load/load load/load 5806 atomic/store/store atomic/store/store 5807 atomic/atomicrmw. atomic/atomicrmw. 5808 - Ensures that all - Ensures that all 5809 memory operations memory operations 5810 to local have have 5811 completed before completed before 5812 performing any performing any 5813 following global following global 5814 memory operations. memory operations. 5815 - Ensures that the - Ensures that the 5816 preceding preceding 5817 local/generic load local/generic load 5818 atomic/atomicrmw atomic/atomicrmw 5819 with an equal or with an equal or 5820 wider sync scope wider sync scope 5821 and memory ordering and memory ordering 5822 stronger than stronger than 5823 unordered (this is unordered (this is 5824 termed the termed the 5825 acquire-fence-paired-atomic acquire-fence-paired-atomic 5826 ) has completed ) has completed 5827 before following before following 5828 global memory global memory 5829 operations. This operations. This 5830 satisfies the satisfies the 5831 requirements of requirements of 5832 acquire. acquire. 5833 - Ensures that all - Ensures that all 5834 previous memory previous memory 5835 operations have operations have 5836 completed before a completed before a 5837 following following 5838 local/generic store local/generic store 5839 atomic/atomicrmw atomic/atomicrmw 5840 with an equal or with an equal or 5841 wider sync scope wider sync scope 5842 and memory ordering and memory ordering 5843 stronger than stronger than 5844 unordered (this is unordered (this is 5845 termed the termed the 5846 release-fence-paired-atomic release-fence-paired-atomic 5847 ). This satisfies the ). This satisfies the 5848 requirements of requirements of 5849 release. release. 5850 - Must happen before 5851 the following 5852 buffer_gl0_inv. 5853 - Ensures that the 5854 acquire-fence-paired 5855 atomic has completed 5856 before invalidating 5857 the 5858 cache. Therefore 5859 any following 5860 locations read must 5861 be no older than 5862 the value read by 5863 the 5864 acquire-fence-paired-atomic. 5865 5866 3. buffer_gl0_inv 5867 5868 - If CU wavefront execution mode, omit. 5869 - Ensures that 5870 following 5871 loads will not see 5872 stale data. 5873 5874 fence acq_rel - agent *none* 1. s_waitcnt lgkmcnt(0) & 1. s_waitcnt lgkmcnt(0) & 5875 - system vmcnt(0) vmcnt(0) & vscnt(0) 5876 5877 - If OpenCL and - If OpenCL and 5878 address space is address space is 5879 not generic, omit not generic, omit 5880 lgkmcnt(0). lgkmcnt(0). 5881 - If OpenCL and 5882 address space is 5883 local, omit 5884 vmcnt(0) and vscnt(0). 5885 - However, since LLVM - However, since LLVM 5886 currently has no currently has no 5887 address space on address space on 5888 the fence need to the fence need to 5889 conservatively conservatively 5890 always generate always generate 5891 (see comment for (see comment for 5892 previous fence). previous fence). 5893 - Could be split into - Could be split into 5894 separate s_waitcnt separate s_waitcnt 5895 vmcnt(0) and vmcnt(0), s_waitcnt 5896 s_waitcnt vscnt(0) and s_waitcnt 5897 lgkmcnt(0) to allow lgkmcnt(0) to allow 5898 them to be them to be 5899 independently moved independently moved 5900 according to the according to the 5901 following rules. following rules. 5902 - s_waitcnt vmcnt(0) - s_waitcnt vmcnt(0) 5903 must happen after must happen after 5904 any preceding any preceding 5905 global/generic global/generic 5906 load/store/load load/load 5907 atomic/store atomic/ 5908 atomic/atomicrmw. atomicrmw-with-return-value. 5909 - s_waitcnt vscnt(0) 5910 must happen after 5911 any preceding 5912 global/generic 5913 store/store atomic/ 5914 atomicrmw-no-return-value. 5915 - s_waitcnt lgkmcnt(0) - s_waitcnt lgkmcnt(0) 5916 must happen after must happen after 5917 any preceding any preceding 5918 local/generic local/generic 5919 load/store/load load/store/load 5920 atomic/store atomic/store 5921 atomic/atomicrmw. atomic/atomicrmw. 5922 - Must happen before - Must happen before 5923 the following the following 5924 buffer_wbinvl1_vol. buffer_gl*_inv. 5925 - Ensures that the - Ensures that the 5926 preceding preceding 5927 global/local/generic global/local/generic 5928 load load 5929 atomic/atomicrmw atomic/atomicrmw 5930 with an equal or with an equal or 5931 wider sync scope wider sync scope 5932 and memory ordering and memory ordering 5933 stronger than stronger than 5934 unordered (this is unordered (this is 5935 termed the termed the 5936 acquire-fence-paired-atomic acquire-fence-paired-atomic 5937 ) has completed ) has completed 5938 before invalidating before invalidating 5939 the cache. This the caches. This 5940 satisfies the satisfies the 5941 requirements of requirements of 5942 acquire. acquire. 5943 - Ensures that all - Ensures that all 5944 previous memory previous memory 5945 operations have operations have 5946 completed before a completed before a 5947 following following 5948 global/local/generic global/local/generic 5949 store store 5950 atomic/atomicrmw atomic/atomicrmw 5951 with an equal or with an equal or 5952 wider sync scope wider sync scope 5953 and memory ordering and memory ordering 5954 stronger than stronger than 5955 unordered (this is unordered (this is 5956 termed the termed the 5957 release-fence-paired-atomic release-fence-paired-atomic 5958 ). This satisfies the ). This satisfies the 5959 requirements of requirements of 5960 release. release. 5961 5962 2. buffer_wbinvl1_vol 2. buffer_gl0_inv; 5963 buffer_gl1_inv 5964 5965 - Must happen before - Must happen before 5966 any following any following 5967 global/generic global/generic 5968 load/load load/load 5969 atomic/store/store atomic/store/store 5970 atomic/atomicrmw. atomic/atomicrmw. 5971 - Ensures that - Ensures that 5972 following loads following loads 5973 will not see stale will not see stale 5974 global data. This global data. This 5975 satisfies the satisfies the 5976 requirements of requirements of 5977 acquire. acquire. 5978 5979 **Sequential Consistent Atomic** 5980 ---------------------------------------------------------------------------------------------------------------------- 5981 load atomic seq_cst - singlethread - global *Same as corresponding *Same as corresponding 5982 - wavefront - local load atomic acquire, load atomic acquire, 5983 - generic except must generated except must generated 5984 all instructions even all instructions even 5985 for OpenCL.* for OpenCL.* 5986 load atomic seq_cst - workgroup - global 1. s_waitcnt lgkmcnt(0) 1. s_waitcnt lgkmcnt(0) & 5987 - generic vmcnt(0) & vscnt(0) 5988 5989 - If CU wavefront execution mode, omit vmcnt and 5990 vscnt. 5991 - Could be split into 5992 separate s_waitcnt 5993 vmcnt(0), s_waitcnt 5994 vscnt(0) and s_waitcnt 5995 lgkmcnt(0) to allow 5996 them to be 5997 independently moved 5998 according to the 5999 following rules. 6000 - Must - waitcnt lgkmcnt(0) must 6001 happen after happen after 6002 preceding preceding 6003 global/generic load local load 6004 atomic/store atomic/store 6005 atomic/atomicrmw atomic/atomicrmw 6006 with memory with memory 6007 ordering of seq_cst ordering of seq_cst 6008 and with equal or and with equal or 6009 wider sync scope. wider sync scope. 6010 (Note that seq_cst (Note that seq_cst 6011 fences have their fences have their 6012 own s_waitcnt own s_waitcnt 6013 lgkmcnt(0) and so do lgkmcnt(0) and so do 6014 not need to be not need to be 6015 considered.) considered.) 6016 - waitcnt vmcnt(0) 6017 Must happen after 6018 preceding 6019 global/generic load 6020 atomic/ 6021 atomicrmw-with-return-value 6022 with memory 6023 ordering of seq_cst 6024 and with equal or 6025 wider sync scope. 6026 (Note that seq_cst 6027 fences have their 6028 own s_waitcnt 6029 vmcnt(0) and so do 6030 not need to be 6031 considered.) 6032 - waitcnt vscnt(0) 6033 Must happen after 6034 preceding 6035 global/generic store 6036 atomic/ 6037 atomicrmw-no-return-value 6038 with memory 6039 ordering of seq_cst 6040 and with equal or 6041 wider sync scope. 6042 (Note that seq_cst 6043 fences have their 6044 own s_waitcnt 6045 vscnt(0) and so do 6046 not need to be 6047 considered.) 6048 - Ensures any - Ensures any 6049 preceding preceding 6050 sequential sequential 6051 consistent local consistent global/local 6052 memory instructions memory instructions 6053 have completed have completed 6054 before executing before executing 6055 this sequentially this sequentially 6056 consistent consistent 6057 instruction. This instruction. This 6058 prevents reordering prevents reordering 6059 a seq_cst store a seq_cst store 6060 followed by a followed by a 6061 seq_cst load. (Note seq_cst load. (Note 6062 that seq_cst is that seq_cst is 6063 stronger than stronger than 6064 acquire/release as acquire/release as 6065 the reordering of the reordering of 6066 load acquire load acquire 6067 followed by a store followed by a store 6068 release is release is 6069 prevented by the prevented by the 6070 waitcnt of waitcnt of 6071 the release, but the release, but 6072 there is nothing there is nothing 6073 preventing a store preventing a store 6074 release followed by release followed by 6075 load acquire from load acquire from 6076 competing out of competing out of 6077 order.) order.) 6078 6079 2. *Following 2. *Following 6080 instructions same as instructions same as 6081 corresponding load corresponding load 6082 atomic acquire, atomic acquire, 6083 except must generated except must generated 6084 all instructions even all instructions even 6085 for OpenCL.* for OpenCL.* 6086 load atomic seq_cst - workgroup - local *Same as corresponding 6087 load atomic acquire, 6088 except must generated 6089 all instructions even 6090 for OpenCL.* 6091 6092 1. s_waitcnt vmcnt(0) & vscnt(0) 6093 6094 - If CU wavefront execution mode, omit. 6095 - Could be split into 6096 separate s_waitcnt 6097 vmcnt(0) and s_waitcnt 6098 vscnt(0) to allow 6099 them to be 6100 independently moved 6101 according to the 6102 following rules. 6103 - waitcnt vmcnt(0) 6104 Must happen after 6105 preceding 6106 global/generic load 6107 atomic/ 6108 atomicrmw-with-return-value 6109 with memory 6110 ordering of seq_cst 6111 and with equal or 6112 wider sync scope. 6113 (Note that seq_cst 6114 fences have their 6115 own s_waitcnt 6116 vmcnt(0) and so do 6117 not need to be 6118 considered.) 6119 - waitcnt vscnt(0) 6120 Must happen after 6121 preceding 6122 global/generic store 6123 atomic/ 6124 atomicrmw-no-return-value 6125 with memory 6126 ordering of seq_cst 6127 and with equal or 6128 wider sync scope. 6129 (Note that seq_cst 6130 fences have their 6131 own s_waitcnt 6132 vscnt(0) and so do 6133 not need to be 6134 considered.) 6135 - Ensures any 6136 preceding 6137 sequential 6138 consistent global 6139 memory instructions 6140 have completed 6141 before executing 6142 this sequentially 6143 consistent 6144 instruction. This 6145 prevents reordering 6146 a seq_cst store 6147 followed by a 6148 seq_cst load. (Note 6149 that seq_cst is 6150 stronger than 6151 acquire/release as 6152 the reordering of 6153 load acquire 6154 followed by a store 6155 release is 6156 prevented by the 6157 waitcnt of 6158 the release, but 6159 there is nothing 6160 preventing a store 6161 release followed by 6162 load acquire from 6163 competing out of 6164 order.) 6165 6166 2. *Following 6167 instructions same as 6168 corresponding load 6169 atomic acquire, 6170 except must generated 6171 all instructions even 6172 for OpenCL.* 6173 6174 load atomic seq_cst - agent - global 1. s_waitcnt lgkmcnt(0) & 1. s_waitcnt lgkmcnt(0) & 6175 - system - generic vmcnt(0) vmcnt(0) & vscnt(0) 6176 6177 - Could be split into - Could be split into 6178 separate s_waitcnt separate s_waitcnt 6179 vmcnt(0) vmcnt(0), s_waitcnt 6180 and s_waitcnt vscnt(0) and s_waitcnt 6181 lgkmcnt(0) to allow lgkmcnt(0) to allow 6182 them to be them to be 6183 independently moved independently moved 6184 according to the according to the 6185 following rules. following rules. 6186 - waitcnt lgkmcnt(0) - waitcnt lgkmcnt(0) 6187 must happen after must happen after 6188 preceding preceding 6189 global/generic load local load 6190 atomic/store atomic/store 6191 atomic/atomicrmw atomic/atomicrmw 6192 with memory with memory 6193 ordering of seq_cst ordering of seq_cst 6194 and with equal or and with equal or 6195 wider sync scope. wider sync scope. 6196 (Note that seq_cst (Note that seq_cst 6197 fences have their fences have their 6198 own s_waitcnt own s_waitcnt 6199 lgkmcnt(0) and so do lgkmcnt(0) and so do 6200 not need to be not need to be 6201 considered.) considered.) 6202 - waitcnt vmcnt(0) - waitcnt vmcnt(0) 6203 must happen after must happen after 6204 preceding preceding 6205 global/generic load global/generic load 6206 atomic/store atomic/ 6207 atomic/atomicrmw atomicrmw-with-return-value 6208 with memory with memory 6209 ordering of seq_cst ordering of seq_cst 6210 and with equal or and with equal or 6211 wider sync scope. wider sync scope. 6212 (Note that seq_cst (Note that seq_cst 6213 fences have their fences have their 6214 own s_waitcnt own s_waitcnt 6215 vmcnt(0) and so do vmcnt(0) and so do 6216 not need to be not need to be 6217 considered.) considered.) 6218 - waitcnt vscnt(0) 6219 Must happen after 6220 preceding 6221 global/generic store 6222 atomic/ 6223 atomicrmw-no-return-value 6224 with memory 6225 ordering of seq_cst 6226 and with equal or 6227 wider sync scope. 6228 (Note that seq_cst 6229 fences have their 6230 own s_waitcnt 6231 vscnt(0) and so do 6232 not need to be 6233 considered.) 6234 - Ensures any - Ensures any 6235 preceding preceding 6236 sequential sequential 6237 consistent global consistent global 6238 memory instructions memory instructions 6239 have completed have completed 6240 before executing before executing 6241 this sequentially this sequentially 6242 consistent consistent 6243 instruction. This instruction. This 6244 prevents reordering prevents reordering 6245 a seq_cst store a seq_cst store 6246 followed by a followed by a 6247 seq_cst load. (Note seq_cst load. (Note 6248 that seq_cst is that seq_cst is 6249 stronger than stronger than 6250 acquire/release as acquire/release as 6251 the reordering of the reordering of 6252 load acquire load acquire 6253 followed by a store followed by a store 6254 release is release is 6255 prevented by the prevented by the 6256 waitcnt of waitcnt of 6257 the release, but the release, but 6258 there is nothing there is nothing 6259 preventing a store preventing a store 6260 release followed by release followed by 6261 load acquire from load acquire from 6262 competing out of competing out of 6263 order.) order.) 6264 6265 2. *Following 2. *Following 6266 instructions same as instructions same as 6267 corresponding load corresponding load 6268 atomic acquire, atomic acquire, 6269 except must generated except must generated 6270 all instructions even all instructions even 6271 for OpenCL.* for OpenCL.* 6272 store atomic seq_cst - singlethread - global *Same as corresponding *Same as corresponding 6273 - wavefront - local store atomic release, store atomic release, 6274 - workgroup - generic except must generated except must generated 6275 all instructions even all instructions even 6276 for OpenCL.* for OpenCL.* 6277 store atomic seq_cst - agent - global *Same as corresponding *Same as corresponding 6278 - system - generic store atomic release, store atomic release, 6279 except must generated except must generated 6280 all instructions even all instructions even 6281 for OpenCL.* for OpenCL.* 6282 atomicrmw seq_cst - singlethread - global *Same as corresponding *Same as corresponding 6283 - wavefront - local atomicrmw acq_rel, atomicrmw acq_rel, 6284 - workgroup - generic except must generated except must generated 6285 all instructions even all instructions even 6286 for OpenCL.* for OpenCL.* 6287 atomicrmw seq_cst - agent - global *Same as corresponding *Same as corresponding 6288 - system - generic atomicrmw acq_rel, atomicrmw acq_rel, 6289 except must generated except must generated 6290 all instructions even all instructions even 6291 for OpenCL.* for OpenCL.* 6292 fence seq_cst - singlethread *none* *Same as corresponding *Same as corresponding 6293 - wavefront fence acq_rel, fence acq_rel, 6294 - workgroup except must generated except must generated 6295 - agent all instructions even all instructions even 6296 - system for OpenCL.* for OpenCL.* 6297 ============ ============ ============== ========== =============================== ================================== 6298 6299The memory order also adds the single thread optimization constrains defined in 6300table 6301:ref:`amdgpu-amdhsa-memory-model-single-thread-optimization-constraints-gfx6-gfx10-table`. 6302 6303 .. table:: AMDHSA Memory Model Single Thread Optimization Constraints GFX6-GFX10 6304 :name: amdgpu-amdhsa-memory-model-single-thread-optimization-constraints-gfx6-gfx10-table 6305 6306 ============ ============================================================== 6307 LLVM Memory Optimization Constraints 6308 Ordering 6309 ============ ============================================================== 6310 unordered *none* 6311 monotonic *none* 6312 acquire - If a load atomic/atomicrmw then no following load/load 6313 atomic/store/ store atomic/atomicrmw/fence instruction can 6314 be moved before the acquire. 6315 - If a fence then same as load atomic, plus no preceding 6316 associated fence-paired-atomic can be moved after the fence. 6317 release - If a store atomic/atomicrmw then no preceding load/load 6318 atomic/store/ store atomic/atomicrmw/fence instruction can 6319 be moved after the release. 6320 - If a fence then same as store atomic, plus no following 6321 associated fence-paired-atomic can be moved before the 6322 fence. 6323 acq_rel Same constraints as both acquire and release. 6324 seq_cst - If a load atomic then same constraints as acquire, plus no 6325 preceding sequentially consistent load atomic/store 6326 atomic/atomicrmw/fence instruction can be moved after the 6327 seq_cst. 6328 - If a store atomic then the same constraints as release, plus 6329 no following sequentially consistent load atomic/store 6330 atomic/atomicrmw/fence instruction can be moved before the 6331 seq_cst. 6332 - If an atomicrmw/fence then same constraints as acq_rel. 6333 ============ ============================================================== 6334 6335Trap Handler ABI 6336~~~~~~~~~~~~~~~~ 6337 6338For code objects generated by AMDGPU backend for HSA [HSA]_ compatible runtimes 6339(such as ROCm [AMD-ROCm]_), the runtime installs a trap handler that supports 6340the ``s_trap`` instruction with the following usage: 6341 6342 .. table:: AMDGPU Trap Handler for AMDHSA OS 6343 :name: amdgpu-trap-handler-for-amdhsa-os-table 6344 6345 =================== =============== =============== ======================= 6346 Usage Code Sequence Trap Handler Description 6347 Inputs 6348 =================== =============== =============== ======================= 6349 reserved ``s_trap 0x00`` Reserved by hardware. 6350 ``debugtrap(arg)`` ``s_trap 0x01`` ``SGPR0-1``: Reserved for HSA 6351 ``queue_ptr`` ``debugtrap`` 6352 ``VGPR0``: intrinsic (not 6353 ``arg`` implemented). 6354 ``llvm.trap`` ``s_trap 0x02`` ``SGPR0-1``: Causes dispatch to be 6355 ``queue_ptr`` terminated and its 6356 associated queue put 6357 into the error state. 6358 ``llvm.debugtrap`` ``s_trap 0x03`` - If debugger not 6359 installed then 6360 behaves as a 6361 no-operation. The 6362 trap handler is 6363 entered and 6364 immediately returns 6365 to continue 6366 execution of the 6367 wavefront. 6368 - If the debugger is 6369 installed, causes 6370 the debug trap to be 6371 reported by the 6372 debugger and the 6373 wavefront is put in 6374 the halt state until 6375 resumed by the 6376 debugger. 6377 reserved ``s_trap 0x04`` Reserved. 6378 reserved ``s_trap 0x05`` Reserved. 6379 reserved ``s_trap 0x06`` Reserved. 6380 debugger breakpoint ``s_trap 0x07`` Reserved for debugger 6381 breakpoints. 6382 reserved ``s_trap 0x08`` Reserved. 6383 reserved ``s_trap 0xfe`` Reserved. 6384 reserved ``s_trap 0xff`` Reserved. 6385 =================== =============== =============== ======================= 6386 6387.. _amdgpu-amdhsa-function-call-convention: 6388 6389Call Convention 6390~~~~~~~~~~~~~~~ 6391 6392.. note:: 6393 6394 This section is currently incomplete and has inakkuracies. It is WIP that will 6395 be updated as information is determined. 6396 6397See :ref:`amdgpu-dwarf-address-space-identifier` for information on swizzled 6398addresses. Unswizzled addresses are normal linear addresses. 6399 6400.. _amdgpu-amdhsa-function-call-convention-kernel-functions: 6401 6402Kernel Functions 6403++++++++++++++++ 6404 6405This section describes the call convention ABI for the outer kernel function. 6406 6407See :ref:`amdgpu-amdhsa-initial-kernel-execution-state` for the kernel call 6408convention. 6409 6410The following is not part of the AMDGPU kernel calling convention but describes 6411how the AMDGPU implements function calls: 6412 64131. Clang decides the kernarg layout to match the *HSA Programmer's Language 6414 Reference* [HSA]_. 6415 6416 - All structs are passed directly. 6417 - Lambda values are passed *TBA*. 6418 6419 .. TODO:: 6420 6421 - Does this really follow HSA rules? Or are structs >16 bytes passed 6422 by-value struct? 6423 - What is ABI for lambda values? 6424 64254. The kernel performs certain setup in its prolog, as described in 6426 :ref:`amdgpu-amdhsa-kernel-prolog`. 6427 6428.. _amdgpu-amdhsa-function-call-convention-non-kernel-functions: 6429 6430Non-Kernel Functions 6431++++++++++++++++++++ 6432 6433This section describes the call convention ABI for functions other than the 6434outer kernel function. 6435 6436If a kernel has function calls then scratch is always allocated and used for 6437the call stack which grows from low address to high address using the swizzled 6438scratch address space. 6439 6440On entry to a function: 6441 64421. SGPR0-3 contain a V# with the following properties (see 6443 :ref:`amdgpu-amdhsa-kernel-prolog-private-segment-buffer`): 6444 6445 * Base address pointing to the beginning of the wavefront scratch backing 6446 memory. 6447 * Swizzled with dword element size and stride of wavefront size elements. 6448 64492. The FLAT_SCRATCH register pair is setup. See 6450 :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`. 64513. GFX6-8: M0 register set to the size of LDS in bytes. See 6452 :ref:`amdgpu-amdhsa-kernel-prolog-m0`. 64534. The EXEC register is set to the lanes active on entry to the function. 64545. MODE register: *TBD* 64556. VGPR0-31 and SGPR4-29 are used to pass function input arguments as described 6456 below. 64577. SGPR30-31 return address (RA). The code address that the function must 6458 return to when it completes. The value is undefined if the function is *no 6459 return*. 64608. SGPR32 is used for the stack pointer (SP). It is an unswizzled scratch 6461 offset relative to the beginning of the wavefront scratch backing memory. 6462 6463 The unswizzled SP can be used with buffer instructions as an unswizzled SGPR 6464 offset with the scratch V# in SGPR0-3 to access the stack in a swizzled 6465 manner. 6466 6467 The unswizzled SP value can be converted into the swizzled SP value by: 6468 6469 | swizzled SP = unswizzled SP / wavefront size 6470 6471 This may be used to obtain the private address space address of stack 6472 objects and to convert this address to a flat address by adding the flat 6473 scratch aperture base address. 6474 6475 The swizzled SP value is always 4 bytes aligned for the ``r600`` 6476 architecture and 16 byte aligned for the ``amdgcn`` architecture. 6477 6478 .. note:: 6479 6480 The ``amdgcn`` value is selected to avoid dynamic stack alignment for the 6481 OpenCL language which has the largest base type defined as 16 bytes. 6482 6483 On entry, the swizzled SP value is the address of the first function 6484 argument passed on the stack. Other stack passed arguments are positive 6485 offsets from the entry swizzled SP value. 6486 6487 The function may use positive offsets beyond the last stack passed argument 6488 for stack allocated local variables and register spill slots. If necessary, 6489 the function may align these to greater alignment than 16 bytes. After these 6490 the function may dynamically allocate space for such things as runtime sized 6491 ``alloca`` local allocations. 6492 6493 If the function calls another function, it will place any stack allocated 6494 arguments after the last local allocation and adjust SGPR32 to the address 6495 after the last local allocation. 6496 64979. All other registers are unspecified. 649810. Any necessary ``waitcnt`` has been performed to ensure memory is available 6499 to the function. 6500 6501On exit from a function: 6502 65031. VGPR0-31 and SGPR4-29 are used to pass function result arguments as 6504 described below. Any registers used are considered clobbered registers. 65052. The following registers are preserved and have the same value as on entry: 6506 6507 * FLAT_SCRATCH 6508 * EXEC 6509 * GFX6-8: M0 6510 * All SGPR and VGPR registers except the clobbered registers of SGPR4-31 and 6511 VGPR0-31. 6512 6513 For the AMDGPU backend, an inter-procedural register allocation (IPRA) 6514 optimization may mark some of clobbered SGPR4-31 and VGPR0-31 registers as 6515 preserved if it can be determined that the called function does not change 6516 their value. 6517 65182. The PC is set to the RA provided on entry. 65193. MODE register: *TBD*. 65204. All other registers are clobbered. 65215. Any necessary ``waitcnt`` has been performed to ensure memory accessed by 6522 function is available to the caller. 6523 6524.. TODO:: 6525 6526 - On gfx908 are all ACC registers clobbered? 6527 6528 - How are function results returned? The address of structured types is passed 6529 by reference, but what about other types? 6530 6531The function input arguments are made up of the formal arguments explicitly 6532declared by the source language function plus the implicit input arguments used 6533by the implementation. 6534 6535The source language input arguments are: 6536 65371. Any source language implicit ``this`` or ``self`` argument comes first as a 6538 pointer type. 65392. Followed by the function formal arguments in left to right source order. 6540 6541The source language result arguments are: 6542 65431. The function result argument. 6544 6545The source language input or result struct type arguments that are less than or 6546equal to 16 bytes, are decomposed recursively into their base type fields, and 6547each field is passed as if a separate argument. For input arguments, if the 6548called function requires the struct to be in memory, for example because its 6549address is taken, then the function body is responsible for allocating a stack 6550location and copying the field arguments into it. Clang terms this *direct 6551struct*. 6552 6553The source language input struct type arguments that are greater than 16 bytes, 6554are passed by reference. The caller is responsible for allocating a stack 6555location to make a copy of the struct value and pass the address as the input 6556argument. The called function is responsible to perform the dereference when 6557accessing the input argument. Clang terms this *by-value struct*. 6558 6559A source language result struct type argument that is greater than 16 bytes, is 6560returned by reference. The caller is responsible for allocating a stack location 6561to hold the result value and passes the address as the last input argument 6562(before the implicit input arguments). In this case there are no result 6563arguments. The called function is responsible to perform the dereference when 6564storing the result value. Clang terms this *structured return (sret)*. 6565 6566*TODO: correct the ``sret`` definition.* 6567 6568.. TODO:: 6569 6570 Is this definition correct? Or is ``sret`` only used if passing in registers, and 6571 pass as non-decomposed struct as stack argument? Or something else? Is the 6572 memory location in the caller stack frame, or a stack memory argument and so 6573 no address is passed as the caller can directly write to the argument stack 6574 location? But then the stack location is still live after return. If an 6575 argument stack location is it the first stack argument or the last one? 6576 6577Lambda argument types are treated as struct types with an implementation defined 6578set of fields. 6579 6580.. TODO:: 6581 6582 Need to specify the ABI for lambda types for AMDGPU. 6583 6584For AMDGPU backend all source language arguments (including the decomposed 6585struct type arguments) are passed in VGPRs unless marked ``inreg`` in which case 6586they are passed in SGPRs. 6587 6588The AMDGPU backend walks the function call graph from the leaves to determine 6589which implicit input arguments are used, propagating to each caller of the 6590function. The used implicit arguments are appended to the function arguments 6591after the source language arguments in the following order: 6592 6593.. TODO:: 6594 6595 Is recursion or external functions supported? 6596 65971. Work-Item ID (1 VGPR) 6598 6599 The X, Y and Z work-item ID are packed into a single VGRP with the following 6600 layout. Only fields actually used by the function are set. The other bits 6601 are undefined. 6602 6603 The values come from the initial kernel execution state. See 6604 :ref:`amdgpu-amdhsa-vgpr-register-set-up-order-table`. 6605 6606 .. table:: Work-item implict argument layout 6607 :name: amdgpu-amdhsa-workitem-implict-argument-layout-table 6608 6609 ======= ======= ============== 6610 Bits Size Field Name 6611 ======= ======= ============== 6612 9:0 10 bits X Work-Item ID 6613 19:10 10 bits Y Work-Item ID 6614 29:20 10 bits Z Work-Item ID 6615 31:30 2 bits Unused 6616 ======= ======= ============== 6617 66182. Dispatch Ptr (2 SGPRs) 6619 6620 The value comes from the initial kernel execution state. See 6621 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 6622 66233. Queue Ptr (2 SGPRs) 6624 6625 The value comes from the initial kernel execution state. See 6626 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 6627 66284. Kernarg Segment Ptr (2 SGPRs) 6629 6630 The value comes from the initial kernel execution state. See 6631 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 6632 66335. Dispatch id (2 SGPRs) 6634 6635 The value comes from the initial kernel execution state. See 6636 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 6637 66386. Work-Group ID X (1 SGPR) 6639 6640 The value comes from the initial kernel execution state. See 6641 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 6642 66437. Work-Group ID Y (1 SGPR) 6644 6645 The value comes from the initial kernel execution state. See 6646 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 6647 66488. Work-Group ID Z (1 SGPR) 6649 6650 The value comes from the initial kernel execution state. See 6651 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 6652 66539. Implicit Argument Ptr (2 SGPRs) 6654 6655 The value is computed by adding an offset to Kernarg Segment Ptr to get the 6656 global address space pointer to the first kernarg implicit argument. 6657 6658The input and result arguments are assigned in order in the following manner: 6659 6660..note:: 6661 6662 There are likely some errors and omissions in the following description that 6663 need correction. 6664 6665 ..TODO:: 6666 6667 Check the clang source code to decipher how function arguments and return 6668 results are handled. Also see the AMDGPU specific values used. 6669 6670* VGPR arguments are assigned to consecutive VGPRs starting at VGPR0 up to 6671 VGPR31. 6672 6673 If there are more arguments than will fit in these registers, the remaining 6674 arguments are allocated on the stack in order on naturally aligned 6675 addresses. 6676 6677 .. TODO:: 6678 6679 How are overly aligned structures allocated on the stack? 6680 6681* SGPR arguments are assigned to consecutive SGPRs starting at SGPR0 up to 6682 SGPR29. 6683 6684 If there are more arguments than will fit in these registers, the remaining 6685 arguments are allocated on the stack in order on naturally aligned 6686 addresses. 6687 6688Note that decomposed struct type arguments may have some fields passed in 6689registers and some in memory. 6690 6691.. TODO:: 6692 6693 So, a struct which can pass some fields as decomposed register arguments, will 6694 pass the rest as decomposed stack elements? But an argument that will not start 6695 in registers will not be decomposed and will be passed as a non-decomposed 6696 stack value? 6697 6698The following is not part of the AMDGPU function calling convention but 6699describes how the AMDGPU implements function calls: 6700 67011. SGPR33 is used as a frame pointer (FP) if necessary. Like the SP it is an 6702 unswizzled scratch address. It is only needed if runtime sized ``alloca`` 6703 are used, or for the reasons defined in ``SIFrameLowering``. 67042. Runtime stack alignment is not currently supported. 6705 6706 .. TODO:: 6707 6708 - If runtime stack alignment is supported, then will an extra argument 6709 pointer register be used? 6710 67112. Allocating SGPR arguments on the stack are not supported. 6712 67133. No CFI is currently generated. See 6714 :ref:`amdgpu-dwarf-call-frame-information`. 6715 6716 ..note:: 6717 6718 CFI will be generated that defines the CFA as the unswizzled address 6719 relative to the wave scratch base in the unswizzled private address space 6720 of the lowest address stack allocated local variable. 6721 6722 ``DW_AT_frame_base`` will be defined as the swizzled address in the 6723 swizzled private address space by dividing the CFA by the wavefront size 6724 (since CFA is always at least dword aligned which matches the scratch 6725 swizzle element size). 6726 6727 If no dynamic stack alignment was performed, the stack allocated arguments 6728 are accessed as negative offsets relative to ``DW_AT_frame_base``, and the 6729 local variables and register spill slots are accessed as positive offsets 6730 relative to ``DW_AT_frame_base``. 6731 67324. Function argument passing is implemented by copying the input physical 6733 registers to virtual registers on entry. The register allocator can spill if 6734 necessary. These are copied back to physical registers at call sites. The 6735 net effect is that each function call can have these values in entirely 6736 distinct locations. The IPRA can help avoid shuffling argument registers. 67375. Call sites are implemented by setting up the arguments at positive offsets 6738 from SP. Then SP is incremented to account for the known frame size before 6739 the call and decremented after the call. 6740 6741 ..note:: 6742 6743 The CFI will reflect the changed calculation needed to compute the CFA 6744 from SP. 6745 67466. 4 byte spill slots are used in the stack frame. One slot is allocated for an 6747 emergency spill slot. Buffer instructions are used for stack accesses and 6748 not the ``flat_scratch`` instruction. 6749 6750 ..TODO:: 6751 6752 Explain when the emergency spill slot is used. 6753 6754.. TODO:: 6755 6756 Possible broken issues: 6757 6758 - Stack arguments must be aligned to required alignment. 6759 - Stack is aligned to max(16, max formal argument alignment) 6760 - Direct argument < 64 bits should check register budget. 6761 - Register budget calculation should respect ``inreg`` for SGPR. 6762 - SGPR overflow is not handled. 6763 - struct with 1 member unpeeling is not checking size of member. 6764 - ``sret`` is after ``this`` pointer. 6765 - Caller is not implementing stack realignment: need an extra pointer. 6766 - Should say AMDGPU passes FP rather than SP. 6767 - Should CFI define CFA as address of locals or arguments. Difference is 6768 apparent when have implemented dynamic alignment. 6769 - If ``SCRATCH`` instruction could allow negative offsets, then can make FP be 6770 highest address of stack frame and use negative offset for locals. Would 6771 allow SP to be the same as FP and could support signal-handler-like as now 6772 have a real SP for the top of the stack. 6773 - How is ``sret`` passed on the stack? In argument stack area? Can it overlay 6774 arguments? 6775 6776AMDPAL 6777------ 6778 6779This section provides code conventions used when the target triple OS is 6780``amdpal`` (see :ref:`amdgpu-target-triples`) for passing runtime parameters 6781from the application/runtime to each invocation of a hardware shader. These 6782parameters include both generic, application-controlled parameters called 6783*user data* as well as system-generated parameters that are a product of the 6784draw or dispatch execution. 6785 6786User Data 6787~~~~~~~~~ 6788 6789Each hardware stage has a set of 32-bit *user data registers* which can be 6790written from a command buffer and then loaded into SGPRs when waves are launched 6791via a subsequent dispatch or draw operation. This is the way most arguments are 6792passed from the application/runtime to a hardware shader. 6793 6794Compute User Data 6795~~~~~~~~~~~~~~~~~ 6796 6797Compute shader user data mappings are simpler than graphics shaders and have a 6798fixed mapping. 6799 6800Note that there are always 10 available *user data entries* in registers - 6801entries beyond that limit must be fetched from memory (via the spill table 6802pointer) by the shader. 6803 6804 .. table:: PAL Compute Shader User Data Registers 6805 :name: pal-compute-user-data-registers 6806 6807 ============= ================================ 6808 User Register Description 6809 ============= ================================ 6810 0 Global Internal Table (32-bit pointer) 6811 1 Per-Shader Internal Table (32-bit pointer) 6812 2 - 11 Application-Controlled User Data (10 32-bit values) 6813 12 Spill Table (32-bit pointer) 6814 13 - 14 Thread Group Count (64-bit pointer) 6815 15 GDS Range 6816 ============= ================================ 6817 6818Graphics User Data 6819~~~~~~~~~~~~~~~~~~ 6820 6821Graphics pipelines support a much more flexible user data mapping: 6822 6823 .. table:: PAL Graphics Shader User Data Registers 6824 :name: pal-graphics-user-data-registers 6825 6826 ============= ================================ 6827 User Register Description 6828 ============= ================================ 6829 0 Global Internal Table (32-bit pointer) 6830 + Per-Shader Internal Table (32-bit pointer) 6831 + 1-15 Application Controlled User Data 6832 (1-15 Contiguous 32-bit Values in Registers) 6833 + Spill Table (32-bit pointer) 6834 + Draw Index (First Stage Only) 6835 + Vertex Offset (First Stage Only) 6836 + Instance Offset (First Stage Only) 6837 ============= ================================ 6838 6839 The placement of the global internal table remains fixed in the first *user 6840 data SGPR register*. Otherwise all parameters are optional, and can be mapped 6841 to any desired *user data SGPR register*, with the following restrictions: 6842 6843 * Draw Index, Vertex Offset, and Instance Offset can only be used by the first 6844 active hardware stage in a graphics pipeline (i.e. where the API vertex 6845 shader runs). 6846 6847 * Application-controlled user data must be mapped into a contiguous range of 6848 user data registers. 6849 6850 * The application-controlled user data range supports compaction remapping, so 6851 only *entries* that are actually consumed by the shader must be assigned to 6852 corresponding *registers*. Note that in order to support an efficient runtime 6853 implementation, the remapping must pack *registers* in the same order as 6854 *entries*, with unused *entries* removed. 6855 6856.. _pal_global_internal_table: 6857 6858Global Internal Table 6859~~~~~~~~~~~~~~~~~~~~~ 6860 6861The global internal table is a table of *shader resource descriptors* (SRDs) 6862that define how certain engine-wide, runtime-managed resources should be 6863accessed from a shader. The majority of these resources have HW-defined formats, 6864and it is up to the compiler to write/read data as required by the target 6865hardware. 6866 6867The following table illustrates the required format: 6868 6869 .. table:: PAL Global Internal Table 6870 :name: pal-git-table 6871 6872 ============= ================================ 6873 Offset Description 6874 ============= ================================ 6875 0-3 Graphics Scratch SRD 6876 4-7 Compute Scratch SRD 6877 8-11 ES/GS Ring Output SRD 6878 12-15 ES/GS Ring Input SRD 6879 16-19 GS/VS Ring Output #0 6880 20-23 GS/VS Ring Output #1 6881 24-27 GS/VS Ring Output #2 6882 28-31 GS/VS Ring Output #3 6883 32-35 GS/VS Ring Input SRD 6884 36-39 Tessellation Factor Buffer SRD 6885 40-43 Off-Chip LDS Buffer SRD 6886 44-47 Off-Chip Param Cache Buffer SRD 6887 48-51 Sample Position Buffer SRD 6888 52 vaRange::ShadowDescriptorTable High Bits 6889 ============= ================================ 6890 6891 The pointer to the global internal table passed to the shader as user data 6892 is a 32-bit pointer. The top 32 bits should be assumed to be the same as 6893 the top 32 bits of the pipeline, so the shader may use the program 6894 counter's top 32 bits. 6895 6896Unspecified OS 6897-------------- 6898 6899This section provides code conventions used when the target triple OS is 6900empty (see :ref:`amdgpu-target-triples`). 6901 6902Trap Handler ABI 6903~~~~~~~~~~~~~~~~ 6904 6905For code objects generated by AMDGPU backend for non-amdhsa OS, the runtime does 6906not install a trap handler. The ``llvm.trap`` and ``llvm.debugtrap`` 6907instructions are handled as follows: 6908 6909 .. table:: AMDGPU Trap Handler for Non-AMDHSA OS 6910 :name: amdgpu-trap-handler-for-non-amdhsa-os-table 6911 6912 =============== =============== =========================================== 6913 Usage Code Sequence Description 6914 =============== =============== =========================================== 6915 llvm.trap s_endpgm Causes wavefront to be terminated. 6916 llvm.debugtrap *none* Compiler warning given that there is no 6917 trap handler installed. 6918 =============== =============== =========================================== 6919 6920Source Languages 6921================ 6922 6923.. _amdgpu-opencl: 6924 6925OpenCL 6926------ 6927 6928When the language is OpenCL the following differences occur: 6929 69301. The OpenCL memory model is used (see :ref:`amdgpu-amdhsa-memory-model`). 69312. The AMDGPU backend appends additional arguments to the kernel's explicit 6932 arguments for the AMDHSA OS (see 6933 :ref:`opencl-kernel-implicit-arguments-appended-for-amdhsa-os-table`). 69343. Additional metadata is generated 6935 (see :ref:`amdgpu-amdhsa-code-object-metadata`). 6936 6937 .. table:: OpenCL kernel implicit arguments appended for AMDHSA OS 6938 :name: opencl-kernel-implicit-arguments-appended-for-amdhsa-os-table 6939 6940 ======== ==== ========= =========================================== 6941 Position Byte Byte Description 6942 Size Alignment 6943 ======== ==== ========= =========================================== 6944 1 8 8 OpenCL Global Offset X 6945 2 8 8 OpenCL Global Offset Y 6946 3 8 8 OpenCL Global Offset Z 6947 4 8 8 OpenCL address of printf buffer 6948 5 8 8 OpenCL address of virtual queue used by 6949 enqueue_kernel. 6950 6 8 8 OpenCL address of AqlWrap struct used by 6951 enqueue_kernel. 6952 7 8 8 Pointer argument used for Multi-gird 6953 synchronization. 6954 ======== ==== ========= =========================================== 6955 6956.. _amdgpu-hcc: 6957 6958HCC 6959--- 6960 6961When the language is HCC the following differences occur: 6962 69631. The HSA memory model is used (see :ref:`amdgpu-amdhsa-memory-model`). 6964 6965.. _amdgpu-assembler: 6966 6967Assembler 6968--------- 6969 6970AMDGPU backend has LLVM-MC based assembler which is currently in development. 6971It supports AMDGCN GFX6-GFX10. 6972 6973This section describes general syntax for instructions and operands. 6974 6975Instructions 6976~~~~~~~~~~~~ 6977 6978An instruction has the following :doc:`syntax<AMDGPUInstructionSyntax>`: 6979 6980 | ``<``\ *opcode*\ ``> <``\ *operand0*\ ``>, <``\ *operand1*\ ``>,... 6981 <``\ *modifier0*\ ``> <``\ *modifier1*\ ``>...`` 6982 6983:doc:`Operands<AMDGPUOperandSyntax>` are comma-separated while 6984:doc:`modifiers<AMDGPUModifierSyntax>` are space-separated. 6985 6986The order of operands and modifiers is fixed. 6987Most modifiers are optional and may be omitted. 6988 6989Links to detailed instruction syntax description may be found in the following 6990table. Note that features under development are not included 6991in this description. 6992 6993 =================================== ======================================= 6994 Core ISA ISA Extensions 6995 =================================== ======================================= 6996 :doc:`GFX7<AMDGPU/AMDGPUAsmGFX7>` \- 6997 :doc:`GFX8<AMDGPU/AMDGPUAsmGFX8>` \- 6998 :doc:`GFX9<AMDGPU/AMDGPUAsmGFX9>` :doc:`gfx900<AMDGPU/AMDGPUAsmGFX900>` 6999 7000 :doc:`gfx902<AMDGPU/AMDGPUAsmGFX900>` 7001 7002 :doc:`gfx904<AMDGPU/AMDGPUAsmGFX904>` 7003 7004 :doc:`gfx906<AMDGPU/AMDGPUAsmGFX906>` 7005 7006 :doc:`gfx908<AMDGPU/AMDGPUAsmGFX908>` 7007 7008 :doc:`gfx909<AMDGPU/AMDGPUAsmGFX900>` 7009 7010 :doc:`GFX10<AMDGPU/AMDGPUAsmGFX10>` :doc:`gfx1011<AMDGPU/AMDGPUAsmGFX1011>` 7011 7012 :doc:`gfx1012<AMDGPU/AMDGPUAsmGFX1011>` 7013 =================================== ======================================= 7014 7015For more information about instructions, their semantics and supported 7016combinations of operands, refer to one of instruction set architecture manuals 7017[AMD-GCN-GFX6]_, [AMD-GCN-GFX7]_, [AMD-GCN-GFX8]_, [AMD-GCN-GFX9]_ and 7018[AMD-GCN-GFX10]_. 7019 7020Operands 7021~~~~~~~~ 7022 7023Detailed description of operands may be found :doc:`here<AMDGPUOperandSyntax>`. 7024 7025Modifiers 7026~~~~~~~~~ 7027 7028Detailed description of modifiers may be found 7029:doc:`here<AMDGPUModifierSyntax>`. 7030 7031Instruction Examples 7032~~~~~~~~~~~~~~~~~~~~ 7033 7034DS 7035++ 7036 7037.. code-block:: nasm 7038 7039 ds_add_u32 v2, v4 offset:16 7040 ds_write_src2_b64 v2 offset0:4 offset1:8 7041 ds_cmpst_f32 v2, v4, v6 7042 ds_min_rtn_f64 v[8:9], v2, v[4:5] 7043 7044For full list of supported instructions, refer to "LDS/GDS instructions" in ISA 7045Manual. 7046 7047FLAT 7048++++ 7049 7050.. code-block:: nasm 7051 7052 flat_load_dword v1, v[3:4] 7053 flat_store_dwordx3 v[3:4], v[5:7] 7054 flat_atomic_swap v1, v[3:4], v5 glc 7055 flat_atomic_cmpswap v1, v[3:4], v[5:6] glc slc 7056 flat_atomic_fmax_x2 v[1:2], v[3:4], v[5:6] glc 7057 7058For full list of supported instructions, refer to "FLAT instructions" in ISA 7059Manual. 7060 7061MUBUF 7062+++++ 7063 7064.. code-block:: nasm 7065 7066 buffer_load_dword v1, off, s[4:7], s1 7067 buffer_store_dwordx4 v[1:4], v2, ttmp[4:7], s1 offen offset:4 glc tfe 7068 buffer_store_format_xy v[1:2], off, s[4:7], s1 7069 buffer_wbinvl1 7070 buffer_atomic_inc v1, v2, s[8:11], s4 idxen offset:4 slc 7071 7072For full list of supported instructions, refer to "MUBUF Instructions" in ISA 7073Manual. 7074 7075SMRD/SMEM 7076+++++++++ 7077 7078.. code-block:: nasm 7079 7080 s_load_dword s1, s[2:3], 0xfc 7081 s_load_dwordx8 s[8:15], s[2:3], s4 7082 s_load_dwordx16 s[88:103], s[2:3], s4 7083 s_dcache_inv_vol 7084 s_memtime s[4:5] 7085 7086For full list of supported instructions, refer to "Scalar Memory Operations" in 7087ISA Manual. 7088 7089SOP1 7090++++ 7091 7092.. code-block:: nasm 7093 7094 s_mov_b32 s1, s2 7095 s_mov_b64 s[0:1], 0x80000000 7096 s_cmov_b32 s1, 200 7097 s_wqm_b64 s[2:3], s[4:5] 7098 s_bcnt0_i32_b64 s1, s[2:3] 7099 s_swappc_b64 s[2:3], s[4:5] 7100 s_cbranch_join s[4:5] 7101 7102For full list of supported instructions, refer to "SOP1 Instructions" in ISA 7103Manual. 7104 7105SOP2 7106++++ 7107 7108.. code-block:: nasm 7109 7110 s_add_u32 s1, s2, s3 7111 s_and_b64 s[2:3], s[4:5], s[6:7] 7112 s_cselect_b32 s1, s2, s3 7113 s_andn2_b32 s2, s4, s6 7114 s_lshr_b64 s[2:3], s[4:5], s6 7115 s_ashr_i32 s2, s4, s6 7116 s_bfm_b64 s[2:3], s4, s6 7117 s_bfe_i64 s[2:3], s[4:5], s6 7118 s_cbranch_g_fork s[4:5], s[6:7] 7119 7120For full list of supported instructions, refer to "SOP2 Instructions" in ISA 7121Manual. 7122 7123SOPC 7124++++ 7125 7126.. code-block:: nasm 7127 7128 s_cmp_eq_i32 s1, s2 7129 s_bitcmp1_b32 s1, s2 7130 s_bitcmp0_b64 s[2:3], s4 7131 s_setvskip s3, s5 7132 7133For full list of supported instructions, refer to "SOPC Instructions" in ISA 7134Manual. 7135 7136SOPP 7137++++ 7138 7139.. code-block:: nasm 7140 7141 s_barrier 7142 s_nop 2 7143 s_endpgm 7144 s_waitcnt 0 ; Wait for all counters to be 0 7145 s_waitcnt vmcnt(0) & expcnt(0) & lgkmcnt(0) ; Equivalent to above 7146 s_waitcnt vmcnt(1) ; Wait for vmcnt counter to be 1. 7147 s_sethalt 9 7148 s_sleep 10 7149 s_sendmsg 0x1 7150 s_sendmsg sendmsg(MSG_INTERRUPT) 7151 s_trap 1 7152 7153For full list of supported instructions, refer to "SOPP Instructions" in ISA 7154Manual. 7155 7156Unless otherwise mentioned, little verification is performed on the operands 7157of SOPP Instructions, so it is up to the programmer to be familiar with the 7158range or acceptable values. 7159 7160VALU 7161++++ 7162 7163For vector ALU instruction opcodes (VOP1, VOP2, VOP3, VOPC, VOP_DPP, VOP_SDWA), 7164the assembler will automatically use optimal encoding based on its operands. To 7165force specific encoding, one can add a suffix to the opcode of the instruction: 7166 7167* _e32 for 32-bit VOP1/VOP2/VOPC 7168* _e64 for 64-bit VOP3 7169* _dpp for VOP_DPP 7170* _sdwa for VOP_SDWA 7171 7172VOP1/VOP2/VOP3/VOPC examples: 7173 7174.. code-block:: nasm 7175 7176 v_mov_b32 v1, v2 7177 v_mov_b32_e32 v1, v2 7178 v_nop 7179 v_cvt_f64_i32_e32 v[1:2], v2 7180 v_floor_f32_e32 v1, v2 7181 v_bfrev_b32_e32 v1, v2 7182 v_add_f32_e32 v1, v2, v3 7183 v_mul_i32_i24_e64 v1, v2, 3 7184 v_mul_i32_i24_e32 v1, -3, v3 7185 v_mul_i32_i24_e32 v1, -100, v3 7186 v_addc_u32 v1, s[0:1], v2, v3, s[2:3] 7187 v_max_f16_e32 v1, v2, v3 7188 7189VOP_DPP examples: 7190 7191.. code-block:: nasm 7192 7193 v_mov_b32 v0, v0 quad_perm:[0,2,1,1] 7194 v_sin_f32 v0, v0 row_shl:1 row_mask:0xa bank_mask:0x1 bound_ctrl:0 7195 v_mov_b32 v0, v0 wave_shl:1 7196 v_mov_b32 v0, v0 row_mirror 7197 v_mov_b32 v0, v0 row_bcast:31 7198 v_mov_b32 v0, v0 quad_perm:[1,3,0,1] row_mask:0xa bank_mask:0x1 bound_ctrl:0 7199 v_add_f32 v0, v0, |v0| row_shl:1 row_mask:0xa bank_mask:0x1 bound_ctrl:0 7200 v_max_f16 v1, v2, v3 row_shl:1 row_mask:0xa bank_mask:0x1 bound_ctrl:0 7201 7202VOP_SDWA examples: 7203 7204.. code-block:: nasm 7205 7206 v_mov_b32 v1, v2 dst_sel:BYTE_0 dst_unused:UNUSED_PRESERVE src0_sel:DWORD 7207 v_min_u32 v200, v200, v1 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:BYTE_1 src1_sel:DWORD 7208 v_sin_f32 v0, v0 dst_unused:UNUSED_PAD src0_sel:WORD_1 7209 v_fract_f32 v0, |v0| dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 7210 v_cmpx_le_u32 vcc, v1, v2 src0_sel:BYTE_2 src1_sel:WORD_0 7211 7212For full list of supported instructions, refer to "Vector ALU instructions". 7213 7214.. TODO:: 7215 7216 Remove once we switch to code object v3 by default. 7217 7218.. _amdgpu-amdhsa-assembler-predefined-symbols-v2: 7219 7220Code Object V2 Predefined Symbols (-mattr=-code-object-v3) 7221~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 7222 7223.. warning:: Code Object V2 is not the default code object version emitted by 7224 this version of LLVM. For a description of the predefined symbols available 7225 with the default configuration (Code Object V3) see 7226 :ref:`amdgpu-amdhsa-assembler-predefined-symbols-v3`. 7227 7228The AMDGPU assembler defines and updates some symbols automatically. These 7229symbols do not affect code generation. 7230 7231.option.machine_version_major 7232+++++++++++++++++++++++++++++ 7233 7234Set to the GFX major generation number of the target being assembled for. For 7235example, when assembling for a "GFX9" target this will be set to the integer 7236value "9". The possible GFX major generation numbers are presented in 7237:ref:`amdgpu-processors`. 7238 7239.option.machine_version_minor 7240+++++++++++++++++++++++++++++ 7241 7242Set to the GFX minor generation number of the target being assembled for. For 7243example, when assembling for a "GFX810" target this will be set to the integer 7244value "1". The possible GFX minor generation numbers are presented in 7245:ref:`amdgpu-processors`. 7246 7247.option.machine_version_stepping 7248++++++++++++++++++++++++++++++++ 7249 7250Set to the GFX stepping generation number of the target being assembled for. 7251For example, when assembling for a "GFX704" target this will be set to the 7252integer value "4". The possible GFX stepping generation numbers are presented 7253in :ref:`amdgpu-processors`. 7254 7255.kernel.vgpr_count 7256++++++++++++++++++ 7257 7258Set to zero each time a 7259:ref:`amdgpu-amdhsa-assembler-directive-amdgpu_hsa_kernel` directive is 7260encountered. At each instruction, if the current value of this symbol is less 7261than or equal to the maximum VPGR number explicitly referenced within that 7262instruction then the symbol value is updated to equal that VGPR number plus 7263one. 7264 7265.kernel.sgpr_count 7266++++++++++++++++++ 7267 7268Set to zero each time a 7269:ref:`amdgpu-amdhsa-assembler-directive-amdgpu_hsa_kernel` directive is 7270encountered. At each instruction, if the current value of this symbol is less 7271than or equal to the maximum VPGR number explicitly referenced within that 7272instruction then the symbol value is updated to equal that SGPR number plus 7273one. 7274 7275.. _amdgpu-amdhsa-assembler-directives-v2: 7276 7277Code Object V2 Directives (-mattr=-code-object-v3) 7278~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 7279 7280.. warning:: Code Object V2 is not the default code object version emitted by 7281 this version of LLVM. For a description of the directives supported with 7282 the default configuration (Code Object V3) see 7283 :ref:`amdgpu-amdhsa-assembler-directives-v3`. 7284 7285AMDGPU ABI defines auxiliary data in output code object. In assembly source, 7286one can specify them with assembler directives. 7287 7288.hsa_code_object_version major, minor 7289+++++++++++++++++++++++++++++++++++++ 7290 7291*major* and *minor* are integers that specify the version of the HSA code 7292object that will be generated by the assembler. 7293 7294.hsa_code_object_isa [major, minor, stepping, vendor, arch] 7295+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 7296 7297 7298*major*, *minor*, and *stepping* are all integers that describe the instruction 7299set architecture (ISA) version of the assembly program. 7300 7301*vendor* and *arch* are quoted strings. *vendor* should always be equal to 7302"AMD" and *arch* should always be equal to "AMDGPU". 7303 7304By default, the assembler will derive the ISA version, *vendor*, and *arch* 7305from the value of the -mcpu option that is passed to the assembler. 7306 7307.. _amdgpu-amdhsa-assembler-directive-amdgpu_hsa_kernel: 7308 7309.amdgpu_hsa_kernel (name) 7310+++++++++++++++++++++++++ 7311 7312This directives specifies that the symbol with given name is a kernel entry 7313point (label) and the object should contain corresponding symbol of type 7314STT_AMDGPU_HSA_KERNEL. 7315 7316.amd_kernel_code_t 7317++++++++++++++++++ 7318 7319This directive marks the beginning of a list of key / value pairs that are used 7320to specify the amd_kernel_code_t object that will be emitted by the assembler. 7321The list must be terminated by the *.end_amd_kernel_code_t* directive. For any 7322amd_kernel_code_t values that are unspecified a default value will be used. The 7323default value for all keys is 0, with the following exceptions: 7324 7325- *amd_code_version_major* defaults to 1. 7326- *amd_kernel_code_version_minor* defaults to 2. 7327- *amd_machine_kind* defaults to 1. 7328- *amd_machine_version_major*, *machine_version_minor*, and 7329 *amd_machine_version_stepping* are derived from the value of the -mcpu option 7330 that is passed to the assembler. 7331- *kernel_code_entry_byte_offset* defaults to 256. 7332- *wavefront_size* defaults 6 for all targets before GFX10. For GFX10 onwards 7333 defaults to 6 if target feature ``wavefrontsize64`` is enabled, otherwise 5. 7334 Note that wavefront size is specified as a power of two, so a value of **n** 7335 means a size of 2^ **n**. 7336- *call_convention* defaults to -1. 7337- *kernarg_segment_alignment*, *group_segment_alignment*, and 7338 *private_segment_alignment* default to 4. Note that alignments are specified 7339 as a power of 2, so a value of **n** means an alignment of 2^ **n**. 7340- *enable_wgp_mode* defaults to 1 if target feature ``cumode`` is disabled for 7341 GFX10 onwards. 7342- *enable_mem_ordered* defaults to 1 for GFX10 onwards. 7343 7344The *.amd_kernel_code_t* directive must be placed immediately after the 7345function label and before any instructions. 7346 7347For a full list of amd_kernel_code_t keys, refer to AMDGPU ABI document, 7348comments in lib/Target/AMDGPU/AmdKernelCodeT.h and test/CodeGen/AMDGPU/hsa.s. 7349 7350.. _amdgpu-amdhsa-assembler-example-v2: 7351 7352Code Object V2 Example Source Code (-mattr=-code-object-v3) 7353~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 7354 7355.. warning:: Code Object V2 is not the default code object version emitted by 7356 this version of LLVM. For a description of the directives supported with 7357 the default configuration (Code Object V3) see 7358 :ref:`amdgpu-amdhsa-assembler-example-v3`. 7359 7360Here is an example of a minimal assembly source file, defining one HSA kernel: 7361 7362.. code:: 7363 :number-lines: 7364 7365 .hsa_code_object_version 1,0 7366 .hsa_code_object_isa 7367 7368 .hsatext 7369 .globl hello_world 7370 .p2align 8 7371 .amdgpu_hsa_kernel hello_world 7372 7373 hello_world: 7374 7375 .amd_kernel_code_t 7376 enable_sgpr_kernarg_segment_ptr = 1 7377 is_ptr64 = 1 7378 compute_pgm_rsrc1_vgprs = 0 7379 compute_pgm_rsrc1_sgprs = 0 7380 compute_pgm_rsrc2_user_sgpr = 2 7381 compute_pgm_rsrc1_wgp_mode = 0 7382 compute_pgm_rsrc1_mem_ordered = 0 7383 compute_pgm_rsrc1_fwd_progress = 1 7384 .end_amd_kernel_code_t 7385 7386 s_load_dwordx2 s[0:1], s[0:1] 0x0 7387 v_mov_b32 v0, 3.14159 7388 s_waitcnt lgkmcnt(0) 7389 v_mov_b32 v1, s0 7390 v_mov_b32 v2, s1 7391 flat_store_dword v[1:2], v0 7392 s_endpgm 7393 .Lfunc_end0: 7394 .size hello_world, .Lfunc_end0-hello_world 7395 7396.. _amdgpu-amdhsa-assembler-predefined-symbols-v3: 7397 7398Code Object V3 Predefined Symbols (-mattr=+code-object-v3) 7399~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 7400 7401The AMDGPU assembler defines and updates some symbols automatically. These 7402symbols do not affect code generation. 7403 7404.amdgcn.gfx_generation_number 7405+++++++++++++++++++++++++++++ 7406 7407Set to the GFX major generation number of the target being assembled for. For 7408example, when assembling for a "GFX9" target this will be set to the integer 7409value "9". The possible GFX major generation numbers are presented in 7410:ref:`amdgpu-processors`. 7411 7412.amdgcn.gfx_generation_minor 7413++++++++++++++++++++++++++++ 7414 7415Set to the GFX minor generation number of the target being assembled for. For 7416example, when assembling for a "GFX810" target this will be set to the integer 7417value "1". The possible GFX minor generation numbers are presented in 7418:ref:`amdgpu-processors`. 7419 7420.amdgcn.gfx_generation_stepping 7421+++++++++++++++++++++++++++++++ 7422 7423Set to the GFX stepping generation number of the target being assembled for. 7424For example, when assembling for a "GFX704" target this will be set to the 7425integer value "4". The possible GFX stepping generation numbers are presented 7426in :ref:`amdgpu-processors`. 7427 7428.. _amdgpu-amdhsa-assembler-symbol-next_free_vgpr: 7429 7430.amdgcn.next_free_vgpr 7431++++++++++++++++++++++ 7432 7433Set to zero before assembly begins. At each instruction, if the current value 7434of this symbol is less than or equal to the maximum VGPR number explicitly 7435referenced within that instruction then the symbol value is updated to equal 7436that VGPR number plus one. 7437 7438May be used to set the `.amdhsa_next_free_vpgr` directive in 7439:ref:`amdhsa-kernel-directives-table`. 7440 7441May be set at any time, e.g. manually set to zero at the start of each kernel. 7442 7443.. _amdgpu-amdhsa-assembler-symbol-next_free_sgpr: 7444 7445.amdgcn.next_free_sgpr 7446++++++++++++++++++++++ 7447 7448Set to zero before assembly begins. At each instruction, if the current value 7449of this symbol is less than or equal the maximum SGPR number explicitly 7450referenced within that instruction then the symbol value is updated to equal 7451that SGPR number plus one. 7452 7453May be used to set the `.amdhsa_next_free_spgr` directive in 7454:ref:`amdhsa-kernel-directives-table`. 7455 7456May be set at any time, e.g. manually set to zero at the start of each kernel. 7457 7458.. _amdgpu-amdhsa-assembler-directives-v3: 7459 7460Code Object V3 Directives (-mattr=+code-object-v3) 7461~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 7462 7463Directives which begin with ``.amdgcn`` are valid for all ``amdgcn`` 7464architecture processors, and are not OS-specific. Directives which begin with 7465``.amdhsa`` are specific to ``amdgcn`` architecture processors when the 7466``amdhsa`` OS is specified. See :ref:`amdgpu-target-triples` and 7467:ref:`amdgpu-processors`. 7468 7469.amdgcn_target <target> 7470+++++++++++++++++++++++ 7471 7472Optional directive which declares the target supported by the containing 7473assembler source file. Valid values are described in 7474:ref:`amdgpu-amdhsa-code-object-target-identification`. Used by the assembler 7475to validate command-line options such as ``-triple``, ``-mcpu``, and those 7476which specify target features. 7477 7478.amdhsa_kernel <name> 7479+++++++++++++++++++++ 7480 7481Creates a correctly aligned AMDHSA kernel descriptor and a symbol, 7482``<name>.kd``, in the current location of the current section. Only valid when 7483the OS is ``amdhsa``. ``<name>`` must be a symbol that labels the first 7484instruction to execute, and does not need to be previously defined. 7485 7486Marks the beginning of a list of directives used to generate the bytes of a 7487kernel descriptor, as described in :ref:`amdgpu-amdhsa-kernel-descriptor`. 7488Directives which may appear in this list are described in 7489:ref:`amdhsa-kernel-directives-table`. Directives may appear in any order, must 7490be valid for the target being assembled for, and cannot be repeated. Directives 7491support the range of values specified by the field they reference in 7492:ref:`amdgpu-amdhsa-kernel-descriptor`. If a directive is not specified, it is 7493assumed to have its default value, unless it is marked as "Required", in which 7494case it is an error to omit the directive. This list of directives is 7495terminated by an ``.end_amdhsa_kernel`` directive. 7496 7497 .. table:: AMDHSA Kernel Assembler Directives 7498 :name: amdhsa-kernel-directives-table 7499 7500 ======================================================== =================== ============ =================== 7501 Directive Default Supported On Description 7502 ======================================================== =================== ============ =================== 7503 ``.amdhsa_group_segment_fixed_size`` 0 GFX6-GFX10 Controls GROUP_SEGMENT_FIXED_SIZE in 7504 :ref:`amdgpu-amdhsa-kernel-descriptor-gfx6-gfx10-table`. 7505 ``.amdhsa_private_segment_fixed_size`` 0 GFX6-GFX10 Controls PRIVATE_SEGMENT_FIXED_SIZE in 7506 :ref:`amdgpu-amdhsa-kernel-descriptor-gfx6-gfx10-table`. 7507 ``.amdhsa_user_sgpr_private_segment_buffer`` 0 GFX6-GFX10 Controls ENABLE_SGPR_PRIVATE_SEGMENT_BUFFER in 7508 :ref:`amdgpu-amdhsa-kernel-descriptor-gfx6-gfx10-table`. 7509 ``.amdhsa_user_sgpr_dispatch_ptr`` 0 GFX6-GFX10 Controls ENABLE_SGPR_DISPATCH_PTR in 7510 :ref:`amdgpu-amdhsa-kernel-descriptor-gfx6-gfx10-table`. 7511 ``.amdhsa_user_sgpr_queue_ptr`` 0 GFX6-GFX10 Controls ENABLE_SGPR_QUEUE_PTR in 7512 :ref:`amdgpu-amdhsa-kernel-descriptor-gfx6-gfx10-table`. 7513 ``.amdhsa_user_sgpr_kernarg_segment_ptr`` 0 GFX6-GFX10 Controls ENABLE_SGPR_KERNARG_SEGMENT_PTR in 7514 :ref:`amdgpu-amdhsa-kernel-descriptor-gfx6-gfx10-table`. 7515 ``.amdhsa_user_sgpr_dispatch_id`` 0 GFX6-GFX10 Controls ENABLE_SGPR_DISPATCH_ID in 7516 :ref:`amdgpu-amdhsa-kernel-descriptor-gfx6-gfx10-table`. 7517 ``.amdhsa_user_sgpr_flat_scratch_init`` 0 GFX6-GFX10 Controls ENABLE_SGPR_FLAT_SCRATCH_INIT in 7518 :ref:`amdgpu-amdhsa-kernel-descriptor-gfx6-gfx10-table`. 7519 ``.amdhsa_user_sgpr_private_segment_size`` 0 GFX6-GFX10 Controls ENABLE_SGPR_PRIVATE_SEGMENT_SIZE in 7520 :ref:`amdgpu-amdhsa-kernel-descriptor-gfx6-gfx10-table`. 7521 ``.amdhsa_wavefront_size32`` Target GFX10 Controls ENABLE_WAVEFRONT_SIZE32 in 7522 Feature :ref:`amdgpu-amdhsa-kernel-descriptor-gfx6-gfx10-table`. 7523 Specific 7524 (-wavefrontsize64) 7525 ``.amdhsa_system_sgpr_private_segment_wavefront_offset`` 0 GFX6-GFX10 Controls ENABLE_SGPR_PRIVATE_SEGMENT_WAVEFRONT_OFFSET in 7526 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 7527 ``.amdhsa_system_sgpr_workgroup_id_x`` 1 GFX6-GFX10 Controls ENABLE_SGPR_WORKGROUP_ID_X in 7528 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 7529 ``.amdhsa_system_sgpr_workgroup_id_y`` 0 GFX6-GFX10 Controls ENABLE_SGPR_WORKGROUP_ID_Y in 7530 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 7531 ``.amdhsa_system_sgpr_workgroup_id_z`` 0 GFX6-GFX10 Controls ENABLE_SGPR_WORKGROUP_ID_Z in 7532 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 7533 ``.amdhsa_system_sgpr_workgroup_info`` 0 GFX6-GFX10 Controls ENABLE_SGPR_WORKGROUP_INFO in 7534 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 7535 ``.amdhsa_system_vgpr_workitem_id`` 0 GFX6-GFX10 Controls ENABLE_VGPR_WORKITEM_ID in 7536 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 7537 Possible values are defined in 7538 :ref:`amdgpu-amdhsa-system-vgpr-work-item-id-enumeration-values-table`. 7539 ``.amdhsa_next_free_vgpr`` Required GFX6-GFX10 Maximum VGPR number explicitly referenced, plus one. 7540 Used to calculate GRANULATED_WORKITEM_VGPR_COUNT in 7541 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 7542 ``.amdhsa_next_free_sgpr`` Required GFX6-GFX10 Maximum SGPR number explicitly referenced, plus one. 7543 Used to calculate GRANULATED_WAVEFRONT_SGPR_COUNT in 7544 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 7545 ``.amdhsa_reserve_vcc`` 1 GFX6-GFX10 Whether the kernel may use the special VCC SGPR. 7546 Used to calculate GRANULATED_WAVEFRONT_SGPR_COUNT in 7547 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 7548 ``.amdhsa_reserve_flat_scratch`` 1 GFX7-GFX10 Whether the kernel may use flat instructions to access 7549 scratch memory. Used to calculate 7550 GRANULATED_WAVEFRONT_SGPR_COUNT in 7551 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 7552 ``.amdhsa_reserve_xnack_mask`` Target GFX8-GFX10 Whether the kernel may trigger XNACK replay. 7553 Feature Used to calculate GRANULATED_WAVEFRONT_SGPR_COUNT in 7554 Specific :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 7555 (+xnack) 7556 ``.amdhsa_float_round_mode_32`` 0 GFX6-GFX10 Controls FLOAT_ROUND_MODE_32 in 7557 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 7558 Possible values are defined in 7559 :ref:`amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table`. 7560 ``.amdhsa_float_round_mode_16_64`` 0 GFX6-GFX10 Controls FLOAT_ROUND_MODE_16_64 in 7561 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 7562 Possible values are defined in 7563 :ref:`amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table`. 7564 ``.amdhsa_float_denorm_mode_32`` 0 GFX6-GFX10 Controls FLOAT_DENORM_MODE_32 in 7565 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 7566 Possible values are defined in 7567 :ref:`amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table`. 7568 ``.amdhsa_float_denorm_mode_16_64`` 3 GFX6-GFX10 Controls FLOAT_DENORM_MODE_16_64 in 7569 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 7570 Possible values are defined in 7571 :ref:`amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table`. 7572 ``.amdhsa_dx10_clamp`` 1 GFX6-GFX10 Controls ENABLE_DX10_CLAMP in 7573 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 7574 ``.amdhsa_ieee_mode`` 1 GFX6-GFX10 Controls ENABLE_IEEE_MODE in 7575 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 7576 ``.amdhsa_fp16_overflow`` 0 GFX9-GFX10 Controls FP16_OVFL in 7577 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 7578 ``.amdhsa_workgroup_processor_mode`` Target GFX10 Controls ENABLE_WGP_MODE in 7579 Feature :ref:`amdgpu-amdhsa-kernel-descriptor-gfx6-gfx10-table`. 7580 Specific 7581 (-cumode) 7582 ``.amdhsa_memory_ordered`` 1 GFX10 Controls MEM_ORDERED in 7583 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 7584 ``.amdhsa_forward_progress`` 0 GFX10 Controls FWD_PROGRESS in 7585 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 7586 ``.amdhsa_exception_fp_ieee_invalid_op`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_INVALID_OPERATION in 7587 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 7588 ``.amdhsa_exception_fp_denorm_src`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_FP_DENORMAL_SOURCE in 7589 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 7590 ``.amdhsa_exception_fp_ieee_div_zero`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_DIVISION_BY_ZERO in 7591 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 7592 ``.amdhsa_exception_fp_ieee_overflow`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_OVERFLOW in 7593 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 7594 ``.amdhsa_exception_fp_ieee_underflow`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_UNDERFLOW in 7595 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 7596 ``.amdhsa_exception_fp_ieee_inexact`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_INEXACT in 7597 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 7598 ``.amdhsa_exception_int_div_zero`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_INT_DIVIDE_BY_ZERO in 7599 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 7600 ======================================================== =================== ============ =================== 7601 7602.amdgpu_metadata 7603++++++++++++++++ 7604 7605Optional directive which declares the contents of the ``NT_AMDGPU_METADATA`` 7606note record (see :ref:`amdgpu-elf-note-records-table-v3`). 7607 7608The contents must be in the [YAML]_ markup format, with the same structure and 7609semantics described in :ref:`amdgpu-amdhsa-code-object-metadata-v3`. 7610 7611This directive is terminated by an ``.end_amdgpu_metadata`` directive. 7612 7613.. _amdgpu-amdhsa-assembler-example-v3: 7614 7615Code Object V3 Example Source Code (-mattr=+code-object-v3) 7616~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 7617 7618Here is an example of a minimal assembly source file, defining one HSA kernel: 7619 7620.. code:: 7621 :number-lines: 7622 7623 .amdgcn_target "amdgcn-amd-amdhsa--gfx900+xnack" // optional 7624 7625 .text 7626 .globl hello_world 7627 .p2align 8 7628 .type hello_world,@function 7629 hello_world: 7630 s_load_dwordx2 s[0:1], s[0:1] 0x0 7631 v_mov_b32 v0, 3.14159 7632 s_waitcnt lgkmcnt(0) 7633 v_mov_b32 v1, s0 7634 v_mov_b32 v2, s1 7635 flat_store_dword v[1:2], v0 7636 s_endpgm 7637 .Lfunc_end0: 7638 .size hello_world, .Lfunc_end0-hello_world 7639 7640 .rodata 7641 .p2align 6 7642 .amdhsa_kernel hello_world 7643 .amdhsa_user_sgpr_kernarg_segment_ptr 1 7644 .amdhsa_next_free_vgpr .amdgcn.next_free_vgpr 7645 .amdhsa_next_free_sgpr .amdgcn.next_free_sgpr 7646 .end_amdhsa_kernel 7647 7648 .amdgpu_metadata 7649 --- 7650 amdhsa.version: 7651 - 1 7652 - 0 7653 amdhsa.kernels: 7654 - .name: hello_world 7655 .symbol: hello_world.kd 7656 .kernarg_segment_size: 48 7657 .group_segment_fixed_size: 0 7658 .private_segment_fixed_size: 0 7659 .kernarg_segment_align: 4 7660 .wavefront_size: 64 7661 .sgpr_count: 2 7662 .vgpr_count: 3 7663 .max_flat_workgroup_size: 256 7664 ... 7665 .end_amdgpu_metadata 7666 7667If an assembly source file contains multiple kernels and/or functions, the 7668:ref:`amdgpu-amdhsa-assembler-symbol-next_free_vgpr` and 7669:ref:`amdgpu-amdhsa-assembler-symbol-next_free_sgpr` symbols may be reset using 7670the ``.set <symbol>, <expression>`` directive. For example, in the case of two 7671kernels, where ``function1`` is only called from ``kernel1`` it is sufficient 7672to group the function with the kernel that calls it and reset the symbols 7673between the two connected components: 7674 7675.. code:: 7676 :number-lines: 7677 7678 .amdgcn_target "amdgcn-amd-amdhsa--gfx900+xnack" // optional 7679 7680 // gpr tracking symbols are implicitly set to zero 7681 7682 .text 7683 .globl kern0 7684 .p2align 8 7685 .type kern0,@function 7686 kern0: 7687 // ... 7688 s_endpgm 7689 .Lkern0_end: 7690 .size kern0, .Lkern0_end-kern0 7691 7692 .rodata 7693 .p2align 6 7694 .amdhsa_kernel kern0 7695 // ... 7696 .amdhsa_next_free_vgpr .amdgcn.next_free_vgpr 7697 .amdhsa_next_free_sgpr .amdgcn.next_free_sgpr 7698 .end_amdhsa_kernel 7699 7700 // reset symbols to begin tracking usage in func1 and kern1 7701 .set .amdgcn.next_free_vgpr, 0 7702 .set .amdgcn.next_free_sgpr, 0 7703 7704 .text 7705 .hidden func1 7706 .global func1 7707 .p2align 2 7708 .type func1,@function 7709 func1: 7710 // ... 7711 s_setpc_b64 s[30:31] 7712 .Lfunc1_end: 7713 .size func1, .Lfunc1_end-func1 7714 7715 .globl kern1 7716 .p2align 8 7717 .type kern1,@function 7718 kern1: 7719 // ... 7720 s_getpc_b64 s[4:5] 7721 s_add_u32 s4, s4, func1@rel32@lo+4 7722 s_addc_u32 s5, s5, func1@rel32@lo+4 7723 s_swappc_b64 s[30:31], s[4:5] 7724 // ... 7725 s_endpgm 7726 .Lkern1_end: 7727 .size kern1, .Lkern1_end-kern1 7728 7729 .rodata 7730 .p2align 6 7731 .amdhsa_kernel kern1 7732 // ... 7733 .amdhsa_next_free_vgpr .amdgcn.next_free_vgpr 7734 .amdhsa_next_free_sgpr .amdgcn.next_free_sgpr 7735 .end_amdhsa_kernel 7736 7737These symbols cannot identify connected components in order to automatically 7738track the usage for each kernel. However, in some cases careful organization of 7739the kernels and functions in the source file means there is minimal additional 7740effort required to accurately calculate GPR usage. 7741 7742Additional Documentation 7743======================== 7744 7745.. [AMD-GCN-GFX6] `AMD Southern Islands Series ISA <http://developer.amd.com/wordpress/media/2012/12/AMD_Southern_Islands_Instruction_Set_Architecture.pdf>`__ 7746.. [AMD-GCN-GFX7] `AMD Sea Islands Series ISA <http://developer.amd.com/wordpress/media/2013/07/AMD_Sea_Islands_Instruction_Set_Architecture.pdf>`_ 7747.. [AMD-GCN-GFX8] `AMD GCN3 Instruction Set Architecture <http://amd-dev.wpengine.netdna-cdn.com/wordpress/media/2013/12/AMD_GCN3_Instruction_Set_Architecture_rev1.1.pdf>`__ 7748.. [AMD-GCN-GFX9] `AMD "Vega" Instruction Set Architecture <http://developer.amd.com/wordpress/media/2013/12/Vega_Shader_ISA_28July2017.pdf>`__ 7749.. [AMD-GCN-GFX10] `AMD "RDNA 1.0" Instruction Set Architecture <https://gpuopen.com/wp-content/uploads/2019/08/RDNA_Shader_ISA_5August2019.pdf>`__ 7750.. [AMD-RADEON-HD-2000-3000] `AMD R6xx shader ISA <http://developer.amd.com/wordpress/media/2012/10/R600_Instruction_Set_Architecture.pdf>`__ 7751.. [AMD-RADEON-HD-4000] `AMD R7xx shader ISA <http://developer.amd.com/wordpress/media/2012/10/R700-Family_Instruction_Set_Architecture.pdf>`__ 7752.. [AMD-RADEON-HD-5000] `AMD Evergreen shader ISA <http://developer.amd.com/wordpress/media/2012/10/AMD_Evergreen-Family_Instruction_Set_Architecture.pdf>`__ 7753.. [AMD-RADEON-HD-6000] `AMD Cayman/Trinity shader ISA <http://developer.amd.com/wordpress/media/2012/10/AMD_HD_6900_Series_Instruction_Set_Architecture.pdf>`__ 7754.. [AMD-ROCm] `AMD ROCm Platform <https://rocm-documentation.readthedocs.io>`__ 7755.. [AMD-ROCm-github] `ROCm github <http://github.com/RadeonOpenCompute>`__ 7756.. [CLANG-ATTR] `Attributes in Clang <https://clang.llvm.org/docs/AttributeReference.html>`__ 7757.. [DWARF] `DWARF Debugging Information Format <http://dwarfstd.org/>`__ 7758.. [ELF] `Executable and Linkable Format (ELF) <http://www.sco.com/developers/gabi/>`__ 7759.. [HRF] `Heterogeneous-race-free Memory Models <http://benedictgaster.org/wp-content/uploads/2014/01/asplos269-FINAL.pdf>`__ 7760.. [HSA] `Heterogeneous System Architecture (HSA) Foundation <http://www.hsafoundation.com/>`__ 7761.. [MsgPack] `Message Pack <http://www.msgpack.org/>`__ 7762.. [OpenCL] `The OpenCL Specification Version 2.0 <http://www.khronos.org/registry/cl/specs/opencl-2.0.pdf>`__ 7763.. [SEMVER] `Semantic Versioning <https://semver.org/>`__ 7764.. [YAML] `YAML Ain't Markup Language (YAML™) Version 1.2 <http://www.yaml.org/spec/1.2/spec.html>`__ 7765