1============================= 2User Guide for AMDGPU Backend 3============================= 4 5.. contents:: 6 :local: 7 8.. toctree:: 9 :hidden: 10 11 AMDGPU/AMDGPUAsmGFX7 12 AMDGPU/AMDGPUAsmGFX8 13 AMDGPU/AMDGPUAsmGFX9 14 AMDGPU/AMDGPUAsmGFX900 15 AMDGPU/AMDGPUAsmGFX904 16 AMDGPU/AMDGPUAsmGFX906 17 AMDGPU/AMDGPUAsmGFX908 18 AMDGPU/AMDGPUAsmGFX10 19 AMDGPU/AMDGPUAsmGFX1011 20 AMDGPUModifierSyntax 21 AMDGPUOperandSyntax 22 AMDGPUInstructionSyntax 23 AMDGPUInstructionNotation 24 AMDGPUDwarfProposalForHeterogeneousDebugging 25 26Introduction 27============ 28 29The AMDGPU backend provides ISA code generation for AMD GPUs, starting with the 30R600 family up until the current GCN families. It lives in the 31``llvm/lib/Target/AMDGPU`` directory. 32 33LLVM 34==== 35 36.. _amdgpu-target-triples: 37 38Target Triples 39-------------- 40 41Use the ``clang -target <Architecture>-<Vendor>-<OS>-<Environment>`` option to 42specify the target triple: 43 44 .. table:: AMDGPU Architectures 45 :name: amdgpu-architecture-table 46 47 ============ ============================================================== 48 Architecture Description 49 ============ ============================================================== 50 ``r600`` AMD GPUs HD2XXX-HD6XXX for graphics and compute shaders. 51 ``amdgcn`` AMD GPUs GCN GFX6 onwards for graphics and compute shaders. 52 ============ ============================================================== 53 54 .. table:: AMDGPU Vendors 55 :name: amdgpu-vendor-table 56 57 ============ ============================================================== 58 Vendor Description 59 ============ ============================================================== 60 ``amd`` Can be used for all AMD GPU usage. 61 ``mesa3d`` Can be used if the OS is ``mesa3d``. 62 ============ ============================================================== 63 64 .. table:: AMDGPU Operating Systems 65 :name: amdgpu-os-table 66 67 ============== ============================================================ 68 OS Description 69 ============== ============================================================ 70 *<empty>* Defaults to the *unknown* OS. 71 ``amdhsa`` Compute kernels executed on HSA [HSA]_ compatible runtimes 72 such as AMD's ROCm [AMD-ROCm]_. 73 ``amdpal`` Graphic shaders and compute kernels executed on AMD PAL 74 runtime. 75 ``mesa3d`` Graphic shaders and compute kernels executed on Mesa 3D 76 runtime. 77 ============== ============================================================ 78 79 .. table:: AMDGPU Environments 80 :name: amdgpu-environment-table 81 82 ============ ============================================================== 83 Environment Description 84 ============ ============================================================== 85 *<empty>* Default. 86 ============ ============================================================== 87 88.. _amdgpu-processors: 89 90Processors 91---------- 92 93Use the ``clang -mcpu <Processor>`` option to specify the AMDGPU processor. The 94names from both the *Processor* and *Alternative Processor* can be used. 95 96 .. table:: AMDGPU Processors 97 :name: amdgpu-processor-table 98 99 =========== =============== ============ ===== ================= ======= ====================== 100 Processor Alternative Target dGPU/ Target ROCm Example 101 Processor Triple APU Features Support Products 102 Architecture Supported 103 [Default] 104 =========== =============== ============ ===== ================= ======= ====================== 105 **Radeon HD 2000/3000 Series (R600)** [AMD-RADEON-HD-2000-3000]_ 106 ----------------------------------------------------------------------------------------------- 107 ``r600`` ``r600`` dGPU 108 ``r630`` ``r600`` dGPU 109 ``rs880`` ``r600`` dGPU 110 ``rv670`` ``r600`` dGPU 111 **Radeon HD 4000 Series (R700)** [AMD-RADEON-HD-4000]_ 112 ----------------------------------------------------------------------------------------------- 113 ``rv710`` ``r600`` dGPU 114 ``rv730`` ``r600`` dGPU 115 ``rv770`` ``r600`` dGPU 116 **Radeon HD 5000 Series (Evergreen)** [AMD-RADEON-HD-5000]_ 117 ----------------------------------------------------------------------------------------------- 118 ``cedar`` ``r600`` dGPU 119 ``cypress`` ``r600`` dGPU 120 ``juniper`` ``r600`` dGPU 121 ``redwood`` ``r600`` dGPU 122 ``sumo`` ``r600`` dGPU 123 **Radeon HD 6000 Series (Northern Islands)** [AMD-RADEON-HD-6000]_ 124 ----------------------------------------------------------------------------------------------- 125 ``barts`` ``r600`` dGPU 126 ``caicos`` ``r600`` dGPU 127 ``cayman`` ``r600`` dGPU 128 ``turks`` ``r600`` dGPU 129 **GCN GFX6 (Southern Islands (SI))** [AMD-GCN-GFX6]_ 130 ----------------------------------------------------------------------------------------------- 131 ``gfx600`` - ``tahiti`` ``amdgcn`` dGPU 132 ``gfx601`` - ``hainan`` ``amdgcn`` dGPU 133 - ``oland`` 134 - ``pitcairn`` 135 - ``verde`` 136 **GCN GFX7 (Sea Islands (CI))** [AMD-GCN-GFX7]_ 137 ----------------------------------------------------------------------------------------------- 138 ``gfx700`` - ``kaveri`` ``amdgcn`` APU - A6-7000 139 - A6 Pro-7050B 140 - A8-7100 141 - A8 Pro-7150B 142 - A10-7300 143 - A10 Pro-7350B 144 - FX-7500 145 - A8-7200P 146 - A10-7400P 147 - FX-7600P 148 ``gfx701`` - ``hawaii`` ``amdgcn`` dGPU ROCm - FirePro W8100 149 - FirePro W9100 150 - FirePro S9150 151 - FirePro S9170 152 ``gfx702`` ``amdgcn`` dGPU ROCm - Radeon R9 290 153 - Radeon R9 290x 154 - Radeon R390 155 - Radeon R390x 156 ``gfx703`` - ``kabini`` ``amdgcn`` APU - E1-2100 157 - ``mullins`` - E1-2200 158 - E1-2500 159 - E2-3000 160 - E2-3800 161 - A4-5000 162 - A4-5100 163 - A6-5200 164 - A4 Pro-3340B 165 ``gfx704`` - ``bonaire`` ``amdgcn`` dGPU - Radeon HD 7790 166 - Radeon HD 8770 167 - R7 260 168 - R7 260X 169 **GCN GFX8 (Volcanic Islands (VI))** [AMD-GCN-GFX8]_ 170 ----------------------------------------------------------------------------------------------- 171 ``gfx801`` - ``carrizo`` ``amdgcn`` APU - xnack - A6-8500P 172 [on] - Pro A6-8500B 173 - A8-8600P 174 - Pro A8-8600B 175 - FX-8800P 176 - Pro A12-8800B 177 \ ``amdgcn`` APU - xnack ROCm - A10-8700P 178 [on] - Pro A10-8700B 179 - A10-8780P 180 \ ``amdgcn`` APU - xnack - A10-9600P 181 [on] - A10-9630P 182 - A12-9700P 183 - A12-9730P 184 - FX-9800P 185 - FX-9830P 186 \ ``amdgcn`` APU - xnack - E2-9010 187 [on] - A6-9210 188 - A9-9410 189 ``gfx802`` - ``iceland`` ``amdgcn`` dGPU - xnack ROCm - FirePro S7150 190 - ``tonga`` [off] - FirePro S7100 191 - FirePro W7100 192 - Radeon R285 193 - Radeon R9 380 194 - Radeon R9 385 195 - Mobile FirePro 196 M7170 197 ``gfx803`` - ``fiji`` ``amdgcn`` dGPU - xnack ROCm - Radeon R9 Nano 198 [off] - Radeon R9 Fury 199 - Radeon R9 FuryX 200 - Radeon Pro Duo 201 - FirePro S9300x2 202 - Radeon Instinct MI8 203 \ - ``polaris10`` ``amdgcn`` dGPU - xnack ROCm - Radeon RX 470 204 [off] - Radeon RX 480 205 - Radeon Instinct MI6 206 \ - ``polaris11`` ``amdgcn`` dGPU - xnack ROCm - Radeon RX 460 207 [off] 208 ``gfx810`` - ``stoney`` ``amdgcn`` APU - xnack 209 [on] 210 **GCN GFX9** [AMD-GCN-GFX9]_ 211 ----------------------------------------------------------------------------------------------- 212 ``gfx900`` ``amdgcn`` dGPU - xnack ROCm - Radeon Vega 213 [off] Frontier Edition 214 - Radeon RX Vega 56 215 - Radeon RX Vega 64 216 - Radeon RX Vega 64 217 Liquid 218 - Radeon Instinct MI25 219 ``gfx902`` ``amdgcn`` APU - xnack - Ryzen 3 2200G 220 [on] - Ryzen 5 2400G 221 ``gfx904`` ``amdgcn`` dGPU - xnack *TBA* 222 [off] 223 .. TODO:: 224 Add product 225 names. 226 ``gfx906`` ``amdgcn`` dGPU - xnack - Radeon Instinct MI50 227 [off] - Radeon Instinct MI60 228 ``gfx908`` ``amdgcn`` dGPU - xnack *TBA* 229 [off] 230 sram-ecc 231 [on] 232 ``gfx909`` ``amdgcn`` APU - xnack *TBA* (Raven Ridge 2) 233 [on] 234 .. TODO:: 235 Add product 236 names. 237 **GCN GFX10** [AMD-GCN-GFX10]_ 238 ----------------------------------------------------------------------------------------------- 239 ``gfx1010`` ``amdgcn`` dGPU - xnack *TBA* 240 [off] 241 - wavefrontsize64 242 [off] 243 - cumode 244 [off] 245 .. TODO:: 246 Add product 247 names. 248 ``gfx1011`` ``amdgcn`` dGPU - xnack *TBA* 249 [off] 250 - wavefrontsize64 251 [off] 252 - cumode 253 [off] 254 .. TODO:: 255 Add product 256 names. 257 ``gfx1012`` ``amdgcn`` dGPU - xnack *TBA* 258 [off] 259 - wavefrontsize64 260 [off] 261 - cumode 262 [off] 263 .. TODO:: 264 Add product 265 names. 266 =========== =============== ============ ===== ================= ======= ====================== 267 268.. _amdgpu-target-features: 269 270Target Features 271--------------- 272 273Target features control how code is generated to support certain 274processor specific features. Not all target features are supported by 275all processors. The runtime must ensure that the features supported by 276the device used to execute the code match the features enabled when 277generating the code. A mismatch of features may result in incorrect 278execution, or a reduction in performance. 279 280The target features supported by each processor, and the default value 281used if not specified explicitly, is listed in 282:ref:`amdgpu-processor-table`. 283 284Use the ``clang -m[no-]<TargetFeature>`` option to specify the AMDGPU 285target features. 286 287For example: 288 289``-mxnack`` 290 Enable the ``xnack`` feature. 291``-mno-xnack`` 292 Disable the ``xnack`` feature. 293 294 .. table:: AMDGPU Target Features 295 :name: amdgpu-target-feature-table 296 297 ====================== ================================================== 298 Target Feature Description 299 ====================== ================================================== 300 -m[no-]xnack Enable/disable generating code that has 301 memory clauses that are compatible with 302 having XNACK replay enabled. 303 304 This is used for demand paging and page 305 migration. If XNACK replay is enabled in 306 the device, then if a page fault occurs 307 the code may execute incorrectly if the 308 ``xnack`` feature is not enabled. Executing 309 code that has the feature enabled on a 310 device that does not have XNACK replay 311 enabled will execute correctly but may 312 be less performant than code with the 313 feature disabled. 314 315 -m[no-]sram-ecc Enable/disable generating code that assumes SRAM 316 ECC is enabled/disabled. 317 318 -m[no-]wavefrontsize64 Control the default wavefront size used when 319 generating code for kernels. When disabled 320 native wavefront size 32 is used, when enabled 321 wavefront size 64 is used. 322 323 -m[no-]cumode Control the default wavefront execution mode used 324 when generating code for kernels. When disabled 325 native WGP wavefront execution mode is used, 326 when enabled CU wavefront execution mode is used 327 (see :ref:`amdgpu-amdhsa-memory-model`). 328 ====================== ================================================== 329 330.. _amdgpu-address-spaces: 331 332Address Spaces 333-------------- 334 335The AMDGPU architecture supports a number of memory address spaces. The address 336space names use the OpenCL standard names, with some additions. 337 338The AMDGPU address spaces correspond to target architecture specific LLVM 339address space numbers used in LLVM IR. 340 341The AMDGPU address spaces are described in 342:ref:`amdgpu-address-spaces-table`. Only 64-bit process address spaces are 343supported for the ``amdgcn`` target. 344 345 .. table:: AMDGPU Address Spaces 346 :name: amdgpu-address-spaces-table 347 348 ================================= =============== =========== ================ ======= ============================ 349 .. 64-Bit Process Address Space 350 --------------------------------- --------------- ----------- ---------------- ------------------------------------ 351 Address Space Name LLVM IR Address HSA Segment Hardware Address NULL Value 352 Space Number Name Name Size 353 ================================= =============== =========== ================ ======= ============================ 354 Generic 0 flat flat 64 0x0000000000000000 355 Global 1 global global 64 0x0000000000000000 356 Region 2 N/A GDS 32 *not implemented for AMDHSA* 357 Local 3 group LDS 32 0xFFFFFFFF 358 Constant 4 constant *same as global* 64 0x0000000000000000 359 Private 5 private scratch 32 0x00000000 360 Constant 32-bit 6 *TODO* 361 Buffer Fat Pointer (experimental) 7 *TODO* 362 ================================= =============== =========== ================ ======= ============================ 363 364**Generic** 365 The generic address space uses the hardware flat address support available in 366 GFX7-GFX10. This uses two fixed ranges of virtual addresses (the private and 367 local apertures), that are outside the range of addressable global memory, to 368 map from a flat address to a private or local address. 369 370 FLAT instructions can take a flat address and access global, private 371 (scratch), and group (LDS) memory depending on if the address is within one 372 of the aperture ranges. Flat access to scratch requires hardware aperture 373 setup and setup in the kernel prologue (see 374 :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`). Flat access to LDS requires 375 hardware aperture setup and M0 (GFX7-GFX8) register setup (see 376 :ref:`amdgpu-amdhsa-kernel-prolog-m0`). 377 378 To convert between a private or group address space address (termed a segment 379 address) and a flat address the base address of the corresponding aperture 380 can be used. For GFX7-GFX8 these are available in the 381 :ref:`amdgpu-amdhsa-hsa-aql-queue` the address of which can be obtained with 382 Queue Ptr SGPR (see :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). For 383 GFX9-GFX10 the aperture base addresses are directly available as inline 384 constant registers ``SRC_SHARED_BASE/LIMIT`` and ``SRC_PRIVATE_BASE/LIMIT``. 385 In 64-bit address mode the aperture sizes are 2^32 bytes and the base is 386 aligned to 2^32 which makes it easier to convert from flat to segment or 387 segment to flat. 388 389 A global address space address has the same value when used as a flat address 390 so no conversion is needed. 391 392**Global and Constant** 393 The global and constant address spaces both use global virtual addresses, 394 which are the same virtual address space used by the CPU. However, some 395 virtual addresses may only be accessible to the CPU, some only accessible 396 by the GPU, and some by both. 397 398 Using the constant address space indicates that the data will not change 399 during the execution of the kernel. This allows scalar read instructions to 400 be used. The vector and scalar L1 caches are invalidated of volatile data 401 before each kernel dispatch execution to allow constant memory to change 402 values between kernel dispatches. 403 404**Region** 405 The region address space uses the hardware Global Data Store (GDS). All 406 wavefronts executing on the same device will access the same memory for any 407 given region address. However, the same region address accessed by wavefronts 408 executing on different devices will access different memory. It is higher 409 performance than global memory. It is allocated by the runtime. The data 410 store (DS) instructions can be used to access it. 411 412**Local** 413 The local address space uses the hardware Local Data Store (LDS) which is 414 automatically allocated when the hardware creates the wavefronts of a 415 work-group, and freed when all the wavefronts of a work-group have 416 terminated. All wavefronts belonging to the same work-group will access the 417 same memory for any given local address. However, the same local address 418 accessed by wavefronts belonging to different work-groups will access 419 different memory. It is higher performance than global memory. The data store 420 (DS) instructions can be used to access it. 421 422**Private** 423 The private address space uses the hardware scratch memory support which 424 automatically allocates memory when it creates a wavefront and frees it when 425 a wavefronts terminates. The memory accessed by a lane of a wavefront for any 426 given private address will be different to the memory accessed by another lane 427 of the same or different wavefront for the same private address. 428 429 If a kernel dispatch uses scratch, then the hardware allocates memory from a 430 pool of backing memory allocated by the runtime for each wavefront. The lanes 431 of the wavefront access this using dword (4 byte) interleaving. The mapping 432 used from private address to backing memory address is: 433 434 ``wavefront-scratch-base + 435 ((private-address / 4) * wavefront-size * 4) + 436 (wavefront-lane-id * 4) + (private-address % 4)`` 437 438 If each lane of a wavefront accesses the same private address, the 439 interleaving results in adjacent dwords being accessed and hence requires 440 fewer cache lines to be fetched. 441 442 There are different ways that the wavefront scratch base address is 443 determined by a wavefront (see 444 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 445 446 Scratch memory can be accessed in an interleaved manner using buffer 447 instructions with the scratch buffer descriptor and per wavefront scratch 448 offset, by the scratch instructions, or by flat instructions. Multi-dword 449 access is not supported except by flat and scratch instructions in 450 GFX9-GFX10. 451 452**Constant 32-bit** 453 *TODO* 454 455**Buffer Fat Pointer** 456 The buffer fat pointer is an experimental address space that is currently 457 unsupported in the backend. It exposes a non-integral pointer that is in 458 the future intended to support the modelling of 128-bit buffer descriptors 459 plus a 32-bit offset into the buffer (in total encapsulating a 160-bit 460 *pointer*), allowing normal LLVM load/store/atomic operations to be used to 461 model the buffer descriptors used heavily in graphics workloads targeting 462 the backend. 463 464.. _amdgpu-memory-scopes: 465 466Memory Scopes 467------------- 468 469This section provides LLVM memory synchronization scopes supported by the AMDGPU 470backend memory model when the target triple OS is ``amdhsa`` (see 471:ref:`amdgpu-amdhsa-memory-model` and :ref:`amdgpu-target-triples`). 472 473The memory model supported is based on the HSA memory model [HSA]_ which is 474based in turn on HRF-indirect with scope inclusion [HRF]_. The happens-before 475relation is transitive over the synchronizes-with relation independent of scope 476and synchronizes-with allows the memory scope instances to be inclusive (see 477table :ref:`amdgpu-amdhsa-llvm-sync-scopes-table`). 478 479This is different to the OpenCL [OpenCL]_ memory model which does not have scope 480inclusion and requires the memory scopes to exactly match. However, this 481is conservatively correct for OpenCL. 482 483 .. table:: AMDHSA LLVM Sync Scopes 484 :name: amdgpu-amdhsa-llvm-sync-scopes-table 485 486 ======================= =================================================== 487 LLVM Sync Scope Description 488 ======================= =================================================== 489 *none* The default: ``system``. 490 491 Synchronizes with, and participates in modification 492 and seq_cst total orderings with, other operations 493 (except image operations) for all address spaces 494 (except private, or generic that accesses private) 495 provided the other operation's sync scope is: 496 497 - ``system``. 498 - ``agent`` and executed by a thread on the same 499 agent. 500 - ``workgroup`` and executed by a thread in the 501 same work-group. 502 - ``wavefront`` and executed by a thread in the 503 same wavefront. 504 505 ``agent`` Synchronizes with, and participates in modification 506 and seq_cst total orderings with, other operations 507 (except image operations) for all address spaces 508 (except private, or generic that accesses private) 509 provided the other operation's sync scope is: 510 511 - ``system`` or ``agent`` and executed by a thread 512 on the same agent. 513 - ``workgroup`` and executed by a thread in the 514 same work-group. 515 - ``wavefront`` and executed by a thread in the 516 same wavefront. 517 518 ``workgroup`` Synchronizes with, and participates in modification 519 and seq_cst total orderings with, other operations 520 (except image operations) for all address spaces 521 (except private, or generic that accesses private) 522 provided the other operation's sync scope is: 523 524 - ``system``, ``agent`` or ``workgroup`` and 525 executed by a thread in the same work-group. 526 - ``wavefront`` and executed by a thread in the 527 same wavefront. 528 529 ``wavefront`` Synchronizes with, and participates in modification 530 and seq_cst total orderings with, other operations 531 (except image operations) for all address spaces 532 (except private, or generic that accesses private) 533 provided the other operation's sync scope is: 534 535 - ``system``, ``agent``, ``workgroup`` or 536 ``wavefront`` and executed by a thread in the 537 same wavefront. 538 539 ``singlethread`` Only synchronizes with and participates in 540 modification and seq_cst total orderings with, 541 other operations (except image operations) running 542 in the same thread for all address spaces (for 543 example, in signal handlers). 544 545 ``one-as`` Same as ``system`` but only synchronizes with other 546 operations within the same address space. 547 548 ``agent-one-as`` Same as ``agent`` but only synchronizes with other 549 operations within the same address space. 550 551 ``workgroup-one-as`` Same as ``workgroup`` but only synchronizes with 552 other operations within the same address space. 553 554 ``wavefront-one-as`` Same as ``wavefront`` but only synchronizes with 555 other operations within the same address space. 556 557 ``singlethread-one-as`` Same as ``singlethread`` but only synchronizes with 558 other operations within the same address space. 559 ======================= =================================================== 560 561LLVM IR Intrinsics 562------------------ 563 564The AMDGPU backend implements the following LLVM IR intrinsics. 565 566*This section is WIP.* 567 568.. TODO:: 569 570 List AMDGPU intrinsics. 571 572LLVM IR Attributes 573------------------ 574 575The AMDGPU backend supports the following LLVM IR attributes. 576 577 .. table:: AMDGPU LLVM IR Attributes 578 :name: amdgpu-llvm-ir-attributes-table 579 580 ======================================= ========================================================== 581 LLVM Attribute Description 582 ======================================= ========================================================== 583 "amdgpu-flat-work-group-size"="min,max" Specify the minimum and maximum flat work group sizes that 584 will be specified when the kernel is dispatched. Generated 585 by the ``amdgpu_flat_work_group_size`` CLANG attribute [CLANG-ATTR]_. 586 "amdgpu-implicitarg-num-bytes"="n" Number of kernel argument bytes to add to the kernel 587 argument block size for the implicit arguments. This 588 varies by OS and language (for OpenCL see 589 :ref:`opencl-kernel-implicit-arguments-appended-for-amdhsa-os-table`). 590 "amdgpu-num-sgpr"="n" Specifies the number of SGPRs to use. Generated by 591 the ``amdgpu_num_sgpr`` CLANG attribute [CLANG-ATTR]_. 592 "amdgpu-num-vgpr"="n" Specifies the number of VGPRs to use. Generated by the 593 ``amdgpu_num_vgpr`` CLANG attribute [CLANG-ATTR]_. 594 "amdgpu-waves-per-eu"="m,n" Specify the minimum and maximum number of waves per 595 execution unit. Generated by the ``amdgpu_waves_per_eu`` 596 CLANG attribute [CLANG-ATTR]_. 597 "amdgpu-ieee" true/false. Specify whether the function expects the IEEE field of the 598 mode register to be set on entry. Overrides the default for 599 the calling convention. 600 "amdgpu-dx10-clamp" true/false. Specify whether the function expects the DX10_CLAMP field of 601 the mode register to be set on entry. Overrides the default 602 for the calling convention. 603 ======================================= ========================================================== 604 605.. _amdgpu-elf-code-object: 606 607ELF Code Object 608=============== 609 610The AMDGPU backend generates a standard ELF [ELF]_ relocatable code object that 611can be linked by ``lld`` to produce a standard ELF shared code object which can 612be loaded and executed on an AMDGPU target. 613 614.. _amdgpu-elf-header: 615 616Header 617------ 618 619The AMDGPU backend uses the following ELF header: 620 621 .. table:: AMDGPU ELF Header 622 :name: amdgpu-elf-header-table 623 624 ========================== =============================== 625 Field Value 626 ========================== =============================== 627 ``e_ident[EI_CLASS]`` ``ELFCLASS64`` 628 ``e_ident[EI_DATA]`` ``ELFDATA2LSB`` 629 ``e_ident[EI_OSABI]`` - ``ELFOSABI_NONE`` 630 - ``ELFOSABI_AMDGPU_HSA`` 631 - ``ELFOSABI_AMDGPU_PAL`` 632 - ``ELFOSABI_AMDGPU_MESA3D`` 633 ``e_ident[EI_ABIVERSION]`` - ``ELFABIVERSION_AMDGPU_HSA`` 634 - ``ELFABIVERSION_AMDGPU_PAL`` 635 - ``ELFABIVERSION_AMDGPU_MESA3D`` 636 ``e_type`` - ``ET_REL`` 637 - ``ET_DYN`` 638 ``e_machine`` ``EM_AMDGPU`` 639 ``e_entry`` 0 640 ``e_flags`` See :ref:`amdgpu-elf-header-e_flags-table` 641 ========================== =============================== 642 643.. 644 645 .. table:: AMDGPU ELF Header Enumeration Values 646 :name: amdgpu-elf-header-enumeration-values-table 647 648 =============================== ===== 649 Name Value 650 =============================== ===== 651 ``EM_AMDGPU`` 224 652 ``ELFOSABI_NONE`` 0 653 ``ELFOSABI_AMDGPU_HSA`` 64 654 ``ELFOSABI_AMDGPU_PAL`` 65 655 ``ELFOSABI_AMDGPU_MESA3D`` 66 656 ``ELFABIVERSION_AMDGPU_HSA`` 1 657 ``ELFABIVERSION_AMDGPU_PAL`` 0 658 ``ELFABIVERSION_AMDGPU_MESA3D`` 0 659 =============================== ===== 660 661``e_ident[EI_CLASS]`` 662 The ELF class is: 663 664 * ``ELFCLASS32`` for ``r600`` architecture. 665 666 * ``ELFCLASS64`` for ``amdgcn`` architecture which only supports 64-bit 667 process address space applications. 668 669``e_ident[EI_DATA]`` 670 All AMDGPU targets use ``ELFDATA2LSB`` for little-endian byte ordering. 671 672``e_ident[EI_OSABI]`` 673 One of the following AMDGPU target architecture specific OS ABIs 674 (see :ref:`amdgpu-os-table`): 675 676 * ``ELFOSABI_NONE`` for *unknown* OS. 677 678 * ``ELFOSABI_AMDGPU_HSA`` for ``amdhsa`` OS. 679 680 * ``ELFOSABI_AMDGPU_PAL`` for ``amdpal`` OS. 681 682 * ``ELFOSABI_AMDGPU_MESA3D`` for ``mesa3D`` OS. 683 684``e_ident[EI_ABIVERSION]`` 685 The ABI version of the AMDGPU target architecture specific OS ABI to which the code 686 object conforms: 687 688 * ``ELFABIVERSION_AMDGPU_HSA`` is used to specify the version of AMD HSA 689 runtime ABI. 690 691 * ``ELFABIVERSION_AMDGPU_PAL`` is used to specify the version of AMD PAL 692 runtime ABI. 693 694 * ``ELFABIVERSION_AMDGPU_MESA3D`` is used to specify the version of AMD MESA 695 3D runtime ABI. 696 697``e_type`` 698 Can be one of the following values: 699 700 701 ``ET_REL`` 702 The type produced by the AMDGPU backend compiler as it is relocatable code 703 object. 704 705 ``ET_DYN`` 706 The type produced by the linker as it is a shared code object. 707 708 The AMD HSA runtime loader requires a ``ET_DYN`` code object. 709 710``e_machine`` 711 The value ``EM_AMDGPU`` is used for the machine for all processors supported 712 by the ``r600`` and ``amdgcn`` architectures (see 713 :ref:`amdgpu-processor-table`). The specific processor is specified in the 714 ``EF_AMDGPU_MACH`` bit field of the ``e_flags`` (see 715 :ref:`amdgpu-elf-header-e_flags-table`). 716 717``e_entry`` 718 The entry point is 0 as the entry points for individual kernels must be 719 selected in order to invoke them through AQL packets. 720 721``e_flags`` 722 The AMDGPU backend uses the following ELF header flags: 723 724 .. table:: AMDGPU ELF Header ``e_flags`` 725 :name: amdgpu-elf-header-e_flags-table 726 727 ================================= ========== ============================= 728 Name Value Description 729 ================================= ========== ============================= 730 **AMDGPU Processor Flag** See :ref:`amdgpu-processor-table`. 731 -------------------------------------------- ----------------------------- 732 ``EF_AMDGPU_MACH`` 0x000000ff AMDGPU processor selection 733 mask for 734 ``EF_AMDGPU_MACH_xxx`` values 735 defined in 736 :ref:`amdgpu-ef-amdgpu-mach-table`. 737 ``EF_AMDGPU_XNACK`` 0x00000100 Indicates if the ``xnack`` 738 target feature is 739 enabled for all code 740 contained in the code object. 741 If the processor 742 does not support the 743 ``xnack`` target 744 feature then must 745 be 0. 746 See 747 :ref:`amdgpu-target-features`. 748 ``EF_AMDGPU_SRAM_ECC`` 0x00000200 Indicates if the ``sram-ecc`` 749 target feature is 750 enabled for all code 751 contained in the code object. 752 If the processor 753 does not support the 754 ``sram-ecc`` target 755 feature then must 756 be 0. 757 See 758 :ref:`amdgpu-target-features`. 759 ================================= ========== ============================= 760 761 .. table:: AMDGPU ``EF_AMDGPU_MACH`` Values 762 :name: amdgpu-ef-amdgpu-mach-table 763 764 ================================= ========== ============================= 765 Name Value Description (see 766 :ref:`amdgpu-processor-table`) 767 ================================= ========== ============================= 768 ``EF_AMDGPU_MACH_NONE`` 0x000 *not specified* 769 ``EF_AMDGPU_MACH_R600_R600`` 0x001 ``r600`` 770 ``EF_AMDGPU_MACH_R600_R630`` 0x002 ``r630`` 771 ``EF_AMDGPU_MACH_R600_RS880`` 0x003 ``rs880`` 772 ``EF_AMDGPU_MACH_R600_RV670`` 0x004 ``rv670`` 773 ``EF_AMDGPU_MACH_R600_RV710`` 0x005 ``rv710`` 774 ``EF_AMDGPU_MACH_R600_RV730`` 0x006 ``rv730`` 775 ``EF_AMDGPU_MACH_R600_RV770`` 0x007 ``rv770`` 776 ``EF_AMDGPU_MACH_R600_CEDAR`` 0x008 ``cedar`` 777 ``EF_AMDGPU_MACH_R600_CYPRESS`` 0x009 ``cypress`` 778 ``EF_AMDGPU_MACH_R600_JUNIPER`` 0x00a ``juniper`` 779 ``EF_AMDGPU_MACH_R600_REDWOOD`` 0x00b ``redwood`` 780 ``EF_AMDGPU_MACH_R600_SUMO`` 0x00c ``sumo`` 781 ``EF_AMDGPU_MACH_R600_BARTS`` 0x00d ``barts`` 782 ``EF_AMDGPU_MACH_R600_CAICOS`` 0x00e ``caicos`` 783 ``EF_AMDGPU_MACH_R600_CAYMAN`` 0x00f ``cayman`` 784 ``EF_AMDGPU_MACH_R600_TURKS`` 0x010 ``turks`` 785 *reserved* 0x011 - Reserved for ``r600`` 786 0x01f architecture processors. 787 ``EF_AMDGPU_MACH_AMDGCN_GFX600`` 0x020 ``gfx600`` 788 ``EF_AMDGPU_MACH_AMDGCN_GFX601`` 0x021 ``gfx601`` 789 ``EF_AMDGPU_MACH_AMDGCN_GFX700`` 0x022 ``gfx700`` 790 ``EF_AMDGPU_MACH_AMDGCN_GFX701`` 0x023 ``gfx701`` 791 ``EF_AMDGPU_MACH_AMDGCN_GFX702`` 0x024 ``gfx702`` 792 ``EF_AMDGPU_MACH_AMDGCN_GFX703`` 0x025 ``gfx703`` 793 ``EF_AMDGPU_MACH_AMDGCN_GFX704`` 0x026 ``gfx704`` 794 *reserved* 0x027 Reserved. 795 ``EF_AMDGPU_MACH_AMDGCN_GFX801`` 0x028 ``gfx801`` 796 ``EF_AMDGPU_MACH_AMDGCN_GFX802`` 0x029 ``gfx802`` 797 ``EF_AMDGPU_MACH_AMDGCN_GFX803`` 0x02a ``gfx803`` 798 ``EF_AMDGPU_MACH_AMDGCN_GFX810`` 0x02b ``gfx810`` 799 ``EF_AMDGPU_MACH_AMDGCN_GFX900`` 0x02c ``gfx900`` 800 ``EF_AMDGPU_MACH_AMDGCN_GFX902`` 0x02d ``gfx902`` 801 ``EF_AMDGPU_MACH_AMDGCN_GFX904`` 0x02e ``gfx904`` 802 ``EF_AMDGPU_MACH_AMDGCN_GFX906`` 0x02f ``gfx906`` 803 ``EF_AMDGPU_MACH_AMDGCN_GFX908`` 0x030 ``gfx908`` 804 ``EF_AMDGPU_MACH_AMDGCN_GFX909`` 0x031 ``gfx909`` 805 *reserved* 0x032 Reserved. 806 ``EF_AMDGPU_MACH_AMDGCN_GFX1010`` 0x033 ``gfx1010`` 807 ``EF_AMDGPU_MACH_AMDGCN_GFX1011`` 0x034 ``gfx1011`` 808 ``EF_AMDGPU_MACH_AMDGCN_GFX1012`` 0x035 ``gfx1012`` 809 ================================= ========== ============================= 810 811Sections 812-------- 813 814An AMDGPU target ELF code object has the standard ELF sections which include: 815 816 .. table:: AMDGPU ELF Sections 817 :name: amdgpu-elf-sections-table 818 819 ================== ================ ================================= 820 Name Type Attributes 821 ================== ================ ================================= 822 ``.bss`` ``SHT_NOBITS`` ``SHF_ALLOC`` + ``SHF_WRITE`` 823 ``.data`` ``SHT_PROGBITS`` ``SHF_ALLOC`` + ``SHF_WRITE`` 824 ``.debug_``\ *\** ``SHT_PROGBITS`` *none* 825 ``.dynamic`` ``SHT_DYNAMIC`` ``SHF_ALLOC`` 826 ``.dynstr`` ``SHT_PROGBITS`` ``SHF_ALLOC`` 827 ``.dynsym`` ``SHT_PROGBITS`` ``SHF_ALLOC`` 828 ``.got`` ``SHT_PROGBITS`` ``SHF_ALLOC`` + ``SHF_WRITE`` 829 ``.hash`` ``SHT_HASH`` ``SHF_ALLOC`` 830 ``.note`` ``SHT_NOTE`` *none* 831 ``.rela``\ *name* ``SHT_RELA`` *none* 832 ``.rela.dyn`` ``SHT_RELA`` *none* 833 ``.rodata`` ``SHT_PROGBITS`` ``SHF_ALLOC`` 834 ``.shstrtab`` ``SHT_STRTAB`` *none* 835 ``.strtab`` ``SHT_STRTAB`` *none* 836 ``.symtab`` ``SHT_SYMTAB`` *none* 837 ``.text`` ``SHT_PROGBITS`` ``SHF_ALLOC`` + ``SHF_EXECINSTR`` 838 ================== ================ ================================= 839 840These sections have their standard meanings (see [ELF]_) and are only generated 841if needed. 842 843``.debug``\ *\** 844 The standard DWARF sections. See :ref:`amdgpu-dwarf-debug-information` for 845 information on the DWARF produced by the AMDGPU backend. 846 847``.dynamic``, ``.dynstr``, ``.dynsym``, ``.hash`` 848 The standard sections used by a dynamic loader. 849 850``.note`` 851 See :ref:`amdgpu-note-records` for the note records supported by the AMDGPU 852 backend. 853 854``.rela``\ *name*, ``.rela.dyn`` 855 For relocatable code objects, *name* is the name of the section that the 856 relocation records apply. For example, ``.rela.text`` is the section name for 857 relocation records associated with the ``.text`` section. 858 859 For linked shared code objects, ``.rela.dyn`` contains all the relocation 860 records from each of the relocatable code object's ``.rela``\ *name* sections. 861 862 See :ref:`amdgpu-relocation-records` for the relocation records supported by 863 the AMDGPU backend. 864 865``.text`` 866 The executable machine code for the kernels and functions they call. Generated 867 as position independent code. See :ref:`amdgpu-code-conventions` for 868 information on conventions used in the isa generation. 869 870.. _amdgpu-note-records: 871 872Note Records 873------------ 874 875The AMDGPU backend code object contains ELF note records in the ``.note`` 876section. The set of generated notes and their semantics depend on the code 877object version; see :ref:`amdgpu-note-records-v2` and 878:ref:`amdgpu-note-records-v3`. 879 880As required by ``ELFCLASS32`` and ``ELFCLASS64``, minimal zero-byte padding 881must be generated after the ``name`` field to ensure the ``desc`` field is 4 882byte aligned. In addition, minimal zero-byte padding must be generated to 883ensure the ``desc`` field size is a multiple of 4 bytes. The ``sh_addralign`` 884field of the ``.note`` section must be at least 4 to indicate at least 8 byte 885alignment. 886 887.. _amdgpu-note-records-v2: 888 889Code Object V2 Note Records (-mattr=-code-object-v3) 890~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 891 892.. warning:: Code Object V2 is not the default code object version emitted by 893 this version of LLVM. For a description of the notes generated with the 894 default configuration (Code Object V3) see :ref:`amdgpu-note-records-v3`. 895 896The AMDGPU backend code object uses the following ELF note record in the 897``.note`` section when compiling for Code Object V2 (-mattr=-code-object-v3). 898 899Additional note records may be present, but any which are not documented here 900are deprecated and should not be used. 901 902 .. table:: AMDGPU Code Object V2 ELF Note Records 903 :name: amdgpu-elf-note-records-table-v2 904 905 ===== ============================== ====================================== 906 Name Type Description 907 ===== ============================== ====================================== 908 "AMD" ``NT_AMD_AMDGPU_HSA_METADATA`` <metadata null terminated string> 909 ===== ============================== ====================================== 910 911.. 912 913 .. table:: AMDGPU Code Object V2 ELF Note Record Enumeration Values 914 :name: amdgpu-elf-note-record-enumeration-values-table-v2 915 916 ============================== ===== 917 Name Value 918 ============================== ===== 919 *reserved* 0-9 920 ``NT_AMD_AMDGPU_HSA_METADATA`` 10 921 *reserved* 11 922 ============================== ===== 923 924``NT_AMD_AMDGPU_HSA_METADATA`` 925 Specifies extensible metadata associated with the code objects executed on HSA 926 [HSA]_ compatible runtimes such as AMD's ROCm [AMD-ROCm]_. It is required when 927 the target triple OS is ``amdhsa`` (see :ref:`amdgpu-target-triples`). See 928 :ref:`amdgpu-amdhsa-code-object-metadata-v2` for the syntax of the code 929 object metadata string. 930 931.. _amdgpu-note-records-v3: 932 933Code Object V3 Note Records (-mattr=+code-object-v3) 934~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 935 936The AMDGPU backend code object uses the following ELF note record in the 937``.note`` section when compiling for Code Object V3 (-mattr=+code-object-v3). 938 939Additional note records may be present, but any which are not documented here 940are deprecated and should not be used. 941 942 .. table:: AMDGPU Code Object V3 ELF Note Records 943 :name: amdgpu-elf-note-records-table-v3 944 945 ======== ============================== ====================================== 946 Name Type Description 947 ======== ============================== ====================================== 948 "AMDGPU" ``NT_AMDGPU_METADATA`` Metadata in Message Pack [MsgPack]_ 949 binary format. 950 ======== ============================== ====================================== 951 952.. 953 954 .. table:: AMDGPU Code Object V3 ELF Note Record Enumeration Values 955 :name: amdgpu-elf-note-record-enumeration-values-table-v3 956 957 ============================== ===== 958 Name Value 959 ============================== ===== 960 *reserved* 0-31 961 ``NT_AMDGPU_METADATA`` 32 962 ============================== ===== 963 964``NT_AMDGPU_METADATA`` 965 Specifies extensible metadata associated with an AMDGPU code 966 object. It is encoded as a map in the Message Pack [MsgPack]_ binary 967 data format. See :ref:`amdgpu-amdhsa-code-object-metadata-v3` for the 968 map keys defined for the ``amdhsa`` OS. 969 970.. _amdgpu-symbols: 971 972Symbols 973------- 974 975Symbols include the following: 976 977 .. table:: AMDGPU ELF Symbols 978 :name: amdgpu-elf-symbols-table 979 980 ===================== ================== ================ ================== 981 Name Type Section Description 982 ===================== ================== ================ ================== 983 *link-name* ``STT_OBJECT`` - ``.data`` Global variable 984 - ``.rodata`` 985 - ``.bss`` 986 *link-name*\ ``.kd`` ``STT_OBJECT`` - ``.rodata`` Kernel descriptor 987 *link-name* ``STT_FUNC`` - ``.text`` Kernel entry point 988 *link-name* ``STT_OBJECT`` - SHN_AMDGPU_LDS Global variable in LDS 989 ===================== ================== ================ ================== 990 991Global variable 992 Global variables both used and defined by the compilation unit. 993 994 If the symbol is defined in the compilation unit then it is allocated in the 995 appropriate section according to if it has initialized data or is readonly. 996 997 If the symbol is external then its section is ``STN_UNDEF`` and the loader 998 will resolve relocations using the definition provided by another code object 999 or explicitly defined by the runtime. 1000 1001 If the symbol resides in local/group memory (LDS) then its section is the 1002 special processor specific section name ``SHN_AMDGPU_LDS``, and the 1003 ``st_value`` field describes alignment requirements as it does for common 1004 symbols. 1005 1006 .. TODO:: 1007 1008 Add description of linked shared object symbols. Seems undefined symbols 1009 are marked as STT_NOTYPE. 1010 1011Kernel descriptor 1012 Every HSA kernel has an associated kernel descriptor. It is the address of the 1013 kernel descriptor that is used in the AQL dispatch packet used to invoke the 1014 kernel, not the kernel entry point. The layout of the HSA kernel descriptor is 1015 defined in :ref:`amdgpu-amdhsa-kernel-descriptor`. 1016 1017Kernel entry point 1018 Every HSA kernel also has a symbol for its machine code entry point. 1019 1020.. _amdgpu-relocation-records: 1021 1022Relocation Records 1023------------------ 1024 1025AMDGPU backend generates ``Elf64_Rela`` relocation records. Supported 1026relocatable fields are: 1027 1028``word32`` 1029 This specifies a 32-bit field occupying 4 bytes with arbitrary byte 1030 alignment. These values use the same byte order as other word values in the 1031 AMDGPU architecture. 1032 1033``word64`` 1034 This specifies a 64-bit field occupying 8 bytes with arbitrary byte 1035 alignment. These values use the same byte order as other word values in the 1036 AMDGPU architecture. 1037 1038Following notations are used for specifying relocation calculations: 1039 1040**A** 1041 Represents the addend used to compute the value of the relocatable field. 1042 1043**G** 1044 Represents the offset into the global offset table at which the relocation 1045 entry's symbol will reside during execution. 1046 1047**GOT** 1048 Represents the address of the global offset table. 1049 1050**P** 1051 Represents the place (section offset for ``et_rel`` or address for ``et_dyn``) 1052 of the storage unit being relocated (computed using ``r_offset``). 1053 1054**S** 1055 Represents the value of the symbol whose index resides in the relocation 1056 entry. Relocations not using this must specify a symbol index of 1057 ``STN_UNDEF``. 1058 1059**B** 1060 Represents the base address of a loaded executable or shared object which is 1061 the difference between the ELF address and the actual load address. 1062 Relocations using this are only valid in executable or shared objects. 1063 1064The following relocation types are supported: 1065 1066 .. table:: AMDGPU ELF Relocation Records 1067 :name: amdgpu-elf-relocation-records-table 1068 1069 ========================== ======= ===== ========== ============================== 1070 Relocation Type Kind Value Field Calculation 1071 ========================== ======= ===== ========== ============================== 1072 ``R_AMDGPU_NONE`` 0 *none* *none* 1073 ``R_AMDGPU_ABS32_LO`` Static, 1 ``word32`` (S + A) & 0xFFFFFFFF 1074 Dynamic 1075 ``R_AMDGPU_ABS32_HI`` Static, 2 ``word32`` (S + A) >> 32 1076 Dynamic 1077 ``R_AMDGPU_ABS64`` Static, 3 ``word64`` S + A 1078 Dynamic 1079 ``R_AMDGPU_REL32`` Static 4 ``word32`` S + A - P 1080 ``R_AMDGPU_REL64`` Static 5 ``word64`` S + A - P 1081 ``R_AMDGPU_ABS32`` Static, 6 ``word32`` S + A 1082 Dynamic 1083 ``R_AMDGPU_GOTPCREL`` Static 7 ``word32`` G + GOT + A - P 1084 ``R_AMDGPU_GOTPCREL32_LO`` Static 8 ``word32`` (G + GOT + A - P) & 0xFFFFFFFF 1085 ``R_AMDGPU_GOTPCREL32_HI`` Static 9 ``word32`` (G + GOT + A - P) >> 32 1086 ``R_AMDGPU_REL32_LO`` Static 10 ``word32`` (S + A - P) & 0xFFFFFFFF 1087 ``R_AMDGPU_REL32_HI`` Static 11 ``word32`` (S + A - P) >> 32 1088 *reserved* 12 1089 ``R_AMDGPU_RELATIVE64`` Dynamic 13 ``word64`` B + A 1090 ========================== ======= ===== ========== ============================== 1091 1092``R_AMDGPU_ABS32_LO`` and ``R_AMDGPU_ABS32_HI`` are only supported by 1093the ``mesa3d`` OS, which does not support ``R_AMDGPU_ABS64``. 1094 1095There is no current OS loader support for 32-bit programs and so 1096``R_AMDGPU_ABS32`` is not used. 1097 1098.. _amdgpu-loaded-code-object-path-uniform-resource-identifier: 1099 1100Loaded Code Object Path Uniform Resource Identifier (URI) 1101--------------------------------------------------------- 1102 1103The AMD GPU code object loader represents the path of the ELF shared object from 1104which the code object was loaded as a textual Unifom Resource Identifier (URI). 1105Note that the code object is the in memory loaded relocated form of the ELF 1106shared object. Multiple code objects may be loaded at different memory 1107addresses in the same process from the same ELF shared object. 1108 1109The loaded code object path URI syntax is defined by the following BNF syntax: 1110 1111.. code:: 1112 1113 code_object_uri ::== file_uri | memory_uri 1114 file_uri ::== "file://" file_path [ range_specifier ] 1115 memory_uri ::== "memory://" process_id range_specifier 1116 range_specifier ::== [ "#" | "?" ] "offset=" number "&" "size=" number 1117 file_path ::== URI_ENCODED_OS_FILE_PATH 1118 process_id ::== DECIMAL_NUMBER 1119 number ::== HEX_NUMBER | DECIMAL_NUMBER | OCTAL_NUMBER 1120 1121**number** 1122 Is a C integral literal where hexadecimal values are prefixed by "0x" or "0X", 1123 and octal values by "0". 1124 1125**file_path** 1126 Is the file's path specified as a URI encoded UTF-8 string. In URI encoding, 1127 every character that is not in the regular expression ``[a-zA-Z0-9/_.~-]`` is 1128 encoded as two uppercase hexidecimal digits proceeded by "%". Directories in 1129 the path are separated by "/". 1130 1131**offset** 1132 Is a 0-based byte offset to the start of the code object. For a file URI, it 1133 is from the start of the file specified by the ``file_path``, and if omitted 1134 defaults to 0. For a memory URI, it is the memory address and is required. 1135 1136**size** 1137 Is the number of bytes in the code object. For a file URI, if omitted it 1138 defaults to the size of the file. It is required for a memory URI. 1139 1140**process_id** 1141 Is the identity of the process owning the memory. For Linux it is the C 1142 unsigned integral decimal literal for the process ID (PID). 1143 1144For example: 1145 1146.. code:: 1147 1148 file:///dir1/dir2/file1 1149 file:///dir3/dir4/file2#offset=0x2000&size=3000 1150 memory://1234#offset=0x20000&size=3000 1151 1152.. _amdgpu-dwarf-debug-information: 1153 1154DWARF Debug Information 1155======================= 1156 1157.. warning:: 1158 1159 This section describes a **provisional proposal** for AMDGPU DWARF [DWARF]_ 1160 that is not currently fully implemented and is subject to change. 1161 1162AMDGPU generates DWARF [DWARF]_ debugging information ELF sections (see 1163:ref:`amdgpu-elf-code-object`) which contain information that maps the code 1164object executable code and data to the source language constructs. It can be 1165used by tools such as debuggers and profilers. It uses features defined in 1166:doc:`AMDGPUDwarfProposalForHeterogeneousDebugging` that are made available in 1167DWARF Version 4 and DWARF Version 5 as an LLVM vendor extension. 1168 1169This section defines the AMDGPU target architecture specific DWARF mappings. 1170 1171.. _amdgpu-dwarf-register-identifier: 1172 1173Register Identifier 1174------------------- 1175 1176This section defines the AMDGPU target architecture register numbers used in 1177DWARF operation expressions (see DWARF Version 5 section 2.5 and 1178:ref:`amdgpu-dwarf-operation-expressions`) and Call Frame Information 1179instructions (see DWARF Version 5 section 6.4 and 1180:ref:`amdgpu-dwarf-call-frame-information`). 1181 1182A single code object can contain code for kernels that have different wavefront 1183sizes. The vector registers and some scalar registers are based on the wavefront 1184size. AMDGPU defines distinct DWARF registers for each wavefront size. This 1185simplifies the consumer of the DWARF so that each register has a fixed size, 1186rather than being dynamic according to the wavefront size mode. Similarly, 1187distinct DWARF registers are defined for those registers that vary in size 1188according to the process address size. This allows a consumer to treat a 1189specific AMDGPU processor as a single architecture regardless of how it is 1190configured at run time. The compiler explicitly specifies the DWARF registers 1191that match the mode in which the code it is generating will be executed. 1192 1193DWARF registers are encoded as numbers, which are mapped to architecture 1194registers. The mapping for AMDGPU is defined in 1195:ref:`amdgpu-dwarf-register-mapping-table`. All AMDGPU targets use the same 1196mapping. 1197 1198.. table:: AMDGPU DWARF Register Mapping 1199 :name: amdgpu-dwarf-register-mapping-table 1200 1201 ============== ================= ======== ================================== 1202 DWARF Register AMDGPU Register Bit Size Description 1203 ============== ================= ======== ================================== 1204 0 PC_32 32 Program Counter (PC) when 1205 executing in a 32-bit process 1206 address space. Used in the CFI to 1207 describe the PC of the calling 1208 frame. 1209 1 EXEC_MASK_32 32 Execution Mask Register when 1210 executing in wavefront 32 mode. 1211 2-15 *Reserved* 1212 16 PC_64 64 Program Counter (PC) when 1213 executing in a 64-bit process 1214 address space. Used in the CFI to 1215 describe the PC of the calling 1216 frame. 1217 17 EXEC_MASK_64 64 Execution Mask Register when 1218 executing in wavefront 64 mode. 1219 18-31 *Reserved* 1220 32-95 SGPR0-SGPR63 32 Scalar General Purpose 1221 Registers. 1222 96-127 *Reserved* 1223 128-511 *Reserved* 1224 512-1023 *Reserved* 1225 1024-1087 *Reserved* 1226 1088-1129 SGPR64-SGPR105 32 Scalar General Purpose Registers 1227 1130-1535 *Reserved* 1228 1536-1791 VGPR0-VGPR255 32*32 Vector General Purpose Registers 1229 when executing in wavefront 32 1230 mode. 1231 1792-2047 *Reserved* 1232 2048-2303 AGPR0-AGPR255 32*32 Vector Accumulation Registers 1233 when executing in wavefront 32 1234 ode. 1235 2304-2559 *Reserved* 1236 2560-2815 VGPR0-VGPR255 64*32 Vector General Purpose Registers 1237 when executing in wavefront 64 1238 mode. 1239 2816-3071 *Reserved* 1240 3072-3327 AGPR0-AGPR255 64*32 Vector Accumulation Registers 1241 when executing in wavefront 64 1242 mode. 1243 3328-3583 *Reserved* 1244 ============== ================= ======== ================================== 1245 1246The vector registers are represented as the full size for the wavefront. They 1247are organized as consecutive dwords (32-bits), one per lane, with the dword at 1248the least significant bit position corresponding to lane 0 and so forth. DWARF 1249location expressions involving the ``DW_OP_LLVM_offset`` and 1250``DW_OP_LLVM_push_lane`` operations are used to select the part of the vector 1251register corresponding to the lane that is executing the current thread of 1252execution in languages that are implemented using a SIMD or SIMT execution 1253model. 1254 1255If the wavefront size is 32 lanes then the wavefront 32 mode register 1256definitions are used. If the wavefront size is 64 lanes then the wavefront 64 1257mode register definitions are used. Some AMDGPU targets support executing in 1258both wavefront 32 and wavefront 64 mode. The register definitions corresponding 1259to the wavefront mode of the generated code will be used. 1260 1261If code is generated to execute in a 32-bit process address space, then the 126232-bit process address space register definitions are used. If code is generated 1263to execute in a 64-bit process address space, then the 64-bit process address 1264space register definitions are used. The ``amdgcn`` target only supports the 126564-bit process address space. 1266 1267.. _amdgpu-dwarf-address-class-identifier: 1268 1269Address Class Identifier 1270------------------------ 1271 1272The DWARF address class represents the source language memory space. See DWARF 1273Version 5 section 2.12 which is updated by the propoal in 1274:ref:`amdgpu-dwarf-segment_addresses`. 1275 1276The DWARF address class mapping used for AMDGPU is defined in 1277:ref:`amdgpu-dwarf-address-class-mapping-table`. 1278 1279.. table:: AMDGPU DWARF Address Class Mapping 1280 :name: amdgpu-dwarf-address-class-mapping-table 1281 1282 ========================= ====== ================= 1283 DWARF AMDGPU 1284 -------------------------------- ----------------- 1285 Address Class Name Value Address Space 1286 ========================= ====== ================= 1287 ``DW_ADDR_none`` 0x0000 Generic (Flat) 1288 ``DW_ADDR_LLVM_global`` 0x0001 Global 1289 ``DW_ADDR_LLVM_constant`` 0x0002 Global 1290 ``DW_ADDR_LLVM_group`` 0x0003 Local (group/LDS) 1291 ``DW_ADDR_LLVM_private`` 0x0004 Private (Scratch) 1292 ``DW_ADDR_AMDGPU_region`` 0x8000 Region (GDS) 1293 ========================= ====== ================= 1294 1295The DWARF address class values defined in the proposal at 1296:ref:`amdgpu-dwarf-segment_addresses` are used. 1297 1298In addition, ``DW_ADDR_AMDGPU_region`` is encoded as a vendor extension. This is 1299available for use for the AMD extension for access to the hardware GDS memory 1300which is scratchpad memory allocated per device. 1301 1302For AMDGPU if no ``DW_AT_address_class`` attribute is present, then the default 1303address class of ``DW_ADDR_none`` is used. 1304 1305See :ref:`amdgpu-dwarf-address-space-identifier` for information on the AMDGPU 1306mapping of DWARF address classes to DWARF address spaces, including address size 1307and NULL value. 1308 1309.. _amdgpu-dwarf-address-space-identifier: 1310 1311Address Space Identifier 1312------------------------ 1313 1314DWARF address spaces correspond to target architecture specific linear 1315addressable memory areas. See DWARF Version 5 section 2.12 and 1316:ref:`amdgpu-dwarf-segment_addresses`. 1317 1318The DWARF address space mapping used for AMDGPU is defined in 1319:ref:`amdgpu-dwarf-address-space-mapping-table`. 1320 1321.. table:: AMDGPU DWARF Address Space Mapping 1322 :name: amdgpu-dwarf-address-space-mapping-table 1323 1324 ======================================= ===== ======= ======== ================= ======================= 1325 DWARF AMDGPU Notes 1326 --------------------------------------- ----- ---------------- ----------------- ----------------------- 1327 Address Space Name Value Address Bit Size Address Space 1328 --------------------------------------- ----- ------- -------- ----------------- ----------------------- 1329 .. 64-bit 32-bit 1330 process process 1331 address address 1332 space space 1333 ======================================= ===== ======= ======== ================= ======================= 1334 ``DW_ASPACE_none`` 0x00 8 4 Global *default address space* 1335 ``DW_ASPACE_AMDGPU_generic`` 0x01 8 4 Generic (Flat) 1336 ``DW_ASPACE_AMDGPU_region`` 0x02 4 4 Region (GDS) 1337 ``DW_ASPACE_AMDGPU_local`` 0x03 4 4 Local (group/LDS) 1338 *Reserved* 0x04 1339 ``DW_ASPACE_AMDGPU_private_lane`` 0x05 4 4 Private (Scratch) *focused lane* 1340 ``DW_ASPACE_AMDGPU_private_wave`` 0x06 4 4 Private (Scratch) *unswizzled wavefront* 1341 *Reserved* 0x07- 1342 0x1F 1343 ``DW_ASPACE_AMDGPU_private_lane<0-63>`` 0x20- 4 4 Private (Scratch) *specific lane* 1344 0x5F 1345 ======================================= ===== ======= ======== ================= ======================= 1346 1347See :ref:`amdgpu-address-spaces` for information on the AMDGPU address spaces 1348including address size and NULL value. 1349 1350The ``DW_ASPACE_none`` address space is the default target architecture address 1351space used in DWARF operations that do not specify an address space. It 1352therefore has to map to the global address space so that the ``DW_OP_addr*`` and 1353related operations can refer to addresses in the program code. 1354 1355The ``DW_ASPACE_AMDGPU_generic`` address space allows location expressions to 1356specify the flat address space. If the address corresponds to an address in the 1357local address space, then it corresponds to the wavefront that is executing the 1358focused thread of execution. If the address corresponds to an address in the 1359private address space, then it corresponds to the lane that is executing the 1360focused thread of execution for languages that are implemented using a SIMD or 1361SIMT execution model. 1362 1363.. note:: 1364 1365 CUDA-like languages such as HIP that do not have address spaces in the 1366 language type system, but do allow variables to be allocated in different 1367 address spaces, need to explicitly specify the ``DW_ASPACE_AMDGPU_generic`` 1368 address space in the DWARF expression operations as the default address space 1369 is the global address space. 1370 1371The ``DW_ASPACE_AMDGPU_local`` address space allows location expressions to 1372specify the local address space corresponding to the wavefront that is executing 1373the focused thread of execution. 1374 1375The ``DW_ASPACE_AMDGPU_private_lane`` address space allows location expressions 1376to specify the private address space corresponding to the lane that is executing 1377the focused thread of execution for languages that are implemented using a SIMD 1378or SIMT execution model. 1379 1380The ``DW_ASPACE_AMDGPU_private_wave`` address space allows location expressions 1381to specify the unswizzled private address space corresponding to the wavefront 1382that is executing the focused thread of execution. The wavefront view of private 1383memory is the per wavefront unswizzled backing memory layout defined in 1384:ref:`amdgpu-address-spaces`, such that address 0 corresponds to the first 1385location for the backing memory of the wavefront (namely the address is not 1386offset by ``wavefront-scratch-base``). The following formula can be used to 1387convert from a ``DW_ASPACE_AMDGPU_private_lane`` address to a 1388``DW_ASPACE_AMDGPU_private_wave`` address: 1389 1390:: 1391 1392 private-address-wavefront = 1393 ((private-address-lane / 4) * wavefront-size * 4) + 1394 (wavefront-lane-id * 4) + (private-address-lane % 4) 1395 1396If the ``DW_ASPACE_AMDGPU_private_lane`` address is dword aligned, and the start 1397of the dwords for each lane starting with lane 0 is required, then this 1398simplifies to: 1399 1400:: 1401 1402 private-address-wavefront = 1403 private-address-lane * wavefront-size 1404 1405A compiler can use the ``DW_ASPACE_AMDGPU_private_wave`` address space to read a 1406complete spilled vector register back into a complete vector register in the 1407CFI. The frame pointer can be a private lane address which is dword aligned, 1408which can be shifted to multiply by the wavefront size, and then used to form a 1409private wavefront address that gives a location for a contiguous set of dwords, 1410one per lane, where the vector register dwords are spilled. The compiler knows 1411the wavefront size since it generates the code. Note that the type of the 1412address may have to be converted as the size of a 1413``DW_ASPACE_AMDGPU_private_lane`` address may be smaller than the size of a 1414``DW_ASPACE_AMDGPU_private_wave`` address. 1415 1416The ``DW_ASPACE_AMDGPU_private_lane<N>`` address space allows location 1417expressions to specify the private address space corresponding to a specific 1418lane N. For example, this can be used when the compiler spills scalar registers 1419to scratch memory, with each scalar register being saved to a different lane's 1420scratch memory. 1421 1422.. _amdgpu-dwarf-lane-identifier: 1423 1424Lane identifier 1425--------------- 1426 1427DWARF lane identifies specify a target architecture lane position for hardware 1428that executes in a SIMD or SIMT manner, and on which a source language maps its 1429threads of execution onto those lanes. The DWARF lane identifier is pushed by 1430the ``DW_OP_LLVM_push_lane`` DWARF expression operation. See DWARF Version 5 1431section 2.5 which is updated by the proposal in 1432:ref:`amdgpu-dwarf-operation-expressions`. 1433 1434For AMDGPU, the lane identifier corresponds to the hardware lane ID of a 1435wavefront. It is numbered from 0 to the wavefront size minus 1. 1436 1437Operation Expressions 1438--------------------- 1439 1440DWARF expressions are used to compute program values and the locations of 1441program objects. See DWARF Version 5 section 2.5 and 1442:ref:`amdgpu-dwarf-operation-expressions`. 1443 1444DWARF location descriptions describe how to access storage which includes memory 1445and registers. When accessing storage on AMDGPU, bytes are ordered with least 1446significant bytes first, and bits are ordered within bytes with least 1447significant bits first. 1448 1449For AMDGPU CFI expressions, ``DW_OP_LLVM_select_bit_piece`` is used to describe 1450unwinding vector registers that are spilled under the execution mask to memory: 1451the zero-single location description is the vector register, and the one-single 1452location description is the spilled memory location description. The 1453``DW_OP_LLVM_form_aspace_address`` is used to specify the address space of the 1454memory location description. 1455 1456In AMDGPU expressions, ``DW_OP_LLVM_select_bit_piece`` is used by the 1457``DW_AT_LLVM_lane_pc`` attribute expression where divergent control flow is 1458controlled by the execution mask. An undefined location description together 1459with ``DW_OP_LLVM_extend`` is used to indicate the lane was not active on entry 1460to the subprogram. See :ref:`amdgpu-dwarf-dw-at-llvm-lane-pc` for an example. 1461 1462Debugger Information Entry Attributes 1463------------------------------------- 1464 1465This section describes how certain debugger information entry attributes are 1466used by AMDGPU. See the sections in DWARF Version 5 section 2 which are updated 1467by the proposal in :ref:`amdgpu-dwarf-debugging-information-entry-attributes`. 1468 1469.. _amdgpu-dwarf-dw-at-llvm-lane-pc: 1470 1471``DW_AT_LLVM_lane_pc`` 1472~~~~~~~~~~~~~~~~~~~~~~ 1473 1474For AMDGPU, the ``DW_AT_LLVM_lane_pc`` attribute is used to specify the program 1475location of the separate lanes of a SIMT thread. 1476 1477If the lane is an active lane then this will be the same as the current program 1478location. 1479 1480If the lane is inactive, but was active on entry to the subprogram, then this is 1481the program location in the subprogram at which execution of the lane is 1482conceptual positioned. 1483 1484If the lane was not active on entry to the subprogram, then this will be the 1485undefined location. A client debugger can check if the lane is part of a valid 1486work-group by checking that the lane is in the range of the associated 1487work-group within the grid, accounting for partial work-groups. If it is not, 1488then the debugger can omit any information for the lane. Otherwise, the debugger 1489may repeatedly unwind the stack and inspect the ``DW_AT_LLVM_lane_pc`` of the 1490calling subprogram until it finds a non-undefined location. Conceptually the 1491lane only has the call frames that it has a non-undefined 1492``DW_AT_LLVM_lane_pc``. 1493 1494The following example illustrates how the AMDGPU backend can generate a DWARF 1495location list expression for the nested ``IF/THEN/ELSE`` structures of the 1496following subprogram pseudo code for a target with 64 lanes per wavefront. 1497 1498.. code:: 1499 :number-lines: 1500 1501 SUBPROGRAM X 1502 BEGIN 1503 a; 1504 IF (c1) THEN 1505 b; 1506 IF (c2) THEN 1507 c; 1508 ELSE 1509 d; 1510 ENDIF 1511 e; 1512 ELSE 1513 f; 1514 ENDIF 1515 g; 1516 END 1517 1518The AMDGPU backend may generate the following pseudo LLVM MIR to manipulate the 1519execution mask (``EXEC``) to linearize the control flow. The condition is 1520evaluated to make a mask of the lanes for which the condition evaluates to true. 1521First the ``THEN`` region is executed by setting the ``EXEC`` mask to the 1522logical ``AND`` of the current ``EXEC`` mask with the condition mask. Then the 1523``ELSE`` region is executed by negating the ``EXEC`` mask and logical ``AND`` of 1524the saved ``EXEC`` mask at the start of the region. After the ``IF/THEN/ELSE`` 1525region the ``EXEC`` mask is restored to the value it had at the beginning of the 1526region. This is shown below. Other approaches are possible, but the basic 1527concept is the same. 1528 1529.. code:: 1530 :number-lines: 1531 1532 $lex_start: 1533 a; 1534 %1 = EXEC 1535 %2 = c1 1536 $lex_1_start: 1537 EXEC = %1 & %2 1538 $if_1_then: 1539 b; 1540 %3 = EXEC 1541 %4 = c2 1542 $lex_1_1_start: 1543 EXEC = %3 & %4 1544 $lex_1_1_then: 1545 c; 1546 EXEC = ~EXEC & %3 1547 $lex_1_1_else: 1548 d; 1549 EXEC = %3 1550 $lex_1_1_end: 1551 e; 1552 EXEC = ~EXEC & %1 1553 $lex_1_else: 1554 f; 1555 EXEC = %1 1556 $lex_1_end: 1557 g; 1558 $lex_end: 1559 1560To create the DWARF location list expression that defines the location 1561description of a vector of lane program locations, the LLVM MIR ``DBG_VALUE`` 1562pseudo instruction can be used to annotate the linearized control flow. This can 1563be done by defining an artificial variable for the lane PC. The DWARF location 1564list expression created for it is used as the value of the 1565``DW_AT_LLVM_lane_pc`` attribute on the subprogram's debugger information entry. 1566 1567A DWARF procedure is defined for each well nested structured control flow region 1568which provides the conceptual lane program location for a lane if it is not 1569active (namely it is divergent). The DWARF operation expression for each region 1570conceptually inherits the value of the immediately enclosing region and modifies 1571it according to the semantics of the region. 1572 1573For an ``IF/THEN/ELSE`` region the divergent program location is at the start of 1574the region for the ``THEN`` region since it is executed first. For the ``ELSE`` 1575region the divergent program location is at the end of the ``IF/THEN/ELSE`` 1576region since the ``THEN`` region has completed. 1577 1578The lane PC artificial variable is assigned at each region transition. It uses 1579the immediately enclosing region's DWARF procedure to compute the program 1580location for each lane assuming they are divergent, and then modifies the result 1581by inserting the current program location for each lane that the ``EXEC`` mask 1582indicates is active. 1583 1584By having separate DWARF procedures for each region, they can be reused to 1585define the value for any nested region. This reduces the total size of the DWARF 1586operation expressions. 1587 1588The following provides an example using pseudo LLVM MIR. 1589 1590.. code:: 1591 :number-lines: 1592 1593 $lex_start: 1594 DEFINE_DWARF %__uint_64 = DW_TAG_base_type[ 1595 DW_AT_name = "__uint64"; 1596 DW_AT_byte_size = 8; 1597 DW_AT_encoding = DW_ATE_unsigned; 1598 ]; 1599 DEFINE_DWARF %__active_lane_pc = DW_TAG_dwarf_procedure[ 1600 DW_AT_name = "__active_lane_pc"; 1601 DW_AT_location = [ 1602 DW_OP_regx PC; 1603 DW_OP_LLVM_extend 64, 64; 1604 DW_OP_regval_type EXEC, %uint_64; 1605 DW_OP_LLVM_select_bit_piece 64, 64; 1606 ]; 1607 ]; 1608 DEFINE_DWARF %__divergent_lane_pc = DW_TAG_dwarf_procedure[ 1609 DW_AT_name = "__divergent_lane_pc"; 1610 DW_AT_location = [ 1611 DW_OP_LLVM_undefined; 1612 DW_OP_LLVM_extend 64, 64; 1613 ]; 1614 ]; 1615 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 1616 DW_OP_call_ref %__divergent_lane_pc; 1617 DW_OP_call_ref %__active_lane_pc; 1618 ]; 1619 a; 1620 %1 = EXEC; 1621 DBG_VALUE %1, $noreg, %__lex_1_save_exec; 1622 %2 = c1; 1623 $lex_1_start: 1624 EXEC = %1 & %2; 1625 $lex_1_then: 1626 DEFINE_DWARF %__divergent_lane_pc_1_then = DW_TAG_dwarf_procedure[ 1627 DW_AT_name = "__divergent_lane_pc_1_then"; 1628 DW_AT_location = DIExpression[ 1629 DW_OP_call_ref %__divergent_lane_pc; 1630 DW_OP_xaddr &lex_1_start; 1631 DW_OP_stack_value; 1632 DW_OP_LLVM_extend 64, 64; 1633 DW_OP_call_ref %__lex_1_save_exec; 1634 DW_OP_deref_type 64, %__uint_64; 1635 DW_OP_LLVM_select_bit_piece 64, 64; 1636 ]; 1637 ]; 1638 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 1639 DW_OP_call_ref %__divergent_lane_pc_1_then; 1640 DW_OP_call_ref %__active_lane_pc; 1641 ]; 1642 b; 1643 %3 = EXEC; 1644 DBG_VALUE %3, %__lex_1_1_save_exec; 1645 %4 = c2; 1646 $lex_1_1_start: 1647 EXEC = %3 & %4; 1648 $lex_1_1_then: 1649 DEFINE_DWARF %__divergent_lane_pc_1_1_then = DW_TAG_dwarf_procedure[ 1650 DW_AT_name = "__divergent_lane_pc_1_1_then"; 1651 DW_AT_location = DIExpression[ 1652 DW_OP_call_ref %__divergent_lane_pc_1_then; 1653 DW_OP_xaddr &lex_1_1_start; 1654 DW_OP_stack_value; 1655 DW_OP_LLVM_extend 64, 64; 1656 DW_OP_call_ref %__lex_1_1_save_exec; 1657 DW_OP_deref_type 64, %__uint_64; 1658 DW_OP_LLVM_select_bit_piece 64, 64; 1659 ]; 1660 ]; 1661 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 1662 DW_OP_call_ref %__divergent_lane_pc_1_1_then; 1663 DW_OP_call_ref %__active_lane_pc; 1664 ]; 1665 c; 1666 EXEC = ~EXEC & %3; 1667 $lex_1_1_else: 1668 DEFINE_DWARF %__divergent_lane_pc_1_1_else = DW_TAG_dwarf_procedure[ 1669 DW_AT_name = "__divergent_lane_pc_1_1_else"; 1670 DW_AT_location = DIExpression[ 1671 DW_OP_call_ref %__divergent_lane_pc_1_then; 1672 DW_OP_xaddr &lex_1_1_end; 1673 DW_OP_stack_value; 1674 DW_OP_LLVM_extend 64, 64; 1675 DW_OP_call_ref %__lex_1_1_save_exec; 1676 DW_OP_deref_type 64, %__uint_64; 1677 DW_OP_LLVM_select_bit_piece 64, 64; 1678 ]; 1679 ]; 1680 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 1681 DW_OP_call_ref %__divergent_lane_pc_1_1_else; 1682 DW_OP_call_ref %__active_lane_pc; 1683 ]; 1684 d; 1685 EXEC = %3; 1686 $lex_1_1_end: 1687 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 1688 DW_OP_call_ref %__divergent_lane_pc; 1689 DW_OP_call_ref %__active_lane_pc; 1690 ]; 1691 e; 1692 EXEC = ~EXEC & %1; 1693 $lex_1_else: 1694 DEFINE_DWARF %__divergent_lane_pc_1_else = DW_TAG_dwarf_procedure[ 1695 DW_AT_name = "__divergent_lane_pc_1_else"; 1696 DW_AT_location = DIExpression[ 1697 DW_OP_call_ref %__divergent_lane_pc; 1698 DW_OP_xaddr &lex_1_end; 1699 DW_OP_stack_value; 1700 DW_OP_LLVM_extend 64, 64; 1701 DW_OP_call_ref %__lex_1_save_exec; 1702 DW_OP_deref_type 64, %__uint_64; 1703 DW_OP_LLVM_select_bit_piece 64, 64; 1704 ]; 1705 ]; 1706 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 1707 DW_OP_call_ref %__divergent_lane_pc_1_else; 1708 DW_OP_call_ref %__active_lane_pc; 1709 ]; 1710 f; 1711 EXEC = %1; 1712 $lex_1_end: 1713 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc DIExpression[ 1714 DW_OP_call_ref %__divergent_lane_pc; 1715 DW_OP_call_ref %__active_lane_pc; 1716 ]; 1717 g; 1718 $lex_end: 1719 1720The DWARF procedure ``%__active_lane_pc`` is used to update the lane pc elements 1721that are active, with the current program location. 1722 1723Artificial variables %__lex_1_save_exec and %__lex_1_1_save_exec are created for 1724the execution masks saved on entry to a region. Using the ``DBG_VALUE`` pseudo 1725instruction, location list entries will be created that describe where the 1726artificial variables are allocated at any given program location. The compiler 1727may allocate them to registers or spill them to memory. 1728 1729The DWARF procedures for each region use the values of the saved execution mask 1730artificial variables to only update the lanes that are active on entry to the 1731region. All other lanes retain the value of the enclosing region where they were 1732last active. If they were not active on entry to the subprogram, then will have 1733the undefined location description. 1734 1735Other structured control flow regions can be handled similarly. For example, 1736loops would set the divergent program location for the region at the end of the 1737loop. Any lanes active will be in the loop, and any lanes not active must have 1738exited the loop. 1739 1740An ``IF/THEN/ELSEIF/ELSEIF/...`` region can be treated as a nest of 1741``IF/THEN/ELSE`` regions. 1742 1743The DWARF procedures can use the active lane artificial variable described in 1744:ref:`amdgpu-dwarf-amdgpu-dw-at-llvm-active-lane` rather than the actual 1745``EXEC`` mask in order to support whole or quad wavefront mode. 1746 1747.. _amdgpu-dwarf-amdgpu-dw-at-llvm-active-lane: 1748 1749``DW_AT_LLVM_active_lane`` 1750~~~~~~~~~~~~~~~~~~~~~~~~~~ 1751 1752The ``DW_AT_LLVM_active_lane`` attribute on a subprogram debugger information 1753entry is used to specify the lanes that are conceptually active for a SIMT 1754thread. 1755 1756The execution mask may be modified to implement whole or quad wavefront mode 1757operations. For example, all lanes may need to temporarily be made active to 1758execute a whole wavefront operation. Such regions would save the ``EXEC`` mask, 1759update it to enable the necessary lanes, perform the operations, and then 1760restore the ``EXEC`` mask from the saved value. While executing the whole 1761wavefront region, the conceptual execution mask is the saved value, not the 1762``EXEC`` value. 1763 1764This is handled by defining an artificial variable for the active lane mask. The 1765active lane mask artificial variable would be the actual ``EXEC`` mask for 1766normal regions, and the saved execution mask for regions where the mask is 1767temporarily updated. The location list expression created for this artificial 1768variable is used to define the value of the ``DW_AT_LLVM_active_lane`` 1769attribute. 1770 1771``DW_AT_LLVM_augmentation`` 1772~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1773 1774For AMDGPU, the ``DW_AT_LLVM_augmentation`` attribute of a compilation unit 1775debugger information entry has the following value for the augmentation string: 1776 1777:: 1778 1779 [amdgpu:v0.0] 1780 1781The "vX.Y" specifies the major X and minor Y version number of the AMDGPU 1782extensions used in the DWARF of the compilation unit. The version number 1783conforms to [SEMVER]_. 1784 1785Call Frame Information 1786---------------------- 1787 1788DWARF Call Frame Information (CFI) describes how a consumer can virtually 1789*unwind* call frames in a running process or core dump. See DWARF Version 5 1790section 6.4 and :ref:`amdgpu-dwarf-call-frame-information`. 1791 1792For AMDGPU, the Common Information Entry (CIE) fields have the following values: 1793 17941. ``augmentation`` string contains the following null-terminated UTF-8 string: 1795 1796 :: 1797 1798 [amd:v0.0] 1799 1800 The ``vX.Y`` specifies the major X and minor Y version number of the AMDGPU 1801 extensions used in this CIE or to the FDEs that use it. The version number 1802 conforms to [SEMVER]_. 1803 18042. ``address_size`` for the ``Global`` address space is defined in 1805 :ref:`amdgpu-dwarf-address-space-identifier`. 1806 18073. ``segment_selector_size`` is 0 as AMDGPU does not use a segment selector. 1808 18094. ``code_alignment_factor`` is 4 bytes. 1810 1811 .. TODO:: 1812 1813 Add to :ref:`amdgpu-processor-table` table. 1814 18155. ``data_alignment_factor`` is 4 bytes. 1816 1817 .. TODO:: 1818 1819 Add to :ref:`amdgpu-processor-table` table. 1820 18216. ``return_address_register`` is ``PC_32`` for 32-bit processes and ``PC_64`` 1822 for 64-bit processes defined in :ref:`amdgpu-dwarf-register-identifier`. 1823 18247. ``initial_instructions`` Since a subprogram X with fewer registers can be 1825 called from subprogram Y that has more allocated, X will not change any of 1826 the extra registers as it cannot access them. Therefore, the default rule 1827 for all columns is ``same value``. 1828 1829For AMDGPU the register number follows the numbering defined in 1830:ref:`amdgpu-dwarf-register-identifier`. 1831 1832For AMDGPU the instructions are variable size. A consumer can subtract 1 from 1833the return address to get the address of a byte within the call site 1834instructions. See DWARF Version 5 section 6.4.4. 1835 1836Accelerated Access 1837------------------ 1838 1839See DWARF Version 5 section 6.1. 1840 1841Lookup By Name Section Header 1842~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1843 1844See DWARF Version 5 section 6.1.1.4.1 and :ref:`amdgpu-dwarf-lookup-by-name`. 1845 1846For AMDGPU the lookup by name section header table: 1847 1848``augmentation_string_size`` (uword) 1849 1850 Set to the length of the ``augmentation_string`` value which is always a 1851 multiple of 4. 1852 1853``augmentation_string`` (sequence of UTF-8 characters) 1854 1855 Contains the following UTF-8 string null padded to a multiple of 4 bytes: 1856 1857 :: 1858 1859 [amdgpu:v0.0] 1860 1861 The "vX.Y" specifies the major X and minor Y version number of the AMDGPU 1862 extensions used in the DWARF of this index. The version number conforms to 1863 [SEMVER]_. 1864 1865 .. note:: 1866 1867 This is different to the DWARF Version 5 definition that requires the first 1868 4 characters to be the vendor ID. But this is consistent with the other 1869 augmentation strings and does allow multiple vendor contributions. However, 1870 backwards compatibility may be more desirable. 1871 1872Lookup By Address Section Header 1873~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1874 1875See DWARF Version 5 section 6.1.2. 1876 1877For AMDGPU the lookup by address section header table: 1878 1879``address_size`` (ubyte) 1880 1881 Match the address size for the ``Global`` address space defined in 1882 :ref:`amdgpu-dwarf-address-space-identifier`. 1883 1884``segment_selector_size`` (ubyte) 1885 1886 AMDGPU does not use a segment selector so this is 0. The entries in the 1887 ``.debug_aranges`` do not have a segment selector. 1888 1889Line Number Information 1890----------------------- 1891 1892See DWARF Version 5 section 6.2 and :ref:`amdgpu-dwarf-line-number-information`. 1893 1894AMDGPU does not use the ``isa`` state machine registers and always sets it to 0. 1895The instruction set must be obtained from the ELF file header ``e_flags`` field 1896in the ``EF_AMDGPU_MACH`` bit position (see :ref:`ELF Header 1897<amdgpu-elf-header>`). See DWARF Version 5 section 6.2.2. 1898 1899.. TODO:: 1900 1901 Should the ``isa`` state machine register be used to indicate if the code is 1902 in wavefront32 or wavefront64 mode? Or used to specify the architecture ISA? 1903 1904For AMDGPU the line number program header fields have the following values (see 1905DWARF Version 5 section 6.2.4): 1906 1907``address_size`` (ubyte) 1908 Matches the address size for the ``Global`` address space defined in 1909 :ref:`amdgpu-dwarf-address-space-identifier`. 1910 1911``segment_selector_size`` (ubyte) 1912 AMDGPU does not use a segment selector so this is 0. 1913 1914``minimum_instruction_length`` (ubyte) 1915 For GFX9-GFX10 this is 4. 1916 1917``maximum_operations_per_instruction`` (ubyte) 1918 For GFX9-GFX10 this is 1. 1919 1920Source text for online-compiled programs (for example, those compiled by the 1921OpenCL language runtime) may be embedded into the DWARF Version 5 line table. 1922See DWARF Version 5 section 6.2.4.1 which is updated by the proposal in 1923:ref:`DW_LNCT_LLVM_source 1924<amdgpu-dwarf-line-number-information-dw-lnct-llvm-source>`. 1925 1926The Clang option used to control source embedding in AMDGPU is defined in 1927:ref:`amdgpu-clang-debug-options-table`. 1928 1929 .. table:: AMDGPU Clang Debug Options 1930 :name: amdgpu-clang-debug-options-table 1931 1932 ==================== ================================================== 1933 Debug Flag Description 1934 ==================== ================================================== 1935 -g[no-]embed-source Enable/disable embedding source text in DWARF 1936 debug sections. Useful for environments where 1937 source cannot be written to disk, such as 1938 when performing online compilation. 1939 ==================== ================================================== 1940 1941For example: 1942 1943``-gembed-source`` 1944 Enable the embedded source. 1945 1946``-gno-embed-source`` 1947 Disable the embedded source. 1948 194932-Bit and 64-Bit DWARF Formats 1950------------------------------- 1951 1952See DWARF Version 5 section 7.4 and 1953:ref:`amdgpu-dwarf-32-bit-and-64-bit-dwarf-formats`. 1954 1955For AMDGPU: 1956 1957* For the ``amdgcn`` target architecture only the 64-bit process address space 1958 is supported. 1959 1960* The producer can generate either 32-bit or 64-bit DWARF format. LLVM generates 1961 the 32-bit DWARF format. 1962 1963Unit Headers 1964------------ 1965 1966For AMDGPU the following values apply for each of the unit headers described in 1967DWARF Version 5 sections 7.5.1.1, 7.5.1.2, and 7.5.1.3: 1968 1969``address_size`` (ubyte) 1970 Matches the address size for the ``Global`` address space defined in 1971 :ref:`amdgpu-dwarf-address-space-identifier`. 1972 1973.. _amdgpu-code-conventions: 1974 1975Code Conventions 1976================ 1977 1978This section provides code conventions used for each supported target triple OS 1979(see :ref:`amdgpu-target-triples`). 1980 1981AMDHSA 1982------ 1983 1984This section provides code conventions used when the target triple OS is 1985``amdhsa`` (see :ref:`amdgpu-target-triples`). 1986 1987.. _amdgpu-amdhsa-code-object-target-identification: 1988 1989Code Object Target Identification 1990~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1991 1992The AMDHSA OS uses the following syntax to specify the code object 1993target as a single string: 1994 1995 ``<Architecture>-<Vendor>-<OS>-<Environment>-<Processor><Target Features>`` 1996 1997Where: 1998 1999 - ``<Architecture>``, ``<Vendor>``, ``<OS>`` and ``<Environment>`` 2000 are the same as the *Target Triple* (see 2001 :ref:`amdgpu-target-triples`). 2002 2003 - ``<Processor>`` is the same as the *Processor* (see 2004 :ref:`amdgpu-processors`). 2005 2006 - ``<Target Features>`` is a list of the enabled *Target Features* 2007 (see :ref:`amdgpu-target-features`), each prefixed by a plus, that 2008 apply to *Processor*. The list must be in the same order as listed 2009 in the table :ref:`amdgpu-target-feature-table`. Note that *Target 2010 Features* must be included in the list if they are enabled even if 2011 that is the default for *Processor*. 2012 2013For example: 2014 2015 ``"amdgcn-amd-amdhsa--gfx902+xnack"`` 2016 2017.. _amdgpu-amdhsa-code-object-metadata: 2018 2019Code Object Metadata 2020~~~~~~~~~~~~~~~~~~~~ 2021 2022The code object metadata specifies extensible metadata associated with the code 2023objects executed on HSA [HSA]_ compatible runtimes such as AMD's ROCm 2024[AMD-ROCm]_. The encoding and semantics of this metadata depends on the code 2025object version; see :ref:`amdgpu-amdhsa-code-object-metadata-v2` and 2026:ref:`amdgpu-amdhsa-code-object-metadata-v3`. 2027 2028Code object metadata is specified in a note record (see 2029:ref:`amdgpu-note-records`) and is required when the target triple OS is 2030``amdhsa`` (see :ref:`amdgpu-target-triples`). It must contain the minimum 2031information necessary to support the ROCM kernel queries. For example, the 2032segment sizes needed in a dispatch packet. In addition, a high-level language 2033runtime may require other information to be included. For example, the AMD 2034OpenCL runtime records kernel argument information. 2035 2036.. _amdgpu-amdhsa-code-object-metadata-v2: 2037 2038Code Object V2 Metadata (-mattr=-code-object-v3) 2039++++++++++++++++++++++++++++++++++++++++++++++++ 2040 2041.. warning:: Code Object V2 is not the default code object version emitted by 2042 this version of LLVM. For a description of the metadata generated with the 2043 default configuration (Code Object V3) see 2044 :ref:`amdgpu-amdhsa-code-object-metadata-v3`. 2045 2046Code object V2 metadata is specified by the ``NT_AMD_AMDGPU_METADATA`` note 2047record (see :ref:`amdgpu-note-records-v2`). 2048 2049The metadata is specified as a YAML formatted string (see [YAML]_ and 2050:doc:`YamlIO`). 2051 2052.. TODO:: 2053 2054 Is the string null terminated? It probably should not if YAML allows it to 2055 contain null characters, otherwise it should be. 2056 2057The metadata is represented as a single YAML document comprised of the mapping 2058defined in table :ref:`amdgpu-amdhsa-code-object-metadata-map-table-v2` and 2059referenced tables. 2060 2061For boolean values, the string values of ``false`` and ``true`` are used for 2062false and true respectively. 2063 2064Additional information can be added to the mappings. To avoid conflicts, any 2065non-AMD key names should be prefixed by "*vendor-name*.". 2066 2067 .. table:: AMDHSA Code Object V2 Metadata Map 2068 :name: amdgpu-amdhsa-code-object-metadata-map-table-v2 2069 2070 ========== ============== ========= ======================================= 2071 String Key Value Type Required? Description 2072 ========== ============== ========= ======================================= 2073 "Version" sequence of Required - The first integer is the major 2074 2 integers version. Currently 1. 2075 - The second integer is the minor 2076 version. Currently 0. 2077 "Printf" sequence of Each string is encoded information 2078 strings about a printf function call. The 2079 encoded information is organized as 2080 fields separated by colon (':'): 2081 2082 ``ID:N:S[0]:S[1]:...:S[N-1]:FormatString`` 2083 2084 where: 2085 2086 ``ID`` 2087 A 32-bit integer as a unique id for 2088 each printf function call 2089 2090 ``N`` 2091 A 32-bit integer equal to the number 2092 of arguments of printf function call 2093 minus 1 2094 2095 ``S[i]`` (where i = 0, 1, ... , N-1) 2096 32-bit integers for the size in bytes 2097 of the i-th FormatString argument of 2098 the printf function call 2099 2100 FormatString 2101 The format string passed to the 2102 printf function call. 2103 "Kernels" sequence of Required Sequence of the mappings for each 2104 mapping kernel in the code object. See 2105 :ref:`amdgpu-amdhsa-code-object-kernel-metadata-map-table-v2` 2106 for the definition of the mapping. 2107 ========== ============== ========= ======================================= 2108 2109.. 2110 2111 .. table:: AMDHSA Code Object V2 Kernel Metadata Map 2112 :name: amdgpu-amdhsa-code-object-kernel-metadata-map-table-v2 2113 2114 ================= ============== ========= ================================ 2115 String Key Value Type Required? Description 2116 ================= ============== ========= ================================ 2117 "Name" string Required Source name of the kernel. 2118 "SymbolName" string Required Name of the kernel 2119 descriptor ELF symbol. 2120 "Language" string Source language of the kernel. 2121 Values include: 2122 2123 - "OpenCL C" 2124 - "OpenCL C++" 2125 - "HCC" 2126 - "OpenMP" 2127 2128 "LanguageVersion" sequence of - The first integer is the major 2129 2 integers version. 2130 - The second integer is the 2131 minor version. 2132 "Attrs" mapping Mapping of kernel attributes. 2133 See 2134 :ref:`amdgpu-amdhsa-code-object-kernel-attribute-metadata-map-table-v2` 2135 for the mapping definition. 2136 "Args" sequence of Sequence of mappings of the 2137 mapping kernel arguments. See 2138 :ref:`amdgpu-amdhsa-code-object-kernel-argument-metadata-map-table-v2` 2139 for the definition of the mapping. 2140 "CodeProps" mapping Mapping of properties related to 2141 the kernel code. See 2142 :ref:`amdgpu-amdhsa-code-object-kernel-code-properties-metadata-map-table-v2` 2143 for the mapping definition. 2144 ================= ============== ========= ================================ 2145 2146.. 2147 2148 .. table:: AMDHSA Code Object V2 Kernel Attribute Metadata Map 2149 :name: amdgpu-amdhsa-code-object-kernel-attribute-metadata-map-table-v2 2150 2151 =================== ============== ========= ============================== 2152 String Key Value Type Required? Description 2153 =================== ============== ========= ============================== 2154 "ReqdWorkGroupSize" sequence of If not 0, 0, 0 then all values 2155 3 integers must be >=1 and the dispatch 2156 work-group size X, Y, Z must 2157 correspond to the specified 2158 values. Defaults to 0, 0, 0. 2159 2160 Corresponds to the OpenCL 2161 ``reqd_work_group_size`` 2162 attribute. 2163 "WorkGroupSizeHint" sequence of The dispatch work-group size 2164 3 integers X, Y, Z is likely to be the 2165 specified values. 2166 2167 Corresponds to the OpenCL 2168 ``work_group_size_hint`` 2169 attribute. 2170 "VecTypeHint" string The name of a scalar or vector 2171 type. 2172 2173 Corresponds to the OpenCL 2174 ``vec_type_hint`` attribute. 2175 2176 "RuntimeHandle" string The external symbol name 2177 associated with a kernel. 2178 OpenCL runtime allocates a 2179 global buffer for the symbol 2180 and saves the kernel's address 2181 to it, which is used for 2182 device side enqueueing. Only 2183 available for device side 2184 enqueued kernels. 2185 =================== ============== ========= ============================== 2186 2187.. 2188 2189 .. table:: AMDHSA Code Object V2 Kernel Argument Metadata Map 2190 :name: amdgpu-amdhsa-code-object-kernel-argument-metadata-map-table-v2 2191 2192 ================= ============== ========= ================================ 2193 String Key Value Type Required? Description 2194 ================= ============== ========= ================================ 2195 "Name" string Kernel argument name. 2196 "TypeName" string Kernel argument type name. 2197 "Size" integer Required Kernel argument size in bytes. 2198 "Align" integer Required Kernel argument alignment in 2199 bytes. Must be a power of two. 2200 "ValueKind" string Required Kernel argument kind that 2201 specifies how to set up the 2202 corresponding argument. 2203 Values include: 2204 2205 "ByValue" 2206 The argument is copied 2207 directly into the kernarg. 2208 2209 "GlobalBuffer" 2210 A global address space pointer 2211 to the buffer data is passed 2212 in the kernarg. 2213 2214 "DynamicSharedPointer" 2215 A group address space pointer 2216 to dynamically allocated LDS 2217 is passed in the kernarg. 2218 2219 "Sampler" 2220 A global address space 2221 pointer to a S# is passed in 2222 the kernarg. 2223 2224 "Image" 2225 A global address space 2226 pointer to a T# is passed in 2227 the kernarg. 2228 2229 "Pipe" 2230 A global address space pointer 2231 to an OpenCL pipe is passed in 2232 the kernarg. 2233 2234 "Queue" 2235 A global address space pointer 2236 to an OpenCL device enqueue 2237 queue is passed in the 2238 kernarg. 2239 2240 "HiddenGlobalOffsetX" 2241 The OpenCL grid dispatch 2242 global offset for the X 2243 dimension is passed in the 2244 kernarg. 2245 2246 "HiddenGlobalOffsetY" 2247 The OpenCL grid dispatch 2248 global offset for the Y 2249 dimension is passed in the 2250 kernarg. 2251 2252 "HiddenGlobalOffsetZ" 2253 The OpenCL grid dispatch 2254 global offset for the Z 2255 dimension is passed in the 2256 kernarg. 2257 2258 "HiddenNone" 2259 An argument that is not used 2260 by the kernel. Space needs to 2261 be left for it, but it does 2262 not need to be set up. 2263 2264 "HiddenPrintfBuffer" 2265 A global address space pointer 2266 to the runtime printf buffer 2267 is passed in kernarg. 2268 2269 "HiddenHostcallBuffer" 2270 A global address space pointer 2271 to the runtime hostcall buffer 2272 is passed in kernarg. 2273 2274 "HiddenDefaultQueue" 2275 A global address space pointer 2276 to the OpenCL device enqueue 2277 queue that should be used by 2278 the kernel by default is 2279 passed in the kernarg. 2280 2281 "HiddenCompletionAction" 2282 A global address space pointer 2283 to help link enqueued kernels into 2284 the ancestor tree for determining 2285 when the parent kernel has finished. 2286 2287 "HiddenMultiGridSyncArg" 2288 A global address space pointer for 2289 multi-grid synchronization is 2290 passed in the kernarg. 2291 2292 "ValueType" string Required Kernel argument value type. Only 2293 present if "ValueKind" is 2294 "ByValue". For vector data 2295 types, the value is for the 2296 element type. Values include: 2297 2298 - "Struct" 2299 - "I8" 2300 - "U8" 2301 - "I16" 2302 - "U16" 2303 - "F16" 2304 - "I32" 2305 - "U32" 2306 - "F32" 2307 - "I64" 2308 - "U64" 2309 - "F64" 2310 2311 .. TODO:: 2312 How can it be determined if a 2313 vector type, and what size 2314 vector? 2315 "PointeeAlign" integer Alignment in bytes of pointee 2316 type for pointer type kernel 2317 argument. Must be a power 2318 of 2. Only present if 2319 "ValueKind" is 2320 "DynamicSharedPointer". 2321 "AddrSpaceQual" string Kernel argument address space 2322 qualifier. Only present if 2323 "ValueKind" is "GlobalBuffer" or 2324 "DynamicSharedPointer". Values 2325 are: 2326 2327 - "Private" 2328 - "Global" 2329 - "Constant" 2330 - "Local" 2331 - "Generic" 2332 - "Region" 2333 2334 .. TODO:: 2335 Is GlobalBuffer only Global 2336 or Constant? Is 2337 DynamicSharedPointer always 2338 Local? Can HCC allow Generic? 2339 How can Private or Region 2340 ever happen? 2341 "AccQual" string Kernel argument access 2342 qualifier. Only present if 2343 "ValueKind" is "Image" or 2344 "Pipe". Values 2345 are: 2346 2347 - "ReadOnly" 2348 - "WriteOnly" 2349 - "ReadWrite" 2350 2351 .. TODO:: 2352 Does this apply to 2353 GlobalBuffer? 2354 "ActualAccQual" string The actual memory accesses 2355 performed by the kernel on the 2356 kernel argument. Only present if 2357 "ValueKind" is "GlobalBuffer", 2358 "Image", or "Pipe". This may be 2359 more restrictive than indicated 2360 by "AccQual" to reflect what the 2361 kernel actual does. If not 2362 present then the runtime must 2363 assume what is implied by 2364 "AccQual" and "IsConst". Values 2365 are: 2366 2367 - "ReadOnly" 2368 - "WriteOnly" 2369 - "ReadWrite" 2370 2371 "IsConst" boolean Indicates if the kernel argument 2372 is const qualified. Only present 2373 if "ValueKind" is 2374 "GlobalBuffer". 2375 2376 "IsRestrict" boolean Indicates if the kernel argument 2377 is restrict qualified. Only 2378 present if "ValueKind" is 2379 "GlobalBuffer". 2380 2381 "IsVolatile" boolean Indicates if the kernel argument 2382 is volatile qualified. Only 2383 present if "ValueKind" is 2384 "GlobalBuffer". 2385 2386 "IsPipe" boolean Indicates if the kernel argument 2387 is pipe qualified. Only present 2388 if "ValueKind" is "Pipe". 2389 2390 .. TODO:: 2391 Can GlobalBuffer be pipe 2392 qualified? 2393 ================= ============== ========= ================================ 2394 2395.. 2396 2397 .. table:: AMDHSA Code Object V2 Kernel Code Properties Metadata Map 2398 :name: amdgpu-amdhsa-code-object-kernel-code-properties-metadata-map-table-v2 2399 2400 ============================ ============== ========= ===================== 2401 String Key Value Type Required? Description 2402 ============================ ============== ========= ===================== 2403 "KernargSegmentSize" integer Required The size in bytes of 2404 the kernarg segment 2405 that holds the values 2406 of the arguments to 2407 the kernel. 2408 "GroupSegmentFixedSize" integer Required The amount of group 2409 segment memory 2410 required by a 2411 work-group in 2412 bytes. This does not 2413 include any 2414 dynamically allocated 2415 group segment memory 2416 that may be added 2417 when the kernel is 2418 dispatched. 2419 "PrivateSegmentFixedSize" integer Required The amount of fixed 2420 private address space 2421 memory required for a 2422 work-item in 2423 bytes. If the kernel 2424 uses a dynamic call 2425 stack then additional 2426 space must be added 2427 to this value for the 2428 call stack. 2429 "KernargSegmentAlign" integer Required The maximum byte 2430 alignment of 2431 arguments in the 2432 kernarg segment. Must 2433 be a power of 2. 2434 "WavefrontSize" integer Required Wavefront size. Must 2435 be a power of 2. 2436 "NumSGPRs" integer Required Number of scalar 2437 registers used by a 2438 wavefront for 2439 GFX6-GFX10. This 2440 includes the special 2441 SGPRs for VCC, Flat 2442 Scratch (GFX7-GFX10) 2443 and XNACK (for 2444 GFX8-GFX10). It does 2445 not include the 16 2446 SGPR added if a trap 2447 handler is 2448 enabled. It is not 2449 rounded up to the 2450 allocation 2451 granularity. 2452 "NumVGPRs" integer Required Number of vector 2453 registers used by 2454 each work-item for 2455 GFX6-GFX10 2456 "MaxFlatWorkGroupSize" integer Required Maximum flat 2457 work-group size 2458 supported by the 2459 kernel in work-items. 2460 Must be >=1 and 2461 consistent with 2462 ReqdWorkGroupSize if 2463 not 0, 0, 0. 2464 "NumSpilledSGPRs" integer Number of stores from 2465 a scalar register to 2466 a register allocator 2467 created spill 2468 location. 2469 "NumSpilledVGPRs" integer Number of stores from 2470 a vector register to 2471 a register allocator 2472 created spill 2473 location. 2474 ============================ ============== ========= ===================== 2475 2476.. _amdgpu-amdhsa-code-object-metadata-v3: 2477 2478Code Object V3 Metadata (-mattr=+code-object-v3) 2479++++++++++++++++++++++++++++++++++++++++++++++++ 2480 2481Code object V3 metadata is specified by the ``NT_AMDGPU_METADATA`` note record 2482(see :ref:`amdgpu-note-records-v3`). 2483 2484The metadata is represented as Message Pack formatted binary data (see 2485[MsgPack]_). The top level is a Message Pack map that includes the 2486keys defined in table 2487:ref:`amdgpu-amdhsa-code-object-metadata-map-table-v3` and referenced 2488tables. 2489 2490Additional information can be added to the maps. To avoid conflicts, 2491any key names should be prefixed by "*vendor-name*." where 2492``vendor-name`` can be the name of the vendor and specific vendor 2493tool that generates the information. The prefix is abbreviated to 2494simply "." when it appears within a map that has been added by the 2495same *vendor-name*. 2496 2497 .. table:: AMDHSA Code Object V3 Metadata Map 2498 :name: amdgpu-amdhsa-code-object-metadata-map-table-v3 2499 2500 ================= ============== ========= ======================================= 2501 String Key Value Type Required? Description 2502 ================= ============== ========= ======================================= 2503 "amdhsa.version" sequence of Required - The first integer is the major 2504 2 integers version. Currently 1. 2505 - The second integer is the minor 2506 version. Currently 0. 2507 "amdhsa.printf" sequence of Each string is encoded information 2508 strings about a printf function call. The 2509 encoded information is organized as 2510 fields separated by colon (':'): 2511 2512 ``ID:N:S[0]:S[1]:...:S[N-1]:FormatString`` 2513 2514 where: 2515 2516 ``ID`` 2517 A 32-bit integer as a unique id for 2518 each printf function call 2519 2520 ``N`` 2521 A 32-bit integer equal to the number 2522 of arguments of printf function call 2523 minus 1 2524 2525 ``S[i]`` (where i = 0, 1, ... , N-1) 2526 32-bit integers for the size in bytes 2527 of the i-th FormatString argument of 2528 the printf function call 2529 2530 FormatString 2531 The format string passed to the 2532 printf function call. 2533 "amdhsa.kernels" sequence of Required Sequence of the maps for each 2534 map kernel in the code object. See 2535 :ref:`amdgpu-amdhsa-code-object-kernel-metadata-map-table-v3` 2536 for the definition of the keys included 2537 in that map. 2538 ================= ============== ========= ======================================= 2539 2540.. 2541 2542 .. table:: AMDHSA Code Object V3 Kernel Metadata Map 2543 :name: amdgpu-amdhsa-code-object-kernel-metadata-map-table-v3 2544 2545 =================================== ============== ========= ================================ 2546 String Key Value Type Required? Description 2547 =================================== ============== ========= ================================ 2548 ".name" string Required Source name of the kernel. 2549 ".symbol" string Required Name of the kernel 2550 descriptor ELF symbol. 2551 ".language" string Source language of the kernel. 2552 Values include: 2553 2554 - "OpenCL C" 2555 - "OpenCL C++" 2556 - "HCC" 2557 - "HIP" 2558 - "OpenMP" 2559 - "Assembler" 2560 2561 ".language_version" sequence of - The first integer is the major 2562 2 integers version. 2563 - The second integer is the 2564 minor version. 2565 ".args" sequence of Sequence of maps of the 2566 map kernel arguments. See 2567 :ref:`amdgpu-amdhsa-code-object-kernel-argument-metadata-map-table-v3` 2568 for the definition of the keys 2569 included in that map. 2570 ".reqd_workgroup_size" sequence of If not 0, 0, 0 then all values 2571 3 integers must be >=1 and the dispatch 2572 work-group size X, Y, Z must 2573 correspond to the specified 2574 values. Defaults to 0, 0, 0. 2575 2576 Corresponds to the OpenCL 2577 ``reqd_work_group_size`` 2578 attribute. 2579 ".workgroup_size_hint" sequence of The dispatch work-group size 2580 3 integers X, Y, Z is likely to be the 2581 specified values. 2582 2583 Corresponds to the OpenCL 2584 ``work_group_size_hint`` 2585 attribute. 2586 ".vec_type_hint" string The name of a scalar or vector 2587 type. 2588 2589 Corresponds to the OpenCL 2590 ``vec_type_hint`` attribute. 2591 2592 ".device_enqueue_symbol" string The external symbol name 2593 associated with a kernel. 2594 OpenCL runtime allocates a 2595 global buffer for the symbol 2596 and saves the kernel's address 2597 to it, which is used for 2598 device side enqueueing. Only 2599 available for device side 2600 enqueued kernels. 2601 ".kernarg_segment_size" integer Required The size in bytes of 2602 the kernarg segment 2603 that holds the values 2604 of the arguments to 2605 the kernel. 2606 ".group_segment_fixed_size" integer Required The amount of group 2607 segment memory 2608 required by a 2609 work-group in 2610 bytes. This does not 2611 include any 2612 dynamically allocated 2613 group segment memory 2614 that may be added 2615 when the kernel is 2616 dispatched. 2617 ".private_segment_fixed_size" integer Required The amount of fixed 2618 private address space 2619 memory required for a 2620 work-item in 2621 bytes. If the kernel 2622 uses a dynamic call 2623 stack then additional 2624 space must be added 2625 to this value for the 2626 call stack. 2627 ".kernarg_segment_align" integer Required The maximum byte 2628 alignment of 2629 arguments in the 2630 kernarg segment. Must 2631 be a power of 2. 2632 ".wavefront_size" integer Required Wavefront size. Must 2633 be a power of 2. 2634 ".sgpr_count" integer Required Number of scalar 2635 registers required by a 2636 wavefront for 2637 GFX6-GFX9. A register 2638 is required if it is 2639 used explicitly, or 2640 if a higher numbered 2641 register is used 2642 explicitly. This 2643 includes the special 2644 SGPRs for VCC, Flat 2645 Scratch (GFX7-GFX9) 2646 and XNACK (for 2647 GFX8-GFX9). It does 2648 not include the 16 2649 SGPR added if a trap 2650 handler is 2651 enabled. It is not 2652 rounded up to the 2653 allocation 2654 granularity. 2655 ".vgpr_count" integer Required Number of vector 2656 registers required by 2657 each work-item for 2658 GFX6-GFX9. A register 2659 is required if it is 2660 used explicitly, or 2661 if a higher numbered 2662 register is used 2663 explicitly. 2664 ".max_flat_workgroup_size" integer Required Maximum flat 2665 work-group size 2666 supported by the 2667 kernel in work-items. 2668 Must be >=1 and 2669 consistent with 2670 ReqdWorkGroupSize if 2671 not 0, 0, 0. 2672 ".sgpr_spill_count" integer Number of stores from 2673 a scalar register to 2674 a register allocator 2675 created spill 2676 location. 2677 ".vgpr_spill_count" integer Number of stores from 2678 a vector register to 2679 a register allocator 2680 created spill 2681 location. 2682 =================================== ============== ========= ================================ 2683 2684.. 2685 2686 .. table:: AMDHSA Code Object V3 Kernel Argument Metadata Map 2687 :name: amdgpu-amdhsa-code-object-kernel-argument-metadata-map-table-v3 2688 2689 ====================== ============== ========= ================================ 2690 String Key Value Type Required? Description 2691 ====================== ============== ========= ================================ 2692 ".name" string Kernel argument name. 2693 ".type_name" string Kernel argument type name. 2694 ".size" integer Required Kernel argument size in bytes. 2695 ".offset" integer Required Kernel argument offset in 2696 bytes. The offset must be a 2697 multiple of the alignment 2698 required by the argument. 2699 ".value_kind" string Required Kernel argument kind that 2700 specifies how to set up the 2701 corresponding argument. 2702 Values include: 2703 2704 "by_value" 2705 The argument is copied 2706 directly into the kernarg. 2707 2708 "global_buffer" 2709 A global address space pointer 2710 to the buffer data is passed 2711 in the kernarg. 2712 2713 "dynamic_shared_pointer" 2714 A group address space pointer 2715 to dynamically allocated LDS 2716 is passed in the kernarg. 2717 2718 "sampler" 2719 A global address space 2720 pointer to a S# is passed in 2721 the kernarg. 2722 2723 "image" 2724 A global address space 2725 pointer to a T# is passed in 2726 the kernarg. 2727 2728 "pipe" 2729 A global address space pointer 2730 to an OpenCL pipe is passed in 2731 the kernarg. 2732 2733 "queue" 2734 A global address space pointer 2735 to an OpenCL device enqueue 2736 queue is passed in the 2737 kernarg. 2738 2739 "hidden_global_offset_x" 2740 The OpenCL grid dispatch 2741 global offset for the X 2742 dimension is passed in the 2743 kernarg. 2744 2745 "hidden_global_offset_y" 2746 The OpenCL grid dispatch 2747 global offset for the Y 2748 dimension is passed in the 2749 kernarg. 2750 2751 "hidden_global_offset_z" 2752 The OpenCL grid dispatch 2753 global offset for the Z 2754 dimension is passed in the 2755 kernarg. 2756 2757 "hidden_none" 2758 An argument that is not used 2759 by the kernel. Space needs to 2760 be left for it, but it does 2761 not need to be set up. 2762 2763 "hidden_printf_buffer" 2764 A global address space pointer 2765 to the runtime printf buffer 2766 is passed in kernarg. 2767 2768 "hidden_hostcall_buffer" 2769 A global address space pointer 2770 to the runtime hostcall buffer 2771 is passed in kernarg. 2772 2773 "hidden_default_queue" 2774 A global address space pointer 2775 to the OpenCL device enqueue 2776 queue that should be used by 2777 the kernel by default is 2778 passed in the kernarg. 2779 2780 "hidden_completion_action" 2781 A global address space pointer 2782 to help link enqueued kernels into 2783 the ancestor tree for determining 2784 when the parent kernel has finished. 2785 2786 "hidden_multigrid_sync_arg" 2787 A global address space pointer for 2788 multi-grid synchronization is 2789 passed in the kernarg. 2790 2791 ".value_type" string Required Kernel argument value type. Only 2792 present if ".value_kind" is 2793 "by_value". For vector data 2794 types, the value is for the 2795 element type. Values include: 2796 2797 - "struct" 2798 - "i8" 2799 - "u8" 2800 - "i16" 2801 - "u16" 2802 - "f16" 2803 - "i32" 2804 - "u32" 2805 - "f32" 2806 - "i64" 2807 - "u64" 2808 - "f64" 2809 2810 .. TODO:: 2811 How can it be determined if a 2812 vector type, and what size 2813 vector? 2814 ".pointee_align" integer Alignment in bytes of pointee 2815 type for pointer type kernel 2816 argument. Must be a power 2817 of 2. Only present if 2818 ".value_kind" is 2819 "dynamic_shared_pointer". 2820 ".address_space" string Kernel argument address space 2821 qualifier. Only present if 2822 ".value_kind" is "global_buffer" or 2823 "dynamic_shared_pointer". Values 2824 are: 2825 2826 - "private" 2827 - "global" 2828 - "constant" 2829 - "local" 2830 - "generic" 2831 - "region" 2832 2833 .. TODO:: 2834 Is "global_buffer" only "global" 2835 or "constant"? Is 2836 "dynamic_shared_pointer" always 2837 "local"? Can HCC allow "generic"? 2838 How can "private" or "region" 2839 ever happen? 2840 ".access" string Kernel argument access 2841 qualifier. Only present if 2842 ".value_kind" is "image" or 2843 "pipe". Values 2844 are: 2845 2846 - "read_only" 2847 - "write_only" 2848 - "read_write" 2849 2850 .. TODO:: 2851 Does this apply to 2852 "global_buffer"? 2853 ".actual_access" string The actual memory accesses 2854 performed by the kernel on the 2855 kernel argument. Only present if 2856 ".value_kind" is "global_buffer", 2857 "image", or "pipe". This may be 2858 more restrictive than indicated 2859 by ".access" to reflect what the 2860 kernel actual does. If not 2861 present then the runtime must 2862 assume what is implied by 2863 ".access" and ".is_const" . Values 2864 are: 2865 2866 - "read_only" 2867 - "write_only" 2868 - "read_write" 2869 2870 ".is_const" boolean Indicates if the kernel argument 2871 is const qualified. Only present 2872 if ".value_kind" is 2873 "global_buffer". 2874 2875 ".is_restrict" boolean Indicates if the kernel argument 2876 is restrict qualified. Only 2877 present if ".value_kind" is 2878 "global_buffer". 2879 2880 ".is_volatile" boolean Indicates if the kernel argument 2881 is volatile qualified. Only 2882 present if ".value_kind" is 2883 "global_buffer". 2884 2885 ".is_pipe" boolean Indicates if the kernel argument 2886 is pipe qualified. Only present 2887 if ".value_kind" is "pipe". 2888 2889 .. TODO:: 2890 Can "global_buffer" be pipe 2891 qualified? 2892 ====================== ============== ========= ================================ 2893 2894.. 2895 2896Kernel Dispatch 2897~~~~~~~~~~~~~~~ 2898 2899The HSA architected queuing language (AQL) defines a user space memory 2900interface that can be used to control the dispatch of kernels, in an agent 2901independent way. An agent can have zero or more AQL queues created for it using 2902the ROCm runtime, in which AQL packets (all of which are 64 bytes) can be 2903placed. See the *HSA Platform System Architecture Specification* [HSA]_ for the 2904AQL queue mechanics and packet layouts. 2905 2906The packet processor of a kernel agent is responsible for detecting and 2907dispatching HSA kernels from the AQL queues associated with it. For AMD GPUs the 2908packet processor is implemented by the hardware command processor (CP), 2909asynchronous dispatch controller (ADC) and shader processor input controller 2910(SPI). 2911 2912The ROCm runtime can be used to allocate an AQL queue object. It uses the kernel 2913mode driver to initialize and register the AQL queue with CP. 2914 2915To dispatch a kernel the following actions are performed. This can occur in the 2916CPU host program, or from an HSA kernel executing on a GPU. 2917 29181. A pointer to an AQL queue for the kernel agent on which the kernel is to be 2919 executed is obtained. 29202. A pointer to the kernel descriptor (see 2921 :ref:`amdgpu-amdhsa-kernel-descriptor`) of the kernel to execute is obtained. 2922 It must be for a kernel that is contained in a code object that that was 2923 loaded by the ROCm runtime on the kernel agent with which the AQL queue is 2924 associated. 29253. Space is allocated for the kernel arguments using the ROCm runtime allocator 2926 for a memory region with the kernarg property for the kernel agent that will 2927 execute the kernel. It must be at least 16-byte aligned. 29284. Kernel argument values are assigned to the kernel argument memory 2929 allocation. The layout is defined in the *HSA Programmer's Language 2930 Reference* [HSA]_. For AMDGPU the kernel execution directly accesses the 2931 kernel argument memory in the same way constant memory is accessed. (Note 2932 that the HSA specification allows an implementation to copy the kernel 2933 argument contents to another location that is accessed by the kernel.) 29345. An AQL kernel dispatch packet is created on the AQL queue. The ROCm runtime 2935 api uses 64-bit atomic operations to reserve space in the AQL queue for the 2936 packet. The packet must be set up, and the final write must use an atomic 2937 store release to set the packet kind to ensure the packet contents are 2938 visible to the kernel agent. AQL defines a doorbell signal mechanism to 2939 notify the kernel agent that the AQL queue has been updated. These rules, and 2940 the layout of the AQL queue and kernel dispatch packet is defined in the *HSA 2941 System Architecture Specification* [HSA]_. 29426. A kernel dispatch packet includes information about the actual dispatch, 2943 such as grid and work-group size, together with information from the code 2944 object about the kernel, such as segment sizes. The ROCm runtime queries on 2945 the kernel symbol can be used to obtain the code object values which are 2946 recorded in the :ref:`amdgpu-amdhsa-code-object-metadata`. 29477. CP executes micro-code and is responsible for detecting and setting up the 2948 GPU to execute the wavefronts of a kernel dispatch. 29498. CP ensures that when the a wavefront starts executing the kernel machine 2950 code, the scalar general purpose registers (SGPR) and vector general purpose 2951 registers (VGPR) are set up as required by the machine code. The required 2952 setup is defined in the :ref:`amdgpu-amdhsa-kernel-descriptor`. The initial 2953 register state is defined in 2954 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`. 29559. The prolog of the kernel machine code (see 2956 :ref:`amdgpu-amdhsa-kernel-prolog`) sets up the machine state as necessary 2957 before continuing executing the machine code that corresponds to the kernel. 295810. When the kernel dispatch has completed execution, CP signals the completion 2959 signal specified in the kernel dispatch packet if not 0. 2960 2961Image and Samplers 2962~~~~~~~~~~~~~~~~~~ 2963 2964Image and sample handles created by the ROCm runtime are 64-bit addresses of a 2965hardware 32-byte V# and 48 byte S# object respectively. In order to support the 2966HSA ``query_sampler`` operations two extra dwords are used to store the HSA BRIG 2967enumeration values for the queries that are not trivially deducible from the S# 2968representation. 2969 2970HSA Signals 2971~~~~~~~~~~~ 2972 2973HSA signal handles created by the ROCm runtime are 64-bit addresses of a 2974structure allocated in memory accessible from both the CPU and GPU. The 2975structure is defined by the ROCm runtime and subject to change between releases 2976(see [AMD-ROCm-github]_). 2977 2978.. _amdgpu-amdhsa-hsa-aql-queue: 2979 2980HSA AQL Queue 2981~~~~~~~~~~~~~ 2982 2983The HSA AQL queue structure is defined by the ROCm runtime and subject to change 2984between releases (see [AMD-ROCm-github]_). For some processors it contains 2985fields needed to implement certain language features such as the flat address 2986aperture bases. It also contains fields used by CP such as managing the 2987allocation of scratch memory. 2988 2989.. _amdgpu-amdhsa-kernel-descriptor: 2990 2991Kernel Descriptor 2992~~~~~~~~~~~~~~~~~ 2993 2994A kernel descriptor consists of the information needed by CP to initiate the 2995execution of a kernel, including the entry point address of the machine code 2996that implements the kernel. 2997 2998Kernel Descriptor for GFX6-GFX10 2999++++++++++++++++++++++++++++++++ 3000 3001CP microcode requires the Kernel descriptor to be allocated on 64-byte 3002alignment. 3003 3004 .. table:: Kernel Descriptor for GFX6-GFX10 3005 :name: amdgpu-amdhsa-kernel-descriptor-gfx6-gfx10-table 3006 3007 ======= ======= =============================== ============================ 3008 Bits Size Field Name Description 3009 ======= ======= =============================== ============================ 3010 31:0 4 bytes GROUP_SEGMENT_FIXED_SIZE The amount of fixed local 3011 address space memory 3012 required for a work-group 3013 in bytes. This does not 3014 include any dynamically 3015 allocated local address 3016 space memory that may be 3017 added when the kernel is 3018 dispatched. 3019 63:32 4 bytes PRIVATE_SEGMENT_FIXED_SIZE The amount of fixed 3020 private address space 3021 memory required for a 3022 work-item in bytes. If 3023 is_dynamic_callstack is 1 3024 then additional space must 3025 be added to this value for 3026 the call stack. 3027 127:64 8 bytes Reserved, must be 0. 3028 191:128 8 bytes KERNEL_CODE_ENTRY_BYTE_OFFSET Byte offset (possibly 3029 negative) from base 3030 address of kernel 3031 descriptor to kernel's 3032 entry point instruction 3033 which must be 256 byte 3034 aligned. 3035 351:272 20 Reserved, must be 0. 3036 bytes 3037 383:352 4 bytes COMPUTE_PGM_RSRC3 GFX6-9 3038 Reserved, must be 0. 3039 GFX10 3040 Compute Shader (CS) 3041 program settings used by 3042 CP to set up 3043 ``COMPUTE_PGM_RSRC3`` 3044 configuration 3045 register. See 3046 :ref:`amdgpu-amdhsa-compute_pgm_rsrc3-gfx10-table`. 3047 415:384 4 bytes COMPUTE_PGM_RSRC1 Compute Shader (CS) 3048 program settings used by 3049 CP to set up 3050 ``COMPUTE_PGM_RSRC1`` 3051 configuration 3052 register. See 3053 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 3054 447:416 4 bytes COMPUTE_PGM_RSRC2 Compute Shader (CS) 3055 program settings used by 3056 CP to set up 3057 ``COMPUTE_PGM_RSRC2`` 3058 configuration 3059 register. See 3060 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 3061 448 1 bit ENABLE_SGPR_PRIVATE_SEGMENT Enable the setup of the 3062 _BUFFER SGPR user data registers 3063 (see 3064 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 3065 3066 The total number of SGPR 3067 user data registers 3068 requested must not exceed 3069 16 and match value in 3070 ``compute_pgm_rsrc2.user_sgpr.user_sgpr_count``. 3071 Any requests beyond 16 3072 will be ignored. 3073 449 1 bit ENABLE_SGPR_DISPATCH_PTR *see above* 3074 450 1 bit ENABLE_SGPR_QUEUE_PTR *see above* 3075 451 1 bit ENABLE_SGPR_KERNARG_SEGMENT_PTR *see above* 3076 452 1 bit ENABLE_SGPR_DISPATCH_ID *see above* 3077 453 1 bit ENABLE_SGPR_FLAT_SCRATCH_INIT *see above* 3078 454 1 bit ENABLE_SGPR_PRIVATE_SEGMENT *see above* 3079 _SIZE 3080 457:455 3 bits Reserved, must be 0. 3081 458 1 bit ENABLE_WAVEFRONT_SIZE32 GFX6-9 3082 Reserved, must be 0. 3083 GFX10 3084 - If 0 execute in 3085 wavefront size 64 mode. 3086 - If 1 execute in 3087 native wavefront size 3088 32 mode. 3089 463:459 5 bits Reserved, must be 0. 3090 511:464 6 bytes Reserved, must be 0. 3091 512 **Total size 64 bytes.** 3092 ======= ==================================================================== 3093 3094.. 3095 3096 .. table:: compute_pgm_rsrc1 for GFX6-GFX10 3097 :name: amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table 3098 3099 ======= ======= =============================== =========================================================================== 3100 Bits Size Field Name Description 3101 ======= ======= =============================== =========================================================================== 3102 5:0 6 bits GRANULATED_WORKITEM_VGPR_COUNT Number of vector register 3103 blocks used by each work-item; 3104 granularity is device 3105 specific: 3106 3107 GFX6-GFX9 3108 - vgprs_used 0..256 3109 - max(0, ceil(vgprs_used / 4) - 1) 3110 GFX10 (wavefront size 64) 3111 - max_vgpr 1..256 3112 - max(0, ceil(vgprs_used / 4) - 1) 3113 GFX10 (wavefront size 32) 3114 - max_vgpr 1..256 3115 - max(0, ceil(vgprs_used / 8) - 1) 3116 3117 Where vgprs_used is defined 3118 as the highest VGPR number 3119 explicitly referenced plus 3120 one. 3121 3122 Used by CP to set up 3123 ``COMPUTE_PGM_RSRC1.VGPRS``. 3124 3125 The 3126 :ref:`amdgpu-assembler` 3127 calculates this 3128 automatically for the 3129 selected processor from 3130 values provided to the 3131 `.amdhsa_kernel` directive 3132 by the 3133 `.amdhsa_next_free_vgpr` 3134 nested directive (see 3135 :ref:`amdhsa-kernel-directives-table`). 3136 9:6 4 bits GRANULATED_WAVEFRONT_SGPR_COUNT Number of scalar register 3137 blocks used by a wavefront; 3138 granularity is device 3139 specific: 3140 3141 GFX6-GFX8 3142 - sgprs_used 0..112 3143 - max(0, ceil(sgprs_used / 8) - 1) 3144 GFX9 3145 - sgprs_used 0..112 3146 - 2 * max(0, ceil(sgprs_used / 16) - 1) 3147 GFX10 3148 Reserved, must be 0. 3149 (128 SGPRs always 3150 allocated.) 3151 3152 Where sgprs_used is 3153 defined as the highest 3154 SGPR number explicitly 3155 referenced plus one, plus 3156 a target specific number 3157 of additional special 3158 SGPRs for VCC, 3159 FLAT_SCRATCH (GFX7+) and 3160 XNACK_MASK (GFX8+), and 3161 any additional 3162 target specific 3163 limitations. It does not 3164 include the 16 SGPRs added 3165 if a trap handler is 3166 enabled. 3167 3168 The target specific 3169 limitations and special 3170 SGPR layout are defined in 3171 the hardware 3172 documentation, which can 3173 be found in the 3174 :ref:`amdgpu-processors` 3175 table. 3176 3177 Used by CP to set up 3178 ``COMPUTE_PGM_RSRC1.SGPRS``. 3179 3180 The 3181 :ref:`amdgpu-assembler` 3182 calculates this 3183 automatically for the 3184 selected processor from 3185 values provided to the 3186 `.amdhsa_kernel` directive 3187 by the 3188 `.amdhsa_next_free_sgpr` 3189 and `.amdhsa_reserve_*` 3190 nested directives (see 3191 :ref:`amdhsa-kernel-directives-table`). 3192 11:10 2 bits PRIORITY Must be 0. 3193 3194 Start executing wavefront 3195 at the specified priority. 3196 3197 CP is responsible for 3198 filling in 3199 ``COMPUTE_PGM_RSRC1.PRIORITY``. 3200 13:12 2 bits FLOAT_ROUND_MODE_32 Wavefront starts execution 3201 with specified rounding 3202 mode for single (32 3203 bit) floating point 3204 precision floating point 3205 operations. 3206 3207 Floating point rounding 3208 mode values are defined in 3209 :ref:`amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table`. 3210 3211 Used by CP to set up 3212 ``COMPUTE_PGM_RSRC1.FLOAT_MODE``. 3213 15:14 2 bits FLOAT_ROUND_MODE_16_64 Wavefront starts execution 3214 with specified rounding 3215 denorm mode for half/double (16 3216 and 64-bit) floating point 3217 precision floating point 3218 operations. 3219 3220 Floating point rounding 3221 mode values are defined in 3222 :ref:`amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table`. 3223 3224 Used by CP to set up 3225 ``COMPUTE_PGM_RSRC1.FLOAT_MODE``. 3226 17:16 2 bits FLOAT_DENORM_MODE_32 Wavefront starts execution 3227 with specified denorm mode 3228 for single (32 3229 bit) floating point 3230 precision floating point 3231 operations. 3232 3233 Floating point denorm mode 3234 values are defined in 3235 :ref:`amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table`. 3236 3237 Used by CP to set up 3238 ``COMPUTE_PGM_RSRC1.FLOAT_MODE``. 3239 19:18 2 bits FLOAT_DENORM_MODE_16_64 Wavefront starts execution 3240 with specified denorm mode 3241 for half/double (16 3242 and 64-bit) floating point 3243 precision floating point 3244 operations. 3245 3246 Floating point denorm mode 3247 values are defined in 3248 :ref:`amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table`. 3249 3250 Used by CP to set up 3251 ``COMPUTE_PGM_RSRC1.FLOAT_MODE``. 3252 20 1 bit PRIV Must be 0. 3253 3254 Start executing wavefront 3255 in privilege trap handler 3256 mode. 3257 3258 CP is responsible for 3259 filling in 3260 ``COMPUTE_PGM_RSRC1.PRIV``. 3261 21 1 bit ENABLE_DX10_CLAMP Wavefront starts execution 3262 with DX10 clamp mode 3263 enabled. Used by the vector 3264 ALU to force DX10 style 3265 treatment of NaN's (when 3266 set, clamp NaN to zero, 3267 otherwise pass NaN 3268 through). 3269 3270 Used by CP to set up 3271 ``COMPUTE_PGM_RSRC1.DX10_CLAMP``. 3272 22 1 bit DEBUG_MODE Must be 0. 3273 3274 Start executing wavefront 3275 in single step mode. 3276 3277 CP is responsible for 3278 filling in 3279 ``COMPUTE_PGM_RSRC1.DEBUG_MODE``. 3280 23 1 bit ENABLE_IEEE_MODE Wavefront starts execution 3281 with IEEE mode 3282 enabled. Floating point 3283 opcodes that support 3284 exception flag gathering 3285 will quiet and propagate 3286 signaling-NaN inputs per 3287 IEEE 754-2008. Min_dx10 and 3288 max_dx10 become IEEE 3289 754-2008 compliant due to 3290 signaling-NaN propagation 3291 and quieting. 3292 3293 Used by CP to set up 3294 ``COMPUTE_PGM_RSRC1.IEEE_MODE``. 3295 24 1 bit BULKY Must be 0. 3296 3297 Only one work-group allowed 3298 to execute on a compute 3299 unit. 3300 3301 CP is responsible for 3302 filling in 3303 ``COMPUTE_PGM_RSRC1.BULKY``. 3304 25 1 bit CDBG_USER Must be 0. 3305 3306 Flag that can be used to 3307 control debugging code. 3308 3309 CP is responsible for 3310 filling in 3311 ``COMPUTE_PGM_RSRC1.CDBG_USER``. 3312 26 1 bit FP16_OVFL GFX6-GFX8 3313 Reserved, must be 0. 3314 GFX9-GFX10 3315 Wavefront starts execution 3316 with specified fp16 overflow 3317 mode. 3318 3319 - If 0, fp16 overflow generates 3320 +/-INF values. 3321 - If 1, fp16 overflow that is the 3322 result of an +/-INF input value 3323 or divide by 0 produces a +/-INF, 3324 otherwise clamps computed 3325 overflow to +/-MAX_FP16 as 3326 appropriate. 3327 3328 Used by CP to set up 3329 ``COMPUTE_PGM_RSRC1.FP16_OVFL``. 3330 28:27 2 bits Reserved, must be 0. 3331 29 1 bit WGP_MODE GFX6-GFX9 3332 Reserved, must be 0. 3333 GFX10 3334 - If 0 execute work-groups in 3335 CU wavefront execution mode. 3336 - If 1 execute work-groups on 3337 in WGP wavefront execution mode. 3338 3339 See :ref:`amdgpu-amdhsa-memory-model`. 3340 3341 Used by CP to set up 3342 ``COMPUTE_PGM_RSRC1.WGP_MODE``. 3343 30 1 bit MEM_ORDERED GFX6-9 3344 Reserved, must be 0. 3345 GFX10 3346 Controls the behavior of the 3347 waitcnt's vmcnt and vscnt 3348 counters. 3349 3350 - If 0 vmcnt reports completion 3351 of load and atomic with return 3352 out of order with sample 3353 instructions, and the vscnt 3354 reports the completion of 3355 store and atomic without 3356 return in order. 3357 - If 1 vmcnt reports completion 3358 of load, atomic with return 3359 and sample instructions in 3360 order, and the vscnt reports 3361 the completion of store and 3362 atomic without return in order. 3363 3364 Used by CP to set up 3365 ``COMPUTE_PGM_RSRC1.MEM_ORDERED``. 3366 31 1 bit FWD_PROGRESS GFX6-9 3367 Reserved, must be 0. 3368 GFX10 3369 - If 0 execute SIMD wavefronts 3370 using oldest first policy. 3371 - If 1 execute SIMD wavefronts to 3372 ensure wavefronts will make some 3373 forward progress. 3374 3375 Used by CP to set up 3376 ``COMPUTE_PGM_RSRC1.FWD_PROGRESS``. 3377 32 **Total size 4 bytes** 3378 ======= =================================================================================================================== 3379 3380.. 3381 3382 .. table:: compute_pgm_rsrc2 for GFX6-GFX10 3383 :name: amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table 3384 3385 ======= ======= =============================== =========================================================================== 3386 Bits Size Field Name Description 3387 ======= ======= =============================== =========================================================================== 3388 0 1 bit ENABLE_SGPR_PRIVATE_SEGMENT Enable the setup of the 3389 _WAVEFRONT_OFFSET SGPR wavefront scratch offset 3390 system register (see 3391 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 3392 3393 Used by CP to set up 3394 ``COMPUTE_PGM_RSRC2.SCRATCH_EN``. 3395 5:1 5 bits USER_SGPR_COUNT The total number of SGPR 3396 user data registers 3397 requested. This number must 3398 match the number of user 3399 data registers enabled. 3400 3401 Used by CP to set up 3402 ``COMPUTE_PGM_RSRC2.USER_SGPR``. 3403 6 1 bit ENABLE_TRAP_HANDLER Must be 0. 3404 3405 This bit represents 3406 ``COMPUTE_PGM_RSRC2.TRAP_PRESENT``, 3407 which is set by the CP if 3408 the runtime has installed a 3409 trap handler. 3410 7 1 bit ENABLE_SGPR_WORKGROUP_ID_X Enable the setup of the 3411 system SGPR register for 3412 the work-group id in the X 3413 dimension (see 3414 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 3415 3416 Used by CP to set up 3417 ``COMPUTE_PGM_RSRC2.TGID_X_EN``. 3418 8 1 bit ENABLE_SGPR_WORKGROUP_ID_Y Enable the setup of the 3419 system SGPR register for 3420 the work-group id in the Y 3421 dimension (see 3422 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 3423 3424 Used by CP to set up 3425 ``COMPUTE_PGM_RSRC2.TGID_Y_EN``. 3426 9 1 bit ENABLE_SGPR_WORKGROUP_ID_Z Enable the setup of the 3427 system SGPR register for 3428 the work-group id in the Z 3429 dimension (see 3430 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 3431 3432 Used by CP to set up 3433 ``COMPUTE_PGM_RSRC2.TGID_Z_EN``. 3434 10 1 bit ENABLE_SGPR_WORKGROUP_INFO Enable the setup of the 3435 system SGPR register for 3436 work-group information (see 3437 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 3438 3439 Used by CP to set up 3440 ``COMPUTE_PGM_RSRC2.TGID_SIZE_EN``. 3441 12:11 2 bits ENABLE_VGPR_WORKITEM_ID Enable the setup of the 3442 VGPR system registers used 3443 for the work-item ID. 3444 :ref:`amdgpu-amdhsa-system-vgpr-work-item-id-enumeration-values-table` 3445 defines the values. 3446 3447 Used by CP to set up 3448 ``COMPUTE_PGM_RSRC2.TIDIG_CMP_CNT``. 3449 13 1 bit ENABLE_EXCEPTION_ADDRESS_WATCH Must be 0. 3450 3451 Wavefront starts execution 3452 with address watch 3453 exceptions enabled which 3454 are generated when L1 has 3455 witnessed a thread access 3456 an *address of 3457 interest*. 3458 3459 CP is responsible for 3460 filling in the address 3461 watch bit in 3462 ``COMPUTE_PGM_RSRC2.EXCP_EN_MSB`` 3463 according to what the 3464 runtime requests. 3465 14 1 bit ENABLE_EXCEPTION_MEMORY Must be 0. 3466 3467 Wavefront starts execution 3468 with memory violation 3469 exceptions exceptions 3470 enabled which are generated 3471 when a memory violation has 3472 occurred for this wavefront from 3473 L1 or LDS 3474 (write-to-read-only-memory, 3475 mis-aligned atomic, LDS 3476 address out of range, 3477 illegal address, etc.). 3478 3479 CP sets the memory 3480 violation bit in 3481 ``COMPUTE_PGM_RSRC2.EXCP_EN_MSB`` 3482 according to what the 3483 runtime requests. 3484 23:15 9 bits GRANULATED_LDS_SIZE Must be 0. 3485 3486 CP uses the rounded value 3487 from the dispatch packet, 3488 not this value, as the 3489 dispatch may contain 3490 dynamically allocated group 3491 segment memory. CP writes 3492 directly to 3493 ``COMPUTE_PGM_RSRC2.LDS_SIZE``. 3494 3495 Amount of group segment 3496 (LDS) to allocate for each 3497 work-group. Granularity is 3498 device specific: 3499 3500 GFX6: 3501 roundup(lds-size / (64 * 4)) 3502 GFX7-GFX10: 3503 roundup(lds-size / (128 * 4)) 3504 3505 24 1 bit ENABLE_EXCEPTION_IEEE_754_FP Wavefront starts execution 3506 _INVALID_OPERATION with specified exceptions 3507 enabled. 3508 3509 Used by CP to set up 3510 ``COMPUTE_PGM_RSRC2.EXCP_EN`` 3511 (set from bits 0..6). 3512 3513 IEEE 754 FP Invalid 3514 Operation 3515 25 1 bit ENABLE_EXCEPTION_FP_DENORMAL FP Denormal one or more 3516 _SOURCE input operands is a 3517 denormal number 3518 26 1 bit ENABLE_EXCEPTION_IEEE_754_FP IEEE 754 FP Division by 3519 _DIVISION_BY_ZERO Zero 3520 27 1 bit ENABLE_EXCEPTION_IEEE_754_FP IEEE 754 FP FP Overflow 3521 _OVERFLOW 3522 28 1 bit ENABLE_EXCEPTION_IEEE_754_FP IEEE 754 FP Underflow 3523 _UNDERFLOW 3524 29 1 bit ENABLE_EXCEPTION_IEEE_754_FP IEEE 754 FP Inexact 3525 _INEXACT 3526 30 1 bit ENABLE_EXCEPTION_INT_DIVIDE_BY Integer Division by Zero 3527 _ZERO (rcp_iflag_f32 instruction 3528 only) 3529 31 1 bit Reserved, must be 0. 3530 32 **Total size 4 bytes.** 3531 ======= =================================================================================================================== 3532 3533.. 3534 3535 .. table:: compute_pgm_rsrc3 for GFX10 3536 :name: amdgpu-amdhsa-compute_pgm_rsrc3-gfx10-table 3537 3538 ======= ======= =============================== =========================================================================== 3539 Bits Size Field Name Description 3540 ======= ======= =============================== =========================================================================== 3541 3:0 4 bits SHARED_VGPR_COUNT Number of shared VGPRs for wavefront size 64. Granularity 8. Value 0-120. 3542 compute_pgm_rsrc1.vgprs + shared_vgpr_cnt cannot exceed 64. 3543 31:4 28 Reserved, must be 0. 3544 bits 3545 32 **Total size 4 bytes.** 3546 ======= =================================================================================================================== 3547 3548.. 3549 3550 .. table:: Floating Point Rounding Mode Enumeration Values 3551 :name: amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table 3552 3553 ====================================== ===== ============================== 3554 Enumeration Name Value Description 3555 ====================================== ===== ============================== 3556 FLOAT_ROUND_MODE_NEAR_EVEN 0 Round Ties To Even 3557 FLOAT_ROUND_MODE_PLUS_INFINITY 1 Round Toward +infinity 3558 FLOAT_ROUND_MODE_MINUS_INFINITY 2 Round Toward -infinity 3559 FLOAT_ROUND_MODE_ZERO 3 Round Toward 0 3560 ====================================== ===== ============================== 3561 3562.. 3563 3564 .. table:: Floating Point Denorm Mode Enumeration Values 3565 :name: amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table 3566 3567 ====================================== ===== ============================== 3568 Enumeration Name Value Description 3569 ====================================== ===== ============================== 3570 FLOAT_DENORM_MODE_FLUSH_SRC_DST 0 Flush Source and Destination 3571 Denorms 3572 FLOAT_DENORM_MODE_FLUSH_DST 1 Flush Output Denorms 3573 FLOAT_DENORM_MODE_FLUSH_SRC 2 Flush Source Denorms 3574 FLOAT_DENORM_MODE_FLUSH_NONE 3 No Flush 3575 ====================================== ===== ============================== 3576 3577.. 3578 3579 .. table:: System VGPR Work-Item ID Enumeration Values 3580 :name: amdgpu-amdhsa-system-vgpr-work-item-id-enumeration-values-table 3581 3582 ======================================== ===== ============================ 3583 Enumeration Name Value Description 3584 ======================================== ===== ============================ 3585 SYSTEM_VGPR_WORKITEM_ID_X 0 Set work-item X dimension 3586 ID. 3587 SYSTEM_VGPR_WORKITEM_ID_X_Y 1 Set work-item X and Y 3588 dimensions ID. 3589 SYSTEM_VGPR_WORKITEM_ID_X_Y_Z 2 Set work-item X, Y and Z 3590 dimensions ID. 3591 SYSTEM_VGPR_WORKITEM_ID_UNDEFINED 3 Undefined. 3592 ======================================== ===== ============================ 3593 3594.. _amdgpu-amdhsa-initial-kernel-execution-state: 3595 3596Initial Kernel Execution State 3597~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 3598 3599This section defines the register state that will be set up by the packet 3600processor prior to the start of execution of every wavefront. This is limited by 3601the constraints of the hardware controllers of CP/ADC/SPI. 3602 3603The order of the SGPR registers is defined, but the compiler can specify which 3604ones are actually setup in the kernel descriptor using the ``enable_sgpr_*`` bit 3605fields (see :ref:`amdgpu-amdhsa-kernel-descriptor`). The register numbers used 3606for enabled registers are dense starting at SGPR0: the first enabled register is 3607SGPR0, the next enabled register is SGPR1 etc.; disabled registers do not have 3608an SGPR number. 3609 3610The initial SGPRs comprise up to 16 User SRGPs that are set by CP and apply to 3611all wavefronts of the grid. It is possible to specify more than 16 User SGPRs 3612using the ``enable_sgpr_*`` bit fields, in which case only the first 16 are 3613actually initialized. These are then immediately followed by the System SGPRs 3614that are set up by ADC/SPI and can have different values for each wavefront of 3615the grid dispatch. 3616 3617SGPR register initial state is defined in 3618:ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 3619 3620 .. table:: SGPR Register Set Up Order 3621 :name: amdgpu-amdhsa-sgpr-register-set-up-order-table 3622 3623 ========== ========================== ====== ============================== 3624 SGPR Order Name Number Description 3625 (kernel descriptor enable of 3626 field) SGPRs 3627 ========== ========================== ====== ============================== 3628 First Private Segment Buffer 4 V# that can be used, together 3629 (enable_sgpr_private with Scratch Wavefront Offset 3630 _segment_buffer) as an offset, to access the 3631 private address space using a 3632 segment address. 3633 3634 CP uses the value provided by 3635 the runtime. 3636 then Dispatch Ptr 2 64-bit address of AQL dispatch 3637 (enable_sgpr_dispatch_ptr) packet for kernel dispatch 3638 actually executing. 3639 then Queue Ptr 2 64-bit address of amd_queue_t 3640 (enable_sgpr_queue_ptr) object for AQL queue on which 3641 the dispatch packet was 3642 queued. 3643 then Kernarg Segment Ptr 2 64-bit address of Kernarg 3644 (enable_sgpr_kernarg segment. This is directly 3645 _segment_ptr) copied from the 3646 kernarg_address in the kernel 3647 dispatch packet. 3648 3649 Having CP load it once avoids 3650 loading it at the beginning of 3651 every wavefront. 3652 then Dispatch Id 2 64-bit Dispatch ID of the 3653 (enable_sgpr_dispatch_id) dispatch packet being 3654 executed. 3655 then Flat Scratch Init 2 This is 2 SGPRs: 3656 (enable_sgpr_flat_scratch 3657 _init) GFX6 3658 Not supported. 3659 GFX7-GFX8 3660 The first SGPR is a 32-bit 3661 byte offset from 3662 ``SH_HIDDEN_PRIVATE_BASE_VIMID`` 3663 to per SPI base of memory 3664 for scratch for the queue 3665 executing the kernel 3666 dispatch. CP obtains this 3667 from the runtime. (The 3668 Scratch Segment Buffer base 3669 address is 3670 ``SH_HIDDEN_PRIVATE_BASE_VIMID`` 3671 plus this offset.) The value 3672 of Scratch Wavefront Offset must 3673 be added to this offset by 3674 the kernel machine code, 3675 right shifted by 8, and 3676 moved to the FLAT_SCRATCH_HI 3677 SGPR register. 3678 FLAT_SCRATCH_HI corresponds 3679 to SGPRn-4 on GFX7, and 3680 SGPRn-6 on GFX8 (where SGPRn 3681 is the highest numbered SGPR 3682 allocated to the wavefront). 3683 FLAT_SCRATCH_HI is 3684 multiplied by 256 (as it is 3685 in units of 256 bytes) and 3686 added to 3687 ``SH_HIDDEN_PRIVATE_BASE_VIMID`` 3688 to calculate the per wavefront 3689 FLAT SCRATCH BASE in flat 3690 memory instructions that 3691 access the scratch 3692 aperture. 3693 3694 The second SGPR is 32-bit 3695 byte size of a single 3696 work-item's scratch memory 3697 usage. CP obtains this from 3698 the runtime, and it is 3699 always a multiple of DWORD. 3700 CP checks that the value in 3701 the kernel dispatch packet 3702 Private Segment Byte Size is 3703 not larger and requests the 3704 runtime to increase the 3705 queue's scratch size if 3706 necessary. The kernel code 3707 must move it to 3708 FLAT_SCRATCH_LO which is 3709 SGPRn-3 on GFX7 and SGPRn-5 3710 on GFX8. FLAT_SCRATCH_LO is 3711 used as the FLAT SCRATCH 3712 SIZE in flat memory 3713 instructions. Having CP load 3714 it once avoids loading it at 3715 the beginning of every 3716 wavefront. 3717 GFX9-GFX10 3718 This is the 3719 64-bit base address of the 3720 per SPI scratch backing 3721 memory managed by SPI for 3722 the queue executing the 3723 kernel dispatch. CP obtains 3724 this from the runtime (and 3725 divides it if there are 3726 multiple Shader Arrays each 3727 with its own SPI). The value 3728 of Scratch Wavefront Offset must 3729 be added by the kernel 3730 machine code and the result 3731 moved to the FLAT_SCRATCH 3732 SGPR which is SGPRn-6 and 3733 SGPRn-5. It is used as the 3734 FLAT SCRATCH BASE in flat 3735 memory instructions. 3736 then Private Segment Size 1 The 32-bit byte size of a 3737 (enable_sgpr_private single 3738 work-item's 3739 scratch_segment_size) memory 3740 allocation. This is the 3741 value from the kernel 3742 dispatch packet Private 3743 Segment Byte Size rounded up 3744 by CP to a multiple of 3745 DWORD. 3746 3747 Having CP load it once avoids 3748 loading it at the beginning of 3749 every wavefront. 3750 3751 This is not used for 3752 GFX7-GFX8 since it is the same 3753 value as the second SGPR of 3754 Flat Scratch Init. However, it 3755 may be needed for GFX9-GFX10 which 3756 changes the meaning of the 3757 Flat Scratch Init value. 3758 then Grid Work-Group Count X 1 32-bit count of the number of 3759 (enable_sgpr_grid work-groups in the X dimension 3760 _workgroup_count_X) for the grid being 3761 executed. Computed from the 3762 fields in the kernel dispatch 3763 packet as ((grid_size.x + 3764 workgroup_size.x - 1) / 3765 workgroup_size.x). 3766 then Grid Work-Group Count Y 1 32-bit count of the number of 3767 (enable_sgpr_grid work-groups in the Y dimension 3768 _workgroup_count_Y && for the grid being 3769 less than 16 previous executed. Computed from the 3770 SGPRs) fields in the kernel dispatch 3771 packet as ((grid_size.y + 3772 workgroup_size.y - 1) / 3773 workgroupSize.y). 3774 3775 Only initialized if <16 3776 previous SGPRs initialized. 3777 then Grid Work-Group Count Z 1 32-bit count of the number of 3778 (enable_sgpr_grid work-groups in the Z dimension 3779 _workgroup_count_Z && for the grid being 3780 less than 16 previous executed. Computed from the 3781 SGPRs) fields in the kernel dispatch 3782 packet as ((grid_size.z + 3783 workgroup_size.z - 1) / 3784 workgroupSize.z). 3785 3786 Only initialized if <16 3787 previous SGPRs initialized. 3788 then Work-Group Id X 1 32-bit work-group id in X 3789 (enable_sgpr_workgroup_id dimension of grid for 3790 _X) wavefront. 3791 then Work-Group Id Y 1 32-bit work-group id in Y 3792 (enable_sgpr_workgroup_id dimension of grid for 3793 _Y) wavefront. 3794 then Work-Group Id Z 1 32-bit work-group id in Z 3795 (enable_sgpr_workgroup_id dimension of grid for 3796 _Z) wavefront. 3797 then Work-Group Info 1 {first_wavefront, 14'b0000, 3798 (enable_sgpr_workgroup ordered_append_term[10:0], 3799 _info) threadgroup_size_in_wavefronts[5:0]} 3800 then Scratch Wavefront Offset 1 32-bit byte offset from base 3801 (enable_sgpr_private of scratch base of queue 3802 _segment_wavefront_offset) executing the kernel 3803 dispatch. Must be used as an 3804 offset with Private 3805 segment address when using 3806 Scratch Segment Buffer. It 3807 must be used to set up FLAT 3808 SCRATCH for flat addressing 3809 (see 3810 :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`). 3811 ========== ========================== ====== ============================== 3812 3813The order of the VGPR registers is defined, but the compiler can specify which 3814ones are actually setup in the kernel descriptor using the ``enable_vgpr*`` bit 3815fields (see :ref:`amdgpu-amdhsa-kernel-descriptor`). The register numbers used 3816for enabled registers are dense starting at VGPR0: the first enabled register is 3817VGPR0, the next enabled register is VGPR1 etc.; disabled registers do not have a 3818VGPR number. 3819 3820VGPR register initial state is defined in 3821:ref:`amdgpu-amdhsa-vgpr-register-set-up-order-table`. 3822 3823 .. table:: VGPR Register Set Up Order 3824 :name: amdgpu-amdhsa-vgpr-register-set-up-order-table 3825 3826 ========== ========================== ====== ============================== 3827 VGPR Order Name Number Description 3828 (kernel descriptor enable of 3829 field) VGPRs 3830 ========== ========================== ====== ============================== 3831 First Work-Item Id X 1 32-bit work item id in X 3832 (Always initialized) dimension of work-group for 3833 wavefront lane. 3834 then Work-Item Id Y 1 32-bit work item id in Y 3835 (enable_vgpr_workitem_id dimension of work-group for 3836 > 0) wavefront lane. 3837 then Work-Item Id Z 1 32-bit work item id in Z 3838 (enable_vgpr_workitem_id dimension of work-group for 3839 > 1) wavefront lane. 3840 ========== ========================== ====== ============================== 3841 3842The setting of registers is done by GPU CP/ADC/SPI hardware as follows: 3843 38441. SGPRs before the Work-Group Ids are set by CP using the 16 User Data 3845 registers. 38462. Work-group Id registers X, Y, Z are set by ADC which supports any 3847 combination including none. 38483. Scratch Wavefront Offset is set by SPI in a per wavefront basis which is why 3849 its value cannot be included with the flat scratch init value which is per 3850 queue. 38514. The VGPRs are set by SPI which only supports specifying either (X), (X, Y) 3852 or (X, Y, Z). 3853 3854Flat Scratch register pair are adjacent SGPRs so they can be moved as a 64-bit 3855value to the hardware required SGPRn-3 and SGPRn-4 respectively. 3856 3857The global segment can be accessed either using buffer instructions (GFX6 which 3858has V# 64-bit address support), flat instructions (GFX7-GFX10), or global 3859instructions (GFX9-GFX10). 3860 3861If buffer operations are used, then the compiler can generate a V# with the 3862following properties: 3863 3864* base address of 0 3865* no swizzle 3866* ATC: 1 if IOMMU present (such as APU) 3867* ptr64: 1 3868* MTYPE set to support memory coherence that matches the runtime (such as CC for 3869 APU and NC for dGPU). 3870 3871.. _amdgpu-amdhsa-kernel-prolog: 3872 3873Kernel Prolog 3874~~~~~~~~~~~~~ 3875 3876The compiler performs initialization in the kernel prologue depending on the 3877target and information about things like stack usage in the kernel and called 3878functions. Some of this initialization requires the compiler to request certain 3879User and System SGPRs be present in the 3880:ref:`amdgpu-amdhsa-initial-kernel-execution-state` via the 3881:ref:`amdgpu-amdhsa-kernel-descriptor`. 3882 3883.. _amdgpu-amdhsa-kernel-prolog-cfi: 3884 3885CFI 3886+++ 3887 38881. The CFI return address is undefined. 3889 38902. The CFI CFA is defined using an expression which evaluates to a location 3891 description that comprises one memory location description for the 3892 ``DW_ASPACE_AMDGPU_private_lane`` address space address ``0``. 3893 3894.. _amdgpu-amdhsa-kernel-prolog-m0: 3895 3896M0 3897++ 3898 3899GFX6-GFX8 3900 The M0 register must be initialized with a value at least the total LDS size 3901 if the kernel may access LDS via DS or flat operations. Total LDS size is 3902 available in dispatch packet. For M0, it is also possible to use maximum 3903 possible value of LDS for given target (0x7FFF for GFX6 and 0xFFFF for 3904 GFX7-GFX8). 3905GFX9-GFX10 3906 The M0 register is not used for range checking LDS accesses and so does not 3907 need to be initialized in the prolog. 3908 3909.. _amdgpu-amdhsa-kernel-prolog-stack-pointer: 3910 3911Stack Pointer 3912+++++++++++++ 3913 3914If the kernel has function calls it must set up the ABI stack pointer described 3915in :ref:`amdgpu-amdhsa-function-call-convention-non-kernel-functions` by setting 3916SGPR32 to the unswizzled scratch offset of the address past the last local 3917allocation. 3918 3919.. _amdgpu-amdhsa-kernel-prolog-frame-pointer: 3920 3921Frame Pointer 3922+++++++++++++ 3923 3924If the kernel needs a frame pointer for the reasons defined in 3925``SIFrameLowering`` then SGPR33 is used and is always set to ``0`` in the 3926kernel prolog. If a frame pointer is not required then all uses of the frame 3927pointer are replaced with immediate ``0`` offsets. 3928 3929.. _amdgpu-amdhsa-kernel-prolog-flat-scratch: 3930 3931Flat Scratch 3932++++++++++++ 3933 3934If the kernel or any function it calls may use flat operations to access 3935scratch memory, the prolog code must set up the FLAT_SCRATCH register pair 3936(FLAT_SCRATCH_LO/FLAT_SCRATCH_HI which are in SGPRn-4/SGPRn-3). Initialization 3937uses Flat Scratch Init and Scratch Wavefront Offset SGPR registers (see 3938:ref:`amdgpu-amdhsa-initial-kernel-execution-state`): 3939 3940GFX6 3941 Flat scratch is not supported. 3942 3943GFX7-GFX8 3944 3945 1. The low word of Flat Scratch Init is 32-bit byte offset from 3946 ``SH_HIDDEN_PRIVATE_BASE_VIMID`` to the base of scratch backing memory 3947 being managed by SPI for the queue executing the kernel dispatch. This is 3948 the same value used in the Scratch Segment Buffer V# base address. The 3949 prolog must add the value of Scratch Wavefront Offset to get the 3950 wavefront's byte scratch backing memory offset from 3951 ``SH_HIDDEN_PRIVATE_BASE_VIMID``. Since FLAT_SCRATCH_LO is in units of 256 3952 bytes, the offset must be right shifted by 8 before moving into 3953 FLAT_SCRATCH_LO. 3954 2. The second word of Flat Scratch Init is 32-bit byte size of a single 3955 work-items scratch memory usage. This is directly loaded from the kernel 3956 dispatch packet Private Segment Byte Size and rounded up to a multiple of 3957 DWORD. Having CP load it once avoids loading it at the beginning of every 3958 wavefront. The prolog must move it to FLAT_SCRATCH_LO for use as FLAT 3959 SCRATCH SIZE. 3960 3961GFX9-GFX10 3962 The Flat Scratch Init is the 64-bit address of the base of scratch backing 3963 memory being managed by SPI for the queue executing the kernel dispatch. The 3964 prolog must add the value of Scratch Wavefront Offset and moved to the 3965 FLAT_SCRATCH pair for use as the flat scratch base in flat memory 3966 instructions. 3967 3968.. _amdgpu-amdhsa-kernel-prolog-private-segment-buffer: 3969 3970Private Segment Buffer 3971++++++++++++++++++++++ 3972 3973A set of four SGPRs beginning at a four-aligned SGPR index are always selected 3974to serve as the scratch V# for the kernel as follows: 3975 3976 - If it is known during instruction selection that there is stack usage, 3977 SGPR0-3 is reserved for use as the scratch V#. Stack usage is assumed if 3978 optimizations are disabled (``-O0``), if stack objects already exist (for 3979 locals, etc.), or if there are any function calls. 3980 3981 - Otherwise, four high numbered SGPRs beginning at a four-aligned SGPR index 3982 are reserved for the tentative scratch V#. These will be used if it is 3983 determined that spilling is needed. 3984 3985 - If no use is made of the tentative scratch V#, then it is unreserved, 3986 and the register count is determined ignoring it. 3987 - If use is made of the tentative scratch V#, then its register numbers 3988 are shifted to the first four-aligned SGPR index after the highest one 3989 allocated by the register allocator, and all uses are updated. The 3990 register count includes them in the shifted location. 3991 - In either case, if the processor has the SGPR allocation bug, the 3992 tentative allocation is not shifted or unreserved in order to ensure 3993 the register count is higher to workaround the bug. 3994 3995 .. note:: 3996 3997 This approach of using a tentative scratch V# and shifting the register 3998 numbers if used avoids having to perform register allocation a second 3999 time if the tentative V# is eliminated. This is more efficient and 4000 avoids the problem that the second register allocation may perform 4001 spilling which will fail as there is no longer a scratch V#. 4002 4003When the kernel prolog code is being emitted it is known whether the scratch V# 4004described above is actually used. If it is, the prolog code must set it up by 4005copying the Private Segment Buffer to the scratch V# registers and then adding 4006the Private Segment Wavefront Offset to the queue base address in the V#. The 4007result is a V# with a base address pointing to the beginning of the wavefront 4008scratch backing memory. 4009 4010The Private Segment Buffer is always requested, but the Private Segment 4011Wavefront Offset is only requested if it is used (see 4012:ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4013 4014.. _amdgpu-amdhsa-memory-model: 4015 4016Memory Model 4017~~~~~~~~~~~~ 4018 4019This section describes the mapping of LLVM memory model onto AMDGPU machine code 4020(see :ref:`memmodel`). 4021 4022The AMDGPU backend supports the memory synchronization scopes specified in 4023:ref:`amdgpu-memory-scopes`. 4024 4025The code sequences used to implement the memory model are defined in table 4026:ref:`amdgpu-amdhsa-memory-model-code-sequences-gfx6-gfx10-table`. 4027 4028The sequences specify the order of instructions that a single thread must 4029execute. The ``s_waitcnt`` and ``buffer_wbinvl1_vol`` are defined with respect 4030to other memory instructions executed by the same thread. This allows them to be 4031moved earlier or later which can allow them to be combined with other instances 4032of the same instruction, or hoisted/sunk out of loops to improve 4033performance. Only the instructions related to the memory model are given; 4034additional ``s_waitcnt`` instructions are required to ensure registers are 4035defined before being used. These may be able to be combined with the memory 4036model ``s_waitcnt`` instructions as described above. 4037 4038The AMDGPU backend supports the following memory models: 4039 4040 HSA Memory Model [HSA]_ 4041 The HSA memory model uses a single happens-before relation for all address 4042 spaces (see :ref:`amdgpu-address-spaces`). 4043 OpenCL Memory Model [OpenCL]_ 4044 The OpenCL memory model which has separate happens-before relations for the 4045 global and local address spaces. Only a fence specifying both global and 4046 local address space, and seq_cst instructions join the relationships. Since 4047 the LLVM ``memfence`` instruction does not allow an address space to be 4048 specified the OpenCL fence has to conservatively assume both local and 4049 global address space was specified. However, optimizations can often be 4050 done to eliminate the additional ``s_waitcnt`` instructions when there are 4051 no intervening memory instructions which access the corresponding address 4052 space. The code sequences in the table indicate what can be omitted for the 4053 OpenCL memory. The target triple environment is used to determine if the 4054 source language is OpenCL (see :ref:`amdgpu-opencl`). 4055 4056``ds/flat_load/store/atomic`` instructions to local memory are termed LDS 4057operations. 4058 4059``buffer/global/flat_load/store/atomic`` instructions to global memory are 4060termed vector memory operations. 4061 4062For GFX6-GFX9: 4063 4064* Each agent has multiple shader arrays (SA). 4065* Each SA has multiple compute units (CU). 4066* Each CU has multiple SIMDs that execute wavefronts. 4067* The wavefronts for a single work-group are executed in the same CU but may be 4068 executed by different SIMDs. 4069* Each CU has a single LDS memory shared by the wavefronts of the work-groups 4070 executing on it. 4071* All LDS operations of a CU are performed as wavefront wide operations in a 4072 global order and involve no caching. Completion is reported to a wavefront in 4073 execution order. 4074* The LDS memory has multiple request queues shared by the SIMDs of a 4075 CU. Therefore, the LDS operations performed by different wavefronts of a 4076 work-group can be reordered relative to each other, which can result in 4077 reordering the visibility of vector memory operations with respect to LDS 4078 operations of other wavefronts in the same work-group. A ``s_waitcnt 4079 lgkmcnt(0)`` is required to ensure synchronization between LDS operations and 4080 vector memory operations between wavefronts of a work-group, but not between 4081 operations performed by the same wavefront. 4082* The vector memory operations are performed as wavefront wide operations and 4083 completion is reported to a wavefront in execution order. The exception is 4084 that for GFX7-GFX9 ``flat_load/store/atomic`` instructions can report out of 4085 vector memory order if they access LDS memory, and out of LDS operation order 4086 if they access global memory. 4087* The vector memory operations access a single vector L1 cache shared by all 4088 SIMDs a CU. Therefore, no special action is required for coherence between the 4089 lanes of a single wavefront, or for coherence between wavefronts in the same 4090 work-group. A ``buffer_wbinvl1_vol`` is required for coherence between 4091 wavefronts executing in different work-groups as they may be executing on 4092 different CUs. 4093* The scalar memory operations access a scalar L1 cache shared by all wavefronts 4094 on a group of CUs. The scalar and vector L1 caches are not coherent. However, 4095 scalar operations are used in a restricted way so do not impact the memory 4096 model. See :ref:`amdgpu-address-spaces`. 4097* The vector and scalar memory operations use an L2 cache shared by all CUs on 4098 the same agent. 4099* The L2 cache has independent channels to service disjoint ranges of virtual 4100 addresses. 4101* Each CU has a separate request queue per channel. Therefore, the vector and 4102 scalar memory operations performed by wavefronts executing in different 4103 work-groups (which may be executing on different CUs) of an agent can be 4104 reordered relative to each other. A ``s_waitcnt vmcnt(0)`` is required to 4105 ensure synchronization between vector memory operations of different CUs. It 4106 ensures a previous vector memory operation has completed before executing a 4107 subsequent vector memory or LDS operation and so can be used to meet the 4108 requirements of acquire and release. 4109* The L2 cache can be kept coherent with other agents on some targets, or ranges 4110 of virtual addresses can be set up to bypass it to ensure system coherence. 4111 4112For GFX10: 4113 4114* Each agent has multiple shader arrays (SA). 4115* Each SA has multiple work-group processors (WGP). 4116* Each WGP has multiple compute units (CU). 4117* Each CU has multiple SIMDs that execute wavefronts. 4118* The wavefronts for a single work-group are executed in the same 4119 WGP. In CU wavefront execution mode the wavefronts may be executed by 4120 different SIMDs in the same CU. In WGP wavefront execution mode the 4121 wavefronts may be executed by different SIMDs in different CUs in the same 4122 WGP. 4123* Each WGP has a single LDS memory shared by the wavefronts of the work-groups 4124 executing on it. 4125* All LDS operations of a WGP are performed as wavefront wide operations in a 4126 global order and involve no caching. Completion is reported to a wavefront in 4127 execution order. 4128* The LDS memory has multiple request queues shared by the SIMDs of a 4129 WGP. Therefore, the LDS operations performed by different wavefronts of a 4130 work-group can be reordered relative to each other, which can result in 4131 reordering the visibility of vector memory operations with respect to LDS 4132 operations of other wavefronts in the same work-group. A ``s_waitcnt 4133 lgkmcnt(0)`` is required to ensure synchronization between LDS operations and 4134 vector memory operations between wavefronts of a work-group, but not between 4135 operations performed by the same wavefront. 4136* The vector memory operations are performed as wavefront wide operations. 4137 Completion of load/store/sample operations are reported to a wavefront in 4138 execution order of other load/store/sample operations performed by that 4139 wavefront. 4140* The vector memory operations access a vector L0 cache. There is a single L0 4141 cache per CU. Each SIMD of a CU accesses the same L0 cache. Therefore, no 4142 special action is required for coherence between the lanes of a single 4143 wavefront. However, a ``BUFFER_GL0_INV`` is required for coherence between 4144 wavefronts executing in the same work-group as they may be executing on SIMDs 4145 of different CUs that access different L0s. A ``BUFFER_GL0_INV`` is also 4146 required for coherence between wavefronts executing in different work-groups 4147 as they may be executing on different WGPs. 4148* The scalar memory operations access a scalar L0 cache shared by all wavefronts 4149 on a WGP. The scalar and vector L0 caches are not coherent. However, scalar 4150 operations are used in a restricted way so do not impact the memory model. See 4151 :ref:`amdgpu-address-spaces`. 4152* The vector and scalar memory L0 caches use an L1 cache shared by all WGPs on 4153 the same SA. Therefore, no special action is required for coherence between 4154 the wavefronts of a single work-group. However, a ``BUFFER_GL1_INV`` is 4155 required for coherence between wavefronts executing in different work-groups 4156 as they may be executing on different SAs that access different L1s. 4157* The L1 caches have independent quadrants to service disjoint ranges of virtual 4158 addresses. 4159* Each L0 cache has a separate request queue per L1 quadrant. Therefore, the 4160 vector and scalar memory operations performed by different wavefronts, whether 4161 executing in the same or different work-groups (which may be executing on 4162 different CUs accessing different L0s), can be reordered relative to each 4163 other. A ``s_waitcnt vmcnt(0) & vscnt(0)`` is required to ensure 4164 synchronization between vector memory operations of different wavefronts. It 4165 ensures a previous vector memory operation has completed before executing a 4166 subsequent vector memory or LDS operation and so can be used to meet the 4167 requirements of acquire, release and sequential consistency. 4168* The L1 caches use an L2 cache shared by all SAs on the same agent. 4169* The L2 cache has independent channels to service disjoint ranges of virtual 4170 addresses. 4171* Each L1 quadrant of a single SA accesses a different L2 channel. Each L1 4172 quadrant has a separate request queue per L2 channel. Therefore, the vector 4173 and scalar memory operations performed by wavefronts executing in different 4174 work-groups (which may be executing on different SAs) of an agent can be 4175 reordered relative to each other. A ``s_waitcnt vmcnt(0) & vscnt(0)`` is 4176 required to ensure synchronization between vector memory operations of 4177 different SAs. It ensures a previous vector memory operation has completed 4178 before executing a subsequent vector memory and so can be used to meet the 4179 requirements of acquire, release and sequential consistency. 4180* The L2 cache can be kept coherent with other agents on some targets, or ranges 4181 of virtual addresses can be set up to bypass it to ensure system coherence. 4182 4183Private address space uses ``buffer_load/store`` using the scratch V# 4184(GFX6-GFX8), or ``scratch_load/store`` (GFX9-GFX10). Since only a single thread 4185is accessing the memory, atomic memory orderings are not meaningful, and all 4186accesses are treated as non-atomic. 4187 4188Constant address space uses ``buffer/global_load`` instructions (or equivalent 4189scalar memory instructions). Since the constant address space contents do not 4190change during the execution of a kernel dispatch it is not legal to perform 4191stores, and atomic memory orderings are not meaningful, and all access are 4192treated as non-atomic. 4193 4194A memory synchronization scope wider than work-group is not meaningful for the 4195group (LDS) address space and is treated as work-group. 4196 4197The memory model does not support the region address space which is treated as 4198non-atomic. 4199 4200Acquire memory ordering is not meaningful on store atomic instructions and is 4201treated as non-atomic. 4202 4203Release memory ordering is not meaningful on load atomic instructions and is 4204treated a non-atomic. 4205 4206Acquire-release memory ordering is not meaningful on load or store atomic 4207instructions and is treated as acquire and release respectively. 4208 4209AMDGPU backend only uses scalar memory operations to access memory that is 4210proven to not change during the execution of the kernel dispatch. This includes 4211constant address space and global address space for program scope const 4212variables. Therefore, the kernel machine code does not have to maintain the 4213scalar L1 cache to ensure it is coherent with the vector L1 cache. The scalar 4214and vector L1 caches are invalidated between kernel dispatches by CP since 4215constant address space data may change between kernel dispatch executions. See 4216:ref:`amdgpu-address-spaces`. 4217 4218The one exception is if scalar writes are used to spill SGPR registers. In this 4219case the AMDGPU backend ensures the memory location used to spill is never 4220accessed by vector memory operations at the same time. If scalar writes are used 4221then a ``s_dcache_wb`` is inserted before the ``s_endpgm`` and before a function 4222return since the locations may be used for vector memory instructions by a 4223future wavefront that uses the same scratch area, or a function call that 4224creates a frame at the same address, respectively. There is no need for a 4225``s_dcache_inv`` as all scalar writes are write-before-read in the same thread. 4226 4227For GFX6-GFX9, scratch backing memory (which is used for the private address 4228space) is accessed with MTYPE NC_NV (non-coherent non-volatile). Since the 4229private address space is only accessed by a single thread, and is always 4230write-before-read, there is never a need to invalidate these entries from the L1 4231cache. Hence all cache invalidates are done as ``*_vol`` to only invalidate the 4232volatile cache lines. 4233 4234For GFX10, scratch backing memory (which is used for the private address space) 4235is accessed with MTYPE NC (non-coherent). Since the private address space is 4236only accessed by a single thread, and is always write-before-read, there is 4237never a need to invalidate these entries from the L0 or L1 caches. 4238 4239For GFX10, wavefronts are executed in native mode with in-order reporting of 4240loads and sample instructions. In this mode vmcnt reports completion of load, 4241atomic with return and sample instructions in order, and the vscnt reports the 4242completion of store and atomic without return in order. See ``MEM_ORDERED`` 4243field in :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 4244 4245In GFX10, wavefronts can be executed in WGP or CU wavefront execution mode: 4246 4247* In WGP wavefront execution mode the wavefronts of a work-group are executed 4248 on the SIMDs of both CUs of the WGP. Therefore, explicit management of the per 4249 CU L0 caches is required for work-group synchronization. Also accesses to L1 4250 at work-group scope need to be explicitly ordered as the accesses from 4251 different CUs are not ordered. 4252* In CU wavefront execution mode the wavefronts of a work-group are executed on 4253 the SIMDs of a single CU of the WGP. Therefore, all global memory access by 4254 the work-group access the same L0 which in turn ensures L1 accesses are 4255 ordered and so do not require explicit management of the caches for 4256 work-group synchronization. 4257 4258See ``WGP_MODE`` field in 4259:ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table` and 4260:ref:`amdgpu-target-features`. 4261 4262On dGPU the kernarg backing memory is accessed as UC (uncached) to avoid needing 4263to invalidate the L2 cache. For GFX6-GFX9, this also causes it to be treated as 4264non-volatile and so is not invalidated by ``*_vol``. On APU it is accessed as CC 4265(cache coherent) and so the L2 cache will be coherent with the CPU and other 4266agents. 4267 4268 .. table:: AMDHSA Memory Model Code Sequences GFX6-GFX10 4269 :name: amdgpu-amdhsa-memory-model-code-sequences-gfx6-gfx10-table 4270 4271 ============ ============ ============== ========== =============================== ================================== 4272 LLVM Instr LLVM Memory LLVM Memory AMDGPU AMDGPU Machine Code AMDGPU Machine Code 4273 Ordering Sync Scope Address GFX6-9 GFX10 4274 Space 4275 ============ ============ ============== ========== =============================== ================================== 4276 **Non-Atomic** 4277 ---------------------------------------------------------------------------------------------------------------------- 4278 load *none* *none* - global - !volatile & !nontemporal - !volatile & !nontemporal 4279 - generic 4280 - private 1. buffer/global/flat_load 1. buffer/global/flat_load 4281 - constant 4282 - volatile & !nontemporal - volatile & !nontemporal 4283 4284 1. buffer/global/flat_load 1. buffer/global/flat_load 4285 glc=1 glc=1 dlc=1 4286 4287 - nontemporal - nontemporal 4288 4289 1. buffer/global/flat_load 1. buffer/global/flat_load 4290 glc=1 slc=1 slc=1 4291 4292 load *none* *none* - local 1. ds_load 1. ds_load 4293 store *none* *none* - global - !nontemporal - !nontemporal 4294 - generic 4295 - private 1. buffer/global/flat_store 1. buffer/global/flat_store 4296 - constant 4297 - nontemporal - nontemporal 4298 4299 1. buffer/global/flat_store 1. buffer/global/flat_store 4300 glc=1 slc=1 slc=1 4301 4302 store *none* *none* - local 1. ds_store 1. ds_store 4303 **Unordered Atomic** 4304 ---------------------------------------------------------------------------------------------------------------------- 4305 load atomic unordered *any* *any* *Same as non-atomic*. *Same as non-atomic*. 4306 store atomic unordered *any* *any* *Same as non-atomic*. *Same as non-atomic*. 4307 atomicrmw unordered *any* *any* *Same as monotonic *Same as monotonic 4308 atomic*. atomic*. 4309 **Monotonic Atomic** 4310 ---------------------------------------------------------------------------------------------------------------------- 4311 load atomic monotonic - singlethread - global 1. buffer/global/flat_load 1. buffer/global/flat_load 4312 - wavefront - generic 4313 load atomic monotonic - workgroup - global 1. buffer/global/flat_load 1. buffer/global/flat_load 4314 - generic glc=1 4315 4316 - If CU wavefront execution mode, omit glc=1. 4317 4318 load atomic monotonic - singlethread - local 1. ds_load 1. ds_load 4319 - wavefront 4320 - workgroup 4321 load atomic monotonic - agent - global 1. buffer/global/flat_load 1. buffer/global/flat_load 4322 - system - generic glc=1 glc=1 dlc=1 4323 store atomic monotonic - singlethread - global 1. buffer/global/flat_store 1. buffer/global/flat_store 4324 - wavefront - generic 4325 - workgroup 4326 - agent 4327 - system 4328 store atomic monotonic - singlethread - local 1. ds_store 1. ds_store 4329 - wavefront 4330 - workgroup 4331 atomicrmw monotonic - singlethread - global 1. buffer/global/flat_atomic 1. buffer/global/flat_atomic 4332 - wavefront - generic 4333 - workgroup 4334 - agent 4335 - system 4336 atomicrmw monotonic - singlethread - local 1. ds_atomic 1. ds_atomic 4337 - wavefront 4338 - workgroup 4339 **Acquire Atomic** 4340 ---------------------------------------------------------------------------------------------------------------------- 4341 load atomic acquire - singlethread - global 1. buffer/global/ds/flat_load 1. buffer/global/ds/flat_load 4342 - wavefront - local 4343 - generic 4344 load atomic acquire - workgroup - global 1. buffer/global/flat_load 1. buffer/global_load glc=1 4345 4346 - If CU wavefront execution mode, omit glc=1. 4347 4348 2. s_waitcnt vmcnt(0) 4349 4350 - If CU wavefront execution mode, omit. 4351 - Must happen before 4352 the following buffer_gl0_inv 4353 and before any following 4354 global/generic 4355 load/load 4356 atomic/store/store 4357 atomic/atomicrmw. 4358 4359 3. buffer_gl0_inv 4360 4361 - If CU wavefront execution mode, omit. 4362 - Ensures that 4363 following 4364 loads will not see 4365 stale data. 4366 4367 load atomic acquire - workgroup - local 1. ds_load 1. ds_load 4368 2. s_waitcnt lgkmcnt(0) 2. s_waitcnt lgkmcnt(0) 4369 4370 - If OpenCL, omit. - If OpenCL, omit. 4371 - Must happen before - Must happen before 4372 any following the following buffer_gl0_inv 4373 global/generic and before any following 4374 load/load global/generic load/load 4375 atomic/store/store atomic/store/store 4376 atomic/atomicrmw. atomic/atomicrmw. 4377 - Ensures any - Ensures any 4378 following global following global 4379 data read is no data read is no 4380 older than the load older than the load 4381 atomic value being atomic value being 4382 acquired. acquired. 4383 4384 3. buffer_gl0_inv 4385 4386 - If CU wavefront execution mode, omit. 4387 - If OpenCL, omit. 4388 - Ensures that 4389 following 4390 loads will not see 4391 stale data. 4392 4393 load atomic acquire - workgroup - generic 1. flat_load 1. flat_load glc=1 4394 4395 - If CU wavefront execution mode, omit glc=1. 4396 4397 2. s_waitcnt lgkmcnt(0) 2. s_waitcnt lgkmcnt(0) & 4398 vmcnt(0) 4399 4400 - If CU wavefront execution mode, omit vmcnt. 4401 - If OpenCL, omit. - If OpenCL, omit 4402 lgkmcnt(0). 4403 - Must happen before - Must happen before 4404 any following the following 4405 global/generic buffer_gl0_inv and any 4406 load/load following global/generic 4407 atomic/store/store load/load 4408 atomic/atomicrmw. atomic/store/store 4409 atomic/atomicrmw. 4410 - Ensures any - Ensures any 4411 following global following global 4412 data read is no data read is no 4413 older than the load older than the load 4414 atomic value being atomic value being 4415 acquired. acquired. 4416 4417 3. buffer_gl0_inv 4418 4419 - If CU wavefront execution mode, omit. 4420 - Ensures that 4421 following 4422 loads will not see 4423 stale data. 4424 4425 load atomic acquire - agent - global 1. buffer/global/flat_load 1. buffer/global_load 4426 - system glc=1 glc=1 dlc=1 4427 2. s_waitcnt vmcnt(0) 2. s_waitcnt vmcnt(0) 4428 4429 - Must happen before - Must happen before 4430 following following 4431 buffer_wbinvl1_vol. buffer_gl*_inv. 4432 - Ensures the load - Ensures the load 4433 has completed has completed 4434 before invalidating before invalidating 4435 the cache. the caches. 4436 4437 3. buffer_wbinvl1_vol 3. buffer_gl0_inv; 4438 buffer_gl1_inv 4439 4440 - Must happen before - Must happen before 4441 any following any following 4442 global/generic global/generic 4443 load/load load/load 4444 atomic/atomicrmw. atomic/atomicrmw. 4445 - Ensures that - Ensures that 4446 following following 4447 loads will not see loads will not see 4448 stale global data. stale global data. 4449 4450 load atomic acquire - agent - generic 1. flat_load glc=1 1. flat_load glc=1 dlc=1 4451 - system 2. s_waitcnt vmcnt(0) & 2. s_waitcnt vmcnt(0) & 4452 lgkmcnt(0) lgkmcnt(0) 4453 4454 - If OpenCL omit - If OpenCL omit 4455 lgkmcnt(0). lgkmcnt(0). 4456 - Must happen before - Must happen before 4457 following following 4458 buffer_wbinvl1_vol. buffer_gl*_invl. 4459 - Ensures the flat_load - Ensures the flat_load 4460 has completed has completed 4461 before invalidating before invalidating 4462 the cache. the caches. 4463 4464 3. buffer_wbinvl1_vol 3. buffer_gl0_inv; 4465 buffer_gl1_inv 4466 4467 - Must happen before - Must happen before 4468 any following any following 4469 global/generic global/generic 4470 load/load load/load 4471 atomic/atomicrmw. atomic/atomicrmw. 4472 - Ensures that - Ensures that 4473 following loads following loads 4474 will not see stale will not see stale 4475 global data. global data. 4476 4477 atomicrmw acquire - singlethread - global 1. buffer/global/ds/flat_atomic 1. buffer/global/ds/flat_atomic 4478 - wavefront - local 4479 - generic 4480 atomicrmw acquire - workgroup - global 1. buffer/global/flat_atomic 1. buffer/global_atomic 4481 2. s_waitcnt vm/vscnt(0) 4482 4483 - If CU wavefront execution mode, omit. 4484 - Use vmcnt if atomic with 4485 return and vscnt if atomic 4486 with no-return. 4487 - Must happen before 4488 the following buffer_gl0_inv 4489 and before any following 4490 global/generic 4491 load/load 4492 atomic/store/store 4493 atomic/atomicrmw. 4494 4495 3. buffer_gl0_inv 4496 4497 - If CU wavefront execution mode, omit. 4498 - Ensures that 4499 following 4500 loads will not see 4501 stale data. 4502 4503 atomicrmw acquire - workgroup - local 1. ds_atomic 1. ds_atomic 4504 2. waitcnt lgkmcnt(0) 2. waitcnt lgkmcnt(0) 4505 4506 - If OpenCL, omit. - If OpenCL, omit. 4507 - Must happen before - Must happen before 4508 any following the following 4509 global/generic buffer_gl0_inv. 4510 load/load 4511 atomic/store/store 4512 atomic/atomicrmw. 4513 - Ensures any - Ensures any 4514 following global following global 4515 data read is no data read is no 4516 older than the older than the 4517 atomicrmw value atomicrmw value 4518 being acquired. being acquired. 4519 4520 3. buffer_gl0_inv 4521 4522 - If OpenCL omit. 4523 - Ensures that 4524 following 4525 loads will not see 4526 stale data. 4527 4528 atomicrmw acquire - workgroup - generic 1. flat_atomic 1. flat_atomic 4529 2. waitcnt lgkmcnt(0) 2. waitcnt lgkmcnt(0) & 4530 vm/vscnt(0) 4531 4532 - If CU wavefront execution mode, omit vm/vscnt. 4533 - If OpenCL, omit. - If OpenCL, omit 4534 waitcnt lgkmcnt(0).. 4535 - Use vmcnt if atomic with 4536 return and vscnt if atomic 4537 with no-return. 4538 waitcnt lgkmcnt(0). 4539 - Must happen before - Must happen before 4540 any following the following 4541 global/generic buffer_gl0_inv. 4542 load/load 4543 atomic/store/store 4544 atomic/atomicrmw. 4545 - Ensures any - Ensures any 4546 following global following global 4547 data read is no data read is no 4548 older than the older than the 4549 atomicrmw value atomicrmw value 4550 being acquired. being acquired. 4551 4552 3. buffer_gl0_inv 4553 4554 - If CU wavefront execution mode, omit. 4555 - Ensures that 4556 following 4557 loads will not see 4558 stale data. 4559 4560 atomicrmw acquire - agent - global 1. buffer/global/flat_atomic 1. buffer/global_atomic 4561 - system 2. s_waitcnt vmcnt(0) 2. s_waitcnt vm/vscnt(0) 4562 4563 - Use vmcnt if atomic with 4564 return and vscnt if atomic 4565 with no-return. 4566 waitcnt lgkmcnt(0). 4567 - Must happen before - Must happen before 4568 following following 4569 buffer_wbinvl1_vol. buffer_gl*_inv. 4570 - Ensures the - Ensures the 4571 atomicrmw has atomicrmw has 4572 completed before completed before 4573 invalidating the invalidating the 4574 cache. caches. 4575 4576 3. buffer_wbinvl1_vol 3. buffer_gl0_inv; 4577 buffer_gl1_inv 4578 4579 - Must happen before - Must happen before 4580 any following any following 4581 global/generic global/generic 4582 load/load load/load 4583 atomic/atomicrmw. atomic/atomicrmw. 4584 - Ensures that - Ensures that 4585 following loads following loads 4586 will not see stale will not see stale 4587 global data. global data. 4588 4589 atomicrmw acquire - agent - generic 1. flat_atomic 1. flat_atomic 4590 - system 2. s_waitcnt vmcnt(0) & 2. s_waitcnt vm/vscnt(0) & 4591 lgkmcnt(0) lgkmcnt(0) 4592 4593 - If OpenCL, omit - If OpenCL, omit 4594 lgkmcnt(0). lgkmcnt(0). 4595 - Use vmcnt if atomic with 4596 return and vscnt if atomic 4597 with no-return. 4598 - Must happen before - Must happen before 4599 following following 4600 buffer_wbinvl1_vol. buffer_gl*_inv. 4601 - Ensures the - Ensures the 4602 atomicrmw has atomicrmw has 4603 completed before completed before 4604 invalidating the invalidating the 4605 cache. caches. 4606 4607 3. buffer_wbinvl1_vol 3. buffer_gl0_inv; 4608 buffer_gl1_inv 4609 4610 - Must happen before - Must happen before 4611 any following any following 4612 global/generic global/generic 4613 load/load load/load 4614 atomic/atomicrmw. atomic/atomicrmw. 4615 - Ensures that - Ensures that 4616 following loads following loads 4617 will not see stale will not see stale 4618 global data. global data. 4619 4620 fence acquire - singlethread *none* *none* *none* 4621 - wavefront 4622 fence acquire - workgroup *none* 1. s_waitcnt lgkmcnt(0) 1. s_waitcnt lgkmcnt(0) & 4623 vmcnt(0) & vscnt(0) 4624 4625 - If CU wavefront execution mode, omit vmcnt and 4626 vscnt. 4627 - If OpenCL and - If OpenCL and 4628 address space is address space is 4629 not generic, omit. not generic, omit 4630 lgkmcnt(0). 4631 - If OpenCL and 4632 address space is 4633 local, omit 4634 vmcnt(0) and vscnt(0). 4635 - However, since LLVM - However, since LLVM 4636 currently has no currently has no 4637 address space on address space on 4638 the fence need to the fence need to 4639 conservatively conservatively 4640 always generate. If always generate. If 4641 fence had an fence had an 4642 address space then address space then 4643 set to address set to address 4644 space of OpenCL space of OpenCL 4645 fence flag, or to fence flag, or to 4646 generic if both generic if both 4647 local and global local and global 4648 flags are flags are 4649 specified. specified. 4650 - Must happen after 4651 any preceding 4652 local/generic load 4653 atomic/atomicrmw 4654 with an equal or 4655 wider sync scope 4656 and memory ordering 4657 stronger than 4658 unordered (this is 4659 termed the 4660 fence-paired-atomic). 4661 - Must happen before 4662 any following 4663 global/generic 4664 load/load 4665 atomic/store/store 4666 atomic/atomicrmw. 4667 - Ensures any 4668 following global 4669 data read is no 4670 older than the 4671 value read by the 4672 fence-paired-atomic. 4673 - Could be split into 4674 separate s_waitcnt 4675 vmcnt(0), s_waitcnt 4676 vscnt(0) and s_waitcnt 4677 lgkmcnt(0) to allow 4678 them to be 4679 independently moved 4680 according to the 4681 following rules. 4682 - s_waitcnt vmcnt(0) 4683 must happen after 4684 any preceding 4685 global/generic load 4686 atomic/ 4687 atomicrmw-with-return-value 4688 with an equal or 4689 wider sync scope 4690 and memory ordering 4691 stronger than 4692 unordered (this is 4693 termed the 4694 fence-paired-atomic). 4695 - s_waitcnt vscnt(0) 4696 must happen after 4697 any preceding 4698 global/generic 4699 atomicrmw-no-return-value 4700 with an equal or 4701 wider sync scope 4702 and memory ordering 4703 stronger than 4704 unordered (this is 4705 termed the 4706 fence-paired-atomic). 4707 - s_waitcnt lgkmcnt(0) 4708 must happen after 4709 any preceding 4710 local/generic load 4711 atomic/atomicrmw 4712 with an equal or 4713 wider sync scope 4714 and memory ordering 4715 stronger than 4716 unordered (this is 4717 termed the 4718 fence-paired-atomic). 4719 - Must happen before 4720 the following 4721 buffer_gl0_inv. 4722 - Ensures that the 4723 fence-paired atomic 4724 has completed 4725 before invalidating 4726 the 4727 cache. Therefore 4728 any following 4729 locations read must 4730 be no older than 4731 the value read by 4732 the 4733 fence-paired-atomic. 4734 4735 3. buffer_gl0_inv 4736 4737 - If CU wavefront execution mode, omit. 4738 - Ensures that 4739 following 4740 loads will not see 4741 stale data. 4742 4743 fence acquire - agent *none* 1. s_waitcnt lgkmcnt(0) & 1. s_waitcnt lgkmcnt(0) & 4744 - system vmcnt(0) vmcnt(0) & vscnt(0) 4745 4746 - If OpenCL and - If OpenCL and 4747 address space is address space is 4748 not generic, omit not generic, omit 4749 lgkmcnt(0). lgkmcnt(0). 4750 - If OpenCL and 4751 address space is 4752 local, omit 4753 vmcnt(0) and vscnt(0). 4754 - However, since LLVM - However, since LLVM 4755 currently has no currently has no 4756 address space on address space on 4757 the fence need to the fence need to 4758 conservatively conservatively 4759 always generate always generate 4760 (see comment for (see comment for 4761 previous fence). previous fence). 4762 - Could be split into 4763 separate s_waitcnt 4764 vmcnt(0) and 4765 s_waitcnt 4766 lgkmcnt(0) to allow 4767 them to be 4768 independently moved 4769 according to the 4770 following rules. 4771 - s_waitcnt vmcnt(0) 4772 must happen after 4773 any preceding 4774 global/generic load 4775 atomic/atomicrmw 4776 with an equal or 4777 wider sync scope 4778 and memory ordering 4779 stronger than 4780 unordered (this is 4781 termed the 4782 fence-paired-atomic). 4783 - s_waitcnt lgkmcnt(0) 4784 must happen after 4785 any preceding 4786 local/generic load 4787 atomic/atomicrmw 4788 with an equal or 4789 wider sync scope 4790 and memory ordering 4791 stronger than 4792 unordered (this is 4793 termed the 4794 fence-paired-atomic). 4795 - Must happen before 4796 the following 4797 buffer_wbinvl1_vol. 4798 - Ensures that the 4799 fence-paired atomic 4800 has completed 4801 before invalidating 4802 the 4803 cache. Therefore 4804 any following 4805 locations read must 4806 be no older than 4807 the value read by 4808 the 4809 fence-paired-atomic. 4810 - Could be split into 4811 separate s_waitcnt 4812 vmcnt(0), s_waitcnt 4813 vscnt(0) and s_waitcnt 4814 lgkmcnt(0) to allow 4815 them to be 4816 independently moved 4817 according to the 4818 following rules. 4819 - s_waitcnt vmcnt(0) 4820 must happen after 4821 any preceding 4822 global/generic load 4823 atomic/ 4824 atomicrmw-with-return-value 4825 with an equal or 4826 wider sync scope 4827 and memory ordering 4828 stronger than 4829 unordered (this is 4830 termed the 4831 fence-paired-atomic). 4832 - s_waitcnt vscnt(0) 4833 must happen after 4834 any preceding 4835 global/generic 4836 atomicrmw-no-return-value 4837 with an equal or 4838 wider sync scope 4839 and memory ordering 4840 stronger than 4841 unordered (this is 4842 termed the 4843 fence-paired-atomic). 4844 - s_waitcnt lgkmcnt(0) 4845 must happen after 4846 any preceding 4847 local/generic load 4848 atomic/atomicrmw 4849 with an equal or 4850 wider sync scope 4851 and memory ordering 4852 stronger than 4853 unordered (this is 4854 termed the 4855 fence-paired-atomic). 4856 - Must happen before 4857 the following 4858 buffer_gl*_inv. 4859 - Ensures that the 4860 fence-paired atomic 4861 has completed 4862 before invalidating 4863 the 4864 caches. Therefore 4865 any following 4866 locations read must 4867 be no older than 4868 the value read by 4869 the 4870 fence-paired-atomic. 4871 4872 2. buffer_wbinvl1_vol 2. buffer_gl0_inv; 4873 buffer_gl1_inv 4874 4875 - Must happen before any - Must happen before any 4876 following global/generic following global/generic 4877 load/load load/load 4878 atomic/store/store atomic/store/store 4879 atomic/atomicrmw. atomic/atomicrmw. 4880 - Ensures that - Ensures that 4881 following loads following loads 4882 will not see stale will not see stale 4883 global data. global data. 4884 4885 **Release Atomic** 4886 ---------------------------------------------------------------------------------------------------------------------- 4887 store atomic release - singlethread - global 1. buffer/global/ds/flat_store 1. buffer/global/ds/flat_store 4888 - wavefront - local 4889 - generic 4890 store atomic release - workgroup - global 1. s_waitcnt lgkmcnt(0) 1. s_waitcnt lgkmcnt(0) & 4891 vmcnt(0) & vscnt(0) 4892 4893 - If CU wavefront execution mode, omit vmcnt and 4894 vscnt. 4895 - If OpenCL, omit. - If OpenCL, omit 4896 lgkmcnt(0). 4897 - Must happen after 4898 any preceding 4899 local/generic 4900 load/store/load 4901 atomic/store 4902 atomic/atomicrmw. 4903 - Could be split into 4904 separate s_waitcnt 4905 vmcnt(0), s_waitcnt 4906 vscnt(0) and s_waitcnt 4907 lgkmcnt(0) to allow 4908 them to be 4909 independently moved 4910 according to the 4911 following rules. 4912 - s_waitcnt vmcnt(0) 4913 must happen after 4914 any preceding 4915 global/generic load/load 4916 atomic/ 4917 atomicrmw-with-return-value. 4918 - s_waitcnt vscnt(0) 4919 must happen after 4920 any preceding 4921 global/generic 4922 store/store 4923 atomic/ 4924 atomicrmw-no-return-value. 4925 - s_waitcnt lgkmcnt(0) 4926 must happen after 4927 any preceding 4928 local/generic 4929 load/store/load 4930 atomic/store 4931 atomic/atomicrmw. 4932 - Must happen before - Must happen before 4933 the following the following 4934 store. store. 4935 - Ensures that all - Ensures that all 4936 memory operations memory operations 4937 to local have have 4938 completed before completed before 4939 performing the performing the 4940 store that is being store that is being 4941 released. released. 4942 4943 2. buffer/global/flat_store 2. buffer/global_store 4944 store atomic release - workgroup - local 1. waitcnt vmcnt(0) & vscnt(0) 4945 4946 - If CU wavefront execution mode, omit. 4947 - If OpenCL, omit. 4948 - Could be split into 4949 separate s_waitcnt 4950 vmcnt(0) and s_waitcnt 4951 vscnt(0) to allow 4952 them to be 4953 independently moved 4954 according to the 4955 following rules. 4956 - s_waitcnt vmcnt(0) 4957 must happen after 4958 any preceding 4959 global/generic load/load 4960 atomic/ 4961 atomicrmw-with-return-value. 4962 - s_waitcnt vscnt(0) 4963 must happen after 4964 any preceding 4965 global/generic 4966 store/store atomic/ 4967 atomicrmw-no-return-value. 4968 - Must happen before 4969 the following 4970 store. 4971 - Ensures that all 4972 global memory 4973 operations have 4974 completed before 4975 performing the 4976 store that is being 4977 released. 4978 4979 1. ds_store 2. ds_store 4980 store atomic release - workgroup - generic 1. s_waitcnt lgkmcnt(0) 1. s_waitcnt lgkmcnt(0) & 4981 vmcnt(0) & vscnt(0) 4982 4983 - If CU wavefront execution mode, omit vmcnt and 4984 vscnt. 4985 - If OpenCL, omit. - If OpenCL, omit 4986 lgkmcnt(0). 4987 - Must happen after 4988 any preceding 4989 local/generic 4990 load/store/load 4991 atomic/store 4992 atomic/atomicrmw. 4993 - Could be split into 4994 separate s_waitcnt 4995 vmcnt(0), s_waitcnt 4996 vscnt(0) and s_waitcnt 4997 lgkmcnt(0) to allow 4998 them to be 4999 independently moved 5000 according to the 5001 following rules. 5002 - s_waitcnt vmcnt(0) 5003 must happen after 5004 any preceding 5005 global/generic load/load 5006 atomic/ 5007 atomicrmw-with-return-value. 5008 - s_waitcnt vscnt(0) 5009 must happen after 5010 any preceding 5011 global/generic 5012 store/store 5013 atomic/ 5014 atomicrmw-no-return-value. 5015 - s_waitcnt lgkmcnt(0) 5016 must happen after 5017 any preceding 5018 local/generic load/store/load 5019 atomic/store atomic/atomicrmw. 5020 - Must happen before - Must happen before 5021 the following the following 5022 store. store. 5023 - Ensures that all - Ensures that all 5024 memory operations memory operations 5025 to local have have 5026 completed before completed before 5027 performing the performing the 5028 store that is being store that is being 5029 released. released. 5030 5031 2. flat_store 2. flat_store 5032 store atomic release - agent - global 1. s_waitcnt lgkmcnt(0) & 1. s_waitcnt lgkmcnt(0) & 5033 - system - generic vmcnt(0) vmcnt(0) & vscnt(0) 5034 5035 - If OpenCL, omit - If OpenCL, omit 5036 lgkmcnt(0). lgkmcnt(0). 5037 - Could be split into - Could be split into 5038 separate s_waitcnt separate s_waitcnt 5039 vmcnt(0) and vmcnt(0), s_waitcnt vscnt(0) 5040 s_waitcnt and s_waitcnt 5041 lgkmcnt(0) to allow lgkmcnt(0) to allow 5042 them to be them to be 5043 independently moved independently moved 5044 according to the according to the 5045 following rules. following rules. 5046 - s_waitcnt vmcnt(0) - s_waitcnt vmcnt(0) 5047 must happen after must happen after 5048 any preceding any preceding 5049 global/generic global/generic 5050 load/store/load load/load 5051 atomic/store atomic/ 5052 atomic/atomicrmw. atomicrmw-with-return-value. 5053 - s_waitcnt vscnt(0) 5054 must happen after 5055 any preceding 5056 global/generic 5057 store/store atomic/ 5058 atomicrmw-no-return-value. 5059 - s_waitcnt lgkmcnt(0) - s_waitcnt lgkmcnt(0) 5060 must happen after must happen after 5061 any preceding any preceding 5062 local/generic local/generic 5063 load/store/load load/store/load 5064 atomic/store atomic/store 5065 atomic/atomicrmw. atomic/atomicrmw. 5066 - Must happen before - Must happen before 5067 the following the following 5068 store. store. 5069 - Ensures that all - Ensures that all 5070 memory operations memory operations 5071 to memory have to memory have 5072 completed before completed before 5073 performing the performing the 5074 store that is being store that is being 5075 released. released. 5076 5077 2. buffer/global/ds/flat_store 2. buffer/global/ds/flat_store 5078 atomicrmw release - singlethread - global 1. buffer/global/ds/flat_atomic 1. buffer/global/ds/flat_atomic 5079 - wavefront - local 5080 - generic 5081 atomicrmw release - workgroup - global 1. s_waitcnt lgkmcnt(0) 1. s_waitcnt lgkmcnt(0) & 5082 vmcnt(0) & vscnt(0) 5083 5084 - If CU wavefront execution mode, omit vmcnt and 5085 vscnt. 5086 - If OpenCL, omit. 5087 5088 - Must happen after 5089 any preceding 5090 local/generic 5091 load/store/load 5092 atomic/store 5093 atomic/atomicrmw. 5094 - Could be split into 5095 separate s_waitcnt 5096 vmcnt(0), s_waitcnt 5097 vscnt(0) and s_waitcnt 5098 lgkmcnt(0) to allow 5099 them to be 5100 independently moved 5101 according to the 5102 following rules. 5103 - s_waitcnt vmcnt(0) 5104 must happen after 5105 any preceding 5106 global/generic load/load 5107 atomic/ 5108 atomicrmw-with-return-value. 5109 - s_waitcnt vscnt(0) 5110 must happen after 5111 any preceding 5112 global/generic 5113 store/store 5114 atomic/ 5115 atomicrmw-no-return-value. 5116 - s_waitcnt lgkmcnt(0) 5117 must happen after 5118 any preceding 5119 local/generic 5120 load/store/load 5121 atomic/store 5122 atomic/atomicrmw. 5123 - Must happen before - Must happen before 5124 the following the following 5125 atomicrmw. atomicrmw. 5126 - Ensures that all - Ensures that all 5127 memory operations memory operations 5128 to local have have 5129 completed before completed before 5130 performing the performing the 5131 atomicrmw that is atomicrmw that is 5132 being released. being released. 5133 5134 2. buffer/global/flat_atomic 2. buffer/global_atomic 5135 atomicrmw release - workgroup - local 1. waitcnt vmcnt(0) & vscnt(0) 5136 5137 - If CU wavefront execution mode, omit. 5138 - If OpenCL, omit. 5139 - Could be split into 5140 separate s_waitcnt 5141 vmcnt(0) and s_waitcnt 5142 vscnt(0) to allow 5143 them to be 5144 independently moved 5145 according to the 5146 following rules. 5147 - s_waitcnt vmcnt(0) 5148 must happen after 5149 any preceding 5150 global/generic load/load 5151 atomic/ 5152 atomicrmw-with-return-value. 5153 - s_waitcnt vscnt(0) 5154 must happen after 5155 any preceding 5156 global/generic 5157 store/store atomic/ 5158 atomicrmw-no-return-value. 5159 - Must happen before 5160 the following 5161 store. 5162 - Ensures that all 5163 global memory 5164 operations have 5165 completed before 5166 performing the 5167 store that is being 5168 released. 5169 5170 1. ds_atomic 2. ds_atomic 5171 atomicrmw release - workgroup - generic 1. s_waitcnt lgkmcnt(0) 1. s_waitcnt lgkmcnt(0) & 5172 vmcnt(0) & vscnt(0) 5173 5174 - If CU wavefront execution mode, omit vmcnt and 5175 vscnt. 5176 - If OpenCL, omit. - If OpenCL, omit 5177 waitcnt lgkmcnt(0). 5178 - Must happen after 5179 any preceding 5180 local/generic 5181 load/store/load 5182 atomic/store 5183 atomic/atomicrmw. 5184 - Could be split into 5185 separate s_waitcnt 5186 vmcnt(0), s_waitcnt 5187 vscnt(0) and s_waitcnt 5188 lgkmcnt(0) to allow 5189 them to be 5190 independently moved 5191 according to the 5192 following rules. 5193 - s_waitcnt vmcnt(0) 5194 must happen after 5195 any preceding 5196 global/generic load/load 5197 atomic/ 5198 atomicrmw-with-return-value. 5199 - s_waitcnt vscnt(0) 5200 must happen after 5201 any preceding 5202 global/generic 5203 store/store 5204 atomic/ 5205 atomicrmw-no-return-value. 5206 - s_waitcnt lgkmcnt(0) 5207 must happen after 5208 any preceding 5209 local/generic load/store/load 5210 atomic/store atomic/atomicrmw. 5211 - Must happen before - Must happen before 5212 the following the following 5213 atomicrmw. atomicrmw. 5214 - Ensures that all - Ensures that all 5215 memory operations memory operations 5216 to local have have 5217 completed before completed before 5218 performing the performing the 5219 atomicrmw that is atomicrmw that is 5220 being released. being released. 5221 5222 2. flat_atomic 2. flat_atomic 5223 atomicrmw release - agent - global 1. s_waitcnt lgkmcnt(0) & 1. s_waitcnt lkkmcnt(0) & 5224 - system - generic vmcnt(0) vmcnt(0) & vscnt(0) 5225 5226 - If OpenCL, omit - If OpenCL, omit 5227 lgkmcnt(0). lgkmcnt(0). 5228 - Could be split into - Could be split into 5229 separate s_waitcnt separate s_waitcnt 5230 vmcnt(0) and vmcnt(0), s_waitcnt 5231 s_waitcnt vscnt(0) and s_waitcnt 5232 lgkmcnt(0) to allow lgkmcnt(0) to allow 5233 them to be them to be 5234 independently moved independently moved 5235 according to the according to the 5236 following rules. following rules. 5237 - s_waitcnt vmcnt(0) - s_waitcnt vmcnt(0) 5238 must happen after must happen after 5239 any preceding any preceding 5240 global/generic global/generic 5241 load/store/load load/load atomic/ 5242 atomic/store atomicrmw-with-return-value. 5243 atomic/atomicrmw. 5244 - s_waitcnt vscnt(0) 5245 must happen after 5246 any preceding 5247 global/generic 5248 store/store atomic/ 5249 atomicrmw-no-return-value. 5250 - s_waitcnt lgkmcnt(0) - s_waitcnt lgkmcnt(0) 5251 must happen after must happen after 5252 any preceding any preceding 5253 local/generic local/generic 5254 load/store/load load/store/load 5255 atomic/store atomic/store 5256 atomic/atomicrmw. atomic/atomicrmw. 5257 - Must happen before - Must happen before 5258 the following the following 5259 atomicrmw. atomicrmw. 5260 - Ensures that all - Ensures that all 5261 memory operations memory operations 5262 to global and local to global and local 5263 have completed have completed 5264 before performing before performing 5265 the atomicrmw that the atomicrmw that 5266 is being released. is being released. 5267 5268 2. buffer/global/ds/flat_atomic 2. buffer/global/ds/flat_atomic 5269 fence release - singlethread *none* *none* *none* 5270 - wavefront 5271 fence release - workgroup *none* 1. s_waitcnt lgkmcnt(0) 1. s_waitcnt lgkmcnt(0) & 5272 vmcnt(0) & vscnt(0) 5273 5274 - If CU wavefront execution mode, omit vmcnt and 5275 vscnt. 5276 - If OpenCL and - If OpenCL and 5277 address space is address space is 5278 not generic, omit. not generic, omit 5279 lgkmcnt(0). 5280 - If OpenCL and 5281 address space is 5282 local, omit 5283 vmcnt(0) and vscnt(0). 5284 - However, since LLVM - However, since LLVM 5285 currently has no currently has no 5286 address space on address space on 5287 the fence need to the fence need to 5288 conservatively conservatively 5289 always generate. If always generate. If 5290 fence had an fence had an 5291 address space then address space then 5292 set to address set to address 5293 space of OpenCL space of OpenCL 5294 fence flag, or to fence flag, or to 5295 generic if both generic if both 5296 local and global local and global 5297 flags are flags are 5298 specified. specified. 5299 - Must happen after 5300 any preceding 5301 local/generic 5302 load/load 5303 atomic/store/store 5304 atomic/atomicrmw. 5305 - Could be split into 5306 separate s_waitcnt 5307 vmcnt(0), s_waitcnt 5308 vscnt(0) and s_waitcnt 5309 lgkmcnt(0) to allow 5310 them to be 5311 independently moved 5312 according to the 5313 following rules. 5314 - s_waitcnt vmcnt(0) 5315 must happen after 5316 any preceding 5317 global/generic 5318 load/load 5319 atomic/ 5320 atomicrmw-with-return-value. 5321 - s_waitcnt vscnt(0) 5322 must happen after 5323 any preceding 5324 global/generic 5325 store/store atomic/ 5326 atomicrmw-no-return-value. 5327 - s_waitcnt lgkmcnt(0) 5328 must happen after 5329 any preceding 5330 local/generic 5331 load/store/load 5332 atomic/store atomic/ 5333 atomicrmw. 5334 - Must happen before - Must happen before 5335 any following store any following store 5336 atomic/atomicrmw atomic/atomicrmw 5337 with an equal or with an equal or 5338 wider sync scope wider sync scope 5339 and memory ordering and memory ordering 5340 stronger than stronger than 5341 unordered (this is unordered (this is 5342 termed the termed the 5343 fence-paired-atomic). fence-paired-atomic). 5344 - Ensures that all - Ensures that all 5345 memory operations memory operations 5346 to local have have 5347 completed before completed before 5348 performing the performing the 5349 following following 5350 fence-paired-atomic. fence-paired-atomic. 5351 5352 fence release - agent *none* 1. s_waitcnt lgkmcnt(0) & 1. s_waitcnt lgkmcnt(0) & 5353 - system vmcnt(0) vmcnt(0) & vscnt(0) 5354 5355 - If OpenCL and - If OpenCL and 5356 address space is address space is 5357 not generic, omit not generic, omit 5358 lgkmcnt(0). lgkmcnt(0). 5359 - If OpenCL and - If OpenCL and 5360 address space is address space is 5361 local, omit local, omit 5362 vmcnt(0). vmcnt(0) and vscnt(0). 5363 - However, since LLVM - However, since LLVM 5364 currently has no currently has no 5365 address space on address space on 5366 the fence need to the fence need to 5367 conservatively conservatively 5368 always generate. If always generate. If 5369 fence had an fence had an 5370 address space then address space then 5371 set to address set to address 5372 space of OpenCL space of OpenCL 5373 fence flag, or to fence flag, or to 5374 generic if both generic if both 5375 local and global local and global 5376 flags are flags are 5377 specified. specified. 5378 - Could be split into - Could be split into 5379 separate s_waitcnt separate s_waitcnt 5380 vmcnt(0) and vmcnt(0), s_waitcnt 5381 s_waitcnt vscnt(0) and s_waitcnt 5382 lgkmcnt(0) to allow lgkmcnt(0) to allow 5383 them to be them to be 5384 independently moved independently moved 5385 according to the according to the 5386 following rules. following rules. 5387 - s_waitcnt vmcnt(0) - s_waitcnt vmcnt(0) 5388 must happen after must happen after 5389 any preceding any preceding 5390 global/generic global/generic 5391 load/store/load load/load atomic/ 5392 atomic/store atomicrmw-with-return-value. 5393 atomic/atomicrmw. 5394 - s_waitcnt vscnt(0) 5395 must happen after 5396 any preceding 5397 global/generic 5398 store/store atomic/ 5399 atomicrmw-no-return-value. 5400 - s_waitcnt lgkmcnt(0) - s_waitcnt lgkmcnt(0) 5401 must happen after must happen after 5402 any preceding any preceding 5403 local/generic local/generic 5404 load/store/load load/store/load 5405 atomic/store atomic/store 5406 atomic/atomicrmw. atomic/atomicrmw. 5407 - Must happen before - Must happen before 5408 any following store any following store 5409 atomic/atomicrmw atomic/atomicrmw 5410 with an equal or with an equal or 5411 wider sync scope wider sync scope 5412 and memory ordering and memory ordering 5413 stronger than stronger than 5414 unordered (this is unordered (this is 5415 termed the termed the 5416 fence-paired-atomic). fence-paired-atomic). 5417 - Ensures that all - Ensures that all 5418 memory operations memory operations 5419 have have 5420 completed before completed before 5421 performing the performing the 5422 following following 5423 fence-paired-atomic. fence-paired-atomic. 5424 5425 **Acquire-Release Atomic** 5426 ---------------------------------------------------------------------------------------------------------------------- 5427 atomicrmw acq_rel - singlethread - global 1. buffer/global/ds/flat_atomic 1. buffer/global/ds/flat_atomic 5428 - wavefront - local 5429 - generic 5430 atomicrmw acq_rel - workgroup - global 1. s_waitcnt lgkmcnt(0) 1. s_waitcnt lgkmcnt(0) & 5431 vmcnt(0) & vscnt(0) 5432 5433 - If CU wavefront execution mode, omit vmcnt and 5434 vscnt. 5435 - If OpenCL, omit. - If OpenCL, omit 5436 s_waitcnt lgkmcnt(0). 5437 - Must happen after - Must happen after 5438 any preceding any preceding 5439 local/generic local/generic 5440 load/store/load load/store/load 5441 atomic/store atomic/store 5442 atomic/atomicrmw. atomic/atomicrmw. 5443 - Could be split into 5444 separate s_waitcnt 5445 vmcnt(0), s_waitcnt 5446 vscnt(0) and s_waitcnt 5447 lgkmcnt(0) to allow 5448 them to be 5449 independently moved 5450 according to the 5451 following rules. 5452 - s_waitcnt vmcnt(0) 5453 must happen after 5454 any preceding 5455 global/generic load/load 5456 atomic/ 5457 atomicrmw-with-return-value. 5458 - s_waitcnt vscnt(0) 5459 must happen after 5460 any preceding 5461 global/generic 5462 store/store 5463 atomic/ 5464 atomicrmw-no-return-value. 5465 - s_waitcnt lgkmcnt(0) 5466 must happen after 5467 any preceding 5468 local/generic load/store/load 5469 atomic/store atomic/atomicrmw. 5470 - Must happen before - Must happen before 5471 the following the following 5472 atomicrmw. atomicrmw. 5473 - Ensures that all - Ensures that all 5474 memory operations memory operations 5475 to local have have 5476 completed before completed before 5477 performing the performing the 5478 atomicrmw that is atomicrmw that is 5479 being released. being released. 5480 5481 2. buffer/global/flat_atomic 2. buffer/global_atomic 5482 3. s_waitcnt vm/vscnt(0) 5483 5484 - If CU wavefront execution mode, omit vm/vscnt. 5485 - Use vmcnt if atomic with 5486 return and vscnt if atomic 5487 with no-return. 5488 waitcnt lgkmcnt(0). 5489 - Must happen before 5490 the following 5491 buffer_gl0_inv. 5492 - Ensures any 5493 following global 5494 data read is no 5495 older than the 5496 atomicrmw value 5497 being acquired. 5498 5499 4. buffer_gl0_inv 5500 5501 - If CU wavefront execution mode, omit. 5502 - Ensures that 5503 following 5504 loads will not see 5505 stale data. 5506 5507 atomicrmw acq_rel - workgroup - local 1. waitcnt vmcnt(0) & vscnt(0) 5508 5509 - If CU wavefront execution mode, omit. 5510 - If OpenCL, omit. 5511 - Could be split into 5512 separate s_waitcnt 5513 vmcnt(0) and s_waitcnt 5514 vscnt(0) to allow 5515 them to be 5516 independently moved 5517 according to the 5518 following rules. 5519 - s_waitcnt vmcnt(0) 5520 must happen after 5521 any preceding 5522 global/generic load/load 5523 atomic/ 5524 atomicrmw-with-return-value. 5525 - s_waitcnt vscnt(0) 5526 must happen after 5527 any preceding 5528 global/generic 5529 store/store atomic/ 5530 atomicrmw-no-return-value. 5531 - Must happen before 5532 the following 5533 store. 5534 - Ensures that all 5535 global memory 5536 operations have 5537 completed before 5538 performing the 5539 store that is being 5540 released. 5541 5542 1. ds_atomic 2. ds_atomic 5543 2. s_waitcnt lgkmcnt(0) 3. s_waitcnt lgkmcnt(0) 5544 5545 - If OpenCL, omit. - If OpenCL, omit. 5546 - Must happen before - Must happen before 5547 any following the following 5548 global/generic buffer_gl0_inv. 5549 load/load 5550 atomic/store/store 5551 atomic/atomicrmw. 5552 - Ensures any - Ensures any 5553 following global following global 5554 data read is no data read is no 5555 older than the load older than the load 5556 atomic value being atomic value being 5557 acquired. acquired. 5558 5559 4. buffer_gl0_inv 5560 5561 - If CU wavefront execution mode, omit. 5562 - If OpenCL omit. 5563 - Ensures that 5564 following 5565 loads will not see 5566 stale data. 5567 5568 atomicrmw acq_rel - workgroup - generic 1. s_waitcnt lgkmcnt(0) 1. s_waitcnt lgkmcnt(0) & 5569 vmcnt(0) & vscnt(0) 5570 5571 - If CU wavefront execution mode, omit vmcnt and 5572 vscnt. 5573 - If OpenCL, omit. - If OpenCL, omit 5574 waitcnt lgkmcnt(0). 5575 - Must happen after 5576 any preceding 5577 local/generic 5578 load/store/load 5579 atomic/store 5580 atomic/atomicrmw. 5581 - Could be split into 5582 separate s_waitcnt 5583 vmcnt(0), s_waitcnt 5584 vscnt(0) and s_waitcnt 5585 lgkmcnt(0) to allow 5586 them to be 5587 independently moved 5588 according to the 5589 following rules. 5590 - s_waitcnt vmcnt(0) 5591 must happen after 5592 any preceding 5593 global/generic load/load 5594 atomic/ 5595 atomicrmw-with-return-value. 5596 - s_waitcnt vscnt(0) 5597 must happen after 5598 any preceding 5599 global/generic 5600 store/store 5601 atomic/ 5602 atomicrmw-no-return-value. 5603 - s_waitcnt lgkmcnt(0) 5604 must happen after 5605 any preceding 5606 local/generic load/store/load 5607 atomic/store atomic/atomicrmw. 5608 - Must happen before - Must happen before 5609 the following the following 5610 atomicrmw. atomicrmw. 5611 - Ensures that all - Ensures that all 5612 memory operations memory operations 5613 to local have have 5614 completed before completed before 5615 performing the performing the 5616 atomicrmw that is atomicrmw that is 5617 being released. being released. 5618 5619 2. flat_atomic 2. flat_atomic 5620 3. s_waitcnt lgkmcnt(0) 3. s_waitcnt lgkmcnt(0) & 5621 vm/vscnt(0) 5622 5623 - If CU wavefront execution mode, omit vm/vscnt. 5624 - If OpenCL, omit. - If OpenCL, omit 5625 waitcnt lgkmcnt(0). 5626 - Must happen before - Must happen before 5627 any following the following 5628 global/generic buffer_gl0_inv. 5629 load/load 5630 atomic/store/store 5631 atomic/atomicrmw. 5632 - Ensures any - Ensures any 5633 following global following global 5634 data read is no data read is no 5635 older than the load older than the load 5636 atomic value being atomic value being 5637 acquired. acquired. 5638 5639 3. buffer_gl0_inv 5640 5641 - If CU wavefront execution mode, omit. 5642 - Ensures that 5643 following 5644 loads will not see 5645 stale data. 5646 5647 atomicrmw acq_rel - agent - global 1. s_waitcnt lgkmcnt(0) & 1. s_waitcnt lgkmcnt(0) & 5648 - system vmcnt(0) vmcnt(0) & vscnt(0) 5649 5650 - If OpenCL, omit - If OpenCL, omit 5651 lgkmcnt(0). lgkmcnt(0). 5652 - Could be split into - Could be split into 5653 separate s_waitcnt separate s_waitcnt 5654 vmcnt(0) and vmcnt(0), s_waitcnt 5655 s_waitcnt vscnt(0) and s_waitcnt 5656 lgkmcnt(0) to allow lgkmcnt(0) to allow 5657 them to be them to be 5658 independently moved independently moved 5659 according to the according to the 5660 following rules. following rules. 5661 - s_waitcnt vmcnt(0) - s_waitcnt vmcnt(0) 5662 must happen after must happen after 5663 any preceding any preceding 5664 global/generic global/generic 5665 load/store/load load/load atomic/ 5666 atomic/store atomicrmw-with-return-value. 5667 atomic/atomicrmw. 5668 - s_waitcnt vscnt(0) 5669 must happen after 5670 any preceding 5671 global/generic 5672 store/store atomic/ 5673 atomicrmw-no-return-value. 5674 - s_waitcnt lgkmcnt(0) - s_waitcnt lgkmcnt(0) 5675 must happen after must happen after 5676 any preceding any preceding 5677 local/generic local/generic 5678 load/store/load load/store/load 5679 atomic/store atomic/store 5680 atomic/atomicrmw. atomic/atomicrmw. 5681 - Must happen before - Must happen before 5682 the following the following 5683 atomicrmw. atomicrmw. 5684 - Ensures that all - Ensures that all 5685 memory operations memory operations 5686 to global have to global have 5687 completed before completed before 5688 performing the performing the 5689 atomicrmw that is atomicrmw that is 5690 being released. being released. 5691 5692 2. buffer/global/flat_atomic 2. buffer/global_atomic 5693 3. s_waitcnt vmcnt(0) 3. s_waitcnt vm/vscnt(0) 5694 5695 - Use vmcnt if atomic with 5696 return and vscnt if atomic 5697 with no-return. 5698 waitcnt lgkmcnt(0). 5699 - Must happen before - Must happen before 5700 following following 5701 buffer_wbinvl1_vol. buffer_gl*_inv. 5702 - Ensures the - Ensures the 5703 atomicrmw has atomicrmw has 5704 completed before completed before 5705 invalidating the invalidating the 5706 cache. caches. 5707 5708 4. buffer_wbinvl1_vol 4. buffer_gl0_inv; 5709 buffer_gl1_inv 5710 5711 - Must happen before - Must happen before 5712 any following any following 5713 global/generic global/generic 5714 load/load load/load 5715 atomic/atomicrmw. atomic/atomicrmw. 5716 - Ensures that - Ensures that 5717 following loads following loads 5718 will not see stale will not see stale 5719 global data. global data. 5720 5721 atomicrmw acq_rel - agent - generic 1. s_waitcnt lgkmcnt(0) & 1. s_waitcnt lgkmcnt(0) & 5722 - system vmcnt(0) vmcnt(0) & vscnt(0) 5723 5724 - If OpenCL, omit - If OpenCL, omit 5725 lgkmcnt(0). lgkmcnt(0). 5726 - Could be split into - Could be split into 5727 separate s_waitcnt separate s_waitcnt 5728 vmcnt(0) and vmcnt(0), s_waitcnt 5729 s_waitcnt vscnt(0) and s_waitcnt 5730 lgkmcnt(0) to allow lgkmcnt(0) to allow 5731 them to be them to be 5732 independently moved independently moved 5733 according to the according to the 5734 following rules. following rules. 5735 - s_waitcnt vmcnt(0) - s_waitcnt vmcnt(0) 5736 must happen after must happen after 5737 any preceding any preceding 5738 global/generic global/generic 5739 load/store/load load/load atomic 5740 atomic/store atomicrmw-with-return-value. 5741 atomic/atomicrmw. 5742 - s_waitcnt vscnt(0) 5743 must happen after 5744 any preceding 5745 global/generic 5746 store/store atomic/ 5747 atomicrmw-no-return-value. 5748 - s_waitcnt lgkmcnt(0) - s_waitcnt lgkmcnt(0) 5749 must happen after must happen after 5750 any preceding any preceding 5751 local/generic local/generic 5752 load/store/load load/store/load 5753 atomic/store atomic/store 5754 atomic/atomicrmw. atomic/atomicrmw. 5755 - Must happen before - Must happen before 5756 the following the following 5757 atomicrmw. atomicrmw. 5758 - Ensures that all - Ensures that all 5759 memory operations memory operations 5760 to global have have 5761 completed before completed before 5762 performing the performing the 5763 atomicrmw that is atomicrmw that is 5764 being released. being released. 5765 5766 2. flat_atomic 2. flat_atomic 5767 3. s_waitcnt vmcnt(0) & 3. s_waitcnt vm/vscnt(0) & 5768 lgkmcnt(0) lgkmcnt(0) 5769 5770 - If OpenCL, omit - If OpenCL, omit 5771 lgkmcnt(0). lgkmcnt(0). 5772 - Use vmcnt if atomic with 5773 return and vscnt if atomic 5774 with no-return. 5775 - Must happen before - Must happen before 5776 following following 5777 buffer_wbinvl1_vol. buffer_gl*_inv. 5778 - Ensures the - Ensures the 5779 atomicrmw has atomicrmw has 5780 completed before completed before 5781 invalidating the invalidating the 5782 cache. caches. 5783 5784 4. buffer_wbinvl1_vol 4. buffer_gl0_inv; 5785 buffer_gl1_inv 5786 5787 - Must happen before - Must happen before 5788 any following any following 5789 global/generic global/generic 5790 load/load load/load 5791 atomic/atomicrmw. atomic/atomicrmw. 5792 - Ensures that - Ensures that 5793 following loads following loads 5794 will not see stale will not see stale 5795 global data. global data. 5796 5797 fence acq_rel - singlethread *none* *none* *none* 5798 - wavefront 5799 fence acq_rel - workgroup *none* 1. s_waitcnt lgkmcnt(0) 1. s_waitcnt lgkmcnt(0) & 5800 vmcnt(0) & vscnt(0) 5801 5802 - If CU wavefront execution mode, omit vmcnt and 5803 vscnt. 5804 - If OpenCL and - If OpenCL and 5805 address space is address space is 5806 not generic, omit. not generic, omit 5807 lgkmcnt(0). 5808 - If OpenCL and 5809 address space is 5810 local, omit 5811 vmcnt(0) and vscnt(0). 5812 - However, - However, 5813 since LLVM since LLVM 5814 currently has no currently has no 5815 address space on address space on 5816 the fence need to the fence need to 5817 conservatively conservatively 5818 always generate always generate 5819 (see comment for (see comment for 5820 previous fence). previous fence). 5821 - Must happen after 5822 any preceding 5823 local/generic 5824 load/load 5825 atomic/store/store 5826 atomic/atomicrmw. 5827 - Could be split into 5828 separate s_waitcnt 5829 vmcnt(0), s_waitcnt 5830 vscnt(0) and s_waitcnt 5831 lgkmcnt(0) to allow 5832 them to be 5833 independently moved 5834 according to the 5835 following rules. 5836 - s_waitcnt vmcnt(0) 5837 must happen after 5838 any preceding 5839 global/generic 5840 load/load 5841 atomic/ 5842 atomicrmw-with-return-value. 5843 - s_waitcnt vscnt(0) 5844 must happen after 5845 any preceding 5846 global/generic 5847 store/store atomic/ 5848 atomicrmw-no-return-value. 5849 - s_waitcnt lgkmcnt(0) 5850 must happen after 5851 any preceding 5852 local/generic 5853 load/store/load 5854 atomic/store atomic/ 5855 atomicrmw. 5856 - Must happen before - Must happen before 5857 any following any following 5858 global/generic global/generic 5859 load/load load/load 5860 atomic/store/store atomic/store/store 5861 atomic/atomicrmw. atomic/atomicrmw. 5862 - Ensures that all - Ensures that all 5863 memory operations memory operations 5864 to local have have 5865 completed before completed before 5866 performing any performing any 5867 following global following global 5868 memory operations. memory operations. 5869 - Ensures that the - Ensures that the 5870 preceding preceding 5871 local/generic load local/generic load 5872 atomic/atomicrmw atomic/atomicrmw 5873 with an equal or with an equal or 5874 wider sync scope wider sync scope 5875 and memory ordering and memory ordering 5876 stronger than stronger than 5877 unordered (this is unordered (this is 5878 termed the termed the 5879 acquire-fence-paired-atomic acquire-fence-paired-atomic 5880 ) has completed ) has completed 5881 before following before following 5882 global memory global memory 5883 operations. This operations. This 5884 satisfies the satisfies the 5885 requirements of requirements of 5886 acquire. acquire. 5887 - Ensures that all - Ensures that all 5888 previous memory previous memory 5889 operations have operations have 5890 completed before a completed before a 5891 following following 5892 local/generic store local/generic store 5893 atomic/atomicrmw atomic/atomicrmw 5894 with an equal or with an equal or 5895 wider sync scope wider sync scope 5896 and memory ordering and memory ordering 5897 stronger than stronger than 5898 unordered (this is unordered (this is 5899 termed the termed the 5900 release-fence-paired-atomic release-fence-paired-atomic 5901 ). This satisfies the ). This satisfies the 5902 requirements of requirements of 5903 release. release. 5904 - Must happen before 5905 the following 5906 buffer_gl0_inv. 5907 - Ensures that the 5908 acquire-fence-paired 5909 atomic has completed 5910 before invalidating 5911 the 5912 cache. Therefore 5913 any following 5914 locations read must 5915 be no older than 5916 the value read by 5917 the 5918 acquire-fence-paired-atomic. 5919 5920 3. buffer_gl0_inv 5921 5922 - If CU wavefront execution mode, omit. 5923 - Ensures that 5924 following 5925 loads will not see 5926 stale data. 5927 5928 fence acq_rel - agent *none* 1. s_waitcnt lgkmcnt(0) & 1. s_waitcnt lgkmcnt(0) & 5929 - system vmcnt(0) vmcnt(0) & vscnt(0) 5930 5931 - If OpenCL and - If OpenCL and 5932 address space is address space is 5933 not generic, omit not generic, omit 5934 lgkmcnt(0). lgkmcnt(0). 5935 - If OpenCL and 5936 address space is 5937 local, omit 5938 vmcnt(0) and vscnt(0). 5939 - However, since LLVM - However, since LLVM 5940 currently has no currently has no 5941 address space on address space on 5942 the fence need to the fence need to 5943 conservatively conservatively 5944 always generate always generate 5945 (see comment for (see comment for 5946 previous fence). previous fence). 5947 - Could be split into - Could be split into 5948 separate s_waitcnt separate s_waitcnt 5949 vmcnt(0) and vmcnt(0), s_waitcnt 5950 s_waitcnt vscnt(0) and s_waitcnt 5951 lgkmcnt(0) to allow lgkmcnt(0) to allow 5952 them to be them to be 5953 independently moved independently moved 5954 according to the according to the 5955 following rules. following rules. 5956 - s_waitcnt vmcnt(0) - s_waitcnt vmcnt(0) 5957 must happen after must happen after 5958 any preceding any preceding 5959 global/generic global/generic 5960 load/store/load load/load 5961 atomic/store atomic/ 5962 atomic/atomicrmw. atomicrmw-with-return-value. 5963 - s_waitcnt vscnt(0) 5964 must happen after 5965 any preceding 5966 global/generic 5967 store/store atomic/ 5968 atomicrmw-no-return-value. 5969 - s_waitcnt lgkmcnt(0) - s_waitcnt lgkmcnt(0) 5970 must happen after must happen after 5971 any preceding any preceding 5972 local/generic local/generic 5973 load/store/load load/store/load 5974 atomic/store atomic/store 5975 atomic/atomicrmw. atomic/atomicrmw. 5976 - Must happen before - Must happen before 5977 the following the following 5978 buffer_wbinvl1_vol. buffer_gl*_inv. 5979 - Ensures that the - Ensures that the 5980 preceding preceding 5981 global/local/generic global/local/generic 5982 load load 5983 atomic/atomicrmw atomic/atomicrmw 5984 with an equal or with an equal or 5985 wider sync scope wider sync scope 5986 and memory ordering and memory ordering 5987 stronger than stronger than 5988 unordered (this is unordered (this is 5989 termed the termed the 5990 acquire-fence-paired-atomic acquire-fence-paired-atomic 5991 ) has completed ) has completed 5992 before invalidating before invalidating 5993 the cache. This the caches. This 5994 satisfies the satisfies the 5995 requirements of requirements of 5996 acquire. acquire. 5997 - Ensures that all - Ensures that all 5998 previous memory previous memory 5999 operations have operations have 6000 completed before a completed before a 6001 following following 6002 global/local/generic global/local/generic 6003 store store 6004 atomic/atomicrmw atomic/atomicrmw 6005 with an equal or with an equal or 6006 wider sync scope wider sync scope 6007 and memory ordering and memory ordering 6008 stronger than stronger than 6009 unordered (this is unordered (this is 6010 termed the termed the 6011 release-fence-paired-atomic release-fence-paired-atomic 6012 ). This satisfies the ). This satisfies the 6013 requirements of requirements of 6014 release. release. 6015 6016 2. buffer_wbinvl1_vol 2. buffer_gl0_inv; 6017 buffer_gl1_inv 6018 6019 - Must happen before - Must happen before 6020 any following any following 6021 global/generic global/generic 6022 load/load load/load 6023 atomic/store/store atomic/store/store 6024 atomic/atomicrmw. atomic/atomicrmw. 6025 - Ensures that - Ensures that 6026 following loads following loads 6027 will not see stale will not see stale 6028 global data. This global data. This 6029 satisfies the satisfies the 6030 requirements of requirements of 6031 acquire. acquire. 6032 6033 **Sequential Consistent Atomic** 6034 ---------------------------------------------------------------------------------------------------------------------- 6035 load atomic seq_cst - singlethread - global *Same as corresponding *Same as corresponding 6036 - wavefront - local load atomic acquire, load atomic acquire, 6037 - generic except must generated except must generated 6038 all instructions even all instructions even 6039 for OpenCL.* for OpenCL.* 6040 load atomic seq_cst - workgroup - global 1. s_waitcnt lgkmcnt(0) 1. s_waitcnt lgkmcnt(0) & 6041 - generic vmcnt(0) & vscnt(0) 6042 6043 - If CU wavefront execution mode, omit vmcnt and 6044 vscnt. 6045 - Could be split into 6046 separate s_waitcnt 6047 vmcnt(0), s_waitcnt 6048 vscnt(0) and s_waitcnt 6049 lgkmcnt(0) to allow 6050 them to be 6051 independently moved 6052 according to the 6053 following rules. 6054 - Must - waitcnt lgkmcnt(0) must 6055 happen after happen after 6056 preceding preceding 6057 global/generic load local load 6058 atomic/store atomic/store 6059 atomic/atomicrmw atomic/atomicrmw 6060 with memory with memory 6061 ordering of seq_cst ordering of seq_cst 6062 and with equal or and with equal or 6063 wider sync scope. wider sync scope. 6064 (Note that seq_cst (Note that seq_cst 6065 fences have their fences have their 6066 own s_waitcnt own s_waitcnt 6067 lgkmcnt(0) and so do lgkmcnt(0) and so do 6068 not need to be not need to be 6069 considered.) considered.) 6070 - waitcnt vmcnt(0) 6071 Must happen after 6072 preceding 6073 global/generic load 6074 atomic/ 6075 atomicrmw-with-return-value 6076 with memory 6077 ordering of seq_cst 6078 and with equal or 6079 wider sync scope. 6080 (Note that seq_cst 6081 fences have their 6082 own s_waitcnt 6083 vmcnt(0) and so do 6084 not need to be 6085 considered.) 6086 - waitcnt vscnt(0) 6087 Must happen after 6088 preceding 6089 global/generic store 6090 atomic/ 6091 atomicrmw-no-return-value 6092 with memory 6093 ordering of seq_cst 6094 and with equal or 6095 wider sync scope. 6096 (Note that seq_cst 6097 fences have their 6098 own s_waitcnt 6099 vscnt(0) and so do 6100 not need to be 6101 considered.) 6102 - Ensures any - Ensures any 6103 preceding preceding 6104 sequential sequential 6105 consistent local consistent global/local 6106 memory instructions memory instructions 6107 have completed have completed 6108 before executing before executing 6109 this sequentially this sequentially 6110 consistent consistent 6111 instruction. This instruction. This 6112 prevents reordering prevents reordering 6113 a seq_cst store a seq_cst store 6114 followed by a followed by a 6115 seq_cst load. (Note seq_cst load. (Note 6116 that seq_cst is that seq_cst is 6117 stronger than stronger than 6118 acquire/release as acquire/release as 6119 the reordering of the reordering of 6120 load acquire load acquire 6121 followed by a store followed by a store 6122 release is release is 6123 prevented by the prevented by the 6124 waitcnt of waitcnt of 6125 the release, but the release, but 6126 there is nothing there is nothing 6127 preventing a store preventing a store 6128 release followed by release followed by 6129 load acquire from load acquire from 6130 competing out of competing out of 6131 order.) order.) 6132 6133 2. *Following 2. *Following 6134 instructions same as instructions same as 6135 corresponding load corresponding load 6136 atomic acquire, atomic acquire, 6137 except must generated except must generated 6138 all instructions even all instructions even 6139 for OpenCL.* for OpenCL.* 6140 load atomic seq_cst - workgroup - local *Same as corresponding 6141 load atomic acquire, 6142 except must generated 6143 all instructions even 6144 for OpenCL.* 6145 6146 1. s_waitcnt vmcnt(0) & vscnt(0) 6147 6148 - If CU wavefront execution mode, omit. 6149 - Could be split into 6150 separate s_waitcnt 6151 vmcnt(0) and s_waitcnt 6152 vscnt(0) to allow 6153 them to be 6154 independently moved 6155 according to the 6156 following rules. 6157 - waitcnt vmcnt(0) 6158 Must happen after 6159 preceding 6160 global/generic load 6161 atomic/ 6162 atomicrmw-with-return-value 6163 with memory 6164 ordering of seq_cst 6165 and with equal or 6166 wider sync scope. 6167 (Note that seq_cst 6168 fences have their 6169 own s_waitcnt 6170 vmcnt(0) and so do 6171 not need to be 6172 considered.) 6173 - waitcnt vscnt(0) 6174 Must happen after 6175 preceding 6176 global/generic store 6177 atomic/ 6178 atomicrmw-no-return-value 6179 with memory 6180 ordering of seq_cst 6181 and with equal or 6182 wider sync scope. 6183 (Note that seq_cst 6184 fences have their 6185 own s_waitcnt 6186 vscnt(0) and so do 6187 not need to be 6188 considered.) 6189 - Ensures any 6190 preceding 6191 sequential 6192 consistent global 6193 memory instructions 6194 have completed 6195 before executing 6196 this sequentially 6197 consistent 6198 instruction. This 6199 prevents reordering 6200 a seq_cst store 6201 followed by a 6202 seq_cst load. (Note 6203 that seq_cst is 6204 stronger than 6205 acquire/release as 6206 the reordering of 6207 load acquire 6208 followed by a store 6209 release is 6210 prevented by the 6211 waitcnt of 6212 the release, but 6213 there is nothing 6214 preventing a store 6215 release followed by 6216 load acquire from 6217 competing out of 6218 order.) 6219 6220 2. *Following 6221 instructions same as 6222 corresponding load 6223 atomic acquire, 6224 except must generated 6225 all instructions even 6226 for OpenCL.* 6227 6228 load atomic seq_cst - agent - global 1. s_waitcnt lgkmcnt(0) & 1. s_waitcnt lgkmcnt(0) & 6229 - system - generic vmcnt(0) vmcnt(0) & vscnt(0) 6230 6231 - Could be split into - Could be split into 6232 separate s_waitcnt separate s_waitcnt 6233 vmcnt(0) vmcnt(0), s_waitcnt 6234 and s_waitcnt vscnt(0) and s_waitcnt 6235 lgkmcnt(0) to allow lgkmcnt(0) to allow 6236 them to be them to be 6237 independently moved independently moved 6238 according to the according to the 6239 following rules. following rules. 6240 - waitcnt lgkmcnt(0) - waitcnt lgkmcnt(0) 6241 must happen after must happen after 6242 preceding preceding 6243 global/generic load local load 6244 atomic/store atomic/store 6245 atomic/atomicrmw atomic/atomicrmw 6246 with memory with memory 6247 ordering of seq_cst ordering of seq_cst 6248 and with equal or and with equal or 6249 wider sync scope. wider sync scope. 6250 (Note that seq_cst (Note that seq_cst 6251 fences have their fences have their 6252 own s_waitcnt own s_waitcnt 6253 lgkmcnt(0) and so do lgkmcnt(0) and so do 6254 not need to be not need to be 6255 considered.) considered.) 6256 - waitcnt vmcnt(0) - waitcnt vmcnt(0) 6257 must happen after must happen after 6258 preceding preceding 6259 global/generic load global/generic load 6260 atomic/store atomic/ 6261 atomic/atomicrmw atomicrmw-with-return-value 6262 with memory with memory 6263 ordering of seq_cst ordering of seq_cst 6264 and with equal or and with equal or 6265 wider sync scope. wider sync scope. 6266 (Note that seq_cst (Note that seq_cst 6267 fences have their fences have their 6268 own s_waitcnt own s_waitcnt 6269 vmcnt(0) and so do vmcnt(0) and so do 6270 not need to be not need to be 6271 considered.) considered.) 6272 - waitcnt vscnt(0) 6273 Must happen after 6274 preceding 6275 global/generic store 6276 atomic/ 6277 atomicrmw-no-return-value 6278 with memory 6279 ordering of seq_cst 6280 and with equal or 6281 wider sync scope. 6282 (Note that seq_cst 6283 fences have their 6284 own s_waitcnt 6285 vscnt(0) and so do 6286 not need to be 6287 considered.) 6288 - Ensures any - Ensures any 6289 preceding preceding 6290 sequential sequential 6291 consistent global consistent global 6292 memory instructions memory instructions 6293 have completed have completed 6294 before executing before executing 6295 this sequentially this sequentially 6296 consistent consistent 6297 instruction. This instruction. This 6298 prevents reordering prevents reordering 6299 a seq_cst store a seq_cst store 6300 followed by a followed by a 6301 seq_cst load. (Note seq_cst load. (Note 6302 that seq_cst is that seq_cst is 6303 stronger than stronger than 6304 acquire/release as acquire/release as 6305 the reordering of the reordering of 6306 load acquire load acquire 6307 followed by a store followed by a store 6308 release is release is 6309 prevented by the prevented by the 6310 waitcnt of waitcnt of 6311 the release, but the release, but 6312 there is nothing there is nothing 6313 preventing a store preventing a store 6314 release followed by release followed by 6315 load acquire from load acquire from 6316 competing out of competing out of 6317 order.) order.) 6318 6319 2. *Following 2. *Following 6320 instructions same as instructions same as 6321 corresponding load corresponding load 6322 atomic acquire, atomic acquire, 6323 except must generated except must generated 6324 all instructions even all instructions even 6325 for OpenCL.* for OpenCL.* 6326 store atomic seq_cst - singlethread - global *Same as corresponding *Same as corresponding 6327 - wavefront - local store atomic release, store atomic release, 6328 - workgroup - generic except must generated except must generated 6329 all instructions even all instructions even 6330 for OpenCL.* for OpenCL.* 6331 store atomic seq_cst - agent - global *Same as corresponding *Same as corresponding 6332 - system - generic store atomic release, store atomic release, 6333 except must generated except must generated 6334 all instructions even all instructions even 6335 for OpenCL.* for OpenCL.* 6336 atomicrmw seq_cst - singlethread - global *Same as corresponding *Same as corresponding 6337 - wavefront - local atomicrmw acq_rel, atomicrmw acq_rel, 6338 - workgroup - generic except must generated except must generated 6339 all instructions even all instructions even 6340 for OpenCL.* for OpenCL.* 6341 atomicrmw seq_cst - agent - global *Same as corresponding *Same as corresponding 6342 - system - generic atomicrmw acq_rel, atomicrmw acq_rel, 6343 except must generated except must generated 6344 all instructions even all instructions even 6345 for OpenCL.* for OpenCL.* 6346 fence seq_cst - singlethread *none* *Same as corresponding *Same as corresponding 6347 - wavefront fence acq_rel, fence acq_rel, 6348 - workgroup except must generated except must generated 6349 - agent all instructions even all instructions even 6350 - system for OpenCL.* for OpenCL.* 6351 ============ ============ ============== ========== =============================== ================================== 6352 6353The memory order also adds the single thread optimization constrains defined in 6354table 6355:ref:`amdgpu-amdhsa-memory-model-single-thread-optimization-constraints-gfx6-gfx10-table`. 6356 6357 .. table:: AMDHSA Memory Model Single Thread Optimization Constraints GFX6-GFX10 6358 :name: amdgpu-amdhsa-memory-model-single-thread-optimization-constraints-gfx6-gfx10-table 6359 6360 ============ ============================================================== 6361 LLVM Memory Optimization Constraints 6362 Ordering 6363 ============ ============================================================== 6364 unordered *none* 6365 monotonic *none* 6366 acquire - If a load atomic/atomicrmw then no following load/load 6367 atomic/store/ store atomic/atomicrmw/fence instruction can 6368 be moved before the acquire. 6369 - If a fence then same as load atomic, plus no preceding 6370 associated fence-paired-atomic can be moved after the fence. 6371 release - If a store atomic/atomicrmw then no preceding load/load 6372 atomic/store/ store atomic/atomicrmw/fence instruction can 6373 be moved after the release. 6374 - If a fence then same as store atomic, plus no following 6375 associated fence-paired-atomic can be moved before the 6376 fence. 6377 acq_rel Same constraints as both acquire and release. 6378 seq_cst - If a load atomic then same constraints as acquire, plus no 6379 preceding sequentially consistent load atomic/store 6380 atomic/atomicrmw/fence instruction can be moved after the 6381 seq_cst. 6382 - If a store atomic then the same constraints as release, plus 6383 no following sequentially consistent load atomic/store 6384 atomic/atomicrmw/fence instruction can be moved before the 6385 seq_cst. 6386 - If an atomicrmw/fence then same constraints as acq_rel. 6387 ============ ============================================================== 6388 6389Trap Handler ABI 6390~~~~~~~~~~~~~~~~ 6391 6392For code objects generated by AMDGPU backend for HSA [HSA]_ compatible runtimes 6393(such as ROCm [AMD-ROCm]_), the runtime installs a trap handler that supports 6394the ``s_trap`` instruction with the following usage: 6395 6396 .. table:: AMDGPU Trap Handler for AMDHSA OS 6397 :name: amdgpu-trap-handler-for-amdhsa-os-table 6398 6399 =================== =============== =============== ======================= 6400 Usage Code Sequence Trap Handler Description 6401 Inputs 6402 =================== =============== =============== ======================= 6403 reserved ``s_trap 0x00`` Reserved by hardware. 6404 ``debugtrap(arg)`` ``s_trap 0x01`` ``SGPR0-1``: Reserved for HSA 6405 ``queue_ptr`` ``debugtrap`` 6406 ``VGPR0``: intrinsic (not 6407 ``arg`` implemented). 6408 ``llvm.trap`` ``s_trap 0x02`` ``SGPR0-1``: Causes dispatch to be 6409 ``queue_ptr`` terminated and its 6410 associated queue put 6411 into the error state. 6412 ``llvm.debugtrap`` ``s_trap 0x03`` - If debugger not 6413 installed then 6414 behaves as a 6415 no-operation. The 6416 trap handler is 6417 entered and 6418 immediately returns 6419 to continue 6420 execution of the 6421 wavefront. 6422 - If the debugger is 6423 installed, causes 6424 the debug trap to be 6425 reported by the 6426 debugger and the 6427 wavefront is put in 6428 the halt state until 6429 resumed by the 6430 debugger. 6431 reserved ``s_trap 0x04`` Reserved. 6432 reserved ``s_trap 0x05`` Reserved. 6433 reserved ``s_trap 0x06`` Reserved. 6434 debugger breakpoint ``s_trap 0x07`` Reserved for debugger 6435 breakpoints. 6436 reserved ``s_trap 0x08`` Reserved. 6437 reserved ``s_trap 0xfe`` Reserved. 6438 reserved ``s_trap 0xff`` Reserved. 6439 =================== =============== =============== ======================= 6440 6441.. _amdgpu-amdhsa-function-call-convention: 6442 6443Call Convention 6444~~~~~~~~~~~~~~~ 6445 6446.. note:: 6447 6448 This section is currently incomplete and has inakkuracies. It is WIP that will 6449 be updated as information is determined. 6450 6451See :ref:`amdgpu-dwarf-address-space-identifier` for information on swizzled 6452addresses. Unswizzled addresses are normal linear addresses. 6453 6454.. _amdgpu-amdhsa-function-call-convention-kernel-functions: 6455 6456Kernel Functions 6457++++++++++++++++ 6458 6459This section describes the call convention ABI for the outer kernel function. 6460 6461See :ref:`amdgpu-amdhsa-initial-kernel-execution-state` for the kernel call 6462convention. 6463 6464The following is not part of the AMDGPU kernel calling convention but describes 6465how the AMDGPU implements function calls: 6466 64671. Clang decides the kernarg layout to match the *HSA Programmer's Language 6468 Reference* [HSA]_. 6469 6470 - All structs are passed directly. 6471 - Lambda values are passed *TBA*. 6472 6473 .. TODO:: 6474 6475 - Does this really follow HSA rules? Or are structs >16 bytes passed 6476 by-value struct? 6477 - What is ABI for lambda values? 6478 64794. The kernel performs certain setup in its prolog, as described in 6480 :ref:`amdgpu-amdhsa-kernel-prolog`. 6481 6482.. _amdgpu-amdhsa-function-call-convention-non-kernel-functions: 6483 6484Non-Kernel Functions 6485++++++++++++++++++++ 6486 6487This section describes the call convention ABI for functions other than the 6488outer kernel function. 6489 6490If a kernel has function calls then scratch is always allocated and used for 6491the call stack which grows from low address to high address using the swizzled 6492scratch address space. 6493 6494On entry to a function: 6495 64961. SGPR0-3 contain a V# with the following properties (see 6497 :ref:`amdgpu-amdhsa-kernel-prolog-private-segment-buffer`): 6498 6499 * Base address pointing to the beginning of the wavefront scratch backing 6500 memory. 6501 * Swizzled with dword element size and stride of wavefront size elements. 6502 65032. The FLAT_SCRATCH register pair is setup. See 6504 :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`. 65053. GFX6-8: M0 register set to the size of LDS in bytes. See 6506 :ref:`amdgpu-amdhsa-kernel-prolog-m0`. 65074. The EXEC register is set to the lanes active on entry to the function. 65085. MODE register: *TBD* 65096. VGPR0-31 and SGPR4-29 are used to pass function input arguments as described 6510 below. 65117. SGPR30-31 return address (RA). The code address that the function must 6512 return to when it completes. The value is undefined if the function is *no 6513 return*. 65148. SGPR32 is used for the stack pointer (SP). It is an unswizzled scratch 6515 offset relative to the beginning of the wavefront scratch backing memory. 6516 6517 The unswizzled SP can be used with buffer instructions as an unswizzled SGPR 6518 offset with the scratch V# in SGPR0-3 to access the stack in a swizzled 6519 manner. 6520 6521 The unswizzled SP value can be converted into the swizzled SP value by: 6522 6523 | swizzled SP = unswizzled SP / wavefront size 6524 6525 This may be used to obtain the private address space address of stack 6526 objects and to convert this address to a flat address by adding the flat 6527 scratch aperture base address. 6528 6529 The swizzled SP value is always 4 bytes aligned for the ``r600`` 6530 architecture and 16 byte aligned for the ``amdgcn`` architecture. 6531 6532 .. note:: 6533 6534 The ``amdgcn`` value is selected to avoid dynamic stack alignment for the 6535 OpenCL language which has the largest base type defined as 16 bytes. 6536 6537 On entry, the swizzled SP value is the address of the first function 6538 argument passed on the stack. Other stack passed arguments are positive 6539 offsets from the entry swizzled SP value. 6540 6541 The function may use positive offsets beyond the last stack passed argument 6542 for stack allocated local variables and register spill slots. If necessary, 6543 the function may align these to greater alignment than 16 bytes. After these 6544 the function may dynamically allocate space for such things as runtime sized 6545 ``alloca`` local allocations. 6546 6547 If the function calls another function, it will place any stack allocated 6548 arguments after the last local allocation and adjust SGPR32 to the address 6549 after the last local allocation. 6550 65519. All other registers are unspecified. 655210. Any necessary ``waitcnt`` has been performed to ensure memory is available 6553 to the function. 6554 6555On exit from a function: 6556 65571. VGPR0-31 and SGPR4-29 are used to pass function result arguments as 6558 described below. Any registers used are considered clobbered registers. 65592. The following registers are preserved and have the same value as on entry: 6560 6561 * FLAT_SCRATCH 6562 * EXEC 6563 * GFX6-8: M0 6564 * All SGPR registers except the clobbered registers of SGPR4-31. 6565 * VGPR40-47 6566 VGPR56-63 6567 VGPR72-79 6568 VGPR88-95 6569 VGPR104-111 6570 VGPR120-127 6571 VGPR136-143 6572 VGPR152-159 6573 VGPR168-175 6574 VGPR184-191 6575 VGPR200-207 6576 VGPR216-223 6577 VGPR232-239 6578 VGPR248-255 6579 6580 *Except the argument registers, the VGPR cloberred and the preserved 6581 registers are intermixed at regular intervals in order to 6582 get a better occupancy.* 6583 6584 For the AMDGPU backend, an inter-procedural register allocation (IPRA) 6585 optimization may mark some of clobbered SGPR and VGPR registers as 6586 preserved if it can be determined that the called function does not change 6587 their value. 6588 65892. The PC is set to the RA provided on entry. 65903. MODE register: *TBD*. 65914. All other registers are clobbered. 65925. Any necessary ``waitcnt`` has been performed to ensure memory accessed by 6593 function is available to the caller. 6594 6595.. TODO:: 6596 6597 - On gfx908 are all ACC registers clobbered? 6598 6599 - How are function results returned? The address of structured types is passed 6600 by reference, but what about other types? 6601 6602The function input arguments are made up of the formal arguments explicitly 6603declared by the source language function plus the implicit input arguments used 6604by the implementation. 6605 6606The source language input arguments are: 6607 66081. Any source language implicit ``this`` or ``self`` argument comes first as a 6609 pointer type. 66102. Followed by the function formal arguments in left to right source order. 6611 6612The source language result arguments are: 6613 66141. The function result argument. 6615 6616The source language input or result struct type arguments that are less than or 6617equal to 16 bytes, are decomposed recursively into their base type fields, and 6618each field is passed as if a separate argument. For input arguments, if the 6619called function requires the struct to be in memory, for example because its 6620address is taken, then the function body is responsible for allocating a stack 6621location and copying the field arguments into it. Clang terms this *direct 6622struct*. 6623 6624The source language input struct type arguments that are greater than 16 bytes, 6625are passed by reference. The caller is responsible for allocating a stack 6626location to make a copy of the struct value and pass the address as the input 6627argument. The called function is responsible to perform the dereference when 6628accessing the input argument. Clang terms this *by-value struct*. 6629 6630A source language result struct type argument that is greater than 16 bytes, is 6631returned by reference. The caller is responsible for allocating a stack location 6632to hold the result value and passes the address as the last input argument 6633(before the implicit input arguments). In this case there are no result 6634arguments. The called function is responsible to perform the dereference when 6635storing the result value. Clang terms this *structured return (sret)*. 6636 6637*TODO: correct the ``sret`` definition.* 6638 6639.. TODO:: 6640 6641 Is this definition correct? Or is ``sret`` only used if passing in registers, and 6642 pass as non-decomposed struct as stack argument? Or something else? Is the 6643 memory location in the caller stack frame, or a stack memory argument and so 6644 no address is passed as the caller can directly write to the argument stack 6645 location? But then the stack location is still live after return. If an 6646 argument stack location is it the first stack argument or the last one? 6647 6648Lambda argument types are treated as struct types with an implementation defined 6649set of fields. 6650 6651.. TODO:: 6652 6653 Need to specify the ABI for lambda types for AMDGPU. 6654 6655For AMDGPU backend all source language arguments (including the decomposed 6656struct type arguments) are passed in VGPRs unless marked ``inreg`` in which case 6657they are passed in SGPRs. 6658 6659The AMDGPU backend walks the function call graph from the leaves to determine 6660which implicit input arguments are used, propagating to each caller of the 6661function. The used implicit arguments are appended to the function arguments 6662after the source language arguments in the following order: 6663 6664.. TODO:: 6665 6666 Is recursion or external functions supported? 6667 66681. Work-Item ID (1 VGPR) 6669 6670 The X, Y and Z work-item ID are packed into a single VGRP with the following 6671 layout. Only fields actually used by the function are set. The other bits 6672 are undefined. 6673 6674 The values come from the initial kernel execution state. See 6675 :ref:`amdgpu-amdhsa-vgpr-register-set-up-order-table`. 6676 6677 .. table:: Work-item implicit argument layout 6678 :name: amdgpu-amdhsa-workitem-implicit-argument-layout-table 6679 6680 ======= ======= ============== 6681 Bits Size Field Name 6682 ======= ======= ============== 6683 9:0 10 bits X Work-Item ID 6684 19:10 10 bits Y Work-Item ID 6685 29:20 10 bits Z Work-Item ID 6686 31:30 2 bits Unused 6687 ======= ======= ============== 6688 66892. Dispatch Ptr (2 SGPRs) 6690 6691 The value comes from the initial kernel execution state. See 6692 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 6693 66943. Queue Ptr (2 SGPRs) 6695 6696 The value comes from the initial kernel execution state. See 6697 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 6698 66994. Kernarg Segment Ptr (2 SGPRs) 6700 6701 The value comes from the initial kernel execution state. See 6702 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 6703 67045. Dispatch id (2 SGPRs) 6705 6706 The value comes from the initial kernel execution state. See 6707 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 6708 67096. Work-Group ID X (1 SGPR) 6710 6711 The value comes from the initial kernel execution state. See 6712 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 6713 67147. Work-Group ID Y (1 SGPR) 6715 6716 The value comes from the initial kernel execution state. See 6717 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 6718 67198. Work-Group ID Z (1 SGPR) 6720 6721 The value comes from the initial kernel execution state. See 6722 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 6723 67249. Implicit Argument Ptr (2 SGPRs) 6725 6726 The value is computed by adding an offset to Kernarg Segment Ptr to get the 6727 global address space pointer to the first kernarg implicit argument. 6728 6729The input and result arguments are assigned in order in the following manner: 6730 6731.. note:: 6732 6733 There are likely some errors and omissions in the following description that 6734 need correction. 6735 6736 .. TODO:: 6737 6738 Check the clang source code to decipher how function arguments and return 6739 results are handled. Also see the AMDGPU specific values used. 6740 6741* VGPR arguments are assigned to consecutive VGPRs starting at VGPR0 up to 6742 VGPR31. 6743 6744 If there are more arguments than will fit in these registers, the remaining 6745 arguments are allocated on the stack in order on naturally aligned 6746 addresses. 6747 6748 .. TODO:: 6749 6750 How are overly aligned structures allocated on the stack? 6751 6752* SGPR arguments are assigned to consecutive SGPRs starting at SGPR0 up to 6753 SGPR29. 6754 6755 If there are more arguments than will fit in these registers, the remaining 6756 arguments are allocated on the stack in order on naturally aligned 6757 addresses. 6758 6759Note that decomposed struct type arguments may have some fields passed in 6760registers and some in memory. 6761 6762.. TODO:: 6763 6764 So, a struct which can pass some fields as decomposed register arguments, will 6765 pass the rest as decomposed stack elements? But an argument that will not start 6766 in registers will not be decomposed and will be passed as a non-decomposed 6767 stack value? 6768 6769The following is not part of the AMDGPU function calling convention but 6770describes how the AMDGPU implements function calls: 6771 67721. SGPR33 is used as a frame pointer (FP) if necessary. Like the SP it is an 6773 unswizzled scratch address. It is only needed if runtime sized ``alloca`` 6774 are used, or for the reasons defined in ``SIFrameLowering``. 67752. Runtime stack alignment is supported. SGPR34 is used as a base pointer (BP) 6776 to access the incoming stack arguments in the function. The BP is needed 6777 only when the function requires the runtime stack alignment. 6778 67793. Allocating SGPR arguments on the stack are not supported. 6780 67814. No CFI is currently generated. See 6782 :ref:`amdgpu-dwarf-call-frame-information`. 6783 6784 .. note:: 6785 6786 CFI will be generated that defines the CFA as the unswizzled address 6787 relative to the wave scratch base in the unswizzled private address space 6788 of the lowest address stack allocated local variable. 6789 6790 ``DW_AT_frame_base`` will be defined as the swizzled address in the 6791 swizzled private address space by dividing the CFA by the wavefront size 6792 (since CFA is always at least dword aligned which matches the scratch 6793 swizzle element size). 6794 6795 If no dynamic stack alignment was performed, the stack allocated arguments 6796 are accessed as negative offsets relative to ``DW_AT_frame_base``, and the 6797 local variables and register spill slots are accessed as positive offsets 6798 relative to ``DW_AT_frame_base``. 6799 68005. Function argument passing is implemented by copying the input physical 6801 registers to virtual registers on entry. The register allocator can spill if 6802 necessary. These are copied back to physical registers at call sites. The 6803 net effect is that each function call can have these values in entirely 6804 distinct locations. The IPRA can help avoid shuffling argument registers. 68056. Call sites are implemented by setting up the arguments at positive offsets 6806 from SP. Then SP is incremented to account for the known frame size before 6807 the call and decremented after the call. 6808 6809 .. note:: 6810 6811 The CFI will reflect the changed calculation needed to compute the CFA 6812 from SP. 6813 68147. 4 byte spill slots are used in the stack frame. One slot is allocated for an 6815 emergency spill slot. Buffer instructions are used for stack accesses and 6816 not the ``flat_scratch`` instruction. 6817 6818 .. TODO:: 6819 6820 Explain when the emergency spill slot is used. 6821 6822.. TODO:: 6823 6824 Possible broken issues: 6825 6826 - Stack arguments must be aligned to required alignment. 6827 - Stack is aligned to max(16, max formal argument alignment) 6828 - Direct argument < 64 bits should check register budget. 6829 - Register budget calculation should respect ``inreg`` for SGPR. 6830 - SGPR overflow is not handled. 6831 - struct with 1 member unpeeling is not checking size of member. 6832 - ``sret`` is after ``this`` pointer. 6833 - Caller is not implementing stack realignment: need an extra pointer. 6834 - Should say AMDGPU passes FP rather than SP. 6835 - Should CFI define CFA as address of locals or arguments. Difference is 6836 apparent when have implemented dynamic alignment. 6837 - If ``SCRATCH`` instruction could allow negative offsets, then can make FP be 6838 highest address of stack frame and use negative offset for locals. Would 6839 allow SP to be the same as FP and could support signal-handler-like as now 6840 have a real SP for the top of the stack. 6841 - How is ``sret`` passed on the stack? In argument stack area? Can it overlay 6842 arguments? 6843 6844AMDPAL 6845------ 6846 6847This section provides code conventions used when the target triple OS is 6848``amdpal`` (see :ref:`amdgpu-target-triples`) for passing runtime parameters 6849from the application/runtime to each invocation of a hardware shader. These 6850parameters include both generic, application-controlled parameters called 6851*user data* as well as system-generated parameters that are a product of the 6852draw or dispatch execution. 6853 6854User Data 6855~~~~~~~~~ 6856 6857Each hardware stage has a set of 32-bit *user data registers* which can be 6858written from a command buffer and then loaded into SGPRs when waves are launched 6859via a subsequent dispatch or draw operation. This is the way most arguments are 6860passed from the application/runtime to a hardware shader. 6861 6862Compute User Data 6863~~~~~~~~~~~~~~~~~ 6864 6865Compute shader user data mappings are simpler than graphics shaders and have a 6866fixed mapping. 6867 6868Note that there are always 10 available *user data entries* in registers - 6869entries beyond that limit must be fetched from memory (via the spill table 6870pointer) by the shader. 6871 6872 .. table:: PAL Compute Shader User Data Registers 6873 :name: pal-compute-user-data-registers 6874 6875 ============= ================================ 6876 User Register Description 6877 ============= ================================ 6878 0 Global Internal Table (32-bit pointer) 6879 1 Per-Shader Internal Table (32-bit pointer) 6880 2 - 11 Application-Controlled User Data (10 32-bit values) 6881 12 Spill Table (32-bit pointer) 6882 13 - 14 Thread Group Count (64-bit pointer) 6883 15 GDS Range 6884 ============= ================================ 6885 6886Graphics User Data 6887~~~~~~~~~~~~~~~~~~ 6888 6889Graphics pipelines support a much more flexible user data mapping: 6890 6891 .. table:: PAL Graphics Shader User Data Registers 6892 :name: pal-graphics-user-data-registers 6893 6894 ============= ================================ 6895 User Register Description 6896 ============= ================================ 6897 0 Global Internal Table (32-bit pointer) 6898 + Per-Shader Internal Table (32-bit pointer) 6899 + 1-15 Application Controlled User Data 6900 (1-15 Contiguous 32-bit Values in Registers) 6901 + Spill Table (32-bit pointer) 6902 + Draw Index (First Stage Only) 6903 + Vertex Offset (First Stage Only) 6904 + Instance Offset (First Stage Only) 6905 ============= ================================ 6906 6907 The placement of the global internal table remains fixed in the first *user 6908 data SGPR register*. Otherwise all parameters are optional, and can be mapped 6909 to any desired *user data SGPR register*, with the following restrictions: 6910 6911 * Draw Index, Vertex Offset, and Instance Offset can only be used by the first 6912 active hardware stage in a graphics pipeline (i.e. where the API vertex 6913 shader runs). 6914 6915 * Application-controlled user data must be mapped into a contiguous range of 6916 user data registers. 6917 6918 * The application-controlled user data range supports compaction remapping, so 6919 only *entries* that are actually consumed by the shader must be assigned to 6920 corresponding *registers*. Note that in order to support an efficient runtime 6921 implementation, the remapping must pack *registers* in the same order as 6922 *entries*, with unused *entries* removed. 6923 6924.. _pal_global_internal_table: 6925 6926Global Internal Table 6927~~~~~~~~~~~~~~~~~~~~~ 6928 6929The global internal table is a table of *shader resource descriptors* (SRDs) 6930that define how certain engine-wide, runtime-managed resources should be 6931accessed from a shader. The majority of these resources have HW-defined formats, 6932and it is up to the compiler to write/read data as required by the target 6933hardware. 6934 6935The following table illustrates the required format: 6936 6937 .. table:: PAL Global Internal Table 6938 :name: pal-git-table 6939 6940 ============= ================================ 6941 Offset Description 6942 ============= ================================ 6943 0-3 Graphics Scratch SRD 6944 4-7 Compute Scratch SRD 6945 8-11 ES/GS Ring Output SRD 6946 12-15 ES/GS Ring Input SRD 6947 16-19 GS/VS Ring Output #0 6948 20-23 GS/VS Ring Output #1 6949 24-27 GS/VS Ring Output #2 6950 28-31 GS/VS Ring Output #3 6951 32-35 GS/VS Ring Input SRD 6952 36-39 Tessellation Factor Buffer SRD 6953 40-43 Off-Chip LDS Buffer SRD 6954 44-47 Off-Chip Param Cache Buffer SRD 6955 48-51 Sample Position Buffer SRD 6956 52 vaRange::ShadowDescriptorTable High Bits 6957 ============= ================================ 6958 6959 The pointer to the global internal table passed to the shader as user data 6960 is a 32-bit pointer. The top 32 bits should be assumed to be the same as 6961 the top 32 bits of the pipeline, so the shader may use the program 6962 counter's top 32 bits. 6963 6964Unspecified OS 6965-------------- 6966 6967This section provides code conventions used when the target triple OS is 6968empty (see :ref:`amdgpu-target-triples`). 6969 6970Trap Handler ABI 6971~~~~~~~~~~~~~~~~ 6972 6973For code objects generated by AMDGPU backend for non-amdhsa OS, the runtime does 6974not install a trap handler. The ``llvm.trap`` and ``llvm.debugtrap`` 6975instructions are handled as follows: 6976 6977 .. table:: AMDGPU Trap Handler for Non-AMDHSA OS 6978 :name: amdgpu-trap-handler-for-non-amdhsa-os-table 6979 6980 =============== =============== =========================================== 6981 Usage Code Sequence Description 6982 =============== =============== =========================================== 6983 llvm.trap s_endpgm Causes wavefront to be terminated. 6984 llvm.debugtrap *none* Compiler warning given that there is no 6985 trap handler installed. 6986 =============== =============== =========================================== 6987 6988Source Languages 6989================ 6990 6991.. _amdgpu-opencl: 6992 6993OpenCL 6994------ 6995 6996When the language is OpenCL the following differences occur: 6997 69981. The OpenCL memory model is used (see :ref:`amdgpu-amdhsa-memory-model`). 69992. The AMDGPU backend appends additional arguments to the kernel's explicit 7000 arguments for the AMDHSA OS (see 7001 :ref:`opencl-kernel-implicit-arguments-appended-for-amdhsa-os-table`). 70023. Additional metadata is generated 7003 (see :ref:`amdgpu-amdhsa-code-object-metadata`). 7004 7005 .. table:: OpenCL kernel implicit arguments appended for AMDHSA OS 7006 :name: opencl-kernel-implicit-arguments-appended-for-amdhsa-os-table 7007 7008 ======== ==== ========= =========================================== 7009 Position Byte Byte Description 7010 Size Alignment 7011 ======== ==== ========= =========================================== 7012 1 8 8 OpenCL Global Offset X 7013 2 8 8 OpenCL Global Offset Y 7014 3 8 8 OpenCL Global Offset Z 7015 4 8 8 OpenCL address of printf buffer 7016 5 8 8 OpenCL address of virtual queue used by 7017 enqueue_kernel. 7018 6 8 8 OpenCL address of AqlWrap struct used by 7019 enqueue_kernel. 7020 7 8 8 Pointer argument used for Multi-gird 7021 synchronization. 7022 ======== ==== ========= =========================================== 7023 7024.. _amdgpu-hcc: 7025 7026HCC 7027--- 7028 7029When the language is HCC the following differences occur: 7030 70311. The HSA memory model is used (see :ref:`amdgpu-amdhsa-memory-model`). 7032 7033.. _amdgpu-assembler: 7034 7035Assembler 7036--------- 7037 7038AMDGPU backend has LLVM-MC based assembler which is currently in development. 7039It supports AMDGCN GFX6-GFX10. 7040 7041This section describes general syntax for instructions and operands. 7042 7043Instructions 7044~~~~~~~~~~~~ 7045 7046An instruction has the following :doc:`syntax<AMDGPUInstructionSyntax>`: 7047 7048 | ``<``\ *opcode*\ ``> <``\ *operand0*\ ``>, <``\ *operand1*\ ``>,... 7049 <``\ *modifier0*\ ``> <``\ *modifier1*\ ``>...`` 7050 7051:doc:`Operands<AMDGPUOperandSyntax>` are comma-separated while 7052:doc:`modifiers<AMDGPUModifierSyntax>` are space-separated. 7053 7054The order of operands and modifiers is fixed. 7055Most modifiers are optional and may be omitted. 7056 7057Links to detailed instruction syntax description may be found in the following 7058table. Note that features under development are not included 7059in this description. 7060 7061 =================================== ======================================= 7062 Core ISA ISA Extensions 7063 =================================== ======================================= 7064 :doc:`GFX7<AMDGPU/AMDGPUAsmGFX7>` \- 7065 :doc:`GFX8<AMDGPU/AMDGPUAsmGFX8>` \- 7066 :doc:`GFX9<AMDGPU/AMDGPUAsmGFX9>` :doc:`gfx900<AMDGPU/AMDGPUAsmGFX900>` 7067 7068 :doc:`gfx902<AMDGPU/AMDGPUAsmGFX900>` 7069 7070 :doc:`gfx904<AMDGPU/AMDGPUAsmGFX904>` 7071 7072 :doc:`gfx906<AMDGPU/AMDGPUAsmGFX906>` 7073 7074 :doc:`gfx908<AMDGPU/AMDGPUAsmGFX908>` 7075 7076 :doc:`gfx909<AMDGPU/AMDGPUAsmGFX900>` 7077 7078 :doc:`GFX10<AMDGPU/AMDGPUAsmGFX10>` :doc:`gfx1011<AMDGPU/AMDGPUAsmGFX1011>` 7079 7080 :doc:`gfx1012<AMDGPU/AMDGPUAsmGFX1011>` 7081 =================================== ======================================= 7082 7083For more information about instructions, their semantics and supported 7084combinations of operands, refer to one of instruction set architecture manuals 7085[AMD-GCN-GFX6]_, [AMD-GCN-GFX7]_, [AMD-GCN-GFX8]_, [AMD-GCN-GFX9]_ and 7086[AMD-GCN-GFX10]_. 7087 7088Operands 7089~~~~~~~~ 7090 7091Detailed description of operands may be found :doc:`here<AMDGPUOperandSyntax>`. 7092 7093Modifiers 7094~~~~~~~~~ 7095 7096Detailed description of modifiers may be found 7097:doc:`here<AMDGPUModifierSyntax>`. 7098 7099Instruction Examples 7100~~~~~~~~~~~~~~~~~~~~ 7101 7102DS 7103++ 7104 7105.. code-block:: nasm 7106 7107 ds_add_u32 v2, v4 offset:16 7108 ds_write_src2_b64 v2 offset0:4 offset1:8 7109 ds_cmpst_f32 v2, v4, v6 7110 ds_min_rtn_f64 v[8:9], v2, v[4:5] 7111 7112For full list of supported instructions, refer to "LDS/GDS instructions" in ISA 7113Manual. 7114 7115FLAT 7116++++ 7117 7118.. code-block:: nasm 7119 7120 flat_load_dword v1, v[3:4] 7121 flat_store_dwordx3 v[3:4], v[5:7] 7122 flat_atomic_swap v1, v[3:4], v5 glc 7123 flat_atomic_cmpswap v1, v[3:4], v[5:6] glc slc 7124 flat_atomic_fmax_x2 v[1:2], v[3:4], v[5:6] glc 7125 7126For full list of supported instructions, refer to "FLAT instructions" in ISA 7127Manual. 7128 7129MUBUF 7130+++++ 7131 7132.. code-block:: nasm 7133 7134 buffer_load_dword v1, off, s[4:7], s1 7135 buffer_store_dwordx4 v[1:4], v2, ttmp[4:7], s1 offen offset:4 glc tfe 7136 buffer_store_format_xy v[1:2], off, s[4:7], s1 7137 buffer_wbinvl1 7138 buffer_atomic_inc v1, v2, s[8:11], s4 idxen offset:4 slc 7139 7140For full list of supported instructions, refer to "MUBUF Instructions" in ISA 7141Manual. 7142 7143SMRD/SMEM 7144+++++++++ 7145 7146.. code-block:: nasm 7147 7148 s_load_dword s1, s[2:3], 0xfc 7149 s_load_dwordx8 s[8:15], s[2:3], s4 7150 s_load_dwordx16 s[88:103], s[2:3], s4 7151 s_dcache_inv_vol 7152 s_memtime s[4:5] 7153 7154For full list of supported instructions, refer to "Scalar Memory Operations" in 7155ISA Manual. 7156 7157SOP1 7158++++ 7159 7160.. code-block:: nasm 7161 7162 s_mov_b32 s1, s2 7163 s_mov_b64 s[0:1], 0x80000000 7164 s_cmov_b32 s1, 200 7165 s_wqm_b64 s[2:3], s[4:5] 7166 s_bcnt0_i32_b64 s1, s[2:3] 7167 s_swappc_b64 s[2:3], s[4:5] 7168 s_cbranch_join s[4:5] 7169 7170For full list of supported instructions, refer to "SOP1 Instructions" in ISA 7171Manual. 7172 7173SOP2 7174++++ 7175 7176.. code-block:: nasm 7177 7178 s_add_u32 s1, s2, s3 7179 s_and_b64 s[2:3], s[4:5], s[6:7] 7180 s_cselect_b32 s1, s2, s3 7181 s_andn2_b32 s2, s4, s6 7182 s_lshr_b64 s[2:3], s[4:5], s6 7183 s_ashr_i32 s2, s4, s6 7184 s_bfm_b64 s[2:3], s4, s6 7185 s_bfe_i64 s[2:3], s[4:5], s6 7186 s_cbranch_g_fork s[4:5], s[6:7] 7187 7188For full list of supported instructions, refer to "SOP2 Instructions" in ISA 7189Manual. 7190 7191SOPC 7192++++ 7193 7194.. code-block:: nasm 7195 7196 s_cmp_eq_i32 s1, s2 7197 s_bitcmp1_b32 s1, s2 7198 s_bitcmp0_b64 s[2:3], s4 7199 s_setvskip s3, s5 7200 7201For full list of supported instructions, refer to "SOPC Instructions" in ISA 7202Manual. 7203 7204SOPP 7205++++ 7206 7207.. code-block:: nasm 7208 7209 s_barrier 7210 s_nop 2 7211 s_endpgm 7212 s_waitcnt 0 ; Wait for all counters to be 0 7213 s_waitcnt vmcnt(0) & expcnt(0) & lgkmcnt(0) ; Equivalent to above 7214 s_waitcnt vmcnt(1) ; Wait for vmcnt counter to be 1. 7215 s_sethalt 9 7216 s_sleep 10 7217 s_sendmsg 0x1 7218 s_sendmsg sendmsg(MSG_INTERRUPT) 7219 s_trap 1 7220 7221For full list of supported instructions, refer to "SOPP Instructions" in ISA 7222Manual. 7223 7224Unless otherwise mentioned, little verification is performed on the operands 7225of SOPP Instructions, so it is up to the programmer to be familiar with the 7226range or acceptable values. 7227 7228VALU 7229++++ 7230 7231For vector ALU instruction opcodes (VOP1, VOP2, VOP3, VOPC, VOP_DPP, VOP_SDWA), 7232the assembler will automatically use optimal encoding based on its operands. To 7233force specific encoding, one can add a suffix to the opcode of the instruction: 7234 7235* _e32 for 32-bit VOP1/VOP2/VOPC 7236* _e64 for 64-bit VOP3 7237* _dpp for VOP_DPP 7238* _sdwa for VOP_SDWA 7239 7240VOP1/VOP2/VOP3/VOPC examples: 7241 7242.. code-block:: nasm 7243 7244 v_mov_b32 v1, v2 7245 v_mov_b32_e32 v1, v2 7246 v_nop 7247 v_cvt_f64_i32_e32 v[1:2], v2 7248 v_floor_f32_e32 v1, v2 7249 v_bfrev_b32_e32 v1, v2 7250 v_add_f32_e32 v1, v2, v3 7251 v_mul_i32_i24_e64 v1, v2, 3 7252 v_mul_i32_i24_e32 v1, -3, v3 7253 v_mul_i32_i24_e32 v1, -100, v3 7254 v_addc_u32 v1, s[0:1], v2, v3, s[2:3] 7255 v_max_f16_e32 v1, v2, v3 7256 7257VOP_DPP examples: 7258 7259.. code-block:: nasm 7260 7261 v_mov_b32 v0, v0 quad_perm:[0,2,1,1] 7262 v_sin_f32 v0, v0 row_shl:1 row_mask:0xa bank_mask:0x1 bound_ctrl:0 7263 v_mov_b32 v0, v0 wave_shl:1 7264 v_mov_b32 v0, v0 row_mirror 7265 v_mov_b32 v0, v0 row_bcast:31 7266 v_mov_b32 v0, v0 quad_perm:[1,3,0,1] row_mask:0xa bank_mask:0x1 bound_ctrl:0 7267 v_add_f32 v0, v0, |v0| row_shl:1 row_mask:0xa bank_mask:0x1 bound_ctrl:0 7268 v_max_f16 v1, v2, v3 row_shl:1 row_mask:0xa bank_mask:0x1 bound_ctrl:0 7269 7270VOP_SDWA examples: 7271 7272.. code-block:: nasm 7273 7274 v_mov_b32 v1, v2 dst_sel:BYTE_0 dst_unused:UNUSED_PRESERVE src0_sel:DWORD 7275 v_min_u32 v200, v200, v1 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:BYTE_1 src1_sel:DWORD 7276 v_sin_f32 v0, v0 dst_unused:UNUSED_PAD src0_sel:WORD_1 7277 v_fract_f32 v0, |v0| dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 7278 v_cmpx_le_u32 vcc, v1, v2 src0_sel:BYTE_2 src1_sel:WORD_0 7279 7280For full list of supported instructions, refer to "Vector ALU instructions". 7281 7282.. TODO:: 7283 7284 Remove once we switch to code object v3 by default. 7285 7286.. _amdgpu-amdhsa-assembler-predefined-symbols-v2: 7287 7288Code Object V2 Predefined Symbols (-mattr=-code-object-v3) 7289~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 7290 7291.. warning:: Code Object V2 is not the default code object version emitted by 7292 this version of LLVM. For a description of the predefined symbols available 7293 with the default configuration (Code Object V3) see 7294 :ref:`amdgpu-amdhsa-assembler-predefined-symbols-v3`. 7295 7296The AMDGPU assembler defines and updates some symbols automatically. These 7297symbols do not affect code generation. 7298 7299.option.machine_version_major 7300+++++++++++++++++++++++++++++ 7301 7302Set to the GFX major generation number of the target being assembled for. For 7303example, when assembling for a "GFX9" target this will be set to the integer 7304value "9". The possible GFX major generation numbers are presented in 7305:ref:`amdgpu-processors`. 7306 7307.option.machine_version_minor 7308+++++++++++++++++++++++++++++ 7309 7310Set to the GFX minor generation number of the target being assembled for. For 7311example, when assembling for a "GFX810" target this will be set to the integer 7312value "1". The possible GFX minor generation numbers are presented in 7313:ref:`amdgpu-processors`. 7314 7315.option.machine_version_stepping 7316++++++++++++++++++++++++++++++++ 7317 7318Set to the GFX stepping generation number of the target being assembled for. 7319For example, when assembling for a "GFX704" target this will be set to the 7320integer value "4". The possible GFX stepping generation numbers are presented 7321in :ref:`amdgpu-processors`. 7322 7323.kernel.vgpr_count 7324++++++++++++++++++ 7325 7326Set to zero each time a 7327:ref:`amdgpu-amdhsa-assembler-directive-amdgpu_hsa_kernel` directive is 7328encountered. At each instruction, if the current value of this symbol is less 7329than or equal to the maximum VPGR number explicitly referenced within that 7330instruction then the symbol value is updated to equal that VGPR number plus 7331one. 7332 7333.kernel.sgpr_count 7334++++++++++++++++++ 7335 7336Set to zero each time a 7337:ref:`amdgpu-amdhsa-assembler-directive-amdgpu_hsa_kernel` directive is 7338encountered. At each instruction, if the current value of this symbol is less 7339than or equal to the maximum VPGR number explicitly referenced within that 7340instruction then the symbol value is updated to equal that SGPR number plus 7341one. 7342 7343.. _amdgpu-amdhsa-assembler-directives-v2: 7344 7345Code Object V2 Directives (-mattr=-code-object-v3) 7346~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 7347 7348.. warning:: Code Object V2 is not the default code object version emitted by 7349 this version of LLVM. For a description of the directives supported with 7350 the default configuration (Code Object V3) see 7351 :ref:`amdgpu-amdhsa-assembler-directives-v3`. 7352 7353AMDGPU ABI defines auxiliary data in output code object. In assembly source, 7354one can specify them with assembler directives. 7355 7356.hsa_code_object_version major, minor 7357+++++++++++++++++++++++++++++++++++++ 7358 7359*major* and *minor* are integers that specify the version of the HSA code 7360object that will be generated by the assembler. 7361 7362.hsa_code_object_isa [major, minor, stepping, vendor, arch] 7363+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 7364 7365 7366*major*, *minor*, and *stepping* are all integers that describe the instruction 7367set architecture (ISA) version of the assembly program. 7368 7369*vendor* and *arch* are quoted strings. *vendor* should always be equal to 7370"AMD" and *arch* should always be equal to "AMDGPU". 7371 7372By default, the assembler will derive the ISA version, *vendor*, and *arch* 7373from the value of the -mcpu option that is passed to the assembler. 7374 7375.. _amdgpu-amdhsa-assembler-directive-amdgpu_hsa_kernel: 7376 7377.amdgpu_hsa_kernel (name) 7378+++++++++++++++++++++++++ 7379 7380This directives specifies that the symbol with given name is a kernel entry 7381point (label) and the object should contain corresponding symbol of type 7382STT_AMDGPU_HSA_KERNEL. 7383 7384.amd_kernel_code_t 7385++++++++++++++++++ 7386 7387This directive marks the beginning of a list of key / value pairs that are used 7388to specify the amd_kernel_code_t object that will be emitted by the assembler. 7389The list must be terminated by the *.end_amd_kernel_code_t* directive. For any 7390amd_kernel_code_t values that are unspecified a default value will be used. The 7391default value for all keys is 0, with the following exceptions: 7392 7393- *amd_code_version_major* defaults to 1. 7394- *amd_kernel_code_version_minor* defaults to 2. 7395- *amd_machine_kind* defaults to 1. 7396- *amd_machine_version_major*, *machine_version_minor*, and 7397 *amd_machine_version_stepping* are derived from the value of the -mcpu option 7398 that is passed to the assembler. 7399- *kernel_code_entry_byte_offset* defaults to 256. 7400- *wavefront_size* defaults 6 for all targets before GFX10. For GFX10 onwards 7401 defaults to 6 if target feature ``wavefrontsize64`` is enabled, otherwise 5. 7402 Note that wavefront size is specified as a power of two, so a value of **n** 7403 means a size of 2^ **n**. 7404- *call_convention* defaults to -1. 7405- *kernarg_segment_alignment*, *group_segment_alignment*, and 7406 *private_segment_alignment* default to 4. Note that alignments are specified 7407 as a power of 2, so a value of **n** means an alignment of 2^ **n**. 7408- *enable_wgp_mode* defaults to 1 if target feature ``cumode`` is disabled for 7409 GFX10 onwards. 7410- *enable_mem_ordered* defaults to 1 for GFX10 onwards. 7411 7412The *.amd_kernel_code_t* directive must be placed immediately after the 7413function label and before any instructions. 7414 7415For a full list of amd_kernel_code_t keys, refer to AMDGPU ABI document, 7416comments in lib/Target/AMDGPU/AmdKernelCodeT.h and test/CodeGen/AMDGPU/hsa.s. 7417 7418.. _amdgpu-amdhsa-assembler-example-v2: 7419 7420Code Object V2 Example Source Code (-mattr=-code-object-v3) 7421~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 7422 7423.. warning:: Code Object V2 is not the default code object version emitted by 7424 this version of LLVM. For a description of the directives supported with 7425 the default configuration (Code Object V3) see 7426 :ref:`amdgpu-amdhsa-assembler-example-v3`. 7427 7428Here is an example of a minimal assembly source file, defining one HSA kernel: 7429 7430.. code:: 7431 :number-lines: 7432 7433 .hsa_code_object_version 1,0 7434 .hsa_code_object_isa 7435 7436 .hsatext 7437 .globl hello_world 7438 .p2align 8 7439 .amdgpu_hsa_kernel hello_world 7440 7441 hello_world: 7442 7443 .amd_kernel_code_t 7444 enable_sgpr_kernarg_segment_ptr = 1 7445 is_ptr64 = 1 7446 compute_pgm_rsrc1_vgprs = 0 7447 compute_pgm_rsrc1_sgprs = 0 7448 compute_pgm_rsrc2_user_sgpr = 2 7449 compute_pgm_rsrc1_wgp_mode = 0 7450 compute_pgm_rsrc1_mem_ordered = 0 7451 compute_pgm_rsrc1_fwd_progress = 1 7452 .end_amd_kernel_code_t 7453 7454 s_load_dwordx2 s[0:1], s[0:1] 0x0 7455 v_mov_b32 v0, 3.14159 7456 s_waitcnt lgkmcnt(0) 7457 v_mov_b32 v1, s0 7458 v_mov_b32 v2, s1 7459 flat_store_dword v[1:2], v0 7460 s_endpgm 7461 .Lfunc_end0: 7462 .size hello_world, .Lfunc_end0-hello_world 7463 7464.. _amdgpu-amdhsa-assembler-predefined-symbols-v3: 7465 7466Code Object V3 Predefined Symbols (-mattr=+code-object-v3) 7467~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 7468 7469The AMDGPU assembler defines and updates some symbols automatically. These 7470symbols do not affect code generation. 7471 7472.amdgcn.gfx_generation_number 7473+++++++++++++++++++++++++++++ 7474 7475Set to the GFX major generation number of the target being assembled for. For 7476example, when assembling for a "GFX9" target this will be set to the integer 7477value "9". The possible GFX major generation numbers are presented in 7478:ref:`amdgpu-processors`. 7479 7480.amdgcn.gfx_generation_minor 7481++++++++++++++++++++++++++++ 7482 7483Set to the GFX minor generation number of the target being assembled for. For 7484example, when assembling for a "GFX810" target this will be set to the integer 7485value "1". The possible GFX minor generation numbers are presented in 7486:ref:`amdgpu-processors`. 7487 7488.amdgcn.gfx_generation_stepping 7489+++++++++++++++++++++++++++++++ 7490 7491Set to the GFX stepping generation number of the target being assembled for. 7492For example, when assembling for a "GFX704" target this will be set to the 7493integer value "4". The possible GFX stepping generation numbers are presented 7494in :ref:`amdgpu-processors`. 7495 7496.. _amdgpu-amdhsa-assembler-symbol-next_free_vgpr: 7497 7498.amdgcn.next_free_vgpr 7499++++++++++++++++++++++ 7500 7501Set to zero before assembly begins. At each instruction, if the current value 7502of this symbol is less than or equal to the maximum VGPR number explicitly 7503referenced within that instruction then the symbol value is updated to equal 7504that VGPR number plus one. 7505 7506May be used to set the `.amdhsa_next_free_vpgr` directive in 7507:ref:`amdhsa-kernel-directives-table`. 7508 7509May be set at any time, e.g. manually set to zero at the start of each kernel. 7510 7511.. _amdgpu-amdhsa-assembler-symbol-next_free_sgpr: 7512 7513.amdgcn.next_free_sgpr 7514++++++++++++++++++++++ 7515 7516Set to zero before assembly begins. At each instruction, if the current value 7517of this symbol is less than or equal the maximum SGPR number explicitly 7518referenced within that instruction then the symbol value is updated to equal 7519that SGPR number plus one. 7520 7521May be used to set the `.amdhsa_next_free_spgr` directive in 7522:ref:`amdhsa-kernel-directives-table`. 7523 7524May be set at any time, e.g. manually set to zero at the start of each kernel. 7525 7526.. _amdgpu-amdhsa-assembler-directives-v3: 7527 7528Code Object V3 Directives (-mattr=+code-object-v3) 7529~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 7530 7531Directives which begin with ``.amdgcn`` are valid for all ``amdgcn`` 7532architecture processors, and are not OS-specific. Directives which begin with 7533``.amdhsa`` are specific to ``amdgcn`` architecture processors when the 7534``amdhsa`` OS is specified. See :ref:`amdgpu-target-triples` and 7535:ref:`amdgpu-processors`. 7536 7537.amdgcn_target <target> 7538+++++++++++++++++++++++ 7539 7540Optional directive which declares the target supported by the containing 7541assembler source file. Valid values are described in 7542:ref:`amdgpu-amdhsa-code-object-target-identification`. Used by the assembler 7543to validate command-line options such as ``-triple``, ``-mcpu``, and those 7544which specify target features. 7545 7546.amdhsa_kernel <name> 7547+++++++++++++++++++++ 7548 7549Creates a correctly aligned AMDHSA kernel descriptor and a symbol, 7550``<name>.kd``, in the current location of the current section. Only valid when 7551the OS is ``amdhsa``. ``<name>`` must be a symbol that labels the first 7552instruction to execute, and does not need to be previously defined. 7553 7554Marks the beginning of a list of directives used to generate the bytes of a 7555kernel descriptor, as described in :ref:`amdgpu-amdhsa-kernel-descriptor`. 7556Directives which may appear in this list are described in 7557:ref:`amdhsa-kernel-directives-table`. Directives may appear in any order, must 7558be valid for the target being assembled for, and cannot be repeated. Directives 7559support the range of values specified by the field they reference in 7560:ref:`amdgpu-amdhsa-kernel-descriptor`. If a directive is not specified, it is 7561assumed to have its default value, unless it is marked as "Required", in which 7562case it is an error to omit the directive. This list of directives is 7563terminated by an ``.end_amdhsa_kernel`` directive. 7564 7565 .. table:: AMDHSA Kernel Assembler Directives 7566 :name: amdhsa-kernel-directives-table 7567 7568 ======================================================== =================== ============ =================== 7569 Directive Default Supported On Description 7570 ======================================================== =================== ============ =================== 7571 ``.amdhsa_group_segment_fixed_size`` 0 GFX6-GFX10 Controls GROUP_SEGMENT_FIXED_SIZE in 7572 :ref:`amdgpu-amdhsa-kernel-descriptor-gfx6-gfx10-table`. 7573 ``.amdhsa_private_segment_fixed_size`` 0 GFX6-GFX10 Controls PRIVATE_SEGMENT_FIXED_SIZE in 7574 :ref:`amdgpu-amdhsa-kernel-descriptor-gfx6-gfx10-table`. 7575 ``.amdhsa_user_sgpr_private_segment_buffer`` 0 GFX6-GFX10 Controls ENABLE_SGPR_PRIVATE_SEGMENT_BUFFER in 7576 :ref:`amdgpu-amdhsa-kernel-descriptor-gfx6-gfx10-table`. 7577 ``.amdhsa_user_sgpr_dispatch_ptr`` 0 GFX6-GFX10 Controls ENABLE_SGPR_DISPATCH_PTR in 7578 :ref:`amdgpu-amdhsa-kernel-descriptor-gfx6-gfx10-table`. 7579 ``.amdhsa_user_sgpr_queue_ptr`` 0 GFX6-GFX10 Controls ENABLE_SGPR_QUEUE_PTR in 7580 :ref:`amdgpu-amdhsa-kernel-descriptor-gfx6-gfx10-table`. 7581 ``.amdhsa_user_sgpr_kernarg_segment_ptr`` 0 GFX6-GFX10 Controls ENABLE_SGPR_KERNARG_SEGMENT_PTR in 7582 :ref:`amdgpu-amdhsa-kernel-descriptor-gfx6-gfx10-table`. 7583 ``.amdhsa_user_sgpr_dispatch_id`` 0 GFX6-GFX10 Controls ENABLE_SGPR_DISPATCH_ID in 7584 :ref:`amdgpu-amdhsa-kernel-descriptor-gfx6-gfx10-table`. 7585 ``.amdhsa_user_sgpr_flat_scratch_init`` 0 GFX6-GFX10 Controls ENABLE_SGPR_FLAT_SCRATCH_INIT in 7586 :ref:`amdgpu-amdhsa-kernel-descriptor-gfx6-gfx10-table`. 7587 ``.amdhsa_user_sgpr_private_segment_size`` 0 GFX6-GFX10 Controls ENABLE_SGPR_PRIVATE_SEGMENT_SIZE in 7588 :ref:`amdgpu-amdhsa-kernel-descriptor-gfx6-gfx10-table`. 7589 ``.amdhsa_wavefront_size32`` Target GFX10 Controls ENABLE_WAVEFRONT_SIZE32 in 7590 Feature :ref:`amdgpu-amdhsa-kernel-descriptor-gfx6-gfx10-table`. 7591 Specific 7592 (-wavefrontsize64) 7593 ``.amdhsa_system_sgpr_private_segment_wavefront_offset`` 0 GFX6-GFX10 Controls ENABLE_SGPR_PRIVATE_SEGMENT_WAVEFRONT_OFFSET in 7594 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 7595 ``.amdhsa_system_sgpr_workgroup_id_x`` 1 GFX6-GFX10 Controls ENABLE_SGPR_WORKGROUP_ID_X in 7596 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 7597 ``.amdhsa_system_sgpr_workgroup_id_y`` 0 GFX6-GFX10 Controls ENABLE_SGPR_WORKGROUP_ID_Y in 7598 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 7599 ``.amdhsa_system_sgpr_workgroup_id_z`` 0 GFX6-GFX10 Controls ENABLE_SGPR_WORKGROUP_ID_Z in 7600 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 7601 ``.amdhsa_system_sgpr_workgroup_info`` 0 GFX6-GFX10 Controls ENABLE_SGPR_WORKGROUP_INFO in 7602 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 7603 ``.amdhsa_system_vgpr_workitem_id`` 0 GFX6-GFX10 Controls ENABLE_VGPR_WORKITEM_ID in 7604 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 7605 Possible values are defined in 7606 :ref:`amdgpu-amdhsa-system-vgpr-work-item-id-enumeration-values-table`. 7607 ``.amdhsa_next_free_vgpr`` Required GFX6-GFX10 Maximum VGPR number explicitly referenced, plus one. 7608 Used to calculate GRANULATED_WORKITEM_VGPR_COUNT in 7609 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 7610 ``.amdhsa_next_free_sgpr`` Required GFX6-GFX10 Maximum SGPR number explicitly referenced, plus one. 7611 Used to calculate GRANULATED_WAVEFRONT_SGPR_COUNT in 7612 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 7613 ``.amdhsa_reserve_vcc`` 1 GFX6-GFX10 Whether the kernel may use the special VCC SGPR. 7614 Used to calculate GRANULATED_WAVEFRONT_SGPR_COUNT in 7615 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 7616 ``.amdhsa_reserve_flat_scratch`` 1 GFX7-GFX10 Whether the kernel may use flat instructions to access 7617 scratch memory. Used to calculate 7618 GRANULATED_WAVEFRONT_SGPR_COUNT in 7619 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 7620 ``.amdhsa_reserve_xnack_mask`` Target GFX8-GFX10 Whether the kernel may trigger XNACK replay. 7621 Feature Used to calculate GRANULATED_WAVEFRONT_SGPR_COUNT in 7622 Specific :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 7623 (+xnack) 7624 ``.amdhsa_float_round_mode_32`` 0 GFX6-GFX10 Controls FLOAT_ROUND_MODE_32 in 7625 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 7626 Possible values are defined in 7627 :ref:`amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table`. 7628 ``.amdhsa_float_round_mode_16_64`` 0 GFX6-GFX10 Controls FLOAT_ROUND_MODE_16_64 in 7629 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 7630 Possible values are defined in 7631 :ref:`amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table`. 7632 ``.amdhsa_float_denorm_mode_32`` 0 GFX6-GFX10 Controls FLOAT_DENORM_MODE_32 in 7633 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 7634 Possible values are defined in 7635 :ref:`amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table`. 7636 ``.amdhsa_float_denorm_mode_16_64`` 3 GFX6-GFX10 Controls FLOAT_DENORM_MODE_16_64 in 7637 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 7638 Possible values are defined in 7639 :ref:`amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table`. 7640 ``.amdhsa_dx10_clamp`` 1 GFX6-GFX10 Controls ENABLE_DX10_CLAMP in 7641 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 7642 ``.amdhsa_ieee_mode`` 1 GFX6-GFX10 Controls ENABLE_IEEE_MODE in 7643 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 7644 ``.amdhsa_fp16_overflow`` 0 GFX9-GFX10 Controls FP16_OVFL in 7645 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 7646 ``.amdhsa_workgroup_processor_mode`` Target GFX10 Controls ENABLE_WGP_MODE in 7647 Feature :ref:`amdgpu-amdhsa-kernel-descriptor-gfx6-gfx10-table`. 7648 Specific 7649 (-cumode) 7650 ``.amdhsa_memory_ordered`` 1 GFX10 Controls MEM_ORDERED in 7651 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 7652 ``.amdhsa_forward_progress`` 0 GFX10 Controls FWD_PROGRESS in 7653 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 7654 ``.amdhsa_exception_fp_ieee_invalid_op`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_INVALID_OPERATION in 7655 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 7656 ``.amdhsa_exception_fp_denorm_src`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_FP_DENORMAL_SOURCE in 7657 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 7658 ``.amdhsa_exception_fp_ieee_div_zero`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_DIVISION_BY_ZERO in 7659 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 7660 ``.amdhsa_exception_fp_ieee_overflow`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_OVERFLOW in 7661 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 7662 ``.amdhsa_exception_fp_ieee_underflow`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_UNDERFLOW in 7663 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 7664 ``.amdhsa_exception_fp_ieee_inexact`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_INEXACT in 7665 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 7666 ``.amdhsa_exception_int_div_zero`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_INT_DIVIDE_BY_ZERO in 7667 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 7668 ======================================================== =================== ============ =================== 7669 7670.amdgpu_metadata 7671++++++++++++++++ 7672 7673Optional directive which declares the contents of the ``NT_AMDGPU_METADATA`` 7674note record (see :ref:`amdgpu-elf-note-records-table-v3`). 7675 7676The contents must be in the [YAML]_ markup format, with the same structure and 7677semantics described in :ref:`amdgpu-amdhsa-code-object-metadata-v3`. 7678 7679This directive is terminated by an ``.end_amdgpu_metadata`` directive. 7680 7681.. _amdgpu-amdhsa-assembler-example-v3: 7682 7683Code Object V3 Example Source Code (-mattr=+code-object-v3) 7684~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 7685 7686Here is an example of a minimal assembly source file, defining one HSA kernel: 7687 7688.. code:: 7689 :number-lines: 7690 7691 .amdgcn_target "amdgcn-amd-amdhsa--gfx900+xnack" // optional 7692 7693 .text 7694 .globl hello_world 7695 .p2align 8 7696 .type hello_world,@function 7697 hello_world: 7698 s_load_dwordx2 s[0:1], s[0:1] 0x0 7699 v_mov_b32 v0, 3.14159 7700 s_waitcnt lgkmcnt(0) 7701 v_mov_b32 v1, s0 7702 v_mov_b32 v2, s1 7703 flat_store_dword v[1:2], v0 7704 s_endpgm 7705 .Lfunc_end0: 7706 .size hello_world, .Lfunc_end0-hello_world 7707 7708 .rodata 7709 .p2align 6 7710 .amdhsa_kernel hello_world 7711 .amdhsa_user_sgpr_kernarg_segment_ptr 1 7712 .amdhsa_next_free_vgpr .amdgcn.next_free_vgpr 7713 .amdhsa_next_free_sgpr .amdgcn.next_free_sgpr 7714 .end_amdhsa_kernel 7715 7716 .amdgpu_metadata 7717 --- 7718 amdhsa.version: 7719 - 1 7720 - 0 7721 amdhsa.kernels: 7722 - .name: hello_world 7723 .symbol: hello_world.kd 7724 .kernarg_segment_size: 48 7725 .group_segment_fixed_size: 0 7726 .private_segment_fixed_size: 0 7727 .kernarg_segment_align: 4 7728 .wavefront_size: 64 7729 .sgpr_count: 2 7730 .vgpr_count: 3 7731 .max_flat_workgroup_size: 256 7732 ... 7733 .end_amdgpu_metadata 7734 7735If an assembly source file contains multiple kernels and/or functions, the 7736:ref:`amdgpu-amdhsa-assembler-symbol-next_free_vgpr` and 7737:ref:`amdgpu-amdhsa-assembler-symbol-next_free_sgpr` symbols may be reset using 7738the ``.set <symbol>, <expression>`` directive. For example, in the case of two 7739kernels, where ``function1`` is only called from ``kernel1`` it is sufficient 7740to group the function with the kernel that calls it and reset the symbols 7741between the two connected components: 7742 7743.. code:: 7744 :number-lines: 7745 7746 .amdgcn_target "amdgcn-amd-amdhsa--gfx900+xnack" // optional 7747 7748 // gpr tracking symbols are implicitly set to zero 7749 7750 .text 7751 .globl kern0 7752 .p2align 8 7753 .type kern0,@function 7754 kern0: 7755 // ... 7756 s_endpgm 7757 .Lkern0_end: 7758 .size kern0, .Lkern0_end-kern0 7759 7760 .rodata 7761 .p2align 6 7762 .amdhsa_kernel kern0 7763 // ... 7764 .amdhsa_next_free_vgpr .amdgcn.next_free_vgpr 7765 .amdhsa_next_free_sgpr .amdgcn.next_free_sgpr 7766 .end_amdhsa_kernel 7767 7768 // reset symbols to begin tracking usage in func1 and kern1 7769 .set .amdgcn.next_free_vgpr, 0 7770 .set .amdgcn.next_free_sgpr, 0 7771 7772 .text 7773 .hidden func1 7774 .global func1 7775 .p2align 2 7776 .type func1,@function 7777 func1: 7778 // ... 7779 s_setpc_b64 s[30:31] 7780 .Lfunc1_end: 7781 .size func1, .Lfunc1_end-func1 7782 7783 .globl kern1 7784 .p2align 8 7785 .type kern1,@function 7786 kern1: 7787 // ... 7788 s_getpc_b64 s[4:5] 7789 s_add_u32 s4, s4, func1@rel32@lo+4 7790 s_addc_u32 s5, s5, func1@rel32@lo+4 7791 s_swappc_b64 s[30:31], s[4:5] 7792 // ... 7793 s_endpgm 7794 .Lkern1_end: 7795 .size kern1, .Lkern1_end-kern1 7796 7797 .rodata 7798 .p2align 6 7799 .amdhsa_kernel kern1 7800 // ... 7801 .amdhsa_next_free_vgpr .amdgcn.next_free_vgpr 7802 .amdhsa_next_free_sgpr .amdgcn.next_free_sgpr 7803 .end_amdhsa_kernel 7804 7805These symbols cannot identify connected components in order to automatically 7806track the usage for each kernel. However, in some cases careful organization of 7807the kernels and functions in the source file means there is minimal additional 7808effort required to accurately calculate GPR usage. 7809 7810Additional Documentation 7811======================== 7812 7813.. [AMD-GCN-GFX6] `AMD Southern Islands Series ISA <http://developer.amd.com/wordpress/media/2012/12/AMD_Southern_Islands_Instruction_Set_Architecture.pdf>`__ 7814.. [AMD-GCN-GFX7] `AMD Sea Islands Series ISA <http://developer.amd.com/wordpress/media/2013/07/AMD_Sea_Islands_Instruction_Set_Architecture.pdf>`_ 7815.. [AMD-GCN-GFX8] `AMD GCN3 Instruction Set Architecture <http://amd-dev.wpengine.netdna-cdn.com/wordpress/media/2013/12/AMD_GCN3_Instruction_Set_Architecture_rev1.1.pdf>`__ 7816.. [AMD-GCN-GFX9] `AMD "Vega" Instruction Set Architecture <http://developer.amd.com/wordpress/media/2013/12/Vega_Shader_ISA_28July2017.pdf>`__ 7817.. [AMD-GCN-GFX10] `AMD "RDNA 1.0" Instruction Set Architecture <https://gpuopen.com/wp-content/uploads/2019/08/RDNA_Shader_ISA_5August2019.pdf>`__ 7818.. [AMD-RADEON-HD-2000-3000] `AMD R6xx shader ISA <http://developer.amd.com/wordpress/media/2012/10/R600_Instruction_Set_Architecture.pdf>`__ 7819.. [AMD-RADEON-HD-4000] `AMD R7xx shader ISA <http://developer.amd.com/wordpress/media/2012/10/R700-Family_Instruction_Set_Architecture.pdf>`__ 7820.. [AMD-RADEON-HD-5000] `AMD Evergreen shader ISA <http://developer.amd.com/wordpress/media/2012/10/AMD_Evergreen-Family_Instruction_Set_Architecture.pdf>`__ 7821.. [AMD-RADEON-HD-6000] `AMD Cayman/Trinity shader ISA <http://developer.amd.com/wordpress/media/2012/10/AMD_HD_6900_Series_Instruction_Set_Architecture.pdf>`__ 7822.. [AMD-ROCm] `AMD ROCm Platform <https://rocm-documentation.readthedocs.io>`__ 7823.. [AMD-ROCm-github] `ROCm github <http://github.com/RadeonOpenCompute>`__ 7824.. [CLANG-ATTR] `Attributes in Clang <https://clang.llvm.org/docs/AttributeReference.html>`__ 7825.. [DWARF] `DWARF Debugging Information Format <http://dwarfstd.org/>`__ 7826.. [ELF] `Executable and Linkable Format (ELF) <http://www.sco.com/developers/gabi/>`__ 7827.. [HRF] `Heterogeneous-race-free Memory Models <http://benedictgaster.org/wp-content/uploads/2014/01/asplos269-FINAL.pdf>`__ 7828.. [HSA] `Heterogeneous System Architecture (HSA) Foundation <http://www.hsafoundation.com/>`__ 7829.. [MsgPack] `Message Pack <http://www.msgpack.org/>`__ 7830.. [OpenCL] `The OpenCL Specification Version 2.0 <http://www.khronos.org/registry/cl/specs/opencl-2.0.pdf>`__ 7831.. [SEMVER] `Semantic Versioning <https://semver.org/>`__ 7832.. [YAML] `YAML Ain't Markup Language (YAML™) Version 1.2 <http://www.yaml.org/spec/1.2/spec.html>`__ 7833