1============================= 2User Guide for AMDGPU Backend 3============================= 4 5.. contents:: 6 :local: 7 8.. toctree:: 9 :hidden: 10 11 AMDGPU/AMDGPUAsmGFX7 12 AMDGPU/AMDGPUAsmGFX8 13 AMDGPU/AMDGPUAsmGFX9 14 AMDGPU/AMDGPUAsmGFX900 15 AMDGPU/AMDGPUAsmGFX904 16 AMDGPU/AMDGPUAsmGFX906 17 AMDGPU/AMDGPUAsmGFX908 18 AMDGPU/AMDGPUAsmGFX10 19 AMDGPU/AMDGPUAsmGFX1011 20 AMDGPUModifierSyntax 21 AMDGPUOperandSyntax 22 AMDGPUInstructionSyntax 23 AMDGPUInstructionNotation 24 AMDGPUDwarfProposalForHeterogeneousDebugging 25 26Introduction 27============ 28 29The AMDGPU backend provides ISA code generation for AMD GPUs, starting with the 30R600 family up until the current GCN families. It lives in the 31``llvm/lib/Target/AMDGPU`` directory. 32 33LLVM 34==== 35 36.. _amdgpu-target-triples: 37 38Target Triples 39-------------- 40 41Use the ``clang -target <Architecture>-<Vendor>-<OS>-<Environment>`` option to 42specify the target triple: 43 44 .. table:: AMDGPU Architectures 45 :name: amdgpu-architecture-table 46 47 ============ ============================================================== 48 Architecture Description 49 ============ ============================================================== 50 ``r600`` AMD GPUs HD2XXX-HD6XXX for graphics and compute shaders. 51 ``amdgcn`` AMD GPUs GCN GFX6 onwards for graphics and compute shaders. 52 ============ ============================================================== 53 54 .. table:: AMDGPU Vendors 55 :name: amdgpu-vendor-table 56 57 ============ ============================================================== 58 Vendor Description 59 ============ ============================================================== 60 ``amd`` Can be used for all AMD GPU usage. 61 ``mesa3d`` Can be used if the OS is ``mesa3d``. 62 ============ ============================================================== 63 64 .. table:: AMDGPU Operating Systems 65 :name: amdgpu-os-table 66 67 ============== ============================================================ 68 OS Description 69 ============== ============================================================ 70 *<empty>* Defaults to the *unknown* OS. 71 ``amdhsa`` Compute kernels executed on HSA [HSA]_ compatible runtimes 72 such as AMD's ROCm [AMD-ROCm]_. 73 ``amdpal`` Graphic shaders and compute kernels executed on AMD PAL 74 runtime. 75 ``mesa3d`` Graphic shaders and compute kernels executed on Mesa 3D 76 runtime. 77 ============== ============================================================ 78 79 .. table:: AMDGPU Environments 80 :name: amdgpu-environment-table 81 82 ============ ============================================================== 83 Environment Description 84 ============ ============================================================== 85 *<empty>* Default. 86 ============ ============================================================== 87 88.. _amdgpu-processors: 89 90Processors 91---------- 92 93Use the ``clang -mcpu <Processor>`` option to specify the AMDGPU processor. The 94names from both the *Processor* and *Alternative Processor* can be used. 95 96 .. table:: AMDGPU Processors 97 :name: amdgpu-processor-table 98 99 =========== =============== ============ ===== ================= ======= ====================== 100 Processor Alternative Target dGPU/ Target ROCm Example 101 Processor Triple APU Features Support Products 102 Architecture Supported 103 [Default] 104 =========== =============== ============ ===== ================= ======= ====================== 105 **Radeon HD 2000/3000 Series (R600)** [AMD-RADEON-HD-2000-3000]_ 106 ----------------------------------------------------------------------------------------------- 107 ``r600`` ``r600`` dGPU 108 ``r630`` ``r600`` dGPU 109 ``rs880`` ``r600`` dGPU 110 ``rv670`` ``r600`` dGPU 111 **Radeon HD 4000 Series (R700)** [AMD-RADEON-HD-4000]_ 112 ----------------------------------------------------------------------------------------------- 113 ``rv710`` ``r600`` dGPU 114 ``rv730`` ``r600`` dGPU 115 ``rv770`` ``r600`` dGPU 116 **Radeon HD 5000 Series (Evergreen)** [AMD-RADEON-HD-5000]_ 117 ----------------------------------------------------------------------------------------------- 118 ``cedar`` ``r600`` dGPU 119 ``cypress`` ``r600`` dGPU 120 ``juniper`` ``r600`` dGPU 121 ``redwood`` ``r600`` dGPU 122 ``sumo`` ``r600`` dGPU 123 **Radeon HD 6000 Series (Northern Islands)** [AMD-RADEON-HD-6000]_ 124 ----------------------------------------------------------------------------------------------- 125 ``barts`` ``r600`` dGPU 126 ``caicos`` ``r600`` dGPU 127 ``cayman`` ``r600`` dGPU 128 ``turks`` ``r600`` dGPU 129 **GCN GFX6 (Southern Islands (SI))** [AMD-GCN-GFX6]_ 130 ----------------------------------------------------------------------------------------------- 131 ``gfx600`` - ``tahiti`` ``amdgcn`` dGPU 132 ``gfx601`` - ``hainan`` ``amdgcn`` dGPU 133 - ``oland`` 134 - ``pitcairn`` 135 - ``verde`` 136 **GCN GFX7 (Sea Islands (CI))** [AMD-GCN-GFX7]_ 137 ----------------------------------------------------------------------------------------------- 138 ``gfx700`` - ``kaveri`` ``amdgcn`` APU - A6-7000 139 - A6 Pro-7050B 140 - A8-7100 141 - A8 Pro-7150B 142 - A10-7300 143 - A10 Pro-7350B 144 - FX-7500 145 - A8-7200P 146 - A10-7400P 147 - FX-7600P 148 ``gfx701`` - ``hawaii`` ``amdgcn`` dGPU ROCm - FirePro W8100 149 - FirePro W9100 150 - FirePro S9150 151 - FirePro S9170 152 ``gfx702`` ``amdgcn`` dGPU ROCm - Radeon R9 290 153 - Radeon R9 290x 154 - Radeon R390 155 - Radeon R390x 156 ``gfx703`` - ``kabini`` ``amdgcn`` APU - E1-2100 157 - ``mullins`` - E1-2200 158 - E1-2500 159 - E2-3000 160 - E2-3800 161 - A4-5000 162 - A4-5100 163 - A6-5200 164 - A4 Pro-3340B 165 ``gfx704`` - ``bonaire`` ``amdgcn`` dGPU - Radeon HD 7790 166 - Radeon HD 8770 167 - R7 260 168 - R7 260X 169 **GCN GFX8 (Volcanic Islands (VI))** [AMD-GCN-GFX8]_ 170 ----------------------------------------------------------------------------------------------- 171 ``gfx801`` - ``carrizo`` ``amdgcn`` APU - xnack - A6-8500P 172 [on] - Pro A6-8500B 173 - A8-8600P 174 - Pro A8-8600B 175 - FX-8800P 176 - Pro A12-8800B 177 \ ``amdgcn`` APU - xnack ROCm - A10-8700P 178 [on] - Pro A10-8700B 179 - A10-8780P 180 \ ``amdgcn`` APU - xnack - A10-9600P 181 [on] - A10-9630P 182 - A12-9700P 183 - A12-9730P 184 - FX-9800P 185 - FX-9830P 186 \ ``amdgcn`` APU - xnack - E2-9010 187 [on] - A6-9210 188 - A9-9410 189 ``gfx802`` - ``iceland`` ``amdgcn`` dGPU - xnack ROCm - FirePro S7150 190 - ``tonga`` [off] - FirePro S7100 191 - FirePro W7100 192 - Radeon R285 193 - Radeon R9 380 194 - Radeon R9 385 195 - Mobile FirePro 196 M7170 197 ``gfx803`` - ``fiji`` ``amdgcn`` dGPU - xnack ROCm - Radeon R9 Nano 198 [off] - Radeon R9 Fury 199 - Radeon R9 FuryX 200 - Radeon Pro Duo 201 - FirePro S9300x2 202 - Radeon Instinct MI8 203 \ - ``polaris10`` ``amdgcn`` dGPU - xnack ROCm - Radeon RX 470 204 [off] - Radeon RX 480 205 - Radeon Instinct MI6 206 \ - ``polaris11`` ``amdgcn`` dGPU - xnack ROCm - Radeon RX 460 207 [off] 208 ``gfx810`` - ``stoney`` ``amdgcn`` APU - xnack 209 [on] 210 **GCN GFX9** [AMD-GCN-GFX9]_ 211 ----------------------------------------------------------------------------------------------- 212 ``gfx900`` ``amdgcn`` dGPU - xnack ROCm - Radeon Vega 213 [off] Frontier Edition 214 - Radeon RX Vega 56 215 - Radeon RX Vega 64 216 - Radeon RX Vega 64 217 Liquid 218 - Radeon Instinct MI25 219 ``gfx902`` ``amdgcn`` APU - xnack - Ryzen 3 2200G 220 [on] - Ryzen 5 2400G 221 ``gfx904`` ``amdgcn`` dGPU - xnack *TBA* 222 [off] 223 .. TODO:: 224 Add product 225 names. 226 ``gfx906`` ``amdgcn`` dGPU - xnack - Radeon Instinct MI50 227 [off] - Radeon Instinct MI60 228 ``gfx908`` ``amdgcn`` dGPU - xnack *TBA* 229 [off] 230 sram-ecc 231 [on] 232 ``gfx909`` ``amdgcn`` APU - xnack *TBA* (Raven Ridge 2) 233 [on] 234 .. TODO:: 235 Add product 236 names. 237 **GCN GFX10** [AMD-GCN-GFX10]_ 238 ----------------------------------------------------------------------------------------------- 239 ``gfx1010`` ``amdgcn`` dGPU - xnack *TBA* 240 [off] 241 - wavefrontsize64 242 [off] 243 - cumode 244 [off] 245 .. TODO:: 246 Add product 247 names. 248 ``gfx1011`` ``amdgcn`` dGPU - xnack *TBA* 249 [off] 250 - wavefrontsize64 251 [off] 252 - cumode 253 [off] 254 .. TODO:: 255 Add product 256 names. 257 ``gfx1012`` ``amdgcn`` dGPU - xnack *TBA* 258 [off] 259 - wavefrontsize64 260 [off] 261 - cumode 262 [off] 263 .. TODO:: 264 Add product 265 names. 266 ``gfx1030`` ``amdgcn`` dGPU - xnack *TBA* 267 [off] 268 - wavefrontsize64 269 [off] 270 - cumode 271 [off] 272 .. TODO 273 Add product 274 names. 275 =========== =============== ============ ===== ================= ======= ====================== 276 277.. _amdgpu-target-features: 278 279Target Features 280--------------- 281 282Target features control how code is generated to support certain 283processor specific features. Not all target features are supported by 284all processors. The runtime must ensure that the features supported by 285the device used to execute the code match the features enabled when 286generating the code. A mismatch of features may result in incorrect 287execution, or a reduction in performance. 288 289The target features supported by each processor, and the default value 290used if not specified explicitly, is listed in 291:ref:`amdgpu-processor-table`. 292 293Use the ``clang -m[no-]<TargetFeature>`` option to specify the AMDGPU 294target features. 295 296For example: 297 298``-mxnack`` 299 Enable the ``xnack`` feature. 300``-mno-xnack`` 301 Disable the ``xnack`` feature. 302 303 .. table:: AMDGPU Target Features 304 :name: amdgpu-target-feature-table 305 306 ====================== ================================================== 307 Target Feature Description 308 ====================== ================================================== 309 -m[no-]xnack Enable/disable generating code that has 310 memory clauses that are compatible with 311 having XNACK replay enabled. 312 313 This is used for demand paging and page 314 migration. If XNACK replay is enabled in 315 the device, then if a page fault occurs 316 the code may execute incorrectly if the 317 ``xnack`` feature is not enabled. Executing 318 code that has the feature enabled on a 319 device that does not have XNACK replay 320 enabled will execute correctly but may 321 be less performant than code with the 322 feature disabled. 323 324 -m[no-]sram-ecc Enable/disable generating code that assumes SRAM 325 ECC is enabled/disabled. 326 327 -m[no-]wavefrontsize64 Control the default wavefront size used when 328 generating code for kernels. When disabled 329 native wavefront size 32 is used, when enabled 330 wavefront size 64 is used. 331 332 -m[no-]cumode Control the default wavefront execution mode used 333 when generating code for kernels. When disabled 334 native WGP wavefront execution mode is used, 335 when enabled CU wavefront execution mode is used 336 (see :ref:`amdgpu-amdhsa-memory-model`). 337 ====================== ================================================== 338 339.. _amdgpu-address-spaces: 340 341Address Spaces 342-------------- 343 344The AMDGPU architecture supports a number of memory address spaces. The address 345space names use the OpenCL standard names, with some additions. 346 347The AMDGPU address spaces correspond to target architecture specific LLVM 348address space numbers used in LLVM IR. 349 350The AMDGPU address spaces are described in 351:ref:`amdgpu-address-spaces-table`. Only 64-bit process address spaces are 352supported for the ``amdgcn`` target. 353 354 .. table:: AMDGPU Address Spaces 355 :name: amdgpu-address-spaces-table 356 357 ================================= =============== =========== ================ ======= ============================ 358 .. 64-Bit Process Address Space 359 --------------------------------- --------------- ----------- ---------------- ------------------------------------ 360 Address Space Name LLVM IR Address HSA Segment Hardware Address NULL Value 361 Space Number Name Name Size 362 ================================= =============== =========== ================ ======= ============================ 363 Generic 0 flat flat 64 0x0000000000000000 364 Global 1 global global 64 0x0000000000000000 365 Region 2 N/A GDS 32 *not implemented for AMDHSA* 366 Local 3 group LDS 32 0xFFFFFFFF 367 Constant 4 constant *same as global* 64 0x0000000000000000 368 Private 5 private scratch 32 0xFFFFFFFF 369 Constant 32-bit 6 *TODO* 0x00000000 370 Buffer Fat Pointer (experimental) 7 *TODO* 371 ================================= =============== =========== ================ ======= ============================ 372 373**Generic** 374 The generic address space uses the hardware flat address support available in 375 GFX7-GFX10. This uses two fixed ranges of virtual addresses (the private and 376 local apertures), that are outside the range of addressable global memory, to 377 map from a flat address to a private or local address. 378 379 FLAT instructions can take a flat address and access global, private 380 (scratch), and group (LDS) memory depending on if the address is within one 381 of the aperture ranges. Flat access to scratch requires hardware aperture 382 setup and setup in the kernel prologue (see 383 :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`). Flat access to LDS requires 384 hardware aperture setup and M0 (GFX7-GFX8) register setup (see 385 :ref:`amdgpu-amdhsa-kernel-prolog-m0`). 386 387 To convert between a private or group address space address (termed a segment 388 address) and a flat address the base address of the corresponding aperture 389 can be used. For GFX7-GFX8 these are available in the 390 :ref:`amdgpu-amdhsa-hsa-aql-queue` the address of which can be obtained with 391 Queue Ptr SGPR (see :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). For 392 GFX9-GFX10 the aperture base addresses are directly available as inline 393 constant registers ``SRC_SHARED_BASE/LIMIT`` and ``SRC_PRIVATE_BASE/LIMIT``. 394 In 64-bit address mode the aperture sizes are 2^32 bytes and the base is 395 aligned to 2^32 which makes it easier to convert from flat to segment or 396 segment to flat. 397 398 A global address space address has the same value when used as a flat address 399 so no conversion is needed. 400 401**Global and Constant** 402 The global and constant address spaces both use global virtual addresses, 403 which are the same virtual address space used by the CPU. However, some 404 virtual addresses may only be accessible to the CPU, some only accessible 405 by the GPU, and some by both. 406 407 Using the constant address space indicates that the data will not change 408 during the execution of the kernel. This allows scalar read instructions to 409 be used. The vector and scalar L1 caches are invalidated of volatile data 410 before each kernel dispatch execution to allow constant memory to change 411 values between kernel dispatches. 412 413**Region** 414 The region address space uses the hardware Global Data Store (GDS). All 415 wavefronts executing on the same device will access the same memory for any 416 given region address. However, the same region address accessed by wavefronts 417 executing on different devices will access different memory. It is higher 418 performance than global memory. It is allocated by the runtime. The data 419 store (DS) instructions can be used to access it. 420 421**Local** 422 The local address space uses the hardware Local Data Store (LDS) which is 423 automatically allocated when the hardware creates the wavefronts of a 424 work-group, and freed when all the wavefronts of a work-group have 425 terminated. All wavefronts belonging to the same work-group will access the 426 same memory for any given local address. However, the same local address 427 accessed by wavefronts belonging to different work-groups will access 428 different memory. It is higher performance than global memory. The data store 429 (DS) instructions can be used to access it. 430 431**Private** 432 The private address space uses the hardware scratch memory support which 433 automatically allocates memory when it creates a wavefront and frees it when 434 a wavefronts terminates. The memory accessed by a lane of a wavefront for any 435 given private address will be different to the memory accessed by another lane 436 of the same or different wavefront for the same private address. 437 438 If a kernel dispatch uses scratch, then the hardware allocates memory from a 439 pool of backing memory allocated by the runtime for each wavefront. The lanes 440 of the wavefront access this using dword (4 byte) interleaving. The mapping 441 used from private address to backing memory address is: 442 443 ``wavefront-scratch-base + 444 ((private-address / 4) * wavefront-size * 4) + 445 (wavefront-lane-id * 4) + (private-address % 4)`` 446 447 If each lane of a wavefront accesses the same private address, the 448 interleaving results in adjacent dwords being accessed and hence requires 449 fewer cache lines to be fetched. 450 451 There are different ways that the wavefront scratch base address is 452 determined by a wavefront (see 453 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 454 455 Scratch memory can be accessed in an interleaved manner using buffer 456 instructions with the scratch buffer descriptor and per wavefront scratch 457 offset, by the scratch instructions, or by flat instructions. Multi-dword 458 access is not supported except by flat and scratch instructions in 459 GFX9-GFX10. 460 461**Constant 32-bit** 462 *TODO* 463 464**Buffer Fat Pointer** 465 The buffer fat pointer is an experimental address space that is currently 466 unsupported in the backend. It exposes a non-integral pointer that is in 467 the future intended to support the modelling of 128-bit buffer descriptors 468 plus a 32-bit offset into the buffer (in total encapsulating a 160-bit 469 *pointer*), allowing normal LLVM load/store/atomic operations to be used to 470 model the buffer descriptors used heavily in graphics workloads targeting 471 the backend. 472 473.. _amdgpu-memory-scopes: 474 475Memory Scopes 476------------- 477 478This section provides LLVM memory synchronization scopes supported by the AMDGPU 479backend memory model when the target triple OS is ``amdhsa`` (see 480:ref:`amdgpu-amdhsa-memory-model` and :ref:`amdgpu-target-triples`). 481 482The memory model supported is based on the HSA memory model [HSA]_ which is 483based in turn on HRF-indirect with scope inclusion [HRF]_. The happens-before 484relation is transitive over the synchronizes-with relation independent of scope 485and synchronizes-with allows the memory scope instances to be inclusive (see 486table :ref:`amdgpu-amdhsa-llvm-sync-scopes-table`). 487 488This is different to the OpenCL [OpenCL]_ memory model which does not have scope 489inclusion and requires the memory scopes to exactly match. However, this 490is conservatively correct for OpenCL. 491 492 .. table:: AMDHSA LLVM Sync Scopes 493 :name: amdgpu-amdhsa-llvm-sync-scopes-table 494 495 ======================= =================================================== 496 LLVM Sync Scope Description 497 ======================= =================================================== 498 *none* The default: ``system``. 499 500 Synchronizes with, and participates in modification 501 and seq_cst total orderings with, other operations 502 (except image operations) for all address spaces 503 (except private, or generic that accesses private) 504 provided the other operation's sync scope is: 505 506 - ``system``. 507 - ``agent`` and executed by a thread on the same 508 agent. 509 - ``workgroup`` and executed by a thread in the 510 same work-group. 511 - ``wavefront`` and executed by a thread in the 512 same wavefront. 513 514 ``agent`` Synchronizes with, and participates in modification 515 and seq_cst total orderings with, other operations 516 (except image operations) for all address spaces 517 (except private, or generic that accesses private) 518 provided the other operation's sync scope is: 519 520 - ``system`` or ``agent`` and executed by a thread 521 on the same agent. 522 - ``workgroup`` and executed by a thread in the 523 same work-group. 524 - ``wavefront`` and executed by a thread in the 525 same wavefront. 526 527 ``workgroup`` Synchronizes with, and participates in modification 528 and seq_cst total orderings with, other operations 529 (except image operations) for all address spaces 530 (except private, or generic that accesses private) 531 provided the other operation's sync scope is: 532 533 - ``system``, ``agent`` or ``workgroup`` and 534 executed by a thread in the same work-group. 535 - ``wavefront`` and executed by a thread in the 536 same wavefront. 537 538 ``wavefront`` Synchronizes with, and participates in modification 539 and seq_cst total orderings with, other operations 540 (except image operations) for all address spaces 541 (except private, or generic that accesses private) 542 provided the other operation's sync scope is: 543 544 - ``system``, ``agent``, ``workgroup`` or 545 ``wavefront`` and executed by a thread in the 546 same wavefront. 547 548 ``singlethread`` Only synchronizes with and participates in 549 modification and seq_cst total orderings with, 550 other operations (except image operations) running 551 in the same thread for all address spaces (for 552 example, in signal handlers). 553 554 ``one-as`` Same as ``system`` but only synchronizes with other 555 operations within the same address space. 556 557 ``agent-one-as`` Same as ``agent`` but only synchronizes with other 558 operations within the same address space. 559 560 ``workgroup-one-as`` Same as ``workgroup`` but only synchronizes with 561 other operations within the same address space. 562 563 ``wavefront-one-as`` Same as ``wavefront`` but only synchronizes with 564 other operations within the same address space. 565 566 ``singlethread-one-as`` Same as ``singlethread`` but only synchronizes with 567 other operations within the same address space. 568 ======================= =================================================== 569 570LLVM IR Intrinsics 571------------------ 572 573The AMDGPU backend implements the following LLVM IR intrinsics. 574 575*This section is WIP.* 576 577.. TODO:: 578 579 List AMDGPU intrinsics. 580 581LLVM IR Attributes 582------------------ 583 584The AMDGPU backend supports the following LLVM IR attributes. 585 586 .. table:: AMDGPU LLVM IR Attributes 587 :name: amdgpu-llvm-ir-attributes-table 588 589 ======================================= ========================================================== 590 LLVM Attribute Description 591 ======================================= ========================================================== 592 "amdgpu-flat-work-group-size"="min,max" Specify the minimum and maximum flat work group sizes that 593 will be specified when the kernel is dispatched. Generated 594 by the ``amdgpu_flat_work_group_size`` CLANG attribute [CLANG-ATTR]_. 595 "amdgpu-implicitarg-num-bytes"="n" Number of kernel argument bytes to add to the kernel 596 argument block size for the implicit arguments. This 597 varies by OS and language (for OpenCL see 598 :ref:`opencl-kernel-implicit-arguments-appended-for-amdhsa-os-table`). 599 "amdgpu-num-sgpr"="n" Specifies the number of SGPRs to use. Generated by 600 the ``amdgpu_num_sgpr`` CLANG attribute [CLANG-ATTR]_. 601 "amdgpu-num-vgpr"="n" Specifies the number of VGPRs to use. Generated by the 602 ``amdgpu_num_vgpr`` CLANG attribute [CLANG-ATTR]_. 603 "amdgpu-waves-per-eu"="m,n" Specify the minimum and maximum number of waves per 604 execution unit. Generated by the ``amdgpu_waves_per_eu`` 605 CLANG attribute [CLANG-ATTR]_. 606 "amdgpu-ieee" true/false. Specify whether the function expects the IEEE field of the 607 mode register to be set on entry. Overrides the default for 608 the calling convention. 609 "amdgpu-dx10-clamp" true/false. Specify whether the function expects the DX10_CLAMP field of 610 the mode register to be set on entry. Overrides the default 611 for the calling convention. 612 ======================================= ========================================================== 613 614.. _amdgpu-elf-code-object: 615 616ELF Code Object 617=============== 618 619The AMDGPU backend generates a standard ELF [ELF]_ relocatable code object that 620can be linked by ``lld`` to produce a standard ELF shared code object which can 621be loaded and executed on an AMDGPU target. 622 623.. _amdgpu-elf-header: 624 625Header 626------ 627 628The AMDGPU backend uses the following ELF header: 629 630 .. table:: AMDGPU ELF Header 631 :name: amdgpu-elf-header-table 632 633 ========================== =============================== 634 Field Value 635 ========================== =============================== 636 ``e_ident[EI_CLASS]`` ``ELFCLASS64`` 637 ``e_ident[EI_DATA]`` ``ELFDATA2LSB`` 638 ``e_ident[EI_OSABI]`` - ``ELFOSABI_NONE`` 639 - ``ELFOSABI_AMDGPU_HSA`` 640 - ``ELFOSABI_AMDGPU_PAL`` 641 - ``ELFOSABI_AMDGPU_MESA3D`` 642 ``e_ident[EI_ABIVERSION]`` - ``ELFABIVERSION_AMDGPU_HSA`` 643 - ``ELFABIVERSION_AMDGPU_PAL`` 644 - ``ELFABIVERSION_AMDGPU_MESA3D`` 645 ``e_type`` - ``ET_REL`` 646 - ``ET_DYN`` 647 ``e_machine`` ``EM_AMDGPU`` 648 ``e_entry`` 0 649 ``e_flags`` See :ref:`amdgpu-elf-header-e_flags-table` 650 ========================== =============================== 651 652.. 653 654 .. table:: AMDGPU ELF Header Enumeration Values 655 :name: amdgpu-elf-header-enumeration-values-table 656 657 =============================== ===== 658 Name Value 659 =============================== ===== 660 ``EM_AMDGPU`` 224 661 ``ELFOSABI_NONE`` 0 662 ``ELFOSABI_AMDGPU_HSA`` 64 663 ``ELFOSABI_AMDGPU_PAL`` 65 664 ``ELFOSABI_AMDGPU_MESA3D`` 66 665 ``ELFABIVERSION_AMDGPU_HSA`` 1 666 ``ELFABIVERSION_AMDGPU_PAL`` 0 667 ``ELFABIVERSION_AMDGPU_MESA3D`` 0 668 =============================== ===== 669 670``e_ident[EI_CLASS]`` 671 The ELF class is: 672 673 * ``ELFCLASS32`` for ``r600`` architecture. 674 675 * ``ELFCLASS64`` for ``amdgcn`` architecture which only supports 64-bit 676 process address space applications. 677 678``e_ident[EI_DATA]`` 679 All AMDGPU targets use ``ELFDATA2LSB`` for little-endian byte ordering. 680 681``e_ident[EI_OSABI]`` 682 One of the following AMDGPU target architecture specific OS ABIs 683 (see :ref:`amdgpu-os-table`): 684 685 * ``ELFOSABI_NONE`` for *unknown* OS. 686 687 * ``ELFOSABI_AMDGPU_HSA`` for ``amdhsa`` OS. 688 689 * ``ELFOSABI_AMDGPU_PAL`` for ``amdpal`` OS. 690 691 * ``ELFOSABI_AMDGPU_MESA3D`` for ``mesa3D`` OS. 692 693``e_ident[EI_ABIVERSION]`` 694 The ABI version of the AMDGPU target architecture specific OS ABI to which the code 695 object conforms: 696 697 * ``ELFABIVERSION_AMDGPU_HSA`` is used to specify the version of AMD HSA 698 runtime ABI. 699 700 * ``ELFABIVERSION_AMDGPU_PAL`` is used to specify the version of AMD PAL 701 runtime ABI. 702 703 * ``ELFABIVERSION_AMDGPU_MESA3D`` is used to specify the version of AMD MESA 704 3D runtime ABI. 705 706``e_type`` 707 Can be one of the following values: 708 709 710 ``ET_REL`` 711 The type produced by the AMDGPU backend compiler as it is relocatable code 712 object. 713 714 ``ET_DYN`` 715 The type produced by the linker as it is a shared code object. 716 717 The AMD HSA runtime loader requires a ``ET_DYN`` code object. 718 719``e_machine`` 720 The value ``EM_AMDGPU`` is used for the machine for all processors supported 721 by the ``r600`` and ``amdgcn`` architectures (see 722 :ref:`amdgpu-processor-table`). The specific processor is specified in the 723 ``EF_AMDGPU_MACH`` bit field of the ``e_flags`` (see 724 :ref:`amdgpu-elf-header-e_flags-table`). 725 726``e_entry`` 727 The entry point is 0 as the entry points for individual kernels must be 728 selected in order to invoke them through AQL packets. 729 730``e_flags`` 731 The AMDGPU backend uses the following ELF header flags: 732 733 .. table:: AMDGPU ELF Header ``e_flags`` 734 :name: amdgpu-elf-header-e_flags-table 735 736 ================================= ========== ============================= 737 Name Value Description 738 ================================= ========== ============================= 739 **AMDGPU Processor Flag** See :ref:`amdgpu-processor-table`. 740 -------------------------------------------- ----------------------------- 741 ``EF_AMDGPU_MACH`` 0x000000ff AMDGPU processor selection 742 mask for 743 ``EF_AMDGPU_MACH_xxx`` values 744 defined in 745 :ref:`amdgpu-ef-amdgpu-mach-table`. 746 ``EF_AMDGPU_XNACK`` 0x00000100 Indicates if the ``xnack`` 747 target feature is 748 enabled for all code 749 contained in the code object. 750 If the processor 751 does not support the 752 ``xnack`` target 753 feature then must 754 be 0. 755 See 756 :ref:`amdgpu-target-features`. 757 ``EF_AMDGPU_SRAM_ECC`` 0x00000200 Indicates if the ``sram-ecc`` 758 target feature is 759 enabled for all code 760 contained in the code object. 761 If the processor 762 does not support the 763 ``sram-ecc`` target 764 feature then must 765 be 0. 766 See 767 :ref:`amdgpu-target-features`. 768 ================================= ========== ============================= 769 770 .. table:: AMDGPU ``EF_AMDGPU_MACH`` Values 771 :name: amdgpu-ef-amdgpu-mach-table 772 773 ================================= ========== ============================= 774 Name Value Description (see 775 :ref:`amdgpu-processor-table`) 776 ================================= ========== ============================= 777 ``EF_AMDGPU_MACH_NONE`` 0x000 *not specified* 778 ``EF_AMDGPU_MACH_R600_R600`` 0x001 ``r600`` 779 ``EF_AMDGPU_MACH_R600_R630`` 0x002 ``r630`` 780 ``EF_AMDGPU_MACH_R600_RS880`` 0x003 ``rs880`` 781 ``EF_AMDGPU_MACH_R600_RV670`` 0x004 ``rv670`` 782 ``EF_AMDGPU_MACH_R600_RV710`` 0x005 ``rv710`` 783 ``EF_AMDGPU_MACH_R600_RV730`` 0x006 ``rv730`` 784 ``EF_AMDGPU_MACH_R600_RV770`` 0x007 ``rv770`` 785 ``EF_AMDGPU_MACH_R600_CEDAR`` 0x008 ``cedar`` 786 ``EF_AMDGPU_MACH_R600_CYPRESS`` 0x009 ``cypress`` 787 ``EF_AMDGPU_MACH_R600_JUNIPER`` 0x00a ``juniper`` 788 ``EF_AMDGPU_MACH_R600_REDWOOD`` 0x00b ``redwood`` 789 ``EF_AMDGPU_MACH_R600_SUMO`` 0x00c ``sumo`` 790 ``EF_AMDGPU_MACH_R600_BARTS`` 0x00d ``barts`` 791 ``EF_AMDGPU_MACH_R600_CAICOS`` 0x00e ``caicos`` 792 ``EF_AMDGPU_MACH_R600_CAYMAN`` 0x00f ``cayman`` 793 ``EF_AMDGPU_MACH_R600_TURKS`` 0x010 ``turks`` 794 *reserved* 0x011 - Reserved for ``r600`` 795 0x01f architecture processors. 796 ``EF_AMDGPU_MACH_AMDGCN_GFX600`` 0x020 ``gfx600`` 797 ``EF_AMDGPU_MACH_AMDGCN_GFX601`` 0x021 ``gfx601`` 798 ``EF_AMDGPU_MACH_AMDGCN_GFX700`` 0x022 ``gfx700`` 799 ``EF_AMDGPU_MACH_AMDGCN_GFX701`` 0x023 ``gfx701`` 800 ``EF_AMDGPU_MACH_AMDGCN_GFX702`` 0x024 ``gfx702`` 801 ``EF_AMDGPU_MACH_AMDGCN_GFX703`` 0x025 ``gfx703`` 802 ``EF_AMDGPU_MACH_AMDGCN_GFX704`` 0x026 ``gfx704`` 803 *reserved* 0x027 Reserved. 804 ``EF_AMDGPU_MACH_AMDGCN_GFX801`` 0x028 ``gfx801`` 805 ``EF_AMDGPU_MACH_AMDGCN_GFX802`` 0x029 ``gfx802`` 806 ``EF_AMDGPU_MACH_AMDGCN_GFX803`` 0x02a ``gfx803`` 807 ``EF_AMDGPU_MACH_AMDGCN_GFX810`` 0x02b ``gfx810`` 808 ``EF_AMDGPU_MACH_AMDGCN_GFX900`` 0x02c ``gfx900`` 809 ``EF_AMDGPU_MACH_AMDGCN_GFX902`` 0x02d ``gfx902`` 810 ``EF_AMDGPU_MACH_AMDGCN_GFX904`` 0x02e ``gfx904`` 811 ``EF_AMDGPU_MACH_AMDGCN_GFX906`` 0x02f ``gfx906`` 812 ``EF_AMDGPU_MACH_AMDGCN_GFX908`` 0x030 ``gfx908`` 813 ``EF_AMDGPU_MACH_AMDGCN_GFX909`` 0x031 ``gfx909`` 814 *reserved* 0x032 Reserved. 815 ``EF_AMDGPU_MACH_AMDGCN_GFX1010`` 0x033 ``gfx1010`` 816 ``EF_AMDGPU_MACH_AMDGCN_GFX1011`` 0x034 ``gfx1011`` 817 ``EF_AMDGPU_MACH_AMDGCN_GFX1012`` 0x035 ``gfx1012`` 818 ``EF_AMDGPU_MACH_AMDGCN_GFX1030`` 0x036 ``gfx1030`` 819 ================================= ========== ============================= 820 821Sections 822-------- 823 824An AMDGPU target ELF code object has the standard ELF sections which include: 825 826 .. table:: AMDGPU ELF Sections 827 :name: amdgpu-elf-sections-table 828 829 ================== ================ ================================= 830 Name Type Attributes 831 ================== ================ ================================= 832 ``.bss`` ``SHT_NOBITS`` ``SHF_ALLOC`` + ``SHF_WRITE`` 833 ``.data`` ``SHT_PROGBITS`` ``SHF_ALLOC`` + ``SHF_WRITE`` 834 ``.debug_``\ *\** ``SHT_PROGBITS`` *none* 835 ``.dynamic`` ``SHT_DYNAMIC`` ``SHF_ALLOC`` 836 ``.dynstr`` ``SHT_PROGBITS`` ``SHF_ALLOC`` 837 ``.dynsym`` ``SHT_PROGBITS`` ``SHF_ALLOC`` 838 ``.got`` ``SHT_PROGBITS`` ``SHF_ALLOC`` + ``SHF_WRITE`` 839 ``.hash`` ``SHT_HASH`` ``SHF_ALLOC`` 840 ``.note`` ``SHT_NOTE`` *none* 841 ``.rela``\ *name* ``SHT_RELA`` *none* 842 ``.rela.dyn`` ``SHT_RELA`` *none* 843 ``.rodata`` ``SHT_PROGBITS`` ``SHF_ALLOC`` 844 ``.shstrtab`` ``SHT_STRTAB`` *none* 845 ``.strtab`` ``SHT_STRTAB`` *none* 846 ``.symtab`` ``SHT_SYMTAB`` *none* 847 ``.text`` ``SHT_PROGBITS`` ``SHF_ALLOC`` + ``SHF_EXECINSTR`` 848 ================== ================ ================================= 849 850These sections have their standard meanings (see [ELF]_) and are only generated 851if needed. 852 853``.debug``\ *\** 854 The standard DWARF sections. See :ref:`amdgpu-dwarf-debug-information` for 855 information on the DWARF produced by the AMDGPU backend. 856 857``.dynamic``, ``.dynstr``, ``.dynsym``, ``.hash`` 858 The standard sections used by a dynamic loader. 859 860``.note`` 861 See :ref:`amdgpu-note-records` for the note records supported by the AMDGPU 862 backend. 863 864``.rela``\ *name*, ``.rela.dyn`` 865 For relocatable code objects, *name* is the name of the section that the 866 relocation records apply. For example, ``.rela.text`` is the section name for 867 relocation records associated with the ``.text`` section. 868 869 For linked shared code objects, ``.rela.dyn`` contains all the relocation 870 records from each of the relocatable code object's ``.rela``\ *name* sections. 871 872 See :ref:`amdgpu-relocation-records` for the relocation records supported by 873 the AMDGPU backend. 874 875``.text`` 876 The executable machine code for the kernels and functions they call. Generated 877 as position independent code. See :ref:`amdgpu-code-conventions` for 878 information on conventions used in the isa generation. 879 880.. _amdgpu-note-records: 881 882Note Records 883------------ 884 885The AMDGPU backend code object contains ELF note records in the ``.note`` 886section. The set of generated notes and their semantics depend on the code 887object version; see :ref:`amdgpu-note-records-v2` and 888:ref:`amdgpu-note-records-v3`. 889 890As required by ``ELFCLASS32`` and ``ELFCLASS64``, minimal zero-byte padding 891must be generated after the ``name`` field to ensure the ``desc`` field is 4 892byte aligned. In addition, minimal zero-byte padding must be generated to 893ensure the ``desc`` field size is a multiple of 4 bytes. The ``sh_addralign`` 894field of the ``.note`` section must be at least 4 to indicate at least 8 byte 895alignment. 896 897.. _amdgpu-note-records-v2: 898 899Code Object V2 Note Records (-mattr=-code-object-v3) 900~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 901 902.. warning:: Code Object V2 is not the default code object version emitted by 903 this version of LLVM. For a description of the notes generated with the 904 default configuration (Code Object V3) see :ref:`amdgpu-note-records-v3`. 905 906The AMDGPU backend code object uses the following ELF note record in the 907``.note`` section when compiling for Code Object V2 (-mattr=-code-object-v3). 908 909Additional note records may be present, but any which are not documented here 910are deprecated and should not be used. 911 912 .. table:: AMDGPU Code Object V2 ELF Note Records 913 :name: amdgpu-elf-note-records-table-v2 914 915 ===== ============================== ====================================== 916 Name Type Description 917 ===== ============================== ====================================== 918 "AMD" ``NT_AMD_AMDGPU_HSA_METADATA`` <metadata null terminated string> 919 ===== ============================== ====================================== 920 921.. 922 923 .. table:: AMDGPU Code Object V2 ELF Note Record Enumeration Values 924 :name: amdgpu-elf-note-record-enumeration-values-table-v2 925 926 ============================== ===== 927 Name Value 928 ============================== ===== 929 *reserved* 0-9 930 ``NT_AMD_AMDGPU_HSA_METADATA`` 10 931 *reserved* 11 932 ============================== ===== 933 934``NT_AMD_AMDGPU_HSA_METADATA`` 935 Specifies extensible metadata associated with the code objects executed on HSA 936 [HSA]_ compatible runtimes such as AMD's ROCm [AMD-ROCm]_. It is required when 937 the target triple OS is ``amdhsa`` (see :ref:`amdgpu-target-triples`). See 938 :ref:`amdgpu-amdhsa-code-object-metadata-v2` for the syntax of the code 939 object metadata string. 940 941.. _amdgpu-note-records-v3: 942 943Code Object V3 Note Records (-mattr=+code-object-v3) 944~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 945 946The AMDGPU backend code object uses the following ELF note record in the 947``.note`` section when compiling for Code Object V3 (-mattr=+code-object-v3). 948 949Additional note records may be present, but any which are not documented here 950are deprecated and should not be used. 951 952 .. table:: AMDGPU Code Object V3 ELF Note Records 953 :name: amdgpu-elf-note-records-table-v3 954 955 ======== ============================== ====================================== 956 Name Type Description 957 ======== ============================== ====================================== 958 "AMDGPU" ``NT_AMDGPU_METADATA`` Metadata in Message Pack [MsgPack]_ 959 binary format. 960 ======== ============================== ====================================== 961 962.. 963 964 .. table:: AMDGPU Code Object V3 ELF Note Record Enumeration Values 965 :name: amdgpu-elf-note-record-enumeration-values-table-v3 966 967 ============================== ===== 968 Name Value 969 ============================== ===== 970 *reserved* 0-31 971 ``NT_AMDGPU_METADATA`` 32 972 ============================== ===== 973 974``NT_AMDGPU_METADATA`` 975 Specifies extensible metadata associated with an AMDGPU code 976 object. It is encoded as a map in the Message Pack [MsgPack]_ binary 977 data format. See :ref:`amdgpu-amdhsa-code-object-metadata-v3` for the 978 map keys defined for the ``amdhsa`` OS. 979 980.. _amdgpu-symbols: 981 982Symbols 983------- 984 985Symbols include the following: 986 987 .. table:: AMDGPU ELF Symbols 988 :name: amdgpu-elf-symbols-table 989 990 ===================== ================== ================ ================== 991 Name Type Section Description 992 ===================== ================== ================ ================== 993 *link-name* ``STT_OBJECT`` - ``.data`` Global variable 994 - ``.rodata`` 995 - ``.bss`` 996 *link-name*\ ``.kd`` ``STT_OBJECT`` - ``.rodata`` Kernel descriptor 997 *link-name* ``STT_FUNC`` - ``.text`` Kernel entry point 998 *link-name* ``STT_OBJECT`` - SHN_AMDGPU_LDS Global variable in LDS 999 ===================== ================== ================ ================== 1000 1001Global variable 1002 Global variables both used and defined by the compilation unit. 1003 1004 If the symbol is defined in the compilation unit then it is allocated in the 1005 appropriate section according to if it has initialized data or is readonly. 1006 1007 If the symbol is external then its section is ``STN_UNDEF`` and the loader 1008 will resolve relocations using the definition provided by another code object 1009 or explicitly defined by the runtime. 1010 1011 If the symbol resides in local/group memory (LDS) then its section is the 1012 special processor specific section name ``SHN_AMDGPU_LDS``, and the 1013 ``st_value`` field describes alignment requirements as it does for common 1014 symbols. 1015 1016 .. TODO:: 1017 1018 Add description of linked shared object symbols. Seems undefined symbols 1019 are marked as STT_NOTYPE. 1020 1021Kernel descriptor 1022 Every HSA kernel has an associated kernel descriptor. It is the address of the 1023 kernel descriptor that is used in the AQL dispatch packet used to invoke the 1024 kernel, not the kernel entry point. The layout of the HSA kernel descriptor is 1025 defined in :ref:`amdgpu-amdhsa-kernel-descriptor`. 1026 1027Kernel entry point 1028 Every HSA kernel also has a symbol for its machine code entry point. 1029 1030.. _amdgpu-relocation-records: 1031 1032Relocation Records 1033------------------ 1034 1035AMDGPU backend generates ``Elf64_Rela`` relocation records. Supported 1036relocatable fields are: 1037 1038``word32`` 1039 This specifies a 32-bit field occupying 4 bytes with arbitrary byte 1040 alignment. These values use the same byte order as other word values in the 1041 AMDGPU architecture. 1042 1043``word64`` 1044 This specifies a 64-bit field occupying 8 bytes with arbitrary byte 1045 alignment. These values use the same byte order as other word values in the 1046 AMDGPU architecture. 1047 1048Following notations are used for specifying relocation calculations: 1049 1050**A** 1051 Represents the addend used to compute the value of the relocatable field. 1052 1053**G** 1054 Represents the offset into the global offset table at which the relocation 1055 entry's symbol will reside during execution. 1056 1057**GOT** 1058 Represents the address of the global offset table. 1059 1060**P** 1061 Represents the place (section offset for ``et_rel`` or address for ``et_dyn``) 1062 of the storage unit being relocated (computed using ``r_offset``). 1063 1064**S** 1065 Represents the value of the symbol whose index resides in the relocation 1066 entry. Relocations not using this must specify a symbol index of 1067 ``STN_UNDEF``. 1068 1069**B** 1070 Represents the base address of a loaded executable or shared object which is 1071 the difference between the ELF address and the actual load address. 1072 Relocations using this are only valid in executable or shared objects. 1073 1074The following relocation types are supported: 1075 1076 .. table:: AMDGPU ELF Relocation Records 1077 :name: amdgpu-elf-relocation-records-table 1078 1079 ========================== ======= ===== ========== ============================== 1080 Relocation Type Kind Value Field Calculation 1081 ========================== ======= ===== ========== ============================== 1082 ``R_AMDGPU_NONE`` 0 *none* *none* 1083 ``R_AMDGPU_ABS32_LO`` Static, 1 ``word32`` (S + A) & 0xFFFFFFFF 1084 Dynamic 1085 ``R_AMDGPU_ABS32_HI`` Static, 2 ``word32`` (S + A) >> 32 1086 Dynamic 1087 ``R_AMDGPU_ABS64`` Static, 3 ``word64`` S + A 1088 Dynamic 1089 ``R_AMDGPU_REL32`` Static 4 ``word32`` S + A - P 1090 ``R_AMDGPU_REL64`` Static 5 ``word64`` S + A - P 1091 ``R_AMDGPU_ABS32`` Static, 6 ``word32`` S + A 1092 Dynamic 1093 ``R_AMDGPU_GOTPCREL`` Static 7 ``word32`` G + GOT + A - P 1094 ``R_AMDGPU_GOTPCREL32_LO`` Static 8 ``word32`` (G + GOT + A - P) & 0xFFFFFFFF 1095 ``R_AMDGPU_GOTPCREL32_HI`` Static 9 ``word32`` (G + GOT + A - P) >> 32 1096 ``R_AMDGPU_REL32_LO`` Static 10 ``word32`` (S + A - P) & 0xFFFFFFFF 1097 ``R_AMDGPU_REL32_HI`` Static 11 ``word32`` (S + A - P) >> 32 1098 *reserved* 12 1099 ``R_AMDGPU_RELATIVE64`` Dynamic 13 ``word64`` B + A 1100 ========================== ======= ===== ========== ============================== 1101 1102``R_AMDGPU_ABS32_LO`` and ``R_AMDGPU_ABS32_HI`` are only supported by 1103the ``mesa3d`` OS, which does not support ``R_AMDGPU_ABS64``. 1104 1105There is no current OS loader support for 32-bit programs and so 1106``R_AMDGPU_ABS32`` is not used. 1107 1108.. _amdgpu-loaded-code-object-path-uniform-resource-identifier: 1109 1110Loaded Code Object Path Uniform Resource Identifier (URI) 1111--------------------------------------------------------- 1112 1113The AMD GPU code object loader represents the path of the ELF shared object from 1114which the code object was loaded as a textual Unifom Resource Identifier (URI). 1115Note that the code object is the in memory loaded relocated form of the ELF 1116shared object. Multiple code objects may be loaded at different memory 1117addresses in the same process from the same ELF shared object. 1118 1119The loaded code object path URI syntax is defined by the following BNF syntax: 1120 1121.. code:: 1122 1123 code_object_uri ::== file_uri | memory_uri 1124 file_uri ::== "file://" file_path [ range_specifier ] 1125 memory_uri ::== "memory://" process_id range_specifier 1126 range_specifier ::== [ "#" | "?" ] "offset=" number "&" "size=" number 1127 file_path ::== URI_ENCODED_OS_FILE_PATH 1128 process_id ::== DECIMAL_NUMBER 1129 number ::== HEX_NUMBER | DECIMAL_NUMBER | OCTAL_NUMBER 1130 1131**number** 1132 Is a C integral literal where hexadecimal values are prefixed by "0x" or "0X", 1133 and octal values by "0". 1134 1135**file_path** 1136 Is the file's path specified as a URI encoded UTF-8 string. In URI encoding, 1137 every character that is not in the regular expression ``[a-zA-Z0-9/_.~-]`` is 1138 encoded as two uppercase hexidecimal digits proceeded by "%". Directories in 1139 the path are separated by "/". 1140 1141**offset** 1142 Is a 0-based byte offset to the start of the code object. For a file URI, it 1143 is from the start of the file specified by the ``file_path``, and if omitted 1144 defaults to 0. For a memory URI, it is the memory address and is required. 1145 1146**size** 1147 Is the number of bytes in the code object. For a file URI, if omitted it 1148 defaults to the size of the file. It is required for a memory URI. 1149 1150**process_id** 1151 Is the identity of the process owning the memory. For Linux it is the C 1152 unsigned integral decimal literal for the process ID (PID). 1153 1154For example: 1155 1156.. code:: 1157 1158 file:///dir1/dir2/file1 1159 file:///dir3/dir4/file2#offset=0x2000&size=3000 1160 memory://1234#offset=0x20000&size=3000 1161 1162.. _amdgpu-dwarf-debug-information: 1163 1164DWARF Debug Information 1165======================= 1166 1167.. warning:: 1168 1169 This section describes a **provisional proposal** for AMDGPU DWARF [DWARF]_ 1170 that is not currently fully implemented and is subject to change. 1171 1172AMDGPU generates DWARF [DWARF]_ debugging information ELF sections (see 1173:ref:`amdgpu-elf-code-object`) which contain information that maps the code 1174object executable code and data to the source language constructs. It can be 1175used by tools such as debuggers and profilers. It uses features defined in 1176:doc:`AMDGPUDwarfProposalForHeterogeneousDebugging` that are made available in 1177DWARF Version 4 and DWARF Version 5 as an LLVM vendor extension. 1178 1179This section defines the AMDGPU target architecture specific DWARF mappings. 1180 1181.. _amdgpu-dwarf-register-identifier: 1182 1183Register Identifier 1184------------------- 1185 1186This section defines the AMDGPU target architecture register numbers used in 1187DWARF operation expressions (see DWARF Version 5 section 2.5 and 1188:ref:`amdgpu-dwarf-operation-expressions`) and Call Frame Information 1189instructions (see DWARF Version 5 section 6.4 and 1190:ref:`amdgpu-dwarf-call-frame-information`). 1191 1192A single code object can contain code for kernels that have different wavefront 1193sizes. The vector registers and some scalar registers are based on the wavefront 1194size. AMDGPU defines distinct DWARF registers for each wavefront size. This 1195simplifies the consumer of the DWARF so that each register has a fixed size, 1196rather than being dynamic according to the wavefront size mode. Similarly, 1197distinct DWARF registers are defined for those registers that vary in size 1198according to the process address size. This allows a consumer to treat a 1199specific AMDGPU processor as a single architecture regardless of how it is 1200configured at run time. The compiler explicitly specifies the DWARF registers 1201that match the mode in which the code it is generating will be executed. 1202 1203DWARF registers are encoded as numbers, which are mapped to architecture 1204registers. The mapping for AMDGPU is defined in 1205:ref:`amdgpu-dwarf-register-mapping-table`. All AMDGPU targets use the same 1206mapping. 1207 1208.. table:: AMDGPU DWARF Register Mapping 1209 :name: amdgpu-dwarf-register-mapping-table 1210 1211 ============== ================= ======== ================================== 1212 DWARF Register AMDGPU Register Bit Size Description 1213 ============== ================= ======== ================================== 1214 0 PC_32 32 Program Counter (PC) when 1215 executing in a 32-bit process 1216 address space. Used in the CFI to 1217 describe the PC of the calling 1218 frame. 1219 1 EXEC_MASK_32 32 Execution Mask Register when 1220 executing in wavefront 32 mode. 1221 2-15 *Reserved* 1222 16 PC_64 64 Program Counter (PC) when 1223 executing in a 64-bit process 1224 address space. Used in the CFI to 1225 describe the PC of the calling 1226 frame. 1227 17 EXEC_MASK_64 64 Execution Mask Register when 1228 executing in wavefront 64 mode. 1229 18-31 *Reserved* 1230 32-95 SGPR0-SGPR63 32 Scalar General Purpose 1231 Registers. 1232 96-127 *Reserved* 1233 128-511 *Reserved* 1234 512-1023 *Reserved* 1235 1024-1087 *Reserved* 1236 1088-1129 SGPR64-SGPR105 32 Scalar General Purpose Registers 1237 1130-1535 *Reserved* 1238 1536-1791 VGPR0-VGPR255 32*32 Vector General Purpose Registers 1239 when executing in wavefront 32 1240 mode. 1241 1792-2047 *Reserved* 1242 2048-2303 AGPR0-AGPR255 32*32 Vector Accumulation Registers 1243 when executing in wavefront 32 1244 ode. 1245 2304-2559 *Reserved* 1246 2560-2815 VGPR0-VGPR255 64*32 Vector General Purpose Registers 1247 when executing in wavefront 64 1248 mode. 1249 2816-3071 *Reserved* 1250 3072-3327 AGPR0-AGPR255 64*32 Vector Accumulation Registers 1251 when executing in wavefront 64 1252 mode. 1253 3328-3583 *Reserved* 1254 ============== ================= ======== ================================== 1255 1256The vector registers are represented as the full size for the wavefront. They 1257are organized as consecutive dwords (32-bits), one per lane, with the dword at 1258the least significant bit position corresponding to lane 0 and so forth. DWARF 1259location expressions involving the ``DW_OP_LLVM_offset`` and 1260``DW_OP_LLVM_push_lane`` operations are used to select the part of the vector 1261register corresponding to the lane that is executing the current thread of 1262execution in languages that are implemented using a SIMD or SIMT execution 1263model. 1264 1265If the wavefront size is 32 lanes then the wavefront 32 mode register 1266definitions are used. If the wavefront size is 64 lanes then the wavefront 64 1267mode register definitions are used. Some AMDGPU targets support executing in 1268both wavefront 32 and wavefront 64 mode. The register definitions corresponding 1269to the wavefront mode of the generated code will be used. 1270 1271If code is generated to execute in a 32-bit process address space, then the 127232-bit process address space register definitions are used. If code is generated 1273to execute in a 64-bit process address space, then the 64-bit process address 1274space register definitions are used. The ``amdgcn`` target only supports the 127564-bit process address space. 1276 1277.. _amdgpu-dwarf-address-class-identifier: 1278 1279Address Class Identifier 1280------------------------ 1281 1282The DWARF address class represents the source language memory space. See DWARF 1283Version 5 section 2.12 which is updated by the propoal in 1284:ref:`amdgpu-dwarf-segment_addresses`. 1285 1286The DWARF address class mapping used for AMDGPU is defined in 1287:ref:`amdgpu-dwarf-address-class-mapping-table`. 1288 1289.. table:: AMDGPU DWARF Address Class Mapping 1290 :name: amdgpu-dwarf-address-class-mapping-table 1291 1292 ========================= ====== ================= 1293 DWARF AMDGPU 1294 -------------------------------- ----------------- 1295 Address Class Name Value Address Space 1296 ========================= ====== ================= 1297 ``DW_ADDR_none`` 0x0000 Generic (Flat) 1298 ``DW_ADDR_LLVM_global`` 0x0001 Global 1299 ``DW_ADDR_LLVM_constant`` 0x0002 Global 1300 ``DW_ADDR_LLVM_group`` 0x0003 Local (group/LDS) 1301 ``DW_ADDR_LLVM_private`` 0x0004 Private (Scratch) 1302 ``DW_ADDR_AMDGPU_region`` 0x8000 Region (GDS) 1303 ========================= ====== ================= 1304 1305The DWARF address class values defined in the proposal at 1306:ref:`amdgpu-dwarf-segment_addresses` are used. 1307 1308In addition, ``DW_ADDR_AMDGPU_region`` is encoded as a vendor extension. This is 1309available for use for the AMD extension for access to the hardware GDS memory 1310which is scratchpad memory allocated per device. 1311 1312For AMDGPU if no ``DW_AT_address_class`` attribute is present, then the default 1313address class of ``DW_ADDR_none`` is used. 1314 1315See :ref:`amdgpu-dwarf-address-space-identifier` for information on the AMDGPU 1316mapping of DWARF address classes to DWARF address spaces, including address size 1317and NULL value. 1318 1319.. _amdgpu-dwarf-address-space-identifier: 1320 1321Address Space Identifier 1322------------------------ 1323 1324DWARF address spaces correspond to target architecture specific linear 1325addressable memory areas. See DWARF Version 5 section 2.12 and 1326:ref:`amdgpu-dwarf-segment_addresses`. 1327 1328The DWARF address space mapping used for AMDGPU is defined in 1329:ref:`amdgpu-dwarf-address-space-mapping-table`. 1330 1331.. table:: AMDGPU DWARF Address Space Mapping 1332 :name: amdgpu-dwarf-address-space-mapping-table 1333 1334 ======================================= ===== ======= ======== ================= ======================= 1335 DWARF AMDGPU Notes 1336 --------------------------------------- ----- ---------------- ----------------- ----------------------- 1337 Address Space Name Value Address Bit Size Address Space 1338 --------------------------------------- ----- ------- -------- ----------------- ----------------------- 1339 .. 64-bit 32-bit 1340 process process 1341 address address 1342 space space 1343 ======================================= ===== ======= ======== ================= ======================= 1344 ``DW_ASPACE_none`` 0x00 8 4 Global *default address space* 1345 ``DW_ASPACE_AMDGPU_generic`` 0x01 8 4 Generic (Flat) 1346 ``DW_ASPACE_AMDGPU_region`` 0x02 4 4 Region (GDS) 1347 ``DW_ASPACE_AMDGPU_local`` 0x03 4 4 Local (group/LDS) 1348 *Reserved* 0x04 1349 ``DW_ASPACE_AMDGPU_private_lane`` 0x05 4 4 Private (Scratch) *focused lane* 1350 ``DW_ASPACE_AMDGPU_private_wave`` 0x06 4 4 Private (Scratch) *unswizzled wavefront* 1351 *Reserved* 0x07- 1352 0x1F 1353 ``DW_ASPACE_AMDGPU_private_lane<0-63>`` 0x20- 4 4 Private (Scratch) *specific lane* 1354 0x5F 1355 ======================================= ===== ======= ======== ================= ======================= 1356 1357See :ref:`amdgpu-address-spaces` for information on the AMDGPU address spaces 1358including address size and NULL value. 1359 1360The ``DW_ASPACE_none`` address space is the default target architecture address 1361space used in DWARF operations that do not specify an address space. It 1362therefore has to map to the global address space so that the ``DW_OP_addr*`` and 1363related operations can refer to addresses in the program code. 1364 1365The ``DW_ASPACE_AMDGPU_generic`` address space allows location expressions to 1366specify the flat address space. If the address corresponds to an address in the 1367local address space, then it corresponds to the wavefront that is executing the 1368focused thread of execution. If the address corresponds to an address in the 1369private address space, then it corresponds to the lane that is executing the 1370focused thread of execution for languages that are implemented using a SIMD or 1371SIMT execution model. 1372 1373.. note:: 1374 1375 CUDA-like languages such as HIP that do not have address spaces in the 1376 language type system, but do allow variables to be allocated in different 1377 address spaces, need to explicitly specify the ``DW_ASPACE_AMDGPU_generic`` 1378 address space in the DWARF expression operations as the default address space 1379 is the global address space. 1380 1381The ``DW_ASPACE_AMDGPU_local`` address space allows location expressions to 1382specify the local address space corresponding to the wavefront that is executing 1383the focused thread of execution. 1384 1385The ``DW_ASPACE_AMDGPU_private_lane`` address space allows location expressions 1386to specify the private address space corresponding to the lane that is executing 1387the focused thread of execution for languages that are implemented using a SIMD 1388or SIMT execution model. 1389 1390The ``DW_ASPACE_AMDGPU_private_wave`` address space allows location expressions 1391to specify the unswizzled private address space corresponding to the wavefront 1392that is executing the focused thread of execution. The wavefront view of private 1393memory is the per wavefront unswizzled backing memory layout defined in 1394:ref:`amdgpu-address-spaces`, such that address 0 corresponds to the first 1395location for the backing memory of the wavefront (namely the address is not 1396offset by ``wavefront-scratch-base``). The following formula can be used to 1397convert from a ``DW_ASPACE_AMDGPU_private_lane`` address to a 1398``DW_ASPACE_AMDGPU_private_wave`` address: 1399 1400:: 1401 1402 private-address-wavefront = 1403 ((private-address-lane / 4) * wavefront-size * 4) + 1404 (wavefront-lane-id * 4) + (private-address-lane % 4) 1405 1406If the ``DW_ASPACE_AMDGPU_private_lane`` address is dword aligned, and the start 1407of the dwords for each lane starting with lane 0 is required, then this 1408simplifies to: 1409 1410:: 1411 1412 private-address-wavefront = 1413 private-address-lane * wavefront-size 1414 1415A compiler can use the ``DW_ASPACE_AMDGPU_private_wave`` address space to read a 1416complete spilled vector register back into a complete vector register in the 1417CFI. The frame pointer can be a private lane address which is dword aligned, 1418which can be shifted to multiply by the wavefront size, and then used to form a 1419private wavefront address that gives a location for a contiguous set of dwords, 1420one per lane, where the vector register dwords are spilled. The compiler knows 1421the wavefront size since it generates the code. Note that the type of the 1422address may have to be converted as the size of a 1423``DW_ASPACE_AMDGPU_private_lane`` address may be smaller than the size of a 1424``DW_ASPACE_AMDGPU_private_wave`` address. 1425 1426The ``DW_ASPACE_AMDGPU_private_lane<N>`` address space allows location 1427expressions to specify the private address space corresponding to a specific 1428lane N. For example, this can be used when the compiler spills scalar registers 1429to scratch memory, with each scalar register being saved to a different lane's 1430scratch memory. 1431 1432.. _amdgpu-dwarf-lane-identifier: 1433 1434Lane identifier 1435--------------- 1436 1437DWARF lane identifies specify a target architecture lane position for hardware 1438that executes in a SIMD or SIMT manner, and on which a source language maps its 1439threads of execution onto those lanes. The DWARF lane identifier is pushed by 1440the ``DW_OP_LLVM_push_lane`` DWARF expression operation. See DWARF Version 5 1441section 2.5 which is updated by the proposal in 1442:ref:`amdgpu-dwarf-operation-expressions`. 1443 1444For AMDGPU, the lane identifier corresponds to the hardware lane ID of a 1445wavefront. It is numbered from 0 to the wavefront size minus 1. 1446 1447Operation Expressions 1448--------------------- 1449 1450DWARF expressions are used to compute program values and the locations of 1451program objects. See DWARF Version 5 section 2.5 and 1452:ref:`amdgpu-dwarf-operation-expressions`. 1453 1454DWARF location descriptions describe how to access storage which includes memory 1455and registers. When accessing storage on AMDGPU, bytes are ordered with least 1456significant bytes first, and bits are ordered within bytes with least 1457significant bits first. 1458 1459For AMDGPU CFI expressions, ``DW_OP_LLVM_select_bit_piece`` is used to describe 1460unwinding vector registers that are spilled under the execution mask to memory: 1461the zero-single location description is the vector register, and the one-single 1462location description is the spilled memory location description. The 1463``DW_OP_LLVM_form_aspace_address`` is used to specify the address space of the 1464memory location description. 1465 1466In AMDGPU expressions, ``DW_OP_LLVM_select_bit_piece`` is used by the 1467``DW_AT_LLVM_lane_pc`` attribute expression where divergent control flow is 1468controlled by the execution mask. An undefined location description together 1469with ``DW_OP_LLVM_extend`` is used to indicate the lane was not active on entry 1470to the subprogram. See :ref:`amdgpu-dwarf-dw-at-llvm-lane-pc` for an example. 1471 1472Debugger Information Entry Attributes 1473------------------------------------- 1474 1475This section describes how certain debugger information entry attributes are 1476used by AMDGPU. See the sections in DWARF Version 5 section 2 which are updated 1477by the proposal in :ref:`amdgpu-dwarf-debugging-information-entry-attributes`. 1478 1479.. _amdgpu-dwarf-dw-at-llvm-lane-pc: 1480 1481``DW_AT_LLVM_lane_pc`` 1482~~~~~~~~~~~~~~~~~~~~~~ 1483 1484For AMDGPU, the ``DW_AT_LLVM_lane_pc`` attribute is used to specify the program 1485location of the separate lanes of a SIMT thread. 1486 1487If the lane is an active lane then this will be the same as the current program 1488location. 1489 1490If the lane is inactive, but was active on entry to the subprogram, then this is 1491the program location in the subprogram at which execution of the lane is 1492conceptual positioned. 1493 1494If the lane was not active on entry to the subprogram, then this will be the 1495undefined location. A client debugger can check if the lane is part of a valid 1496work-group by checking that the lane is in the range of the associated 1497work-group within the grid, accounting for partial work-groups. If it is not, 1498then the debugger can omit any information for the lane. Otherwise, the debugger 1499may repeatedly unwind the stack and inspect the ``DW_AT_LLVM_lane_pc`` of the 1500calling subprogram until it finds a non-undefined location. Conceptually the 1501lane only has the call frames that it has a non-undefined 1502``DW_AT_LLVM_lane_pc``. 1503 1504The following example illustrates how the AMDGPU backend can generate a DWARF 1505location list expression for the nested ``IF/THEN/ELSE`` structures of the 1506following subprogram pseudo code for a target with 64 lanes per wavefront. 1507 1508.. code:: 1509 :number-lines: 1510 1511 SUBPROGRAM X 1512 BEGIN 1513 a; 1514 IF (c1) THEN 1515 b; 1516 IF (c2) THEN 1517 c; 1518 ELSE 1519 d; 1520 ENDIF 1521 e; 1522 ELSE 1523 f; 1524 ENDIF 1525 g; 1526 END 1527 1528The AMDGPU backend may generate the following pseudo LLVM MIR to manipulate the 1529execution mask (``EXEC``) to linearize the control flow. The condition is 1530evaluated to make a mask of the lanes for which the condition evaluates to true. 1531First the ``THEN`` region is executed by setting the ``EXEC`` mask to the 1532logical ``AND`` of the current ``EXEC`` mask with the condition mask. Then the 1533``ELSE`` region is executed by negating the ``EXEC`` mask and logical ``AND`` of 1534the saved ``EXEC`` mask at the start of the region. After the ``IF/THEN/ELSE`` 1535region the ``EXEC`` mask is restored to the value it had at the beginning of the 1536region. This is shown below. Other approaches are possible, but the basic 1537concept is the same. 1538 1539.. code:: 1540 :number-lines: 1541 1542 $lex_start: 1543 a; 1544 %1 = EXEC 1545 %2 = c1 1546 $lex_1_start: 1547 EXEC = %1 & %2 1548 $if_1_then: 1549 b; 1550 %3 = EXEC 1551 %4 = c2 1552 $lex_1_1_start: 1553 EXEC = %3 & %4 1554 $lex_1_1_then: 1555 c; 1556 EXEC = ~EXEC & %3 1557 $lex_1_1_else: 1558 d; 1559 EXEC = %3 1560 $lex_1_1_end: 1561 e; 1562 EXEC = ~EXEC & %1 1563 $lex_1_else: 1564 f; 1565 EXEC = %1 1566 $lex_1_end: 1567 g; 1568 $lex_end: 1569 1570To create the DWARF location list expression that defines the location 1571description of a vector of lane program locations, the LLVM MIR ``DBG_VALUE`` 1572pseudo instruction can be used to annotate the linearized control flow. This can 1573be done by defining an artificial variable for the lane PC. The DWARF location 1574list expression created for it is used as the value of the 1575``DW_AT_LLVM_lane_pc`` attribute on the subprogram's debugger information entry. 1576 1577A DWARF procedure is defined for each well nested structured control flow region 1578which provides the conceptual lane program location for a lane if it is not 1579active (namely it is divergent). The DWARF operation expression for each region 1580conceptually inherits the value of the immediately enclosing region and modifies 1581it according to the semantics of the region. 1582 1583For an ``IF/THEN/ELSE`` region the divergent program location is at the start of 1584the region for the ``THEN`` region since it is executed first. For the ``ELSE`` 1585region the divergent program location is at the end of the ``IF/THEN/ELSE`` 1586region since the ``THEN`` region has completed. 1587 1588The lane PC artificial variable is assigned at each region transition. It uses 1589the immediately enclosing region's DWARF procedure to compute the program 1590location for each lane assuming they are divergent, and then modifies the result 1591by inserting the current program location for each lane that the ``EXEC`` mask 1592indicates is active. 1593 1594By having separate DWARF procedures for each region, they can be reused to 1595define the value for any nested region. This reduces the total size of the DWARF 1596operation expressions. 1597 1598The following provides an example using pseudo LLVM MIR. 1599 1600.. code:: 1601 :number-lines: 1602 1603 $lex_start: 1604 DEFINE_DWARF %__uint_64 = DW_TAG_base_type[ 1605 DW_AT_name = "__uint64"; 1606 DW_AT_byte_size = 8; 1607 DW_AT_encoding = DW_ATE_unsigned; 1608 ]; 1609 DEFINE_DWARF %__active_lane_pc = DW_TAG_dwarf_procedure[ 1610 DW_AT_name = "__active_lane_pc"; 1611 DW_AT_location = [ 1612 DW_OP_regx PC; 1613 DW_OP_LLVM_extend 64, 64; 1614 DW_OP_regval_type EXEC, %uint_64; 1615 DW_OP_LLVM_select_bit_piece 64, 64; 1616 ]; 1617 ]; 1618 DEFINE_DWARF %__divergent_lane_pc = DW_TAG_dwarf_procedure[ 1619 DW_AT_name = "__divergent_lane_pc"; 1620 DW_AT_location = [ 1621 DW_OP_LLVM_undefined; 1622 DW_OP_LLVM_extend 64, 64; 1623 ]; 1624 ]; 1625 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 1626 DW_OP_call_ref %__divergent_lane_pc; 1627 DW_OP_call_ref %__active_lane_pc; 1628 ]; 1629 a; 1630 %1 = EXEC; 1631 DBG_VALUE %1, $noreg, %__lex_1_save_exec; 1632 %2 = c1; 1633 $lex_1_start: 1634 EXEC = %1 & %2; 1635 $lex_1_then: 1636 DEFINE_DWARF %__divergent_lane_pc_1_then = DW_TAG_dwarf_procedure[ 1637 DW_AT_name = "__divergent_lane_pc_1_then"; 1638 DW_AT_location = DIExpression[ 1639 DW_OP_call_ref %__divergent_lane_pc; 1640 DW_OP_xaddr &lex_1_start; 1641 DW_OP_stack_value; 1642 DW_OP_LLVM_extend 64, 64; 1643 DW_OP_call_ref %__lex_1_save_exec; 1644 DW_OP_deref_type 64, %__uint_64; 1645 DW_OP_LLVM_select_bit_piece 64, 64; 1646 ]; 1647 ]; 1648 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 1649 DW_OP_call_ref %__divergent_lane_pc_1_then; 1650 DW_OP_call_ref %__active_lane_pc; 1651 ]; 1652 b; 1653 %3 = EXEC; 1654 DBG_VALUE %3, %__lex_1_1_save_exec; 1655 %4 = c2; 1656 $lex_1_1_start: 1657 EXEC = %3 & %4; 1658 $lex_1_1_then: 1659 DEFINE_DWARF %__divergent_lane_pc_1_1_then = DW_TAG_dwarf_procedure[ 1660 DW_AT_name = "__divergent_lane_pc_1_1_then"; 1661 DW_AT_location = DIExpression[ 1662 DW_OP_call_ref %__divergent_lane_pc_1_then; 1663 DW_OP_xaddr &lex_1_1_start; 1664 DW_OP_stack_value; 1665 DW_OP_LLVM_extend 64, 64; 1666 DW_OP_call_ref %__lex_1_1_save_exec; 1667 DW_OP_deref_type 64, %__uint_64; 1668 DW_OP_LLVM_select_bit_piece 64, 64; 1669 ]; 1670 ]; 1671 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 1672 DW_OP_call_ref %__divergent_lane_pc_1_1_then; 1673 DW_OP_call_ref %__active_lane_pc; 1674 ]; 1675 c; 1676 EXEC = ~EXEC & %3; 1677 $lex_1_1_else: 1678 DEFINE_DWARF %__divergent_lane_pc_1_1_else = DW_TAG_dwarf_procedure[ 1679 DW_AT_name = "__divergent_lane_pc_1_1_else"; 1680 DW_AT_location = DIExpression[ 1681 DW_OP_call_ref %__divergent_lane_pc_1_then; 1682 DW_OP_xaddr &lex_1_1_end; 1683 DW_OP_stack_value; 1684 DW_OP_LLVM_extend 64, 64; 1685 DW_OP_call_ref %__lex_1_1_save_exec; 1686 DW_OP_deref_type 64, %__uint_64; 1687 DW_OP_LLVM_select_bit_piece 64, 64; 1688 ]; 1689 ]; 1690 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 1691 DW_OP_call_ref %__divergent_lane_pc_1_1_else; 1692 DW_OP_call_ref %__active_lane_pc; 1693 ]; 1694 d; 1695 EXEC = %3; 1696 $lex_1_1_end: 1697 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 1698 DW_OP_call_ref %__divergent_lane_pc; 1699 DW_OP_call_ref %__active_lane_pc; 1700 ]; 1701 e; 1702 EXEC = ~EXEC & %1; 1703 $lex_1_else: 1704 DEFINE_DWARF %__divergent_lane_pc_1_else = DW_TAG_dwarf_procedure[ 1705 DW_AT_name = "__divergent_lane_pc_1_else"; 1706 DW_AT_location = DIExpression[ 1707 DW_OP_call_ref %__divergent_lane_pc; 1708 DW_OP_xaddr &lex_1_end; 1709 DW_OP_stack_value; 1710 DW_OP_LLVM_extend 64, 64; 1711 DW_OP_call_ref %__lex_1_save_exec; 1712 DW_OP_deref_type 64, %__uint_64; 1713 DW_OP_LLVM_select_bit_piece 64, 64; 1714 ]; 1715 ]; 1716 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 1717 DW_OP_call_ref %__divergent_lane_pc_1_else; 1718 DW_OP_call_ref %__active_lane_pc; 1719 ]; 1720 f; 1721 EXEC = %1; 1722 $lex_1_end: 1723 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc DIExpression[ 1724 DW_OP_call_ref %__divergent_lane_pc; 1725 DW_OP_call_ref %__active_lane_pc; 1726 ]; 1727 g; 1728 $lex_end: 1729 1730The DWARF procedure ``%__active_lane_pc`` is used to update the lane pc elements 1731that are active, with the current program location. 1732 1733Artificial variables %__lex_1_save_exec and %__lex_1_1_save_exec are created for 1734the execution masks saved on entry to a region. Using the ``DBG_VALUE`` pseudo 1735instruction, location list entries will be created that describe where the 1736artificial variables are allocated at any given program location. The compiler 1737may allocate them to registers or spill them to memory. 1738 1739The DWARF procedures for each region use the values of the saved execution mask 1740artificial variables to only update the lanes that are active on entry to the 1741region. All other lanes retain the value of the enclosing region where they were 1742last active. If they were not active on entry to the subprogram, then will have 1743the undefined location description. 1744 1745Other structured control flow regions can be handled similarly. For example, 1746loops would set the divergent program location for the region at the end of the 1747loop. Any lanes active will be in the loop, and any lanes not active must have 1748exited the loop. 1749 1750An ``IF/THEN/ELSEIF/ELSEIF/...`` region can be treated as a nest of 1751``IF/THEN/ELSE`` regions. 1752 1753The DWARF procedures can use the active lane artificial variable described in 1754:ref:`amdgpu-dwarf-amdgpu-dw-at-llvm-active-lane` rather than the actual 1755``EXEC`` mask in order to support whole or quad wavefront mode. 1756 1757.. _amdgpu-dwarf-amdgpu-dw-at-llvm-active-lane: 1758 1759``DW_AT_LLVM_active_lane`` 1760~~~~~~~~~~~~~~~~~~~~~~~~~~ 1761 1762The ``DW_AT_LLVM_active_lane`` attribute on a subprogram debugger information 1763entry is used to specify the lanes that are conceptually active for a SIMT 1764thread. 1765 1766The execution mask may be modified to implement whole or quad wavefront mode 1767operations. For example, all lanes may need to temporarily be made active to 1768execute a whole wavefront operation. Such regions would save the ``EXEC`` mask, 1769update it to enable the necessary lanes, perform the operations, and then 1770restore the ``EXEC`` mask from the saved value. While executing the whole 1771wavefront region, the conceptual execution mask is the saved value, not the 1772``EXEC`` value. 1773 1774This is handled by defining an artificial variable for the active lane mask. The 1775active lane mask artificial variable would be the actual ``EXEC`` mask for 1776normal regions, and the saved execution mask for regions where the mask is 1777temporarily updated. The location list expression created for this artificial 1778variable is used to define the value of the ``DW_AT_LLVM_active_lane`` 1779attribute. 1780 1781``DW_AT_LLVM_augmentation`` 1782~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1783 1784For AMDGPU, the ``DW_AT_LLVM_augmentation`` attribute of a compilation unit 1785debugger information entry has the following value for the augmentation string: 1786 1787:: 1788 1789 [amdgpu:v0.0] 1790 1791The "vX.Y" specifies the major X and minor Y version number of the AMDGPU 1792extensions used in the DWARF of the compilation unit. The version number 1793conforms to [SEMVER]_. 1794 1795Call Frame Information 1796---------------------- 1797 1798DWARF Call Frame Information (CFI) describes how a consumer can virtually 1799*unwind* call frames in a running process or core dump. See DWARF Version 5 1800section 6.4 and :ref:`amdgpu-dwarf-call-frame-information`. 1801 1802For AMDGPU, the Common Information Entry (CIE) fields have the following values: 1803 18041. ``augmentation`` string contains the following null-terminated UTF-8 string: 1805 1806 :: 1807 1808 [amd:v0.0] 1809 1810 The ``vX.Y`` specifies the major X and minor Y version number of the AMDGPU 1811 extensions used in this CIE or to the FDEs that use it. The version number 1812 conforms to [SEMVER]_. 1813 18142. ``address_size`` for the ``Global`` address space is defined in 1815 :ref:`amdgpu-dwarf-address-space-identifier`. 1816 18173. ``segment_selector_size`` is 0 as AMDGPU does not use a segment selector. 1818 18194. ``code_alignment_factor`` is 4 bytes. 1820 1821 .. TODO:: 1822 1823 Add to :ref:`amdgpu-processor-table` table. 1824 18255. ``data_alignment_factor`` is 4 bytes. 1826 1827 .. TODO:: 1828 1829 Add to :ref:`amdgpu-processor-table` table. 1830 18316. ``return_address_register`` is ``PC_32`` for 32-bit processes and ``PC_64`` 1832 for 64-bit processes defined in :ref:`amdgpu-dwarf-register-identifier`. 1833 18347. ``initial_instructions`` Since a subprogram X with fewer registers can be 1835 called from subprogram Y that has more allocated, X will not change any of 1836 the extra registers as it cannot access them. Therefore, the default rule 1837 for all columns is ``same value``. 1838 1839For AMDGPU the register number follows the numbering defined in 1840:ref:`amdgpu-dwarf-register-identifier`. 1841 1842For AMDGPU the instructions are variable size. A consumer can subtract 1 from 1843the return address to get the address of a byte within the call site 1844instructions. See DWARF Version 5 section 6.4.4. 1845 1846Accelerated Access 1847------------------ 1848 1849See DWARF Version 5 section 6.1. 1850 1851Lookup By Name Section Header 1852~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1853 1854See DWARF Version 5 section 6.1.1.4.1 and :ref:`amdgpu-dwarf-lookup-by-name`. 1855 1856For AMDGPU the lookup by name section header table: 1857 1858``augmentation_string_size`` (uword) 1859 1860 Set to the length of the ``augmentation_string`` value which is always a 1861 multiple of 4. 1862 1863``augmentation_string`` (sequence of UTF-8 characters) 1864 1865 Contains the following UTF-8 string null padded to a multiple of 4 bytes: 1866 1867 :: 1868 1869 [amdgpu:v0.0] 1870 1871 The "vX.Y" specifies the major X and minor Y version number of the AMDGPU 1872 extensions used in the DWARF of this index. The version number conforms to 1873 [SEMVER]_. 1874 1875 .. note:: 1876 1877 This is different to the DWARF Version 5 definition that requires the first 1878 4 characters to be the vendor ID. But this is consistent with the other 1879 augmentation strings and does allow multiple vendor contributions. However, 1880 backwards compatibility may be more desirable. 1881 1882Lookup By Address Section Header 1883~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1884 1885See DWARF Version 5 section 6.1.2. 1886 1887For AMDGPU the lookup by address section header table: 1888 1889``address_size`` (ubyte) 1890 1891 Match the address size for the ``Global`` address space defined in 1892 :ref:`amdgpu-dwarf-address-space-identifier`. 1893 1894``segment_selector_size`` (ubyte) 1895 1896 AMDGPU does not use a segment selector so this is 0. The entries in the 1897 ``.debug_aranges`` do not have a segment selector. 1898 1899Line Number Information 1900----------------------- 1901 1902See DWARF Version 5 section 6.2 and :ref:`amdgpu-dwarf-line-number-information`. 1903 1904AMDGPU does not use the ``isa`` state machine registers and always sets it to 0. 1905The instruction set must be obtained from the ELF file header ``e_flags`` field 1906in the ``EF_AMDGPU_MACH`` bit position (see :ref:`ELF Header 1907<amdgpu-elf-header>`). See DWARF Version 5 section 6.2.2. 1908 1909.. TODO:: 1910 1911 Should the ``isa`` state machine register be used to indicate if the code is 1912 in wavefront32 or wavefront64 mode? Or used to specify the architecture ISA? 1913 1914For AMDGPU the line number program header fields have the following values (see 1915DWARF Version 5 section 6.2.4): 1916 1917``address_size`` (ubyte) 1918 Matches the address size for the ``Global`` address space defined in 1919 :ref:`amdgpu-dwarf-address-space-identifier`. 1920 1921``segment_selector_size`` (ubyte) 1922 AMDGPU does not use a segment selector so this is 0. 1923 1924``minimum_instruction_length`` (ubyte) 1925 For GFX9-GFX10 this is 4. 1926 1927``maximum_operations_per_instruction`` (ubyte) 1928 For GFX9-GFX10 this is 1. 1929 1930Source text for online-compiled programs (for example, those compiled by the 1931OpenCL language runtime) may be embedded into the DWARF Version 5 line table. 1932See DWARF Version 5 section 6.2.4.1 which is updated by the proposal in 1933:ref:`DW_LNCT_LLVM_source 1934<amdgpu-dwarf-line-number-information-dw-lnct-llvm-source>`. 1935 1936The Clang option used to control source embedding in AMDGPU is defined in 1937:ref:`amdgpu-clang-debug-options-table`. 1938 1939 .. table:: AMDGPU Clang Debug Options 1940 :name: amdgpu-clang-debug-options-table 1941 1942 ==================== ================================================== 1943 Debug Flag Description 1944 ==================== ================================================== 1945 -g[no-]embed-source Enable/disable embedding source text in DWARF 1946 debug sections. Useful for environments where 1947 source cannot be written to disk, such as 1948 when performing online compilation. 1949 ==================== ================================================== 1950 1951For example: 1952 1953``-gembed-source`` 1954 Enable the embedded source. 1955 1956``-gno-embed-source`` 1957 Disable the embedded source. 1958 195932-Bit and 64-Bit DWARF Formats 1960------------------------------- 1961 1962See DWARF Version 5 section 7.4 and 1963:ref:`amdgpu-dwarf-32-bit-and-64-bit-dwarf-formats`. 1964 1965For AMDGPU: 1966 1967* For the ``amdgcn`` target architecture only the 64-bit process address space 1968 is supported. 1969 1970* The producer can generate either 32-bit or 64-bit DWARF format. LLVM generates 1971 the 32-bit DWARF format. 1972 1973Unit Headers 1974------------ 1975 1976For AMDGPU the following values apply for each of the unit headers described in 1977DWARF Version 5 sections 7.5.1.1, 7.5.1.2, and 7.5.1.3: 1978 1979``address_size`` (ubyte) 1980 Matches the address size for the ``Global`` address space defined in 1981 :ref:`amdgpu-dwarf-address-space-identifier`. 1982 1983.. _amdgpu-code-conventions: 1984 1985Code Conventions 1986================ 1987 1988This section provides code conventions used for each supported target triple OS 1989(see :ref:`amdgpu-target-triples`). 1990 1991AMDHSA 1992------ 1993 1994This section provides code conventions used when the target triple OS is 1995``amdhsa`` (see :ref:`amdgpu-target-triples`). 1996 1997.. _amdgpu-amdhsa-code-object-target-identification: 1998 1999Code Object Target Identification 2000~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2001 2002The AMDHSA OS uses the following syntax to specify the code object 2003target as a single string: 2004 2005 ``<Architecture>-<Vendor>-<OS>-<Environment>-<Processor><Target Features>`` 2006 2007Where: 2008 2009 - ``<Architecture>``, ``<Vendor>``, ``<OS>`` and ``<Environment>`` 2010 are the same as the *Target Triple* (see 2011 :ref:`amdgpu-target-triples`). 2012 2013 - ``<Processor>`` is the same as the *Processor* (see 2014 :ref:`amdgpu-processors`). 2015 2016 - ``<Target Features>`` is a list of the enabled *Target Features* 2017 (see :ref:`amdgpu-target-features`), each prefixed by a plus, that 2018 apply to *Processor*. The list must be in the same order as listed 2019 in the table :ref:`amdgpu-target-feature-table`. Note that *Target 2020 Features* must be included in the list if they are enabled even if 2021 that is the default for *Processor*. 2022 2023For example: 2024 2025 ``"amdgcn-amd-amdhsa--gfx902+xnack"`` 2026 2027.. _amdgpu-amdhsa-code-object-metadata: 2028 2029Code Object Metadata 2030~~~~~~~~~~~~~~~~~~~~ 2031 2032The code object metadata specifies extensible metadata associated with the code 2033objects executed on HSA [HSA]_ compatible runtimes such as AMD's ROCm 2034[AMD-ROCm]_. The encoding and semantics of this metadata depends on the code 2035object version; see :ref:`amdgpu-amdhsa-code-object-metadata-v2` and 2036:ref:`amdgpu-amdhsa-code-object-metadata-v3`. 2037 2038Code object metadata is specified in a note record (see 2039:ref:`amdgpu-note-records`) and is required when the target triple OS is 2040``amdhsa`` (see :ref:`amdgpu-target-triples`). It must contain the minimum 2041information necessary to support the ROCM kernel queries. For example, the 2042segment sizes needed in a dispatch packet. In addition, a high-level language 2043runtime may require other information to be included. For example, the AMD 2044OpenCL runtime records kernel argument information. 2045 2046.. _amdgpu-amdhsa-code-object-metadata-v2: 2047 2048Code Object V2 Metadata (-mattr=-code-object-v3) 2049++++++++++++++++++++++++++++++++++++++++++++++++ 2050 2051.. warning:: Code Object V2 is not the default code object version emitted by 2052 this version of LLVM. For a description of the metadata generated with the 2053 default configuration (Code Object V3) see 2054 :ref:`amdgpu-amdhsa-code-object-metadata-v3`. 2055 2056Code object V2 metadata is specified by the ``NT_AMD_AMDGPU_METADATA`` note 2057record (see :ref:`amdgpu-note-records-v2`). 2058 2059The metadata is specified as a YAML formatted string (see [YAML]_ and 2060:doc:`YamlIO`). 2061 2062.. TODO:: 2063 2064 Is the string null terminated? It probably should not if YAML allows it to 2065 contain null characters, otherwise it should be. 2066 2067The metadata is represented as a single YAML document comprised of the mapping 2068defined in table :ref:`amdgpu-amdhsa-code-object-metadata-map-table-v2` and 2069referenced tables. 2070 2071For boolean values, the string values of ``false`` and ``true`` are used for 2072false and true respectively. 2073 2074Additional information can be added to the mappings. To avoid conflicts, any 2075non-AMD key names should be prefixed by "*vendor-name*.". 2076 2077 .. table:: AMDHSA Code Object V2 Metadata Map 2078 :name: amdgpu-amdhsa-code-object-metadata-map-table-v2 2079 2080 ========== ============== ========= ======================================= 2081 String Key Value Type Required? Description 2082 ========== ============== ========= ======================================= 2083 "Version" sequence of Required - The first integer is the major 2084 2 integers version. Currently 1. 2085 - The second integer is the minor 2086 version. Currently 0. 2087 "Printf" sequence of Each string is encoded information 2088 strings about a printf function call. The 2089 encoded information is organized as 2090 fields separated by colon (':'): 2091 2092 ``ID:N:S[0]:S[1]:...:S[N-1]:FormatString`` 2093 2094 where: 2095 2096 ``ID`` 2097 A 32-bit integer as a unique id for 2098 each printf function call 2099 2100 ``N`` 2101 A 32-bit integer equal to the number 2102 of arguments of printf function call 2103 minus 1 2104 2105 ``S[i]`` (where i = 0, 1, ... , N-1) 2106 32-bit integers for the size in bytes 2107 of the i-th FormatString argument of 2108 the printf function call 2109 2110 FormatString 2111 The format string passed to the 2112 printf function call. 2113 "Kernels" sequence of Required Sequence of the mappings for each 2114 mapping kernel in the code object. See 2115 :ref:`amdgpu-amdhsa-code-object-kernel-metadata-map-table-v2` 2116 for the definition of the mapping. 2117 ========== ============== ========= ======================================= 2118 2119.. 2120 2121 .. table:: AMDHSA Code Object V2 Kernel Metadata Map 2122 :name: amdgpu-amdhsa-code-object-kernel-metadata-map-table-v2 2123 2124 ================= ============== ========= ================================ 2125 String Key Value Type Required? Description 2126 ================= ============== ========= ================================ 2127 "Name" string Required Source name of the kernel. 2128 "SymbolName" string Required Name of the kernel 2129 descriptor ELF symbol. 2130 "Language" string Source language of the kernel. 2131 Values include: 2132 2133 - "OpenCL C" 2134 - "OpenCL C++" 2135 - "HCC" 2136 - "OpenMP" 2137 2138 "LanguageVersion" sequence of - The first integer is the major 2139 2 integers version. 2140 - The second integer is the 2141 minor version. 2142 "Attrs" mapping Mapping of kernel attributes. 2143 See 2144 :ref:`amdgpu-amdhsa-code-object-kernel-attribute-metadata-map-table-v2` 2145 for the mapping definition. 2146 "Args" sequence of Sequence of mappings of the 2147 mapping kernel arguments. See 2148 :ref:`amdgpu-amdhsa-code-object-kernel-argument-metadata-map-table-v2` 2149 for the definition of the mapping. 2150 "CodeProps" mapping Mapping of properties related to 2151 the kernel code. See 2152 :ref:`amdgpu-amdhsa-code-object-kernel-code-properties-metadata-map-table-v2` 2153 for the mapping definition. 2154 ================= ============== ========= ================================ 2155 2156.. 2157 2158 .. table:: AMDHSA Code Object V2 Kernel Attribute Metadata Map 2159 :name: amdgpu-amdhsa-code-object-kernel-attribute-metadata-map-table-v2 2160 2161 =================== ============== ========= ============================== 2162 String Key Value Type Required? Description 2163 =================== ============== ========= ============================== 2164 "ReqdWorkGroupSize" sequence of If not 0, 0, 0 then all values 2165 3 integers must be >=1 and the dispatch 2166 work-group size X, Y, Z must 2167 correspond to the specified 2168 values. Defaults to 0, 0, 0. 2169 2170 Corresponds to the OpenCL 2171 ``reqd_work_group_size`` 2172 attribute. 2173 "WorkGroupSizeHint" sequence of The dispatch work-group size 2174 3 integers X, Y, Z is likely to be the 2175 specified values. 2176 2177 Corresponds to the OpenCL 2178 ``work_group_size_hint`` 2179 attribute. 2180 "VecTypeHint" string The name of a scalar or vector 2181 type. 2182 2183 Corresponds to the OpenCL 2184 ``vec_type_hint`` attribute. 2185 2186 "RuntimeHandle" string The external symbol name 2187 associated with a kernel. 2188 OpenCL runtime allocates a 2189 global buffer for the symbol 2190 and saves the kernel's address 2191 to it, which is used for 2192 device side enqueueing. Only 2193 available for device side 2194 enqueued kernels. 2195 =================== ============== ========= ============================== 2196 2197.. 2198 2199 .. table:: AMDHSA Code Object V2 Kernel Argument Metadata Map 2200 :name: amdgpu-amdhsa-code-object-kernel-argument-metadata-map-table-v2 2201 2202 ================= ============== ========= ================================ 2203 String Key Value Type Required? Description 2204 ================= ============== ========= ================================ 2205 "Name" string Kernel argument name. 2206 "TypeName" string Kernel argument type name. 2207 "Size" integer Required Kernel argument size in bytes. 2208 "Align" integer Required Kernel argument alignment in 2209 bytes. Must be a power of two. 2210 "ValueKind" string Required Kernel argument kind that 2211 specifies how to set up the 2212 corresponding argument. 2213 Values include: 2214 2215 "ByValue" 2216 The argument is copied 2217 directly into the kernarg. 2218 2219 "GlobalBuffer" 2220 A global address space pointer 2221 to the buffer data is passed 2222 in the kernarg. 2223 2224 "DynamicSharedPointer" 2225 A group address space pointer 2226 to dynamically allocated LDS 2227 is passed in the kernarg. 2228 2229 "Sampler" 2230 A global address space 2231 pointer to a S# is passed in 2232 the kernarg. 2233 2234 "Image" 2235 A global address space 2236 pointer to a T# is passed in 2237 the kernarg. 2238 2239 "Pipe" 2240 A global address space pointer 2241 to an OpenCL pipe is passed in 2242 the kernarg. 2243 2244 "Queue" 2245 A global address space pointer 2246 to an OpenCL device enqueue 2247 queue is passed in the 2248 kernarg. 2249 2250 "HiddenGlobalOffsetX" 2251 The OpenCL grid dispatch 2252 global offset for the X 2253 dimension is passed in the 2254 kernarg. 2255 2256 "HiddenGlobalOffsetY" 2257 The OpenCL grid dispatch 2258 global offset for the Y 2259 dimension is passed in the 2260 kernarg. 2261 2262 "HiddenGlobalOffsetZ" 2263 The OpenCL grid dispatch 2264 global offset for the Z 2265 dimension is passed in the 2266 kernarg. 2267 2268 "HiddenNone" 2269 An argument that is not used 2270 by the kernel. Space needs to 2271 be left for it, but it does 2272 not need to be set up. 2273 2274 "HiddenPrintfBuffer" 2275 A global address space pointer 2276 to the runtime printf buffer 2277 is passed in kernarg. 2278 2279 "HiddenHostcallBuffer" 2280 A global address space pointer 2281 to the runtime hostcall buffer 2282 is passed in kernarg. 2283 2284 "HiddenDefaultQueue" 2285 A global address space pointer 2286 to the OpenCL device enqueue 2287 queue that should be used by 2288 the kernel by default is 2289 passed in the kernarg. 2290 2291 "HiddenCompletionAction" 2292 A global address space pointer 2293 to help link enqueued kernels into 2294 the ancestor tree for determining 2295 when the parent kernel has finished. 2296 2297 "HiddenMultiGridSyncArg" 2298 A global address space pointer for 2299 multi-grid synchronization is 2300 passed in the kernarg. 2301 2302 "ValueType" string Required Kernel argument value type. Only 2303 present if "ValueKind" is 2304 "ByValue". For vector data 2305 types, the value is for the 2306 element type. Values include: 2307 2308 - "Struct" 2309 - "I8" 2310 - "U8" 2311 - "I16" 2312 - "U16" 2313 - "F16" 2314 - "I32" 2315 - "U32" 2316 - "F32" 2317 - "I64" 2318 - "U64" 2319 - "F64" 2320 2321 .. TODO:: 2322 How can it be determined if a 2323 vector type, and what size 2324 vector? 2325 "PointeeAlign" integer Alignment in bytes of pointee 2326 type for pointer type kernel 2327 argument. Must be a power 2328 of 2. Only present if 2329 "ValueKind" is 2330 "DynamicSharedPointer". 2331 "AddrSpaceQual" string Kernel argument address space 2332 qualifier. Only present if 2333 "ValueKind" is "GlobalBuffer" or 2334 "DynamicSharedPointer". Values 2335 are: 2336 2337 - "Private" 2338 - "Global" 2339 - "Constant" 2340 - "Local" 2341 - "Generic" 2342 - "Region" 2343 2344 .. TODO:: 2345 Is GlobalBuffer only Global 2346 or Constant? Is 2347 DynamicSharedPointer always 2348 Local? Can HCC allow Generic? 2349 How can Private or Region 2350 ever happen? 2351 "AccQual" string Kernel argument access 2352 qualifier. Only present if 2353 "ValueKind" is "Image" or 2354 "Pipe". Values 2355 are: 2356 2357 - "ReadOnly" 2358 - "WriteOnly" 2359 - "ReadWrite" 2360 2361 .. TODO:: 2362 Does this apply to 2363 GlobalBuffer? 2364 "ActualAccQual" string The actual memory accesses 2365 performed by the kernel on the 2366 kernel argument. Only present if 2367 "ValueKind" is "GlobalBuffer", 2368 "Image", or "Pipe". This may be 2369 more restrictive than indicated 2370 by "AccQual" to reflect what the 2371 kernel actual does. If not 2372 present then the runtime must 2373 assume what is implied by 2374 "AccQual" and "IsConst". Values 2375 are: 2376 2377 - "ReadOnly" 2378 - "WriteOnly" 2379 - "ReadWrite" 2380 2381 "IsConst" boolean Indicates if the kernel argument 2382 is const qualified. Only present 2383 if "ValueKind" is 2384 "GlobalBuffer". 2385 2386 "IsRestrict" boolean Indicates if the kernel argument 2387 is restrict qualified. Only 2388 present if "ValueKind" is 2389 "GlobalBuffer". 2390 2391 "IsVolatile" boolean Indicates if the kernel argument 2392 is volatile qualified. Only 2393 present if "ValueKind" is 2394 "GlobalBuffer". 2395 2396 "IsPipe" boolean Indicates if the kernel argument 2397 is pipe qualified. Only present 2398 if "ValueKind" is "Pipe". 2399 2400 .. TODO:: 2401 Can GlobalBuffer be pipe 2402 qualified? 2403 ================= ============== ========= ================================ 2404 2405.. 2406 2407 .. table:: AMDHSA Code Object V2 Kernel Code Properties Metadata Map 2408 :name: amdgpu-amdhsa-code-object-kernel-code-properties-metadata-map-table-v2 2409 2410 ============================ ============== ========= ===================== 2411 String Key Value Type Required? Description 2412 ============================ ============== ========= ===================== 2413 "KernargSegmentSize" integer Required The size in bytes of 2414 the kernarg segment 2415 that holds the values 2416 of the arguments to 2417 the kernel. 2418 "GroupSegmentFixedSize" integer Required The amount of group 2419 segment memory 2420 required by a 2421 work-group in 2422 bytes. This does not 2423 include any 2424 dynamically allocated 2425 group segment memory 2426 that may be added 2427 when the kernel is 2428 dispatched. 2429 "PrivateSegmentFixedSize" integer Required The amount of fixed 2430 private address space 2431 memory required for a 2432 work-item in 2433 bytes. If the kernel 2434 uses a dynamic call 2435 stack then additional 2436 space must be added 2437 to this value for the 2438 call stack. 2439 "KernargSegmentAlign" integer Required The maximum byte 2440 alignment of 2441 arguments in the 2442 kernarg segment. Must 2443 be a power of 2. 2444 "WavefrontSize" integer Required Wavefront size. Must 2445 be a power of 2. 2446 "NumSGPRs" integer Required Number of scalar 2447 registers used by a 2448 wavefront for 2449 GFX6-GFX10. This 2450 includes the special 2451 SGPRs for VCC, Flat 2452 Scratch (GFX7-GFX10) 2453 and XNACK (for 2454 GFX8-GFX10). It does 2455 not include the 16 2456 SGPR added if a trap 2457 handler is 2458 enabled. It is not 2459 rounded up to the 2460 allocation 2461 granularity. 2462 "NumVGPRs" integer Required Number of vector 2463 registers used by 2464 each work-item for 2465 GFX6-GFX10 2466 "MaxFlatWorkGroupSize" integer Required Maximum flat 2467 work-group size 2468 supported by the 2469 kernel in work-items. 2470 Must be >=1 and 2471 consistent with 2472 ReqdWorkGroupSize if 2473 not 0, 0, 0. 2474 "NumSpilledSGPRs" integer Number of stores from 2475 a scalar register to 2476 a register allocator 2477 created spill 2478 location. 2479 "NumSpilledVGPRs" integer Number of stores from 2480 a vector register to 2481 a register allocator 2482 created spill 2483 location. 2484 ============================ ============== ========= ===================== 2485 2486.. _amdgpu-amdhsa-code-object-metadata-v3: 2487 2488Code Object V3 Metadata (-mattr=+code-object-v3) 2489++++++++++++++++++++++++++++++++++++++++++++++++ 2490 2491Code object V3 metadata is specified by the ``NT_AMDGPU_METADATA`` note record 2492(see :ref:`amdgpu-note-records-v3`). 2493 2494The metadata is represented as Message Pack formatted binary data (see 2495[MsgPack]_). The top level is a Message Pack map that includes the 2496keys defined in table 2497:ref:`amdgpu-amdhsa-code-object-metadata-map-table-v3` and referenced 2498tables. 2499 2500Additional information can be added to the maps. To avoid conflicts, 2501any key names should be prefixed by "*vendor-name*." where 2502``vendor-name`` can be the name of the vendor and specific vendor 2503tool that generates the information. The prefix is abbreviated to 2504simply "." when it appears within a map that has been added by the 2505same *vendor-name*. 2506 2507 .. table:: AMDHSA Code Object V3 Metadata Map 2508 :name: amdgpu-amdhsa-code-object-metadata-map-table-v3 2509 2510 ================= ============== ========= ======================================= 2511 String Key Value Type Required? Description 2512 ================= ============== ========= ======================================= 2513 "amdhsa.version" sequence of Required - The first integer is the major 2514 2 integers version. Currently 1. 2515 - The second integer is the minor 2516 version. Currently 0. 2517 "amdhsa.printf" sequence of Each string is encoded information 2518 strings about a printf function call. The 2519 encoded information is organized as 2520 fields separated by colon (':'): 2521 2522 ``ID:N:S[0]:S[1]:...:S[N-1]:FormatString`` 2523 2524 where: 2525 2526 ``ID`` 2527 A 32-bit integer as a unique id for 2528 each printf function call 2529 2530 ``N`` 2531 A 32-bit integer equal to the number 2532 of arguments of printf function call 2533 minus 1 2534 2535 ``S[i]`` (where i = 0, 1, ... , N-1) 2536 32-bit integers for the size in bytes 2537 of the i-th FormatString argument of 2538 the printf function call 2539 2540 FormatString 2541 The format string passed to the 2542 printf function call. 2543 "amdhsa.kernels" sequence of Required Sequence of the maps for each 2544 map kernel in the code object. See 2545 :ref:`amdgpu-amdhsa-code-object-kernel-metadata-map-table-v3` 2546 for the definition of the keys included 2547 in that map. 2548 ================= ============== ========= ======================================= 2549 2550.. 2551 2552 .. table:: AMDHSA Code Object V3 Kernel Metadata Map 2553 :name: amdgpu-amdhsa-code-object-kernel-metadata-map-table-v3 2554 2555 =================================== ============== ========= ================================ 2556 String Key Value Type Required? Description 2557 =================================== ============== ========= ================================ 2558 ".name" string Required Source name of the kernel. 2559 ".symbol" string Required Name of the kernel 2560 descriptor ELF symbol. 2561 ".language" string Source language of the kernel. 2562 Values include: 2563 2564 - "OpenCL C" 2565 - "OpenCL C++" 2566 - "HCC" 2567 - "HIP" 2568 - "OpenMP" 2569 - "Assembler" 2570 2571 ".language_version" sequence of - The first integer is the major 2572 2 integers version. 2573 - The second integer is the 2574 minor version. 2575 ".args" sequence of Sequence of maps of the 2576 map kernel arguments. See 2577 :ref:`amdgpu-amdhsa-code-object-kernel-argument-metadata-map-table-v3` 2578 for the definition of the keys 2579 included in that map. 2580 ".reqd_workgroup_size" sequence of If not 0, 0, 0 then all values 2581 3 integers must be >=1 and the dispatch 2582 work-group size X, Y, Z must 2583 correspond to the specified 2584 values. Defaults to 0, 0, 0. 2585 2586 Corresponds to the OpenCL 2587 ``reqd_work_group_size`` 2588 attribute. 2589 ".workgroup_size_hint" sequence of The dispatch work-group size 2590 3 integers X, Y, Z is likely to be the 2591 specified values. 2592 2593 Corresponds to the OpenCL 2594 ``work_group_size_hint`` 2595 attribute. 2596 ".vec_type_hint" string The name of a scalar or vector 2597 type. 2598 2599 Corresponds to the OpenCL 2600 ``vec_type_hint`` attribute. 2601 2602 ".device_enqueue_symbol" string The external symbol name 2603 associated with a kernel. 2604 OpenCL runtime allocates a 2605 global buffer for the symbol 2606 and saves the kernel's address 2607 to it, which is used for 2608 device side enqueueing. Only 2609 available for device side 2610 enqueued kernels. 2611 ".kernarg_segment_size" integer Required The size in bytes of 2612 the kernarg segment 2613 that holds the values 2614 of the arguments to 2615 the kernel. 2616 ".group_segment_fixed_size" integer Required The amount of group 2617 segment memory 2618 required by a 2619 work-group in 2620 bytes. This does not 2621 include any 2622 dynamically allocated 2623 group segment memory 2624 that may be added 2625 when the kernel is 2626 dispatched. 2627 ".private_segment_fixed_size" integer Required The amount of fixed 2628 private address space 2629 memory required for a 2630 work-item in 2631 bytes. If the kernel 2632 uses a dynamic call 2633 stack then additional 2634 space must be added 2635 to this value for the 2636 call stack. 2637 ".kernarg_segment_align" integer Required The maximum byte 2638 alignment of 2639 arguments in the 2640 kernarg segment. Must 2641 be a power of 2. 2642 ".wavefront_size" integer Required Wavefront size. Must 2643 be a power of 2. 2644 ".sgpr_count" integer Required Number of scalar 2645 registers required by a 2646 wavefront for 2647 GFX6-GFX9. A register 2648 is required if it is 2649 used explicitly, or 2650 if a higher numbered 2651 register is used 2652 explicitly. This 2653 includes the special 2654 SGPRs for VCC, Flat 2655 Scratch (GFX7-GFX9) 2656 and XNACK (for 2657 GFX8-GFX9). It does 2658 not include the 16 2659 SGPR added if a trap 2660 handler is 2661 enabled. It is not 2662 rounded up to the 2663 allocation 2664 granularity. 2665 ".vgpr_count" integer Required Number of vector 2666 registers required by 2667 each work-item for 2668 GFX6-GFX9. A register 2669 is required if it is 2670 used explicitly, or 2671 if a higher numbered 2672 register is used 2673 explicitly. 2674 ".max_flat_workgroup_size" integer Required Maximum flat 2675 work-group size 2676 supported by the 2677 kernel in work-items. 2678 Must be >=1 and 2679 consistent with 2680 ReqdWorkGroupSize if 2681 not 0, 0, 0. 2682 ".sgpr_spill_count" integer Number of stores from 2683 a scalar register to 2684 a register allocator 2685 created spill 2686 location. 2687 ".vgpr_spill_count" integer Number of stores from 2688 a vector register to 2689 a register allocator 2690 created spill 2691 location. 2692 =================================== ============== ========= ================================ 2693 2694.. 2695 2696 .. table:: AMDHSA Code Object V3 Kernel Argument Metadata Map 2697 :name: amdgpu-amdhsa-code-object-kernel-argument-metadata-map-table-v3 2698 2699 ====================== ============== ========= ================================ 2700 String Key Value Type Required? Description 2701 ====================== ============== ========= ================================ 2702 ".name" string Kernel argument name. 2703 ".type_name" string Kernel argument type name. 2704 ".size" integer Required Kernel argument size in bytes. 2705 ".offset" integer Required Kernel argument offset in 2706 bytes. The offset must be a 2707 multiple of the alignment 2708 required by the argument. 2709 ".value_kind" string Required Kernel argument kind that 2710 specifies how to set up the 2711 corresponding argument. 2712 Values include: 2713 2714 "by_value" 2715 The argument is copied 2716 directly into the kernarg. 2717 2718 "global_buffer" 2719 A global address space pointer 2720 to the buffer data is passed 2721 in the kernarg. 2722 2723 "dynamic_shared_pointer" 2724 A group address space pointer 2725 to dynamically allocated LDS 2726 is passed in the kernarg. 2727 2728 "sampler" 2729 A global address space 2730 pointer to a S# is passed in 2731 the kernarg. 2732 2733 "image" 2734 A global address space 2735 pointer to a T# is passed in 2736 the kernarg. 2737 2738 "pipe" 2739 A global address space pointer 2740 to an OpenCL pipe is passed in 2741 the kernarg. 2742 2743 "queue" 2744 A global address space pointer 2745 to an OpenCL device enqueue 2746 queue is passed in the 2747 kernarg. 2748 2749 "hidden_global_offset_x" 2750 The OpenCL grid dispatch 2751 global offset for the X 2752 dimension is passed in the 2753 kernarg. 2754 2755 "hidden_global_offset_y" 2756 The OpenCL grid dispatch 2757 global offset for the Y 2758 dimension is passed in the 2759 kernarg. 2760 2761 "hidden_global_offset_z" 2762 The OpenCL grid dispatch 2763 global offset for the Z 2764 dimension is passed in the 2765 kernarg. 2766 2767 "hidden_none" 2768 An argument that is not used 2769 by the kernel. Space needs to 2770 be left for it, but it does 2771 not need to be set up. 2772 2773 "hidden_printf_buffer" 2774 A global address space pointer 2775 to the runtime printf buffer 2776 is passed in kernarg. 2777 2778 "hidden_hostcall_buffer" 2779 A global address space pointer 2780 to the runtime hostcall buffer 2781 is passed in kernarg. 2782 2783 "hidden_default_queue" 2784 A global address space pointer 2785 to the OpenCL device enqueue 2786 queue that should be used by 2787 the kernel by default is 2788 passed in the kernarg. 2789 2790 "hidden_completion_action" 2791 A global address space pointer 2792 to help link enqueued kernels into 2793 the ancestor tree for determining 2794 when the parent kernel has finished. 2795 2796 "hidden_multigrid_sync_arg" 2797 A global address space pointer for 2798 multi-grid synchronization is 2799 passed in the kernarg. 2800 2801 ".value_type" string Required Kernel argument value type. Only 2802 present if ".value_kind" is 2803 "by_value". For vector data 2804 types, the value is for the 2805 element type. Values include: 2806 2807 - "struct" 2808 - "i8" 2809 - "u8" 2810 - "i16" 2811 - "u16" 2812 - "f16" 2813 - "i32" 2814 - "u32" 2815 - "f32" 2816 - "i64" 2817 - "u64" 2818 - "f64" 2819 2820 .. TODO:: 2821 How can it be determined if a 2822 vector type, and what size 2823 vector? 2824 ".pointee_align" integer Alignment in bytes of pointee 2825 type for pointer type kernel 2826 argument. Must be a power 2827 of 2. Only present if 2828 ".value_kind" is 2829 "dynamic_shared_pointer". 2830 ".address_space" string Kernel argument address space 2831 qualifier. Only present if 2832 ".value_kind" is "global_buffer" or 2833 "dynamic_shared_pointer". Values 2834 are: 2835 2836 - "private" 2837 - "global" 2838 - "constant" 2839 - "local" 2840 - "generic" 2841 - "region" 2842 2843 .. TODO:: 2844 Is "global_buffer" only "global" 2845 or "constant"? Is 2846 "dynamic_shared_pointer" always 2847 "local"? Can HCC allow "generic"? 2848 How can "private" or "region" 2849 ever happen? 2850 ".access" string Kernel argument access 2851 qualifier. Only present if 2852 ".value_kind" is "image" or 2853 "pipe". Values 2854 are: 2855 2856 - "read_only" 2857 - "write_only" 2858 - "read_write" 2859 2860 .. TODO:: 2861 Does this apply to 2862 "global_buffer"? 2863 ".actual_access" string The actual memory accesses 2864 performed by the kernel on the 2865 kernel argument. Only present if 2866 ".value_kind" is "global_buffer", 2867 "image", or "pipe". This may be 2868 more restrictive than indicated 2869 by ".access" to reflect what the 2870 kernel actual does. If not 2871 present then the runtime must 2872 assume what is implied by 2873 ".access" and ".is_const" . Values 2874 are: 2875 2876 - "read_only" 2877 - "write_only" 2878 - "read_write" 2879 2880 ".is_const" boolean Indicates if the kernel argument 2881 is const qualified. Only present 2882 if ".value_kind" is 2883 "global_buffer". 2884 2885 ".is_restrict" boolean Indicates if the kernel argument 2886 is restrict qualified. Only 2887 present if ".value_kind" is 2888 "global_buffer". 2889 2890 ".is_volatile" boolean Indicates if the kernel argument 2891 is volatile qualified. Only 2892 present if ".value_kind" is 2893 "global_buffer". 2894 2895 ".is_pipe" boolean Indicates if the kernel argument 2896 is pipe qualified. Only present 2897 if ".value_kind" is "pipe". 2898 2899 .. TODO:: 2900 Can "global_buffer" be pipe 2901 qualified? 2902 ====================== ============== ========= ================================ 2903 2904.. 2905 2906Kernel Dispatch 2907~~~~~~~~~~~~~~~ 2908 2909The HSA architected queuing language (AQL) defines a user space memory 2910interface that can be used to control the dispatch of kernels, in an agent 2911independent way. An agent can have zero or more AQL queues created for it using 2912the ROCm runtime, in which AQL packets (all of which are 64 bytes) can be 2913placed. See the *HSA Platform System Architecture Specification* [HSA]_ for the 2914AQL queue mechanics and packet layouts. 2915 2916The packet processor of a kernel agent is responsible for detecting and 2917dispatching HSA kernels from the AQL queues associated with it. For AMD GPUs the 2918packet processor is implemented by the hardware command processor (CP), 2919asynchronous dispatch controller (ADC) and shader processor input controller 2920(SPI). 2921 2922The ROCm runtime can be used to allocate an AQL queue object. It uses the kernel 2923mode driver to initialize and register the AQL queue with CP. 2924 2925To dispatch a kernel the following actions are performed. This can occur in the 2926CPU host program, or from an HSA kernel executing on a GPU. 2927 29281. A pointer to an AQL queue for the kernel agent on which the kernel is to be 2929 executed is obtained. 29302. A pointer to the kernel descriptor (see 2931 :ref:`amdgpu-amdhsa-kernel-descriptor`) of the kernel to execute is obtained. 2932 It must be for a kernel that is contained in a code object that that was 2933 loaded by the ROCm runtime on the kernel agent with which the AQL queue is 2934 associated. 29353. Space is allocated for the kernel arguments using the ROCm runtime allocator 2936 for a memory region with the kernarg property for the kernel agent that will 2937 execute the kernel. It must be at least 16-byte aligned. 29384. Kernel argument values are assigned to the kernel argument memory 2939 allocation. The layout is defined in the *HSA Programmer's Language 2940 Reference* [HSA]_. For AMDGPU the kernel execution directly accesses the 2941 kernel argument memory in the same way constant memory is accessed. (Note 2942 that the HSA specification allows an implementation to copy the kernel 2943 argument contents to another location that is accessed by the kernel.) 29445. An AQL kernel dispatch packet is created on the AQL queue. The ROCm runtime 2945 api uses 64-bit atomic operations to reserve space in the AQL queue for the 2946 packet. The packet must be set up, and the final write must use an atomic 2947 store release to set the packet kind to ensure the packet contents are 2948 visible to the kernel agent. AQL defines a doorbell signal mechanism to 2949 notify the kernel agent that the AQL queue has been updated. These rules, and 2950 the layout of the AQL queue and kernel dispatch packet is defined in the *HSA 2951 System Architecture Specification* [HSA]_. 29526. A kernel dispatch packet includes information about the actual dispatch, 2953 such as grid and work-group size, together with information from the code 2954 object about the kernel, such as segment sizes. The ROCm runtime queries on 2955 the kernel symbol can be used to obtain the code object values which are 2956 recorded in the :ref:`amdgpu-amdhsa-code-object-metadata`. 29577. CP executes micro-code and is responsible for detecting and setting up the 2958 GPU to execute the wavefronts of a kernel dispatch. 29598. CP ensures that when the a wavefront starts executing the kernel machine 2960 code, the scalar general purpose registers (SGPR) and vector general purpose 2961 registers (VGPR) are set up as required by the machine code. The required 2962 setup is defined in the :ref:`amdgpu-amdhsa-kernel-descriptor`. The initial 2963 register state is defined in 2964 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`. 29659. The prolog of the kernel machine code (see 2966 :ref:`amdgpu-amdhsa-kernel-prolog`) sets up the machine state as necessary 2967 before continuing executing the machine code that corresponds to the kernel. 296810. When the kernel dispatch has completed execution, CP signals the completion 2969 signal specified in the kernel dispatch packet if not 0. 2970 2971Image and Samplers 2972~~~~~~~~~~~~~~~~~~ 2973 2974Image and sample handles created by the ROCm runtime are 64-bit addresses of a 2975hardware 32-byte V# and 48 byte S# object respectively. In order to support the 2976HSA ``query_sampler`` operations two extra dwords are used to store the HSA BRIG 2977enumeration values for the queries that are not trivially deducible from the S# 2978representation. 2979 2980HSA Signals 2981~~~~~~~~~~~ 2982 2983HSA signal handles created by the ROCm runtime are 64-bit addresses of a 2984structure allocated in memory accessible from both the CPU and GPU. The 2985structure is defined by the ROCm runtime and subject to change between releases 2986(see [AMD-ROCm-github]_). 2987 2988.. _amdgpu-amdhsa-hsa-aql-queue: 2989 2990HSA AQL Queue 2991~~~~~~~~~~~~~ 2992 2993The HSA AQL queue structure is defined by the ROCm runtime and subject to change 2994between releases (see [AMD-ROCm-github]_). For some processors it contains 2995fields needed to implement certain language features such as the flat address 2996aperture bases. It also contains fields used by CP such as managing the 2997allocation of scratch memory. 2998 2999.. _amdgpu-amdhsa-kernel-descriptor: 3000 3001Kernel Descriptor 3002~~~~~~~~~~~~~~~~~ 3003 3004A kernel descriptor consists of the information needed by CP to initiate the 3005execution of a kernel, including the entry point address of the machine code 3006that implements the kernel. 3007 3008Kernel Descriptor for GFX6-GFX10 3009++++++++++++++++++++++++++++++++ 3010 3011CP microcode requires the Kernel descriptor to be allocated on 64-byte 3012alignment. 3013 3014 .. table:: Kernel Descriptor for GFX6-GFX10 3015 :name: amdgpu-amdhsa-kernel-descriptor-gfx6-gfx10-table 3016 3017 ======= ======= =============================== ============================ 3018 Bits Size Field Name Description 3019 ======= ======= =============================== ============================ 3020 31:0 4 bytes GROUP_SEGMENT_FIXED_SIZE The amount of fixed local 3021 address space memory 3022 required for a work-group 3023 in bytes. This does not 3024 include any dynamically 3025 allocated local address 3026 space memory that may be 3027 added when the kernel is 3028 dispatched. 3029 63:32 4 bytes PRIVATE_SEGMENT_FIXED_SIZE The amount of fixed 3030 private address space 3031 memory required for a 3032 work-item in bytes. If 3033 is_dynamic_callstack is 1 3034 then additional space must 3035 be added to this value for 3036 the call stack. 3037 127:64 8 bytes Reserved, must be 0. 3038 191:128 8 bytes KERNEL_CODE_ENTRY_BYTE_OFFSET Byte offset (possibly 3039 negative) from base 3040 address of kernel 3041 descriptor to kernel's 3042 entry point instruction 3043 which must be 256 byte 3044 aligned. 3045 351:272 20 Reserved, must be 0. 3046 bytes 3047 383:352 4 bytes COMPUTE_PGM_RSRC3 GFX6-9 3048 Reserved, must be 0. 3049 GFX10 3050 Compute Shader (CS) 3051 program settings used by 3052 CP to set up 3053 ``COMPUTE_PGM_RSRC3`` 3054 configuration 3055 register. See 3056 :ref:`amdgpu-amdhsa-compute_pgm_rsrc3-gfx10-table`. 3057 415:384 4 bytes COMPUTE_PGM_RSRC1 Compute Shader (CS) 3058 program settings used by 3059 CP to set up 3060 ``COMPUTE_PGM_RSRC1`` 3061 configuration 3062 register. See 3063 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 3064 447:416 4 bytes COMPUTE_PGM_RSRC2 Compute Shader (CS) 3065 program settings used by 3066 CP to set up 3067 ``COMPUTE_PGM_RSRC2`` 3068 configuration 3069 register. See 3070 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 3071 448 1 bit ENABLE_SGPR_PRIVATE_SEGMENT Enable the setup of the 3072 _BUFFER SGPR user data registers 3073 (see 3074 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 3075 3076 The total number of SGPR 3077 user data registers 3078 requested must not exceed 3079 16 and match value in 3080 ``compute_pgm_rsrc2.user_sgpr.user_sgpr_count``. 3081 Any requests beyond 16 3082 will be ignored. 3083 449 1 bit ENABLE_SGPR_DISPATCH_PTR *see above* 3084 450 1 bit ENABLE_SGPR_QUEUE_PTR *see above* 3085 451 1 bit ENABLE_SGPR_KERNARG_SEGMENT_PTR *see above* 3086 452 1 bit ENABLE_SGPR_DISPATCH_ID *see above* 3087 453 1 bit ENABLE_SGPR_FLAT_SCRATCH_INIT *see above* 3088 454 1 bit ENABLE_SGPR_PRIVATE_SEGMENT *see above* 3089 _SIZE 3090 457:455 3 bits Reserved, must be 0. 3091 458 1 bit ENABLE_WAVEFRONT_SIZE32 GFX6-9 3092 Reserved, must be 0. 3093 GFX10 3094 - If 0 execute in 3095 wavefront size 64 mode. 3096 - If 1 execute in 3097 native wavefront size 3098 32 mode. 3099 463:459 5 bits Reserved, must be 0. 3100 511:464 6 bytes Reserved, must be 0. 3101 512 **Total size 64 bytes.** 3102 ======= ==================================================================== 3103 3104.. 3105 3106 .. table:: compute_pgm_rsrc1 for GFX6-GFX10 3107 :name: amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table 3108 3109 ======= ======= =============================== =========================================================================== 3110 Bits Size Field Name Description 3111 ======= ======= =============================== =========================================================================== 3112 5:0 6 bits GRANULATED_WORKITEM_VGPR_COUNT Number of vector register 3113 blocks used by each work-item; 3114 granularity is device 3115 specific: 3116 3117 GFX6-GFX9 3118 - vgprs_used 0..256 3119 - max(0, ceil(vgprs_used / 4) - 1) 3120 GFX10 (wavefront size 64) 3121 - max_vgpr 1..256 3122 - max(0, ceil(vgprs_used / 4) - 1) 3123 GFX10 (wavefront size 32) 3124 - max_vgpr 1..256 3125 - max(0, ceil(vgprs_used / 8) - 1) 3126 3127 Where vgprs_used is defined 3128 as the highest VGPR number 3129 explicitly referenced plus 3130 one. 3131 3132 Used by CP to set up 3133 ``COMPUTE_PGM_RSRC1.VGPRS``. 3134 3135 The 3136 :ref:`amdgpu-assembler` 3137 calculates this 3138 automatically for the 3139 selected processor from 3140 values provided to the 3141 `.amdhsa_kernel` directive 3142 by the 3143 `.amdhsa_next_free_vgpr` 3144 nested directive (see 3145 :ref:`amdhsa-kernel-directives-table`). 3146 9:6 4 bits GRANULATED_WAVEFRONT_SGPR_COUNT Number of scalar register 3147 blocks used by a wavefront; 3148 granularity is device 3149 specific: 3150 3151 GFX6-GFX8 3152 - sgprs_used 0..112 3153 - max(0, ceil(sgprs_used / 8) - 1) 3154 GFX9 3155 - sgprs_used 0..112 3156 - 2 * max(0, ceil(sgprs_used / 16) - 1) 3157 GFX10 3158 Reserved, must be 0. 3159 (128 SGPRs always 3160 allocated.) 3161 3162 Where sgprs_used is 3163 defined as the highest 3164 SGPR number explicitly 3165 referenced plus one, plus 3166 a target specific number 3167 of additional special 3168 SGPRs for VCC, 3169 FLAT_SCRATCH (GFX7+) and 3170 XNACK_MASK (GFX8+), and 3171 any additional 3172 target specific 3173 limitations. It does not 3174 include the 16 SGPRs added 3175 if a trap handler is 3176 enabled. 3177 3178 The target specific 3179 limitations and special 3180 SGPR layout are defined in 3181 the hardware 3182 documentation, which can 3183 be found in the 3184 :ref:`amdgpu-processors` 3185 table. 3186 3187 Used by CP to set up 3188 ``COMPUTE_PGM_RSRC1.SGPRS``. 3189 3190 The 3191 :ref:`amdgpu-assembler` 3192 calculates this 3193 automatically for the 3194 selected processor from 3195 values provided to the 3196 `.amdhsa_kernel` directive 3197 by the 3198 `.amdhsa_next_free_sgpr` 3199 and `.amdhsa_reserve_*` 3200 nested directives (see 3201 :ref:`amdhsa-kernel-directives-table`). 3202 11:10 2 bits PRIORITY Must be 0. 3203 3204 Start executing wavefront 3205 at the specified priority. 3206 3207 CP is responsible for 3208 filling in 3209 ``COMPUTE_PGM_RSRC1.PRIORITY``. 3210 13:12 2 bits FLOAT_ROUND_MODE_32 Wavefront starts execution 3211 with specified rounding 3212 mode for single (32 3213 bit) floating point 3214 precision floating point 3215 operations. 3216 3217 Floating point rounding 3218 mode values are defined in 3219 :ref:`amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table`. 3220 3221 Used by CP to set up 3222 ``COMPUTE_PGM_RSRC1.FLOAT_MODE``. 3223 15:14 2 bits FLOAT_ROUND_MODE_16_64 Wavefront starts execution 3224 with specified rounding 3225 denorm mode for half/double (16 3226 and 64-bit) floating point 3227 precision floating point 3228 operations. 3229 3230 Floating point rounding 3231 mode values are defined in 3232 :ref:`amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table`. 3233 3234 Used by CP to set up 3235 ``COMPUTE_PGM_RSRC1.FLOAT_MODE``. 3236 17:16 2 bits FLOAT_DENORM_MODE_32 Wavefront starts execution 3237 with specified denorm mode 3238 for single (32 3239 bit) floating point 3240 precision floating point 3241 operations. 3242 3243 Floating point denorm mode 3244 values are defined in 3245 :ref:`amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table`. 3246 3247 Used by CP to set up 3248 ``COMPUTE_PGM_RSRC1.FLOAT_MODE``. 3249 19:18 2 bits FLOAT_DENORM_MODE_16_64 Wavefront starts execution 3250 with specified denorm mode 3251 for half/double (16 3252 and 64-bit) floating point 3253 precision floating point 3254 operations. 3255 3256 Floating point denorm mode 3257 values are defined in 3258 :ref:`amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table`. 3259 3260 Used by CP to set up 3261 ``COMPUTE_PGM_RSRC1.FLOAT_MODE``. 3262 20 1 bit PRIV Must be 0. 3263 3264 Start executing wavefront 3265 in privilege trap handler 3266 mode. 3267 3268 CP is responsible for 3269 filling in 3270 ``COMPUTE_PGM_RSRC1.PRIV``. 3271 21 1 bit ENABLE_DX10_CLAMP Wavefront starts execution 3272 with DX10 clamp mode 3273 enabled. Used by the vector 3274 ALU to force DX10 style 3275 treatment of NaN's (when 3276 set, clamp NaN to zero, 3277 otherwise pass NaN 3278 through). 3279 3280 Used by CP to set up 3281 ``COMPUTE_PGM_RSRC1.DX10_CLAMP``. 3282 22 1 bit DEBUG_MODE Must be 0. 3283 3284 Start executing wavefront 3285 in single step mode. 3286 3287 CP is responsible for 3288 filling in 3289 ``COMPUTE_PGM_RSRC1.DEBUG_MODE``. 3290 23 1 bit ENABLE_IEEE_MODE Wavefront starts execution 3291 with IEEE mode 3292 enabled. Floating point 3293 opcodes that support 3294 exception flag gathering 3295 will quiet and propagate 3296 signaling-NaN inputs per 3297 IEEE 754-2008. Min_dx10 and 3298 max_dx10 become IEEE 3299 754-2008 compliant due to 3300 signaling-NaN propagation 3301 and quieting. 3302 3303 Used by CP to set up 3304 ``COMPUTE_PGM_RSRC1.IEEE_MODE``. 3305 24 1 bit BULKY Must be 0. 3306 3307 Only one work-group allowed 3308 to execute on a compute 3309 unit. 3310 3311 CP is responsible for 3312 filling in 3313 ``COMPUTE_PGM_RSRC1.BULKY``. 3314 25 1 bit CDBG_USER Must be 0. 3315 3316 Flag that can be used to 3317 control debugging code. 3318 3319 CP is responsible for 3320 filling in 3321 ``COMPUTE_PGM_RSRC1.CDBG_USER``. 3322 26 1 bit FP16_OVFL GFX6-GFX8 3323 Reserved, must be 0. 3324 GFX9-GFX10 3325 Wavefront starts execution 3326 with specified fp16 overflow 3327 mode. 3328 3329 - If 0, fp16 overflow generates 3330 +/-INF values. 3331 - If 1, fp16 overflow that is the 3332 result of an +/-INF input value 3333 or divide by 0 produces a +/-INF, 3334 otherwise clamps computed 3335 overflow to +/-MAX_FP16 as 3336 appropriate. 3337 3338 Used by CP to set up 3339 ``COMPUTE_PGM_RSRC1.FP16_OVFL``. 3340 28:27 2 bits Reserved, must be 0. 3341 29 1 bit WGP_MODE GFX6-GFX9 3342 Reserved, must be 0. 3343 GFX10 3344 - If 0 execute work-groups in 3345 CU wavefront execution mode. 3346 - If 1 execute work-groups on 3347 in WGP wavefront execution mode. 3348 3349 See :ref:`amdgpu-amdhsa-memory-model`. 3350 3351 Used by CP to set up 3352 ``COMPUTE_PGM_RSRC1.WGP_MODE``. 3353 30 1 bit MEM_ORDERED GFX6-9 3354 Reserved, must be 0. 3355 GFX10 3356 Controls the behavior of the 3357 waitcnt's vmcnt and vscnt 3358 counters. 3359 3360 - If 0 vmcnt reports completion 3361 of load and atomic with return 3362 out of order with sample 3363 instructions, and the vscnt 3364 reports the completion of 3365 store and atomic without 3366 return in order. 3367 - If 1 vmcnt reports completion 3368 of load, atomic with return 3369 and sample instructions in 3370 order, and the vscnt reports 3371 the completion of store and 3372 atomic without return in order. 3373 3374 Used by CP to set up 3375 ``COMPUTE_PGM_RSRC1.MEM_ORDERED``. 3376 31 1 bit FWD_PROGRESS GFX6-9 3377 Reserved, must be 0. 3378 GFX10 3379 - If 0 execute SIMD wavefronts 3380 using oldest first policy. 3381 - If 1 execute SIMD wavefronts to 3382 ensure wavefronts will make some 3383 forward progress. 3384 3385 Used by CP to set up 3386 ``COMPUTE_PGM_RSRC1.FWD_PROGRESS``. 3387 32 **Total size 4 bytes** 3388 ======= =================================================================================================================== 3389 3390.. 3391 3392 .. table:: compute_pgm_rsrc2 for GFX6-GFX10 3393 :name: amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table 3394 3395 ======= ======= =============================== =========================================================================== 3396 Bits Size Field Name Description 3397 ======= ======= =============================== =========================================================================== 3398 0 1 bit ENABLE_SGPR_PRIVATE_SEGMENT Enable the setup of the 3399 _WAVEFRONT_OFFSET SGPR wavefront scratch offset 3400 system register (see 3401 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 3402 3403 Used by CP to set up 3404 ``COMPUTE_PGM_RSRC2.SCRATCH_EN``. 3405 5:1 5 bits USER_SGPR_COUNT The total number of SGPR 3406 user data registers 3407 requested. This number must 3408 match the number of user 3409 data registers enabled. 3410 3411 Used by CP to set up 3412 ``COMPUTE_PGM_RSRC2.USER_SGPR``. 3413 6 1 bit ENABLE_TRAP_HANDLER Must be 0. 3414 3415 This bit represents 3416 ``COMPUTE_PGM_RSRC2.TRAP_PRESENT``, 3417 which is set by the CP if 3418 the runtime has installed a 3419 trap handler. 3420 7 1 bit ENABLE_SGPR_WORKGROUP_ID_X Enable the setup of the 3421 system SGPR register for 3422 the work-group id in the X 3423 dimension (see 3424 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 3425 3426 Used by CP to set up 3427 ``COMPUTE_PGM_RSRC2.TGID_X_EN``. 3428 8 1 bit ENABLE_SGPR_WORKGROUP_ID_Y Enable the setup of the 3429 system SGPR register for 3430 the work-group id in the Y 3431 dimension (see 3432 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 3433 3434 Used by CP to set up 3435 ``COMPUTE_PGM_RSRC2.TGID_Y_EN``. 3436 9 1 bit ENABLE_SGPR_WORKGROUP_ID_Z Enable the setup of the 3437 system SGPR register for 3438 the work-group id in the Z 3439 dimension (see 3440 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 3441 3442 Used by CP to set up 3443 ``COMPUTE_PGM_RSRC2.TGID_Z_EN``. 3444 10 1 bit ENABLE_SGPR_WORKGROUP_INFO Enable the setup of the 3445 system SGPR register for 3446 work-group information (see 3447 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 3448 3449 Used by CP to set up 3450 ``COMPUTE_PGM_RSRC2.TGID_SIZE_EN``. 3451 12:11 2 bits ENABLE_VGPR_WORKITEM_ID Enable the setup of the 3452 VGPR system registers used 3453 for the work-item ID. 3454 :ref:`amdgpu-amdhsa-system-vgpr-work-item-id-enumeration-values-table` 3455 defines the values. 3456 3457 Used by CP to set up 3458 ``COMPUTE_PGM_RSRC2.TIDIG_CMP_CNT``. 3459 13 1 bit ENABLE_EXCEPTION_ADDRESS_WATCH Must be 0. 3460 3461 Wavefront starts execution 3462 with address watch 3463 exceptions enabled which 3464 are generated when L1 has 3465 witnessed a thread access 3466 an *address of 3467 interest*. 3468 3469 CP is responsible for 3470 filling in the address 3471 watch bit in 3472 ``COMPUTE_PGM_RSRC2.EXCP_EN_MSB`` 3473 according to what the 3474 runtime requests. 3475 14 1 bit ENABLE_EXCEPTION_MEMORY Must be 0. 3476 3477 Wavefront starts execution 3478 with memory violation 3479 exceptions exceptions 3480 enabled which are generated 3481 when a memory violation has 3482 occurred for this wavefront from 3483 L1 or LDS 3484 (write-to-read-only-memory, 3485 mis-aligned atomic, LDS 3486 address out of range, 3487 illegal address, etc.). 3488 3489 CP sets the memory 3490 violation bit in 3491 ``COMPUTE_PGM_RSRC2.EXCP_EN_MSB`` 3492 according to what the 3493 runtime requests. 3494 23:15 9 bits GRANULATED_LDS_SIZE Must be 0. 3495 3496 CP uses the rounded value 3497 from the dispatch packet, 3498 not this value, as the 3499 dispatch may contain 3500 dynamically allocated group 3501 segment memory. CP writes 3502 directly to 3503 ``COMPUTE_PGM_RSRC2.LDS_SIZE``. 3504 3505 Amount of group segment 3506 (LDS) to allocate for each 3507 work-group. Granularity is 3508 device specific: 3509 3510 GFX6: 3511 roundup(lds-size / (64 * 4)) 3512 GFX7-GFX10: 3513 roundup(lds-size / (128 * 4)) 3514 3515 24 1 bit ENABLE_EXCEPTION_IEEE_754_FP Wavefront starts execution 3516 _INVALID_OPERATION with specified exceptions 3517 enabled. 3518 3519 Used by CP to set up 3520 ``COMPUTE_PGM_RSRC2.EXCP_EN`` 3521 (set from bits 0..6). 3522 3523 IEEE 754 FP Invalid 3524 Operation 3525 25 1 bit ENABLE_EXCEPTION_FP_DENORMAL FP Denormal one or more 3526 _SOURCE input operands is a 3527 denormal number 3528 26 1 bit ENABLE_EXCEPTION_IEEE_754_FP IEEE 754 FP Division by 3529 _DIVISION_BY_ZERO Zero 3530 27 1 bit ENABLE_EXCEPTION_IEEE_754_FP IEEE 754 FP FP Overflow 3531 _OVERFLOW 3532 28 1 bit ENABLE_EXCEPTION_IEEE_754_FP IEEE 754 FP Underflow 3533 _UNDERFLOW 3534 29 1 bit ENABLE_EXCEPTION_IEEE_754_FP IEEE 754 FP Inexact 3535 _INEXACT 3536 30 1 bit ENABLE_EXCEPTION_INT_DIVIDE_BY Integer Division by Zero 3537 _ZERO (rcp_iflag_f32 instruction 3538 only) 3539 31 1 bit Reserved, must be 0. 3540 32 **Total size 4 bytes.** 3541 ======= =================================================================================================================== 3542 3543.. 3544 3545 .. table:: compute_pgm_rsrc3 for GFX10 3546 :name: amdgpu-amdhsa-compute_pgm_rsrc3-gfx10-table 3547 3548 ======= ======= =============================== =========================================================================== 3549 Bits Size Field Name Description 3550 ======= ======= =============================== =========================================================================== 3551 3:0 4 bits SHARED_VGPR_COUNT Number of shared VGPRs for wavefront size 64. Granularity 8. Value 0-120. 3552 compute_pgm_rsrc1.vgprs + shared_vgpr_cnt cannot exceed 64. 3553 31:4 28 Reserved, must be 0. 3554 bits 3555 32 **Total size 4 bytes.** 3556 ======= =================================================================================================================== 3557 3558.. 3559 3560 .. table:: Floating Point Rounding Mode Enumeration Values 3561 :name: amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table 3562 3563 ====================================== ===== ============================== 3564 Enumeration Name Value Description 3565 ====================================== ===== ============================== 3566 FLOAT_ROUND_MODE_NEAR_EVEN 0 Round Ties To Even 3567 FLOAT_ROUND_MODE_PLUS_INFINITY 1 Round Toward +infinity 3568 FLOAT_ROUND_MODE_MINUS_INFINITY 2 Round Toward -infinity 3569 FLOAT_ROUND_MODE_ZERO 3 Round Toward 0 3570 ====================================== ===== ============================== 3571 3572.. 3573 3574 .. table:: Floating Point Denorm Mode Enumeration Values 3575 :name: amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table 3576 3577 ====================================== ===== ============================== 3578 Enumeration Name Value Description 3579 ====================================== ===== ============================== 3580 FLOAT_DENORM_MODE_FLUSH_SRC_DST 0 Flush Source and Destination 3581 Denorms 3582 FLOAT_DENORM_MODE_FLUSH_DST 1 Flush Output Denorms 3583 FLOAT_DENORM_MODE_FLUSH_SRC 2 Flush Source Denorms 3584 FLOAT_DENORM_MODE_FLUSH_NONE 3 No Flush 3585 ====================================== ===== ============================== 3586 3587.. 3588 3589 .. table:: System VGPR Work-Item ID Enumeration Values 3590 :name: amdgpu-amdhsa-system-vgpr-work-item-id-enumeration-values-table 3591 3592 ======================================== ===== ============================ 3593 Enumeration Name Value Description 3594 ======================================== ===== ============================ 3595 SYSTEM_VGPR_WORKITEM_ID_X 0 Set work-item X dimension 3596 ID. 3597 SYSTEM_VGPR_WORKITEM_ID_X_Y 1 Set work-item X and Y 3598 dimensions ID. 3599 SYSTEM_VGPR_WORKITEM_ID_X_Y_Z 2 Set work-item X, Y and Z 3600 dimensions ID. 3601 SYSTEM_VGPR_WORKITEM_ID_UNDEFINED 3 Undefined. 3602 ======================================== ===== ============================ 3603 3604.. _amdgpu-amdhsa-initial-kernel-execution-state: 3605 3606Initial Kernel Execution State 3607~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 3608 3609This section defines the register state that will be set up by the packet 3610processor prior to the start of execution of every wavefront. This is limited by 3611the constraints of the hardware controllers of CP/ADC/SPI. 3612 3613The order of the SGPR registers is defined, but the compiler can specify which 3614ones are actually setup in the kernel descriptor using the ``enable_sgpr_*`` bit 3615fields (see :ref:`amdgpu-amdhsa-kernel-descriptor`). The register numbers used 3616for enabled registers are dense starting at SGPR0: the first enabled register is 3617SGPR0, the next enabled register is SGPR1 etc.; disabled registers do not have 3618an SGPR number. 3619 3620The initial SGPRs comprise up to 16 User SRGPs that are set by CP and apply to 3621all wavefronts of the grid. It is possible to specify more than 16 User SGPRs 3622using the ``enable_sgpr_*`` bit fields, in which case only the first 16 are 3623actually initialized. These are then immediately followed by the System SGPRs 3624that are set up by ADC/SPI and can have different values for each wavefront of 3625the grid dispatch. 3626 3627SGPR register initial state is defined in 3628:ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 3629 3630 .. table:: SGPR Register Set Up Order 3631 :name: amdgpu-amdhsa-sgpr-register-set-up-order-table 3632 3633 ========== ========================== ====== ============================== 3634 SGPR Order Name Number Description 3635 (kernel descriptor enable of 3636 field) SGPRs 3637 ========== ========================== ====== ============================== 3638 First Private Segment Buffer 4 V# that can be used, together 3639 (enable_sgpr_private with Scratch Wavefront Offset 3640 _segment_buffer) as an offset, to access the 3641 private address space using a 3642 segment address. 3643 3644 CP uses the value provided by 3645 the runtime. 3646 then Dispatch Ptr 2 64-bit address of AQL dispatch 3647 (enable_sgpr_dispatch_ptr) packet for kernel dispatch 3648 actually executing. 3649 then Queue Ptr 2 64-bit address of amd_queue_t 3650 (enable_sgpr_queue_ptr) object for AQL queue on which 3651 the dispatch packet was 3652 queued. 3653 then Kernarg Segment Ptr 2 64-bit address of Kernarg 3654 (enable_sgpr_kernarg segment. This is directly 3655 _segment_ptr) copied from the 3656 kernarg_address in the kernel 3657 dispatch packet. 3658 3659 Having CP load it once avoids 3660 loading it at the beginning of 3661 every wavefront. 3662 then Dispatch Id 2 64-bit Dispatch ID of the 3663 (enable_sgpr_dispatch_id) dispatch packet being 3664 executed. 3665 then Flat Scratch Init 2 This is 2 SGPRs: 3666 (enable_sgpr_flat_scratch 3667 _init) GFX6 3668 Not supported. 3669 GFX7-GFX8 3670 The first SGPR is a 32-bit 3671 byte offset from 3672 ``SH_HIDDEN_PRIVATE_BASE_VIMID`` 3673 to per SPI base of memory 3674 for scratch for the queue 3675 executing the kernel 3676 dispatch. CP obtains this 3677 from the runtime. (The 3678 Scratch Segment Buffer base 3679 address is 3680 ``SH_HIDDEN_PRIVATE_BASE_VIMID`` 3681 plus this offset.) The value 3682 of Scratch Wavefront Offset must 3683 be added to this offset by 3684 the kernel machine code, 3685 right shifted by 8, and 3686 moved to the FLAT_SCRATCH_HI 3687 SGPR register. 3688 FLAT_SCRATCH_HI corresponds 3689 to SGPRn-4 on GFX7, and 3690 SGPRn-6 on GFX8 (where SGPRn 3691 is the highest numbered SGPR 3692 allocated to the wavefront). 3693 FLAT_SCRATCH_HI is 3694 multiplied by 256 (as it is 3695 in units of 256 bytes) and 3696 added to 3697 ``SH_HIDDEN_PRIVATE_BASE_VIMID`` 3698 to calculate the per wavefront 3699 FLAT SCRATCH BASE in flat 3700 memory instructions that 3701 access the scratch 3702 aperture. 3703 3704 The second SGPR is 32-bit 3705 byte size of a single 3706 work-item's scratch memory 3707 usage. CP obtains this from 3708 the runtime, and it is 3709 always a multiple of DWORD. 3710 CP checks that the value in 3711 the kernel dispatch packet 3712 Private Segment Byte Size is 3713 not larger and requests the 3714 runtime to increase the 3715 queue's scratch size if 3716 necessary. The kernel code 3717 must move it to 3718 FLAT_SCRATCH_LO which is 3719 SGPRn-3 on GFX7 and SGPRn-5 3720 on GFX8. FLAT_SCRATCH_LO is 3721 used as the FLAT SCRATCH 3722 SIZE in flat memory 3723 instructions. Having CP load 3724 it once avoids loading it at 3725 the beginning of every 3726 wavefront. 3727 GFX9-GFX10 3728 This is the 3729 64-bit base address of the 3730 per SPI scratch backing 3731 memory managed by SPI for 3732 the queue executing the 3733 kernel dispatch. CP obtains 3734 this from the runtime (and 3735 divides it if there are 3736 multiple Shader Arrays each 3737 with its own SPI). The value 3738 of Scratch Wavefront Offset must 3739 be added by the kernel 3740 machine code and the result 3741 moved to the FLAT_SCRATCH 3742 SGPR which is SGPRn-6 and 3743 SGPRn-5. It is used as the 3744 FLAT SCRATCH BASE in flat 3745 memory instructions. 3746 then Private Segment Size 1 The 32-bit byte size of a 3747 (enable_sgpr_private single 3748 work-item's 3749 scratch_segment_size) memory 3750 allocation. This is the 3751 value from the kernel 3752 dispatch packet Private 3753 Segment Byte Size rounded up 3754 by CP to a multiple of 3755 DWORD. 3756 3757 Having CP load it once avoids 3758 loading it at the beginning of 3759 every wavefront. 3760 3761 This is not used for 3762 GFX7-GFX8 since it is the same 3763 value as the second SGPR of 3764 Flat Scratch Init. However, it 3765 may be needed for GFX9-GFX10 which 3766 changes the meaning of the 3767 Flat Scratch Init value. 3768 then Grid Work-Group Count X 1 32-bit count of the number of 3769 (enable_sgpr_grid work-groups in the X dimension 3770 _workgroup_count_X) for the grid being 3771 executed. Computed from the 3772 fields in the kernel dispatch 3773 packet as ((grid_size.x + 3774 workgroup_size.x - 1) / 3775 workgroup_size.x). 3776 then Grid Work-Group Count Y 1 32-bit count of the number of 3777 (enable_sgpr_grid work-groups in the Y dimension 3778 _workgroup_count_Y && for the grid being 3779 less than 16 previous executed. Computed from the 3780 SGPRs) fields in the kernel dispatch 3781 packet as ((grid_size.y + 3782 workgroup_size.y - 1) / 3783 workgroupSize.y). 3784 3785 Only initialized if <16 3786 previous SGPRs initialized. 3787 then Grid Work-Group Count Z 1 32-bit count of the number of 3788 (enable_sgpr_grid work-groups in the Z dimension 3789 _workgroup_count_Z && for the grid being 3790 less than 16 previous executed. Computed from the 3791 SGPRs) fields in the kernel dispatch 3792 packet as ((grid_size.z + 3793 workgroup_size.z - 1) / 3794 workgroupSize.z). 3795 3796 Only initialized if <16 3797 previous SGPRs initialized. 3798 then Work-Group Id X 1 32-bit work-group id in X 3799 (enable_sgpr_workgroup_id dimension of grid for 3800 _X) wavefront. 3801 then Work-Group Id Y 1 32-bit work-group id in Y 3802 (enable_sgpr_workgroup_id dimension of grid for 3803 _Y) wavefront. 3804 then Work-Group Id Z 1 32-bit work-group id in Z 3805 (enable_sgpr_workgroup_id dimension of grid for 3806 _Z) wavefront. 3807 then Work-Group Info 1 {first_wavefront, 14'b0000, 3808 (enable_sgpr_workgroup ordered_append_term[10:0], 3809 _info) threadgroup_size_in_wavefronts[5:0]} 3810 then Scratch Wavefront Offset 1 32-bit byte offset from base 3811 (enable_sgpr_private of scratch base of queue 3812 _segment_wavefront_offset) executing the kernel 3813 dispatch. Must be used as an 3814 offset with Private 3815 segment address when using 3816 Scratch Segment Buffer. It 3817 must be used to set up FLAT 3818 SCRATCH for flat addressing 3819 (see 3820 :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`). 3821 ========== ========================== ====== ============================== 3822 3823The order of the VGPR registers is defined, but the compiler can specify which 3824ones are actually setup in the kernel descriptor using the ``enable_vgpr*`` bit 3825fields (see :ref:`amdgpu-amdhsa-kernel-descriptor`). The register numbers used 3826for enabled registers are dense starting at VGPR0: the first enabled register is 3827VGPR0, the next enabled register is VGPR1 etc.; disabled registers do not have a 3828VGPR number. 3829 3830VGPR register initial state is defined in 3831:ref:`amdgpu-amdhsa-vgpr-register-set-up-order-table`. 3832 3833 .. table:: VGPR Register Set Up Order 3834 :name: amdgpu-amdhsa-vgpr-register-set-up-order-table 3835 3836 ========== ========================== ====== ============================== 3837 VGPR Order Name Number Description 3838 (kernel descriptor enable of 3839 field) VGPRs 3840 ========== ========================== ====== ============================== 3841 First Work-Item Id X 1 32-bit work item id in X 3842 (Always initialized) dimension of work-group for 3843 wavefront lane. 3844 then Work-Item Id Y 1 32-bit work item id in Y 3845 (enable_vgpr_workitem_id dimension of work-group for 3846 > 0) wavefront lane. 3847 then Work-Item Id Z 1 32-bit work item id in Z 3848 (enable_vgpr_workitem_id dimension of work-group for 3849 > 1) wavefront lane. 3850 ========== ========================== ====== ============================== 3851 3852The setting of registers is done by GPU CP/ADC/SPI hardware as follows: 3853 38541. SGPRs before the Work-Group Ids are set by CP using the 16 User Data 3855 registers. 38562. Work-group Id registers X, Y, Z are set by ADC which supports any 3857 combination including none. 38583. Scratch Wavefront Offset is set by SPI in a per wavefront basis which is why 3859 its value cannot be included with the flat scratch init value which is per 3860 queue. 38614. The VGPRs are set by SPI which only supports specifying either (X), (X, Y) 3862 or (X, Y, Z). 3863 3864Flat Scratch register pair are adjacent SGPRs so they can be moved as a 64-bit 3865value to the hardware required SGPRn-3 and SGPRn-4 respectively. 3866 3867The global segment can be accessed either using buffer instructions (GFX6 which 3868has V# 64-bit address support), flat instructions (GFX7-GFX10), or global 3869instructions (GFX9-GFX10). 3870 3871If buffer operations are used, then the compiler can generate a V# with the 3872following properties: 3873 3874* base address of 0 3875* no swizzle 3876* ATC: 1 if IOMMU present (such as APU) 3877* ptr64: 1 3878* MTYPE set to support memory coherence that matches the runtime (such as CC for 3879 APU and NC for dGPU). 3880 3881.. _amdgpu-amdhsa-kernel-prolog: 3882 3883Kernel Prolog 3884~~~~~~~~~~~~~ 3885 3886The compiler performs initialization in the kernel prologue depending on the 3887target and information about things like stack usage in the kernel and called 3888functions. Some of this initialization requires the compiler to request certain 3889User and System SGPRs be present in the 3890:ref:`amdgpu-amdhsa-initial-kernel-execution-state` via the 3891:ref:`amdgpu-amdhsa-kernel-descriptor`. 3892 3893.. _amdgpu-amdhsa-kernel-prolog-cfi: 3894 3895CFI 3896+++ 3897 38981. The CFI return address is undefined. 3899 39002. The CFI CFA is defined using an expression which evaluates to a location 3901 description that comprises one memory location description for the 3902 ``DW_ASPACE_AMDGPU_private_lane`` address space address ``0``. 3903 3904.. _amdgpu-amdhsa-kernel-prolog-m0: 3905 3906M0 3907++ 3908 3909GFX6-GFX8 3910 The M0 register must be initialized with a value at least the total LDS size 3911 if the kernel may access LDS via DS or flat operations. Total LDS size is 3912 available in dispatch packet. For M0, it is also possible to use maximum 3913 possible value of LDS for given target (0x7FFF for GFX6 and 0xFFFF for 3914 GFX7-GFX8). 3915GFX9-GFX10 3916 The M0 register is not used for range checking LDS accesses and so does not 3917 need to be initialized in the prolog. 3918 3919.. _amdgpu-amdhsa-kernel-prolog-stack-pointer: 3920 3921Stack Pointer 3922+++++++++++++ 3923 3924If the kernel has function calls it must set up the ABI stack pointer described 3925in :ref:`amdgpu-amdhsa-function-call-convention-non-kernel-functions` by setting 3926SGPR32 to the unswizzled scratch offset of the address past the last local 3927allocation. 3928 3929.. _amdgpu-amdhsa-kernel-prolog-frame-pointer: 3930 3931Frame Pointer 3932+++++++++++++ 3933 3934If the kernel needs a frame pointer for the reasons defined in 3935``SIFrameLowering`` then SGPR33 is used and is always set to ``0`` in the 3936kernel prolog. If a frame pointer is not required then all uses of the frame 3937pointer are replaced with immediate ``0`` offsets. 3938 3939.. _amdgpu-amdhsa-kernel-prolog-flat-scratch: 3940 3941Flat Scratch 3942++++++++++++ 3943 3944If the kernel or any function it calls may use flat operations to access 3945scratch memory, the prolog code must set up the FLAT_SCRATCH register pair 3946(FLAT_SCRATCH_LO/FLAT_SCRATCH_HI which are in SGPRn-4/SGPRn-3). Initialization 3947uses Flat Scratch Init and Scratch Wavefront Offset SGPR registers (see 3948:ref:`amdgpu-amdhsa-initial-kernel-execution-state`): 3949 3950GFX6 3951 Flat scratch is not supported. 3952 3953GFX7-GFX8 3954 3955 1. The low word of Flat Scratch Init is 32-bit byte offset from 3956 ``SH_HIDDEN_PRIVATE_BASE_VIMID`` to the base of scratch backing memory 3957 being managed by SPI for the queue executing the kernel dispatch. This is 3958 the same value used in the Scratch Segment Buffer V# base address. The 3959 prolog must add the value of Scratch Wavefront Offset to get the 3960 wavefront's byte scratch backing memory offset from 3961 ``SH_HIDDEN_PRIVATE_BASE_VIMID``. Since FLAT_SCRATCH_LO is in units of 256 3962 bytes, the offset must be right shifted by 8 before moving into 3963 FLAT_SCRATCH_LO. 3964 2. The second word of Flat Scratch Init is 32-bit byte size of a single 3965 work-items scratch memory usage. This is directly loaded from the kernel 3966 dispatch packet Private Segment Byte Size and rounded up to a multiple of 3967 DWORD. Having CP load it once avoids loading it at the beginning of every 3968 wavefront. The prolog must move it to FLAT_SCRATCH_LO for use as FLAT 3969 SCRATCH SIZE. 3970 3971GFX9-GFX10 3972 The Flat Scratch Init is the 64-bit address of the base of scratch backing 3973 memory being managed by SPI for the queue executing the kernel dispatch. The 3974 prolog must add the value of Scratch Wavefront Offset and moved to the 3975 FLAT_SCRATCH pair for use as the flat scratch base in flat memory 3976 instructions. 3977 3978.. _amdgpu-amdhsa-kernel-prolog-private-segment-buffer: 3979 3980Private Segment Buffer 3981++++++++++++++++++++++ 3982 3983A set of four SGPRs beginning at a four-aligned SGPR index are always selected 3984to serve as the scratch V# for the kernel as follows: 3985 3986 - If it is known during instruction selection that there is stack usage, 3987 SGPR0-3 is reserved for use as the scratch V#. Stack usage is assumed if 3988 optimizations are disabled (``-O0``), if stack objects already exist (for 3989 locals, etc.), or if there are any function calls. 3990 3991 - Otherwise, four high numbered SGPRs beginning at a four-aligned SGPR index 3992 are reserved for the tentative scratch V#. These will be used if it is 3993 determined that spilling is needed. 3994 3995 - If no use is made of the tentative scratch V#, then it is unreserved, 3996 and the register count is determined ignoring it. 3997 - If use is made of the tentative scratch V#, then its register numbers 3998 are shifted to the first four-aligned SGPR index after the highest one 3999 allocated by the register allocator, and all uses are updated. The 4000 register count includes them in the shifted location. 4001 - In either case, if the processor has the SGPR allocation bug, the 4002 tentative allocation is not shifted or unreserved in order to ensure 4003 the register count is higher to workaround the bug. 4004 4005 .. note:: 4006 4007 This approach of using a tentative scratch V# and shifting the register 4008 numbers if used avoids having to perform register allocation a second 4009 time if the tentative V# is eliminated. This is more efficient and 4010 avoids the problem that the second register allocation may perform 4011 spilling which will fail as there is no longer a scratch V#. 4012 4013When the kernel prolog code is being emitted it is known whether the scratch V# 4014described above is actually used. If it is, the prolog code must set it up by 4015copying the Private Segment Buffer to the scratch V# registers and then adding 4016the Private Segment Wavefront Offset to the queue base address in the V#. The 4017result is a V# with a base address pointing to the beginning of the wavefront 4018scratch backing memory. 4019 4020The Private Segment Buffer is always requested, but the Private Segment 4021Wavefront Offset is only requested if it is used (see 4022:ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4023 4024.. _amdgpu-amdhsa-memory-model: 4025 4026Memory Model 4027~~~~~~~~~~~~ 4028 4029This section describes the mapping of LLVM memory model onto AMDGPU machine code 4030(see :ref:`memmodel`). 4031 4032The AMDGPU backend supports the memory synchronization scopes specified in 4033:ref:`amdgpu-memory-scopes`. 4034 4035The code sequences used to implement the memory model are defined in table 4036:ref:`amdgpu-amdhsa-memory-model-code-sequences-gfx6-gfx10-table`. 4037 4038The sequences specify the order of instructions that a single thread must 4039execute. The ``s_waitcnt`` and ``buffer_wbinvl1_vol`` are defined with respect 4040to other memory instructions executed by the same thread. This allows them to be 4041moved earlier or later which can allow them to be combined with other instances 4042of the same instruction, or hoisted/sunk out of loops to improve 4043performance. Only the instructions related to the memory model are given; 4044additional ``s_waitcnt`` instructions are required to ensure registers are 4045defined before being used. These may be able to be combined with the memory 4046model ``s_waitcnt`` instructions as described above. 4047 4048The AMDGPU backend supports the following memory models: 4049 4050 HSA Memory Model [HSA]_ 4051 The HSA memory model uses a single happens-before relation for all address 4052 spaces (see :ref:`amdgpu-address-spaces`). 4053 OpenCL Memory Model [OpenCL]_ 4054 The OpenCL memory model which has separate happens-before relations for the 4055 global and local address spaces. Only a fence specifying both global and 4056 local address space, and seq_cst instructions join the relationships. Since 4057 the LLVM ``memfence`` instruction does not allow an address space to be 4058 specified the OpenCL fence has to conservatively assume both local and 4059 global address space was specified. However, optimizations can often be 4060 done to eliminate the additional ``s_waitcnt`` instructions when there are 4061 no intervening memory instructions which access the corresponding address 4062 space. The code sequences in the table indicate what can be omitted for the 4063 OpenCL memory. The target triple environment is used to determine if the 4064 source language is OpenCL (see :ref:`amdgpu-opencl`). 4065 4066``ds/flat_load/store/atomic`` instructions to local memory are termed LDS 4067operations. 4068 4069``buffer/global/flat_load/store/atomic`` instructions to global memory are 4070termed vector memory operations. 4071 4072For GFX6-GFX9: 4073 4074* Each agent has multiple shader arrays (SA). 4075* Each SA has multiple compute units (CU). 4076* Each CU has multiple SIMDs that execute wavefronts. 4077* The wavefronts for a single work-group are executed in the same CU but may be 4078 executed by different SIMDs. 4079* Each CU has a single LDS memory shared by the wavefronts of the work-groups 4080 executing on it. 4081* All LDS operations of a CU are performed as wavefront wide operations in a 4082 global order and involve no caching. Completion is reported to a wavefront in 4083 execution order. 4084* The LDS memory has multiple request queues shared by the SIMDs of a 4085 CU. Therefore, the LDS operations performed by different wavefronts of a 4086 work-group can be reordered relative to each other, which can result in 4087 reordering the visibility of vector memory operations with respect to LDS 4088 operations of other wavefronts in the same work-group. A ``s_waitcnt 4089 lgkmcnt(0)`` is required to ensure synchronization between LDS operations and 4090 vector memory operations between wavefronts of a work-group, but not between 4091 operations performed by the same wavefront. 4092* The vector memory operations are performed as wavefront wide operations and 4093 completion is reported to a wavefront in execution order. The exception is 4094 that for GFX7-GFX9 ``flat_load/store/atomic`` instructions can report out of 4095 vector memory order if they access LDS memory, and out of LDS operation order 4096 if they access global memory. 4097* The vector memory operations access a single vector L1 cache shared by all 4098 SIMDs a CU. Therefore, no special action is required for coherence between the 4099 lanes of a single wavefront, or for coherence between wavefronts in the same 4100 work-group. A ``buffer_wbinvl1_vol`` is required for coherence between 4101 wavefronts executing in different work-groups as they may be executing on 4102 different CUs. 4103* The scalar memory operations access a scalar L1 cache shared by all wavefronts 4104 on a group of CUs. The scalar and vector L1 caches are not coherent. However, 4105 scalar operations are used in a restricted way so do not impact the memory 4106 model. See :ref:`amdgpu-address-spaces`. 4107* The vector and scalar memory operations use an L2 cache shared by all CUs on 4108 the same agent. 4109* The L2 cache has independent channels to service disjoint ranges of virtual 4110 addresses. 4111* Each CU has a separate request queue per channel. Therefore, the vector and 4112 scalar memory operations performed by wavefronts executing in different 4113 work-groups (which may be executing on different CUs) of an agent can be 4114 reordered relative to each other. A ``s_waitcnt vmcnt(0)`` is required to 4115 ensure synchronization between vector memory operations of different CUs. It 4116 ensures a previous vector memory operation has completed before executing a 4117 subsequent vector memory or LDS operation and so can be used to meet the 4118 requirements of acquire and release. 4119* The L2 cache can be kept coherent with other agents on some targets, or ranges 4120 of virtual addresses can be set up to bypass it to ensure system coherence. 4121 4122For GFX10: 4123 4124* Each agent has multiple shader arrays (SA). 4125* Each SA has multiple work-group processors (WGP). 4126* Each WGP has multiple compute units (CU). 4127* Each CU has multiple SIMDs that execute wavefronts. 4128* The wavefronts for a single work-group are executed in the same 4129 WGP. In CU wavefront execution mode the wavefronts may be executed by 4130 different SIMDs in the same CU. In WGP wavefront execution mode the 4131 wavefronts may be executed by different SIMDs in different CUs in the same 4132 WGP. 4133* Each WGP has a single LDS memory shared by the wavefronts of the work-groups 4134 executing on it. 4135* All LDS operations of a WGP are performed as wavefront wide operations in a 4136 global order and involve no caching. Completion is reported to a wavefront in 4137 execution order. 4138* The LDS memory has multiple request queues shared by the SIMDs of a 4139 WGP. Therefore, the LDS operations performed by different wavefronts of a 4140 work-group can be reordered relative to each other, which can result in 4141 reordering the visibility of vector memory operations with respect to LDS 4142 operations of other wavefronts in the same work-group. A ``s_waitcnt 4143 lgkmcnt(0)`` is required to ensure synchronization between LDS operations and 4144 vector memory operations between wavefronts of a work-group, but not between 4145 operations performed by the same wavefront. 4146* The vector memory operations are performed as wavefront wide operations. 4147 Completion of load/store/sample operations are reported to a wavefront in 4148 execution order of other load/store/sample operations performed by that 4149 wavefront. 4150* The vector memory operations access a vector L0 cache. There is a single L0 4151 cache per CU. Each SIMD of a CU accesses the same L0 cache. Therefore, no 4152 special action is required for coherence between the lanes of a single 4153 wavefront. However, a ``BUFFER_GL0_INV`` is required for coherence between 4154 wavefronts executing in the same work-group as they may be executing on SIMDs 4155 of different CUs that access different L0s. A ``BUFFER_GL0_INV`` is also 4156 required for coherence between wavefronts executing in different work-groups 4157 as they may be executing on different WGPs. 4158* The scalar memory operations access a scalar L0 cache shared by all wavefronts 4159 on a WGP. The scalar and vector L0 caches are not coherent. However, scalar 4160 operations are used in a restricted way so do not impact the memory model. See 4161 :ref:`amdgpu-address-spaces`. 4162* The vector and scalar memory L0 caches use an L1 cache shared by all WGPs on 4163 the same SA. Therefore, no special action is required for coherence between 4164 the wavefronts of a single work-group. However, a ``BUFFER_GL1_INV`` is 4165 required for coherence between wavefronts executing in different work-groups 4166 as they may be executing on different SAs that access different L1s. 4167* The L1 caches have independent quadrants to service disjoint ranges of virtual 4168 addresses. 4169* Each L0 cache has a separate request queue per L1 quadrant. Therefore, the 4170 vector and scalar memory operations performed by different wavefronts, whether 4171 executing in the same or different work-groups (which may be executing on 4172 different CUs accessing different L0s), can be reordered relative to each 4173 other. A ``s_waitcnt vmcnt(0) & vscnt(0)`` is required to ensure 4174 synchronization between vector memory operations of different wavefronts. It 4175 ensures a previous vector memory operation has completed before executing a 4176 subsequent vector memory or LDS operation and so can be used to meet the 4177 requirements of acquire, release and sequential consistency. 4178* The L1 caches use an L2 cache shared by all SAs on the same agent. 4179* The L2 cache has independent channels to service disjoint ranges of virtual 4180 addresses. 4181* Each L1 quadrant of a single SA accesses a different L2 channel. Each L1 4182 quadrant has a separate request queue per L2 channel. Therefore, the vector 4183 and scalar memory operations performed by wavefronts executing in different 4184 work-groups (which may be executing on different SAs) of an agent can be 4185 reordered relative to each other. A ``s_waitcnt vmcnt(0) & vscnt(0)`` is 4186 required to ensure synchronization between vector memory operations of 4187 different SAs. It ensures a previous vector memory operation has completed 4188 before executing a subsequent vector memory and so can be used to meet the 4189 requirements of acquire, release and sequential consistency. 4190* The L2 cache can be kept coherent with other agents on some targets, or ranges 4191 of virtual addresses can be set up to bypass it to ensure system coherence. 4192 4193Private address space uses ``buffer_load/store`` using the scratch V# 4194(GFX6-GFX8), or ``scratch_load/store`` (GFX9-GFX10). Since only a single thread 4195is accessing the memory, atomic memory orderings are not meaningful, and all 4196accesses are treated as non-atomic. 4197 4198Constant address space uses ``buffer/global_load`` instructions (or equivalent 4199scalar memory instructions). Since the constant address space contents do not 4200change during the execution of a kernel dispatch it is not legal to perform 4201stores, and atomic memory orderings are not meaningful, and all access are 4202treated as non-atomic. 4203 4204A memory synchronization scope wider than work-group is not meaningful for the 4205group (LDS) address space and is treated as work-group. 4206 4207The memory model does not support the region address space which is treated as 4208non-atomic. 4209 4210Acquire memory ordering is not meaningful on store atomic instructions and is 4211treated as non-atomic. 4212 4213Release memory ordering is not meaningful on load atomic instructions and is 4214treated a non-atomic. 4215 4216Acquire-release memory ordering is not meaningful on load or store atomic 4217instructions and is treated as acquire and release respectively. 4218 4219AMDGPU backend only uses scalar memory operations to access memory that is 4220proven to not change during the execution of the kernel dispatch. This includes 4221constant address space and global address space for program scope const 4222variables. Therefore, the kernel machine code does not have to maintain the 4223scalar L1 cache to ensure it is coherent with the vector L1 cache. The scalar 4224and vector L1 caches are invalidated between kernel dispatches by CP since 4225constant address space data may change between kernel dispatch executions. See 4226:ref:`amdgpu-address-spaces`. 4227 4228The one exception is if scalar writes are used to spill SGPR registers. In this 4229case the AMDGPU backend ensures the memory location used to spill is never 4230accessed by vector memory operations at the same time. If scalar writes are used 4231then a ``s_dcache_wb`` is inserted before the ``s_endpgm`` and before a function 4232return since the locations may be used for vector memory instructions by a 4233future wavefront that uses the same scratch area, or a function call that 4234creates a frame at the same address, respectively. There is no need for a 4235``s_dcache_inv`` as all scalar writes are write-before-read in the same thread. 4236 4237For GFX6-GFX9, scratch backing memory (which is used for the private address 4238space) is accessed with MTYPE NC_NV (non-coherent non-volatile). Since the 4239private address space is only accessed by a single thread, and is always 4240write-before-read, there is never a need to invalidate these entries from the L1 4241cache. Hence all cache invalidates are done as ``*_vol`` to only invalidate the 4242volatile cache lines. 4243 4244For GFX10, scratch backing memory (which is used for the private address space) 4245is accessed with MTYPE NC (non-coherent). Since the private address space is 4246only accessed by a single thread, and is always write-before-read, there is 4247never a need to invalidate these entries from the L0 or L1 caches. 4248 4249For GFX10, wavefronts are executed in native mode with in-order reporting of 4250loads and sample instructions. In this mode vmcnt reports completion of load, 4251atomic with return and sample instructions in order, and the vscnt reports the 4252completion of store and atomic without return in order. See ``MEM_ORDERED`` 4253field in :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 4254 4255In GFX10, wavefronts can be executed in WGP or CU wavefront execution mode: 4256 4257* In WGP wavefront execution mode the wavefronts of a work-group are executed 4258 on the SIMDs of both CUs of the WGP. Therefore, explicit management of the per 4259 CU L0 caches is required for work-group synchronization. Also accesses to L1 4260 at work-group scope need to be explicitly ordered as the accesses from 4261 different CUs are not ordered. 4262* In CU wavefront execution mode the wavefronts of a work-group are executed on 4263 the SIMDs of a single CU of the WGP. Therefore, all global memory access by 4264 the work-group access the same L0 which in turn ensures L1 accesses are 4265 ordered and so do not require explicit management of the caches for 4266 work-group synchronization. 4267 4268See ``WGP_MODE`` field in 4269:ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table` and 4270:ref:`amdgpu-target-features`. 4271 4272On dGPU the kernarg backing memory is accessed as UC (uncached) to avoid needing 4273to invalidate the L2 cache. For GFX6-GFX9, this also causes it to be treated as 4274non-volatile and so is not invalidated by ``*_vol``. On APU it is accessed as CC 4275(cache coherent) and so the L2 cache will be coherent with the CPU and other 4276agents. 4277 4278 .. table:: AMDHSA Memory Model Code Sequences GFX6-GFX10 4279 :name: amdgpu-amdhsa-memory-model-code-sequences-gfx6-gfx10-table 4280 4281 ============ ============ ============== ========== =============================== ================================== 4282 LLVM Instr LLVM Memory LLVM Memory AMDGPU AMDGPU Machine Code AMDGPU Machine Code 4283 Ordering Sync Scope Address GFX6-9 GFX10 4284 Space 4285 ============ ============ ============== ========== =============================== ================================== 4286 **Non-Atomic** 4287 ---------------------------------------------------------------------------------------------------------------------- 4288 load *none* *none* - global - !volatile & !nontemporal - !volatile & !nontemporal 4289 - generic 4290 - private 1. buffer/global/flat_load 1. buffer/global/flat_load 4291 - constant 4292 - volatile & !nontemporal - volatile & !nontemporal 4293 4294 1. buffer/global/flat_load 1. buffer/global/flat_load 4295 glc=1 glc=1 dlc=1 4296 4297 - nontemporal - nontemporal 4298 4299 1. buffer/global/flat_load 1. buffer/global/flat_load 4300 glc=1 slc=1 slc=1 4301 4302 load *none* *none* - local 1. ds_load 1. ds_load 4303 store *none* *none* - global - !nontemporal - !nontemporal 4304 - generic 4305 - private 1. buffer/global/flat_store 1. buffer/global/flat_store 4306 - constant 4307 - nontemporal - nontemporal 4308 4309 1. buffer/global/flat_store 1. buffer/global/flat_store 4310 glc=1 slc=1 slc=1 4311 4312 store *none* *none* - local 1. ds_store 1. ds_store 4313 **Unordered Atomic** 4314 ---------------------------------------------------------------------------------------------------------------------- 4315 load atomic unordered *any* *any* *Same as non-atomic*. *Same as non-atomic*. 4316 store atomic unordered *any* *any* *Same as non-atomic*. *Same as non-atomic*. 4317 atomicrmw unordered *any* *any* *Same as monotonic *Same as monotonic 4318 atomic*. atomic*. 4319 **Monotonic Atomic** 4320 ---------------------------------------------------------------------------------------------------------------------- 4321 load atomic monotonic - singlethread - global 1. buffer/global/flat_load 1. buffer/global/flat_load 4322 - wavefront - generic 4323 load atomic monotonic - workgroup - global 1. buffer/global/flat_load 1. buffer/global/flat_load 4324 - generic glc=1 4325 4326 - If CU wavefront execution mode, omit glc=1. 4327 4328 load atomic monotonic - singlethread - local 1. ds_load 1. ds_load 4329 - wavefront 4330 - workgroup 4331 load atomic monotonic - agent - global 1. buffer/global/flat_load 1. buffer/global/flat_load 4332 - system - generic glc=1 glc=1 dlc=1 4333 store atomic monotonic - singlethread - global 1. buffer/global/flat_store 1. buffer/global/flat_store 4334 - wavefront - generic 4335 - workgroup 4336 - agent 4337 - system 4338 store atomic monotonic - singlethread - local 1. ds_store 1. ds_store 4339 - wavefront 4340 - workgroup 4341 atomicrmw monotonic - singlethread - global 1. buffer/global/flat_atomic 1. buffer/global/flat_atomic 4342 - wavefront - generic 4343 - workgroup 4344 - agent 4345 - system 4346 atomicrmw monotonic - singlethread - local 1. ds_atomic 1. ds_atomic 4347 - wavefront 4348 - workgroup 4349 **Acquire Atomic** 4350 ---------------------------------------------------------------------------------------------------------------------- 4351 load atomic acquire - singlethread - global 1. buffer/global/ds/flat_load 1. buffer/global/ds/flat_load 4352 - wavefront - local 4353 - generic 4354 load atomic acquire - workgroup - global 1. buffer/global/flat_load 1. buffer/global_load glc=1 4355 4356 - If CU wavefront execution mode, omit glc=1. 4357 4358 2. s_waitcnt vmcnt(0) 4359 4360 - If CU wavefront execution mode, omit. 4361 - Must happen before 4362 the following buffer_gl0_inv 4363 and before any following 4364 global/generic 4365 load/load 4366 atomic/store/store 4367 atomic/atomicrmw. 4368 4369 3. buffer_gl0_inv 4370 4371 - If CU wavefront execution mode, omit. 4372 - Ensures that 4373 following 4374 loads will not see 4375 stale data. 4376 4377 load atomic acquire - workgroup - local 1. ds_load 1. ds_load 4378 2. s_waitcnt lgkmcnt(0) 2. s_waitcnt lgkmcnt(0) 4379 4380 - If OpenCL, omit. - If OpenCL, omit. 4381 - Must happen before - Must happen before 4382 any following the following buffer_gl0_inv 4383 global/generic and before any following 4384 load/load global/generic load/load 4385 atomic/store/store atomic/store/store 4386 atomic/atomicrmw. atomic/atomicrmw. 4387 - Ensures any - Ensures any 4388 following global following global 4389 data read is no data read is no 4390 older than the load older than the load 4391 atomic value being atomic value being 4392 acquired. acquired. 4393 4394 3. buffer_gl0_inv 4395 4396 - If CU wavefront execution mode, omit. 4397 - If OpenCL, omit. 4398 - Ensures that 4399 following 4400 loads will not see 4401 stale data. 4402 4403 load atomic acquire - workgroup - generic 1. flat_load 1. flat_load glc=1 4404 4405 - If CU wavefront execution mode, omit glc=1. 4406 4407 2. s_waitcnt lgkmcnt(0) 2. s_waitcnt lgkmcnt(0) & 4408 vmcnt(0) 4409 4410 - If CU wavefront execution mode, omit vmcnt. 4411 - If OpenCL, omit. - If OpenCL, omit 4412 lgkmcnt(0). 4413 - Must happen before - Must happen before 4414 any following the following 4415 global/generic buffer_gl0_inv and any 4416 load/load following global/generic 4417 atomic/store/store load/load 4418 atomic/atomicrmw. atomic/store/store 4419 atomic/atomicrmw. 4420 - Ensures any - Ensures any 4421 following global following global 4422 data read is no data read is no 4423 older than the load older than the load 4424 atomic value being atomic value being 4425 acquired. acquired. 4426 4427 3. buffer_gl0_inv 4428 4429 - If CU wavefront execution mode, omit. 4430 - Ensures that 4431 following 4432 loads will not see 4433 stale data. 4434 4435 load atomic acquire - agent - global 1. buffer/global/flat_load 1. buffer/global_load 4436 - system glc=1 glc=1 dlc=1 4437 2. s_waitcnt vmcnt(0) 2. s_waitcnt vmcnt(0) 4438 4439 - Must happen before - Must happen before 4440 following following 4441 buffer_wbinvl1_vol. buffer_gl*_inv. 4442 - Ensures the load - Ensures the load 4443 has completed has completed 4444 before invalidating before invalidating 4445 the cache. the caches. 4446 4447 3. buffer_wbinvl1_vol 3. buffer_gl0_inv; 4448 buffer_gl1_inv 4449 4450 - Must happen before - Must happen before 4451 any following any following 4452 global/generic global/generic 4453 load/load load/load 4454 atomic/atomicrmw. atomic/atomicrmw. 4455 - Ensures that - Ensures that 4456 following following 4457 loads will not see loads will not see 4458 stale global data. stale global data. 4459 4460 load atomic acquire - agent - generic 1. flat_load glc=1 1. flat_load glc=1 dlc=1 4461 - system 2. s_waitcnt vmcnt(0) & 2. s_waitcnt vmcnt(0) & 4462 lgkmcnt(0) lgkmcnt(0) 4463 4464 - If OpenCL omit - If OpenCL omit 4465 lgkmcnt(0). lgkmcnt(0). 4466 - Must happen before - Must happen before 4467 following following 4468 buffer_wbinvl1_vol. buffer_gl*_invl. 4469 - Ensures the flat_load - Ensures the flat_load 4470 has completed has completed 4471 before invalidating before invalidating 4472 the cache. the caches. 4473 4474 3. buffer_wbinvl1_vol 3. buffer_gl0_inv; 4475 buffer_gl1_inv 4476 4477 - Must happen before - Must happen before 4478 any following any following 4479 global/generic global/generic 4480 load/load load/load 4481 atomic/atomicrmw. atomic/atomicrmw. 4482 - Ensures that - Ensures that 4483 following loads following loads 4484 will not see stale will not see stale 4485 global data. global data. 4486 4487 atomicrmw acquire - singlethread - global 1. buffer/global/ds/flat_atomic 1. buffer/global/ds/flat_atomic 4488 - wavefront - local 4489 - generic 4490 atomicrmw acquire - workgroup - global 1. buffer/global/flat_atomic 1. buffer/global_atomic 4491 2. s_waitcnt vm/vscnt(0) 4492 4493 - If CU wavefront execution mode, omit. 4494 - Use vmcnt if atomic with 4495 return and vscnt if atomic 4496 with no-return. 4497 - Must happen before 4498 the following buffer_gl0_inv 4499 and before any following 4500 global/generic 4501 load/load 4502 atomic/store/store 4503 atomic/atomicrmw. 4504 4505 3. buffer_gl0_inv 4506 4507 - If CU wavefront execution mode, omit. 4508 - Ensures that 4509 following 4510 loads will not see 4511 stale data. 4512 4513 atomicrmw acquire - workgroup - local 1. ds_atomic 1. ds_atomic 4514 2. waitcnt lgkmcnt(0) 2. waitcnt lgkmcnt(0) 4515 4516 - If OpenCL, omit. - If OpenCL, omit. 4517 - Must happen before - Must happen before 4518 any following the following 4519 global/generic buffer_gl0_inv. 4520 load/load 4521 atomic/store/store 4522 atomic/atomicrmw. 4523 - Ensures any - Ensures any 4524 following global following global 4525 data read is no data read is no 4526 older than the older than the 4527 atomicrmw value atomicrmw value 4528 being acquired. being acquired. 4529 4530 3. buffer_gl0_inv 4531 4532 - If OpenCL omit. 4533 - Ensures that 4534 following 4535 loads will not see 4536 stale data. 4537 4538 atomicrmw acquire - workgroup - generic 1. flat_atomic 1. flat_atomic 4539 2. waitcnt lgkmcnt(0) 2. waitcnt lgkmcnt(0) & 4540 vm/vscnt(0) 4541 4542 - If CU wavefront execution mode, omit vm/vscnt. 4543 - If OpenCL, omit. - If OpenCL, omit 4544 waitcnt lgkmcnt(0).. 4545 - Use vmcnt if atomic with 4546 return and vscnt if atomic 4547 with no-return. 4548 waitcnt lgkmcnt(0). 4549 - Must happen before - Must happen before 4550 any following the following 4551 global/generic buffer_gl0_inv. 4552 load/load 4553 atomic/store/store 4554 atomic/atomicrmw. 4555 - Ensures any - Ensures any 4556 following global following global 4557 data read is no data read is no 4558 older than the older than the 4559 atomicrmw value atomicrmw value 4560 being acquired. being acquired. 4561 4562 3. buffer_gl0_inv 4563 4564 - If CU wavefront execution mode, omit. 4565 - Ensures that 4566 following 4567 loads will not see 4568 stale data. 4569 4570 atomicrmw acquire - agent - global 1. buffer/global/flat_atomic 1. buffer/global_atomic 4571 - system 2. s_waitcnt vmcnt(0) 2. s_waitcnt vm/vscnt(0) 4572 4573 - Use vmcnt if atomic with 4574 return and vscnt if atomic 4575 with no-return. 4576 waitcnt lgkmcnt(0). 4577 - Must happen before - Must happen before 4578 following following 4579 buffer_wbinvl1_vol. buffer_gl*_inv. 4580 - Ensures the - Ensures the 4581 atomicrmw has atomicrmw has 4582 completed before completed before 4583 invalidating the invalidating the 4584 cache. caches. 4585 4586 3. buffer_wbinvl1_vol 3. buffer_gl0_inv; 4587 buffer_gl1_inv 4588 4589 - Must happen before - Must happen before 4590 any following any following 4591 global/generic global/generic 4592 load/load load/load 4593 atomic/atomicrmw. atomic/atomicrmw. 4594 - Ensures that - Ensures that 4595 following loads following loads 4596 will not see stale will not see stale 4597 global data. global data. 4598 4599 atomicrmw acquire - agent - generic 1. flat_atomic 1. flat_atomic 4600 - system 2. s_waitcnt vmcnt(0) & 2. s_waitcnt vm/vscnt(0) & 4601 lgkmcnt(0) lgkmcnt(0) 4602 4603 - If OpenCL, omit - If OpenCL, omit 4604 lgkmcnt(0). lgkmcnt(0). 4605 - Use vmcnt if atomic with 4606 return and vscnt if atomic 4607 with no-return. 4608 - Must happen before - Must happen before 4609 following following 4610 buffer_wbinvl1_vol. buffer_gl*_inv. 4611 - Ensures the - Ensures the 4612 atomicrmw has atomicrmw has 4613 completed before completed before 4614 invalidating the invalidating the 4615 cache. caches. 4616 4617 3. buffer_wbinvl1_vol 3. buffer_gl0_inv; 4618 buffer_gl1_inv 4619 4620 - Must happen before - Must happen before 4621 any following any following 4622 global/generic global/generic 4623 load/load load/load 4624 atomic/atomicrmw. atomic/atomicrmw. 4625 - Ensures that - Ensures that 4626 following loads following loads 4627 will not see stale will not see stale 4628 global data. global data. 4629 4630 fence acquire - singlethread *none* *none* *none* 4631 - wavefront 4632 fence acquire - workgroup *none* 1. s_waitcnt lgkmcnt(0) 1. s_waitcnt lgkmcnt(0) & 4633 vmcnt(0) & vscnt(0) 4634 4635 - If CU wavefront execution mode, omit vmcnt and 4636 vscnt. 4637 - If OpenCL and - If OpenCL and 4638 address space is address space is 4639 not generic, omit. not generic, omit 4640 lgkmcnt(0). 4641 - If OpenCL and 4642 address space is 4643 local, omit 4644 vmcnt(0) and vscnt(0). 4645 - However, since LLVM - However, since LLVM 4646 currently has no currently has no 4647 address space on address space on 4648 the fence need to the fence need to 4649 conservatively conservatively 4650 always generate. If always generate. If 4651 fence had an fence had an 4652 address space then address space then 4653 set to address set to address 4654 space of OpenCL space of OpenCL 4655 fence flag, or to fence flag, or to 4656 generic if both generic if both 4657 local and global local and global 4658 flags are flags are 4659 specified. specified. 4660 - Must happen after 4661 any preceding 4662 local/generic load 4663 atomic/atomicrmw 4664 with an equal or 4665 wider sync scope 4666 and memory ordering 4667 stronger than 4668 unordered (this is 4669 termed the 4670 fence-paired-atomic). 4671 - Must happen before 4672 any following 4673 global/generic 4674 load/load 4675 atomic/store/store 4676 atomic/atomicrmw. 4677 - Ensures any 4678 following global 4679 data read is no 4680 older than the 4681 value read by the 4682 fence-paired-atomic. 4683 - Could be split into 4684 separate s_waitcnt 4685 vmcnt(0), s_waitcnt 4686 vscnt(0) and s_waitcnt 4687 lgkmcnt(0) to allow 4688 them to be 4689 independently moved 4690 according to the 4691 following rules. 4692 - s_waitcnt vmcnt(0) 4693 must happen after 4694 any preceding 4695 global/generic load 4696 atomic/ 4697 atomicrmw-with-return-value 4698 with an equal or 4699 wider sync scope 4700 and memory ordering 4701 stronger than 4702 unordered (this is 4703 termed the 4704 fence-paired-atomic). 4705 - s_waitcnt vscnt(0) 4706 must happen after 4707 any preceding 4708 global/generic 4709 atomicrmw-no-return-value 4710 with an equal or 4711 wider sync scope 4712 and memory ordering 4713 stronger than 4714 unordered (this is 4715 termed the 4716 fence-paired-atomic). 4717 - s_waitcnt lgkmcnt(0) 4718 must happen after 4719 any preceding 4720 local/generic load 4721 atomic/atomicrmw 4722 with an equal or 4723 wider sync scope 4724 and memory ordering 4725 stronger than 4726 unordered (this is 4727 termed the 4728 fence-paired-atomic). 4729 - Must happen before 4730 the following 4731 buffer_gl0_inv. 4732 - Ensures that the 4733 fence-paired atomic 4734 has completed 4735 before invalidating 4736 the 4737 cache. Therefore 4738 any following 4739 locations read must 4740 be no older than 4741 the value read by 4742 the 4743 fence-paired-atomic. 4744 4745 3. buffer_gl0_inv 4746 4747 - If CU wavefront execution mode, omit. 4748 - Ensures that 4749 following 4750 loads will not see 4751 stale data. 4752 4753 fence acquire - agent *none* 1. s_waitcnt lgkmcnt(0) & 1. s_waitcnt lgkmcnt(0) & 4754 - system vmcnt(0) vmcnt(0) & vscnt(0) 4755 4756 - If OpenCL and - If OpenCL and 4757 address space is address space is 4758 not generic, omit not generic, omit 4759 lgkmcnt(0). lgkmcnt(0). 4760 - If OpenCL and 4761 address space is 4762 local, omit 4763 vmcnt(0) and vscnt(0). 4764 - However, since LLVM - However, since LLVM 4765 currently has no currently has no 4766 address space on address space on 4767 the fence need to the fence need to 4768 conservatively conservatively 4769 always generate always generate 4770 (see comment for (see comment for 4771 previous fence). previous fence). 4772 - Could be split into 4773 separate s_waitcnt 4774 vmcnt(0) and 4775 s_waitcnt 4776 lgkmcnt(0) to allow 4777 them to be 4778 independently moved 4779 according to the 4780 following rules. 4781 - s_waitcnt vmcnt(0) 4782 must happen after 4783 any preceding 4784 global/generic load 4785 atomic/atomicrmw 4786 with an equal or 4787 wider sync scope 4788 and memory ordering 4789 stronger than 4790 unordered (this is 4791 termed the 4792 fence-paired-atomic). 4793 - s_waitcnt lgkmcnt(0) 4794 must happen after 4795 any preceding 4796 local/generic load 4797 atomic/atomicrmw 4798 with an equal or 4799 wider sync scope 4800 and memory ordering 4801 stronger than 4802 unordered (this is 4803 termed the 4804 fence-paired-atomic). 4805 - Must happen before 4806 the following 4807 buffer_wbinvl1_vol. 4808 - Ensures that the 4809 fence-paired atomic 4810 has completed 4811 before invalidating 4812 the 4813 cache. Therefore 4814 any following 4815 locations read must 4816 be no older than 4817 the value read by 4818 the 4819 fence-paired-atomic. 4820 - Could be split into 4821 separate s_waitcnt 4822 vmcnt(0), s_waitcnt 4823 vscnt(0) and s_waitcnt 4824 lgkmcnt(0) to allow 4825 them to be 4826 independently moved 4827 according to the 4828 following rules. 4829 - s_waitcnt vmcnt(0) 4830 must happen after 4831 any preceding 4832 global/generic load 4833 atomic/ 4834 atomicrmw-with-return-value 4835 with an equal or 4836 wider sync scope 4837 and memory ordering 4838 stronger than 4839 unordered (this is 4840 termed the 4841 fence-paired-atomic). 4842 - s_waitcnt vscnt(0) 4843 must happen after 4844 any preceding 4845 global/generic 4846 atomicrmw-no-return-value 4847 with an equal or 4848 wider sync scope 4849 and memory ordering 4850 stronger than 4851 unordered (this is 4852 termed the 4853 fence-paired-atomic). 4854 - s_waitcnt lgkmcnt(0) 4855 must happen after 4856 any preceding 4857 local/generic load 4858 atomic/atomicrmw 4859 with an equal or 4860 wider sync scope 4861 and memory ordering 4862 stronger than 4863 unordered (this is 4864 termed the 4865 fence-paired-atomic). 4866 - Must happen before 4867 the following 4868 buffer_gl*_inv. 4869 - Ensures that the 4870 fence-paired atomic 4871 has completed 4872 before invalidating 4873 the 4874 caches. Therefore 4875 any following 4876 locations read must 4877 be no older than 4878 the value read by 4879 the 4880 fence-paired-atomic. 4881 4882 2. buffer_wbinvl1_vol 2. buffer_gl0_inv; 4883 buffer_gl1_inv 4884 4885 - Must happen before any - Must happen before any 4886 following global/generic following global/generic 4887 load/load load/load 4888 atomic/store/store atomic/store/store 4889 atomic/atomicrmw. atomic/atomicrmw. 4890 - Ensures that - Ensures that 4891 following loads following loads 4892 will not see stale will not see stale 4893 global data. global data. 4894 4895 **Release Atomic** 4896 ---------------------------------------------------------------------------------------------------------------------- 4897 store atomic release - singlethread - global 1. buffer/global/ds/flat_store 1. buffer/global/ds/flat_store 4898 - wavefront - local 4899 - generic 4900 store atomic release - workgroup - global 1. s_waitcnt lgkmcnt(0) 1. s_waitcnt lgkmcnt(0) & 4901 vmcnt(0) & vscnt(0) 4902 4903 - If CU wavefront execution mode, omit vmcnt and 4904 vscnt. 4905 - If OpenCL, omit. - If OpenCL, omit 4906 lgkmcnt(0). 4907 - Must happen after 4908 any preceding 4909 local/generic 4910 load/store/load 4911 atomic/store 4912 atomic/atomicrmw. 4913 - Could be split into 4914 separate s_waitcnt 4915 vmcnt(0), s_waitcnt 4916 vscnt(0) and s_waitcnt 4917 lgkmcnt(0) to allow 4918 them to be 4919 independently moved 4920 according to the 4921 following rules. 4922 - s_waitcnt vmcnt(0) 4923 must happen after 4924 any preceding 4925 global/generic load/load 4926 atomic/ 4927 atomicrmw-with-return-value. 4928 - s_waitcnt vscnt(0) 4929 must happen after 4930 any preceding 4931 global/generic 4932 store/store 4933 atomic/ 4934 atomicrmw-no-return-value. 4935 - s_waitcnt lgkmcnt(0) 4936 must happen after 4937 any preceding 4938 local/generic 4939 load/store/load 4940 atomic/store 4941 atomic/atomicrmw. 4942 - Must happen before - Must happen before 4943 the following the following 4944 store. store. 4945 - Ensures that all - Ensures that all 4946 memory operations memory operations 4947 to local have have 4948 completed before completed before 4949 performing the performing the 4950 store that is being store that is being 4951 released. released. 4952 4953 2. buffer/global/flat_store 2. buffer/global_store 4954 store atomic release - workgroup - local 1. waitcnt vmcnt(0) & vscnt(0) 4955 4956 - If CU wavefront execution mode, omit. 4957 - If OpenCL, omit. 4958 - Could be split into 4959 separate s_waitcnt 4960 vmcnt(0) and s_waitcnt 4961 vscnt(0) to allow 4962 them to be 4963 independently moved 4964 according to the 4965 following rules. 4966 - s_waitcnt vmcnt(0) 4967 must happen after 4968 any preceding 4969 global/generic load/load 4970 atomic/ 4971 atomicrmw-with-return-value. 4972 - s_waitcnt vscnt(0) 4973 must happen after 4974 any preceding 4975 global/generic 4976 store/store atomic/ 4977 atomicrmw-no-return-value. 4978 - Must happen before 4979 the following 4980 store. 4981 - Ensures that all 4982 global memory 4983 operations have 4984 completed before 4985 performing the 4986 store that is being 4987 released. 4988 4989 1. ds_store 2. ds_store 4990 store atomic release - workgroup - generic 1. s_waitcnt lgkmcnt(0) 1. s_waitcnt lgkmcnt(0) & 4991 vmcnt(0) & vscnt(0) 4992 4993 - If CU wavefront execution mode, omit vmcnt and 4994 vscnt. 4995 - If OpenCL, omit. - If OpenCL, omit 4996 lgkmcnt(0). 4997 - Must happen after 4998 any preceding 4999 local/generic 5000 load/store/load 5001 atomic/store 5002 atomic/atomicrmw. 5003 - Could be split into 5004 separate s_waitcnt 5005 vmcnt(0), s_waitcnt 5006 vscnt(0) and s_waitcnt 5007 lgkmcnt(0) to allow 5008 them to be 5009 independently moved 5010 according to the 5011 following rules. 5012 - s_waitcnt vmcnt(0) 5013 must happen after 5014 any preceding 5015 global/generic load/load 5016 atomic/ 5017 atomicrmw-with-return-value. 5018 - s_waitcnt vscnt(0) 5019 must happen after 5020 any preceding 5021 global/generic 5022 store/store 5023 atomic/ 5024 atomicrmw-no-return-value. 5025 - s_waitcnt lgkmcnt(0) 5026 must happen after 5027 any preceding 5028 local/generic load/store/load 5029 atomic/store atomic/atomicrmw. 5030 - Must happen before - Must happen before 5031 the following the following 5032 store. store. 5033 - Ensures that all - Ensures that all 5034 memory operations memory operations 5035 to local have have 5036 completed before completed before 5037 performing the performing the 5038 store that is being store that is being 5039 released. released. 5040 5041 2. flat_store 2. flat_store 5042 store atomic release - agent - global 1. s_waitcnt lgkmcnt(0) & 1. s_waitcnt lgkmcnt(0) & 5043 - system - generic vmcnt(0) vmcnt(0) & vscnt(0) 5044 5045 - If OpenCL, omit - If OpenCL, omit 5046 lgkmcnt(0). lgkmcnt(0). 5047 - Could be split into - Could be split into 5048 separate s_waitcnt separate s_waitcnt 5049 vmcnt(0) and vmcnt(0), s_waitcnt vscnt(0) 5050 s_waitcnt and s_waitcnt 5051 lgkmcnt(0) to allow lgkmcnt(0) to allow 5052 them to be them to be 5053 independently moved independently moved 5054 according to the according to the 5055 following rules. following rules. 5056 - s_waitcnt vmcnt(0) - s_waitcnt vmcnt(0) 5057 must happen after must happen after 5058 any preceding any preceding 5059 global/generic global/generic 5060 load/store/load load/load 5061 atomic/store atomic/ 5062 atomic/atomicrmw. atomicrmw-with-return-value. 5063 - s_waitcnt vscnt(0) 5064 must happen after 5065 any preceding 5066 global/generic 5067 store/store atomic/ 5068 atomicrmw-no-return-value. 5069 - s_waitcnt lgkmcnt(0) - s_waitcnt lgkmcnt(0) 5070 must happen after must happen after 5071 any preceding any preceding 5072 local/generic local/generic 5073 load/store/load load/store/load 5074 atomic/store atomic/store 5075 atomic/atomicrmw. atomic/atomicrmw. 5076 - Must happen before - Must happen before 5077 the following the following 5078 store. store. 5079 - Ensures that all - Ensures that all 5080 memory operations memory operations 5081 to memory have to memory have 5082 completed before completed before 5083 performing the performing the 5084 store that is being store that is being 5085 released. released. 5086 5087 2. buffer/global/ds/flat_store 2. buffer/global/ds/flat_store 5088 atomicrmw release - singlethread - global 1. buffer/global/ds/flat_atomic 1. buffer/global/ds/flat_atomic 5089 - wavefront - local 5090 - generic 5091 atomicrmw release - workgroup - global 1. s_waitcnt lgkmcnt(0) 1. s_waitcnt lgkmcnt(0) & 5092 vmcnt(0) & vscnt(0) 5093 5094 - If CU wavefront execution mode, omit vmcnt and 5095 vscnt. 5096 - If OpenCL, omit. 5097 5098 - Must happen after 5099 any preceding 5100 local/generic 5101 load/store/load 5102 atomic/store 5103 atomic/atomicrmw. 5104 - Could be split into 5105 separate s_waitcnt 5106 vmcnt(0), s_waitcnt 5107 vscnt(0) and s_waitcnt 5108 lgkmcnt(0) to allow 5109 them to be 5110 independently moved 5111 according to the 5112 following rules. 5113 - s_waitcnt vmcnt(0) 5114 must happen after 5115 any preceding 5116 global/generic load/load 5117 atomic/ 5118 atomicrmw-with-return-value. 5119 - s_waitcnt vscnt(0) 5120 must happen after 5121 any preceding 5122 global/generic 5123 store/store 5124 atomic/ 5125 atomicrmw-no-return-value. 5126 - s_waitcnt lgkmcnt(0) 5127 must happen after 5128 any preceding 5129 local/generic 5130 load/store/load 5131 atomic/store 5132 atomic/atomicrmw. 5133 - Must happen before - Must happen before 5134 the following the following 5135 atomicrmw. atomicrmw. 5136 - Ensures that all - Ensures that all 5137 memory operations memory operations 5138 to local have have 5139 completed before completed before 5140 performing the performing the 5141 atomicrmw that is atomicrmw that is 5142 being released. being released. 5143 5144 2. buffer/global/flat_atomic 2. buffer/global_atomic 5145 atomicrmw release - workgroup - local 1. waitcnt vmcnt(0) & vscnt(0) 5146 5147 - If CU wavefront execution mode, omit. 5148 - If OpenCL, omit. 5149 - Could be split into 5150 separate s_waitcnt 5151 vmcnt(0) and s_waitcnt 5152 vscnt(0) to allow 5153 them to be 5154 independently moved 5155 according to the 5156 following rules. 5157 - s_waitcnt vmcnt(0) 5158 must happen after 5159 any preceding 5160 global/generic load/load 5161 atomic/ 5162 atomicrmw-with-return-value. 5163 - s_waitcnt vscnt(0) 5164 must happen after 5165 any preceding 5166 global/generic 5167 store/store atomic/ 5168 atomicrmw-no-return-value. 5169 - Must happen before 5170 the following 5171 store. 5172 - Ensures that all 5173 global memory 5174 operations have 5175 completed before 5176 performing the 5177 store that is being 5178 released. 5179 5180 1. ds_atomic 2. ds_atomic 5181 atomicrmw release - workgroup - generic 1. s_waitcnt lgkmcnt(0) 1. s_waitcnt lgkmcnt(0) & 5182 vmcnt(0) & vscnt(0) 5183 5184 - If CU wavefront execution mode, omit vmcnt and 5185 vscnt. 5186 - If OpenCL, omit. - If OpenCL, omit 5187 waitcnt lgkmcnt(0). 5188 - Must happen after 5189 any preceding 5190 local/generic 5191 load/store/load 5192 atomic/store 5193 atomic/atomicrmw. 5194 - Could be split into 5195 separate s_waitcnt 5196 vmcnt(0), s_waitcnt 5197 vscnt(0) and s_waitcnt 5198 lgkmcnt(0) to allow 5199 them to be 5200 independently moved 5201 according to the 5202 following rules. 5203 - s_waitcnt vmcnt(0) 5204 must happen after 5205 any preceding 5206 global/generic load/load 5207 atomic/ 5208 atomicrmw-with-return-value. 5209 - s_waitcnt vscnt(0) 5210 must happen after 5211 any preceding 5212 global/generic 5213 store/store 5214 atomic/ 5215 atomicrmw-no-return-value. 5216 - s_waitcnt lgkmcnt(0) 5217 must happen after 5218 any preceding 5219 local/generic load/store/load 5220 atomic/store atomic/atomicrmw. 5221 - Must happen before - Must happen before 5222 the following the following 5223 atomicrmw. atomicrmw. 5224 - Ensures that all - Ensures that all 5225 memory operations memory operations 5226 to local have have 5227 completed before completed before 5228 performing the performing the 5229 atomicrmw that is atomicrmw that is 5230 being released. being released. 5231 5232 2. flat_atomic 2. flat_atomic 5233 atomicrmw release - agent - global 1. s_waitcnt lgkmcnt(0) & 1. s_waitcnt lkkmcnt(0) & 5234 - system - generic vmcnt(0) vmcnt(0) & vscnt(0) 5235 5236 - If OpenCL, omit - If OpenCL, omit 5237 lgkmcnt(0). lgkmcnt(0). 5238 - Could be split into - Could be split into 5239 separate s_waitcnt separate s_waitcnt 5240 vmcnt(0) and vmcnt(0), s_waitcnt 5241 s_waitcnt vscnt(0) and s_waitcnt 5242 lgkmcnt(0) to allow lgkmcnt(0) to allow 5243 them to be them to be 5244 independently moved independently moved 5245 according to the according to the 5246 following rules. following rules. 5247 - s_waitcnt vmcnt(0) - s_waitcnt vmcnt(0) 5248 must happen after must happen after 5249 any preceding any preceding 5250 global/generic global/generic 5251 load/store/load load/load atomic/ 5252 atomic/store atomicrmw-with-return-value. 5253 atomic/atomicrmw. 5254 - s_waitcnt vscnt(0) 5255 must happen after 5256 any preceding 5257 global/generic 5258 store/store atomic/ 5259 atomicrmw-no-return-value. 5260 - s_waitcnt lgkmcnt(0) - s_waitcnt lgkmcnt(0) 5261 must happen after must happen after 5262 any preceding any preceding 5263 local/generic local/generic 5264 load/store/load load/store/load 5265 atomic/store atomic/store 5266 atomic/atomicrmw. atomic/atomicrmw. 5267 - Must happen before - Must happen before 5268 the following the following 5269 atomicrmw. atomicrmw. 5270 - Ensures that all - Ensures that all 5271 memory operations memory operations 5272 to global and local to global and local 5273 have completed have completed 5274 before performing before performing 5275 the atomicrmw that the atomicrmw that 5276 is being released. is being released. 5277 5278 2. buffer/global/ds/flat_atomic 2. buffer/global/ds/flat_atomic 5279 fence release - singlethread *none* *none* *none* 5280 - wavefront 5281 fence release - workgroup *none* 1. s_waitcnt lgkmcnt(0) 1. s_waitcnt lgkmcnt(0) & 5282 vmcnt(0) & vscnt(0) 5283 5284 - If CU wavefront execution mode, omit vmcnt and 5285 vscnt. 5286 - If OpenCL and - If OpenCL and 5287 address space is address space is 5288 not generic, omit. not generic, omit 5289 lgkmcnt(0). 5290 - If OpenCL and 5291 address space is 5292 local, omit 5293 vmcnt(0) and vscnt(0). 5294 - However, since LLVM - However, since LLVM 5295 currently has no currently has no 5296 address space on address space on 5297 the fence need to the fence need to 5298 conservatively conservatively 5299 always generate. If always generate. If 5300 fence had an fence had an 5301 address space then address space then 5302 set to address set to address 5303 space of OpenCL space of OpenCL 5304 fence flag, or to fence flag, or to 5305 generic if both generic if both 5306 local and global local and global 5307 flags are flags are 5308 specified. specified. 5309 - Must happen after 5310 any preceding 5311 local/generic 5312 load/load 5313 atomic/store/store 5314 atomic/atomicrmw. 5315 - Could be split into 5316 separate s_waitcnt 5317 vmcnt(0), s_waitcnt 5318 vscnt(0) and s_waitcnt 5319 lgkmcnt(0) to allow 5320 them to be 5321 independently moved 5322 according to the 5323 following rules. 5324 - s_waitcnt vmcnt(0) 5325 must happen after 5326 any preceding 5327 global/generic 5328 load/load 5329 atomic/ 5330 atomicrmw-with-return-value. 5331 - s_waitcnt vscnt(0) 5332 must happen after 5333 any preceding 5334 global/generic 5335 store/store atomic/ 5336 atomicrmw-no-return-value. 5337 - s_waitcnt lgkmcnt(0) 5338 must happen after 5339 any preceding 5340 local/generic 5341 load/store/load 5342 atomic/store atomic/ 5343 atomicrmw. 5344 - Must happen before - Must happen before 5345 any following store any following store 5346 atomic/atomicrmw atomic/atomicrmw 5347 with an equal or with an equal or 5348 wider sync scope wider sync scope 5349 and memory ordering and memory ordering 5350 stronger than stronger than 5351 unordered (this is unordered (this is 5352 termed the termed the 5353 fence-paired-atomic). fence-paired-atomic). 5354 - Ensures that all - Ensures that all 5355 memory operations memory operations 5356 to local have have 5357 completed before completed before 5358 performing the performing the 5359 following following 5360 fence-paired-atomic. fence-paired-atomic. 5361 5362 fence release - agent *none* 1. s_waitcnt lgkmcnt(0) & 1. s_waitcnt lgkmcnt(0) & 5363 - system vmcnt(0) vmcnt(0) & vscnt(0) 5364 5365 - If OpenCL and - If OpenCL and 5366 address space is address space is 5367 not generic, omit not generic, omit 5368 lgkmcnt(0). lgkmcnt(0). 5369 - If OpenCL and - If OpenCL and 5370 address space is address space is 5371 local, omit local, omit 5372 vmcnt(0). vmcnt(0) and vscnt(0). 5373 - However, since LLVM - However, since LLVM 5374 currently has no currently has no 5375 address space on address space on 5376 the fence need to the fence need to 5377 conservatively conservatively 5378 always generate. If always generate. If 5379 fence had an fence had an 5380 address space then address space then 5381 set to address set to address 5382 space of OpenCL space of OpenCL 5383 fence flag, or to fence flag, or to 5384 generic if both generic if both 5385 local and global local and global 5386 flags are flags are 5387 specified. specified. 5388 - Could be split into - Could be split into 5389 separate s_waitcnt separate s_waitcnt 5390 vmcnt(0) and vmcnt(0), s_waitcnt 5391 s_waitcnt vscnt(0) and s_waitcnt 5392 lgkmcnt(0) to allow lgkmcnt(0) to allow 5393 them to be them to be 5394 independently moved independently moved 5395 according to the according to the 5396 following rules. following rules. 5397 - s_waitcnt vmcnt(0) - s_waitcnt vmcnt(0) 5398 must happen after must happen after 5399 any preceding any preceding 5400 global/generic global/generic 5401 load/store/load load/load atomic/ 5402 atomic/store atomicrmw-with-return-value. 5403 atomic/atomicrmw. 5404 - s_waitcnt vscnt(0) 5405 must happen after 5406 any preceding 5407 global/generic 5408 store/store atomic/ 5409 atomicrmw-no-return-value. 5410 - s_waitcnt lgkmcnt(0) - s_waitcnt lgkmcnt(0) 5411 must happen after must happen after 5412 any preceding any preceding 5413 local/generic local/generic 5414 load/store/load load/store/load 5415 atomic/store atomic/store 5416 atomic/atomicrmw. atomic/atomicrmw. 5417 - Must happen before - Must happen before 5418 any following store any following store 5419 atomic/atomicrmw atomic/atomicrmw 5420 with an equal or with an equal or 5421 wider sync scope wider sync scope 5422 and memory ordering and memory ordering 5423 stronger than stronger than 5424 unordered (this is unordered (this is 5425 termed the termed the 5426 fence-paired-atomic). fence-paired-atomic). 5427 - Ensures that all - Ensures that all 5428 memory operations memory operations 5429 have have 5430 completed before completed before 5431 performing the performing the 5432 following following 5433 fence-paired-atomic. fence-paired-atomic. 5434 5435 **Acquire-Release Atomic** 5436 ---------------------------------------------------------------------------------------------------------------------- 5437 atomicrmw acq_rel - singlethread - global 1. buffer/global/ds/flat_atomic 1. buffer/global/ds/flat_atomic 5438 - wavefront - local 5439 - generic 5440 atomicrmw acq_rel - workgroup - global 1. s_waitcnt lgkmcnt(0) 1. s_waitcnt lgkmcnt(0) & 5441 vmcnt(0) & vscnt(0) 5442 5443 - If CU wavefront execution mode, omit vmcnt and 5444 vscnt. 5445 - If OpenCL, omit. - If OpenCL, omit 5446 s_waitcnt lgkmcnt(0). 5447 - Must happen after - Must happen after 5448 any preceding any preceding 5449 local/generic local/generic 5450 load/store/load load/store/load 5451 atomic/store atomic/store 5452 atomic/atomicrmw. atomic/atomicrmw. 5453 - Could be split into 5454 separate s_waitcnt 5455 vmcnt(0), s_waitcnt 5456 vscnt(0) and s_waitcnt 5457 lgkmcnt(0) to allow 5458 them to be 5459 independently moved 5460 according to the 5461 following rules. 5462 - s_waitcnt vmcnt(0) 5463 must happen after 5464 any preceding 5465 global/generic load/load 5466 atomic/ 5467 atomicrmw-with-return-value. 5468 - s_waitcnt vscnt(0) 5469 must happen after 5470 any preceding 5471 global/generic 5472 store/store 5473 atomic/ 5474 atomicrmw-no-return-value. 5475 - s_waitcnt lgkmcnt(0) 5476 must happen after 5477 any preceding 5478 local/generic load/store/load 5479 atomic/store atomic/atomicrmw. 5480 - Must happen before - Must happen before 5481 the following the following 5482 atomicrmw. atomicrmw. 5483 - Ensures that all - Ensures that all 5484 memory operations memory operations 5485 to local have have 5486 completed before completed before 5487 performing the performing the 5488 atomicrmw that is atomicrmw that is 5489 being released. being released. 5490 5491 2. buffer/global/flat_atomic 2. buffer/global_atomic 5492 3. s_waitcnt vm/vscnt(0) 5493 5494 - If CU wavefront execution mode, omit vm/vscnt. 5495 - Use vmcnt if atomic with 5496 return and vscnt if atomic 5497 with no-return. 5498 waitcnt lgkmcnt(0). 5499 - Must happen before 5500 the following 5501 buffer_gl0_inv. 5502 - Ensures any 5503 following global 5504 data read is no 5505 older than the 5506 atomicrmw value 5507 being acquired. 5508 5509 4. buffer_gl0_inv 5510 5511 - If CU wavefront execution mode, omit. 5512 - Ensures that 5513 following 5514 loads will not see 5515 stale data. 5516 5517 atomicrmw acq_rel - workgroup - local 1. waitcnt vmcnt(0) & vscnt(0) 5518 5519 - If CU wavefront execution mode, omit. 5520 - If OpenCL, omit. 5521 - Could be split into 5522 separate s_waitcnt 5523 vmcnt(0) and s_waitcnt 5524 vscnt(0) to allow 5525 them to be 5526 independently moved 5527 according to the 5528 following rules. 5529 - s_waitcnt vmcnt(0) 5530 must happen after 5531 any preceding 5532 global/generic load/load 5533 atomic/ 5534 atomicrmw-with-return-value. 5535 - s_waitcnt vscnt(0) 5536 must happen after 5537 any preceding 5538 global/generic 5539 store/store atomic/ 5540 atomicrmw-no-return-value. 5541 - Must happen before 5542 the following 5543 store. 5544 - Ensures that all 5545 global memory 5546 operations have 5547 completed before 5548 performing the 5549 store that is being 5550 released. 5551 5552 1. ds_atomic 2. ds_atomic 5553 2. s_waitcnt lgkmcnt(0) 3. s_waitcnt lgkmcnt(0) 5554 5555 - If OpenCL, omit. - If OpenCL, omit. 5556 - Must happen before - Must happen before 5557 any following the following 5558 global/generic buffer_gl0_inv. 5559 load/load 5560 atomic/store/store 5561 atomic/atomicrmw. 5562 - Ensures any - Ensures any 5563 following global following global 5564 data read is no data read is no 5565 older than the load older than the load 5566 atomic value being atomic value being 5567 acquired. acquired. 5568 5569 4. buffer_gl0_inv 5570 5571 - If CU wavefront execution mode, omit. 5572 - If OpenCL omit. 5573 - Ensures that 5574 following 5575 loads will not see 5576 stale data. 5577 5578 atomicrmw acq_rel - workgroup - generic 1. s_waitcnt lgkmcnt(0) 1. s_waitcnt lgkmcnt(0) & 5579 vmcnt(0) & vscnt(0) 5580 5581 - If CU wavefront execution mode, omit vmcnt and 5582 vscnt. 5583 - If OpenCL, omit. - If OpenCL, omit 5584 waitcnt lgkmcnt(0). 5585 - Must happen after 5586 any preceding 5587 local/generic 5588 load/store/load 5589 atomic/store 5590 atomic/atomicrmw. 5591 - Could be split into 5592 separate s_waitcnt 5593 vmcnt(0), s_waitcnt 5594 vscnt(0) and s_waitcnt 5595 lgkmcnt(0) to allow 5596 them to be 5597 independently moved 5598 according to the 5599 following rules. 5600 - s_waitcnt vmcnt(0) 5601 must happen after 5602 any preceding 5603 global/generic load/load 5604 atomic/ 5605 atomicrmw-with-return-value. 5606 - s_waitcnt vscnt(0) 5607 must happen after 5608 any preceding 5609 global/generic 5610 store/store 5611 atomic/ 5612 atomicrmw-no-return-value. 5613 - s_waitcnt lgkmcnt(0) 5614 must happen after 5615 any preceding 5616 local/generic load/store/load 5617 atomic/store atomic/atomicrmw. 5618 - Must happen before - Must happen before 5619 the following the following 5620 atomicrmw. atomicrmw. 5621 - Ensures that all - Ensures that all 5622 memory operations memory operations 5623 to local have have 5624 completed before completed before 5625 performing the performing the 5626 atomicrmw that is atomicrmw that is 5627 being released. being released. 5628 5629 2. flat_atomic 2. flat_atomic 5630 3. s_waitcnt lgkmcnt(0) 3. s_waitcnt lgkmcnt(0) & 5631 vm/vscnt(0) 5632 5633 - If CU wavefront execution mode, omit vm/vscnt. 5634 - If OpenCL, omit. - If OpenCL, omit 5635 waitcnt lgkmcnt(0). 5636 - Must happen before - Must happen before 5637 any following the following 5638 global/generic buffer_gl0_inv. 5639 load/load 5640 atomic/store/store 5641 atomic/atomicrmw. 5642 - Ensures any - Ensures any 5643 following global following global 5644 data read is no data read is no 5645 older than the load older than the load 5646 atomic value being atomic value being 5647 acquired. acquired. 5648 5649 3. buffer_gl0_inv 5650 5651 - If CU wavefront execution mode, omit. 5652 - Ensures that 5653 following 5654 loads will not see 5655 stale data. 5656 5657 atomicrmw acq_rel - agent - global 1. s_waitcnt lgkmcnt(0) & 1. s_waitcnt lgkmcnt(0) & 5658 - system vmcnt(0) vmcnt(0) & vscnt(0) 5659 5660 - If OpenCL, omit - If OpenCL, omit 5661 lgkmcnt(0). lgkmcnt(0). 5662 - Could be split into - Could be split into 5663 separate s_waitcnt separate s_waitcnt 5664 vmcnt(0) and vmcnt(0), s_waitcnt 5665 s_waitcnt vscnt(0) and s_waitcnt 5666 lgkmcnt(0) to allow lgkmcnt(0) to allow 5667 them to be them to be 5668 independently moved independently moved 5669 according to the according to the 5670 following rules. following rules. 5671 - s_waitcnt vmcnt(0) - s_waitcnt vmcnt(0) 5672 must happen after must happen after 5673 any preceding any preceding 5674 global/generic global/generic 5675 load/store/load load/load atomic/ 5676 atomic/store atomicrmw-with-return-value. 5677 atomic/atomicrmw. 5678 - s_waitcnt vscnt(0) 5679 must happen after 5680 any preceding 5681 global/generic 5682 store/store atomic/ 5683 atomicrmw-no-return-value. 5684 - s_waitcnt lgkmcnt(0) - s_waitcnt lgkmcnt(0) 5685 must happen after must happen after 5686 any preceding any preceding 5687 local/generic local/generic 5688 load/store/load load/store/load 5689 atomic/store atomic/store 5690 atomic/atomicrmw. atomic/atomicrmw. 5691 - Must happen before - Must happen before 5692 the following the following 5693 atomicrmw. atomicrmw. 5694 - Ensures that all - Ensures that all 5695 memory operations memory operations 5696 to global have to global have 5697 completed before completed before 5698 performing the performing the 5699 atomicrmw that is atomicrmw that is 5700 being released. being released. 5701 5702 2. buffer/global/flat_atomic 2. buffer/global_atomic 5703 3. s_waitcnt vmcnt(0) 3. s_waitcnt vm/vscnt(0) 5704 5705 - Use vmcnt if atomic with 5706 return and vscnt if atomic 5707 with no-return. 5708 waitcnt lgkmcnt(0). 5709 - Must happen before - Must happen before 5710 following following 5711 buffer_wbinvl1_vol. buffer_gl*_inv. 5712 - Ensures the - Ensures the 5713 atomicrmw has atomicrmw has 5714 completed before completed before 5715 invalidating the invalidating the 5716 cache. caches. 5717 5718 4. buffer_wbinvl1_vol 4. buffer_gl0_inv; 5719 buffer_gl1_inv 5720 5721 - Must happen before - Must happen before 5722 any following any following 5723 global/generic global/generic 5724 load/load load/load 5725 atomic/atomicrmw. atomic/atomicrmw. 5726 - Ensures that - Ensures that 5727 following loads following loads 5728 will not see stale will not see stale 5729 global data. global data. 5730 5731 atomicrmw acq_rel - agent - generic 1. s_waitcnt lgkmcnt(0) & 1. s_waitcnt lgkmcnt(0) & 5732 - system vmcnt(0) vmcnt(0) & vscnt(0) 5733 5734 - If OpenCL, omit - If OpenCL, omit 5735 lgkmcnt(0). lgkmcnt(0). 5736 - Could be split into - Could be split into 5737 separate s_waitcnt separate s_waitcnt 5738 vmcnt(0) and vmcnt(0), s_waitcnt 5739 s_waitcnt vscnt(0) and s_waitcnt 5740 lgkmcnt(0) to allow lgkmcnt(0) to allow 5741 them to be them to be 5742 independently moved independently moved 5743 according to the according to the 5744 following rules. following rules. 5745 - s_waitcnt vmcnt(0) - s_waitcnt vmcnt(0) 5746 must happen after must happen after 5747 any preceding any preceding 5748 global/generic global/generic 5749 load/store/load load/load atomic 5750 atomic/store atomicrmw-with-return-value. 5751 atomic/atomicrmw. 5752 - s_waitcnt vscnt(0) 5753 must happen after 5754 any preceding 5755 global/generic 5756 store/store atomic/ 5757 atomicrmw-no-return-value. 5758 - s_waitcnt lgkmcnt(0) - s_waitcnt lgkmcnt(0) 5759 must happen after must happen after 5760 any preceding any preceding 5761 local/generic local/generic 5762 load/store/load load/store/load 5763 atomic/store atomic/store 5764 atomic/atomicrmw. atomic/atomicrmw. 5765 - Must happen before - Must happen before 5766 the following the following 5767 atomicrmw. atomicrmw. 5768 - Ensures that all - Ensures that all 5769 memory operations memory operations 5770 to global have have 5771 completed before completed before 5772 performing the performing the 5773 atomicrmw that is atomicrmw that is 5774 being released. being released. 5775 5776 2. flat_atomic 2. flat_atomic 5777 3. s_waitcnt vmcnt(0) & 3. s_waitcnt vm/vscnt(0) & 5778 lgkmcnt(0) lgkmcnt(0) 5779 5780 - If OpenCL, omit - If OpenCL, omit 5781 lgkmcnt(0). lgkmcnt(0). 5782 - Use vmcnt if atomic with 5783 return and vscnt if atomic 5784 with no-return. 5785 - Must happen before - Must happen before 5786 following following 5787 buffer_wbinvl1_vol. buffer_gl*_inv. 5788 - Ensures the - Ensures the 5789 atomicrmw has atomicrmw has 5790 completed before completed before 5791 invalidating the invalidating the 5792 cache. caches. 5793 5794 4. buffer_wbinvl1_vol 4. buffer_gl0_inv; 5795 buffer_gl1_inv 5796 5797 - Must happen before - Must happen before 5798 any following any following 5799 global/generic global/generic 5800 load/load load/load 5801 atomic/atomicrmw. atomic/atomicrmw. 5802 - Ensures that - Ensures that 5803 following loads following loads 5804 will not see stale will not see stale 5805 global data. global data. 5806 5807 fence acq_rel - singlethread *none* *none* *none* 5808 - wavefront 5809 fence acq_rel - workgroup *none* 1. s_waitcnt lgkmcnt(0) 1. s_waitcnt lgkmcnt(0) & 5810 vmcnt(0) & vscnt(0) 5811 5812 - If CU wavefront execution mode, omit vmcnt and 5813 vscnt. 5814 - If OpenCL and - If OpenCL and 5815 address space is address space is 5816 not generic, omit. not generic, omit 5817 lgkmcnt(0). 5818 - If OpenCL and 5819 address space is 5820 local, omit 5821 vmcnt(0) and vscnt(0). 5822 - However, - However, 5823 since LLVM since LLVM 5824 currently has no currently has no 5825 address space on address space on 5826 the fence need to the fence need to 5827 conservatively conservatively 5828 always generate always generate 5829 (see comment for (see comment for 5830 previous fence). previous fence). 5831 - Must happen after 5832 any preceding 5833 local/generic 5834 load/load 5835 atomic/store/store 5836 atomic/atomicrmw. 5837 - Could be split into 5838 separate s_waitcnt 5839 vmcnt(0), s_waitcnt 5840 vscnt(0) and s_waitcnt 5841 lgkmcnt(0) to allow 5842 them to be 5843 independently moved 5844 according to the 5845 following rules. 5846 - s_waitcnt vmcnt(0) 5847 must happen after 5848 any preceding 5849 global/generic 5850 load/load 5851 atomic/ 5852 atomicrmw-with-return-value. 5853 - s_waitcnt vscnt(0) 5854 must happen after 5855 any preceding 5856 global/generic 5857 store/store atomic/ 5858 atomicrmw-no-return-value. 5859 - s_waitcnt lgkmcnt(0) 5860 must happen after 5861 any preceding 5862 local/generic 5863 load/store/load 5864 atomic/store atomic/ 5865 atomicrmw. 5866 - Must happen before - Must happen before 5867 any following any following 5868 global/generic global/generic 5869 load/load load/load 5870 atomic/store/store atomic/store/store 5871 atomic/atomicrmw. atomic/atomicrmw. 5872 - Ensures that all - Ensures that all 5873 memory operations memory operations 5874 to local have have 5875 completed before completed before 5876 performing any performing any 5877 following global following global 5878 memory operations. memory operations. 5879 - Ensures that the - Ensures that the 5880 preceding preceding 5881 local/generic load local/generic load 5882 atomic/atomicrmw atomic/atomicrmw 5883 with an equal or with an equal or 5884 wider sync scope wider sync scope 5885 and memory ordering and memory ordering 5886 stronger than stronger than 5887 unordered (this is unordered (this is 5888 termed the termed the 5889 acquire-fence-paired-atomic acquire-fence-paired-atomic 5890 ) has completed ) has completed 5891 before following before following 5892 global memory global memory 5893 operations. This operations. This 5894 satisfies the satisfies the 5895 requirements of requirements of 5896 acquire. acquire. 5897 - Ensures that all - Ensures that all 5898 previous memory previous memory 5899 operations have operations have 5900 completed before a completed before a 5901 following following 5902 local/generic store local/generic store 5903 atomic/atomicrmw atomic/atomicrmw 5904 with an equal or with an equal or 5905 wider sync scope wider sync scope 5906 and memory ordering and memory ordering 5907 stronger than stronger than 5908 unordered (this is unordered (this is 5909 termed the termed the 5910 release-fence-paired-atomic release-fence-paired-atomic 5911 ). This satisfies the ). This satisfies the 5912 requirements of requirements of 5913 release. release. 5914 - Must happen before 5915 the following 5916 buffer_gl0_inv. 5917 - Ensures that the 5918 acquire-fence-paired 5919 atomic has completed 5920 before invalidating 5921 the 5922 cache. Therefore 5923 any following 5924 locations read must 5925 be no older than 5926 the value read by 5927 the 5928 acquire-fence-paired-atomic. 5929 5930 3. buffer_gl0_inv 5931 5932 - If CU wavefront execution mode, omit. 5933 - Ensures that 5934 following 5935 loads will not see 5936 stale data. 5937 5938 fence acq_rel - agent *none* 1. s_waitcnt lgkmcnt(0) & 1. s_waitcnt lgkmcnt(0) & 5939 - system vmcnt(0) vmcnt(0) & vscnt(0) 5940 5941 - If OpenCL and - If OpenCL and 5942 address space is address space is 5943 not generic, omit not generic, omit 5944 lgkmcnt(0). lgkmcnt(0). 5945 - If OpenCL and 5946 address space is 5947 local, omit 5948 vmcnt(0) and vscnt(0). 5949 - However, since LLVM - However, since LLVM 5950 currently has no currently has no 5951 address space on address space on 5952 the fence need to the fence need to 5953 conservatively conservatively 5954 always generate always generate 5955 (see comment for (see comment for 5956 previous fence). previous fence). 5957 - Could be split into - Could be split into 5958 separate s_waitcnt separate s_waitcnt 5959 vmcnt(0) and vmcnt(0), s_waitcnt 5960 s_waitcnt vscnt(0) and s_waitcnt 5961 lgkmcnt(0) to allow lgkmcnt(0) to allow 5962 them to be them to be 5963 independently moved independently moved 5964 according to the according to the 5965 following rules. following rules. 5966 - s_waitcnt vmcnt(0) - s_waitcnt vmcnt(0) 5967 must happen after must happen after 5968 any preceding any preceding 5969 global/generic global/generic 5970 load/store/load load/load 5971 atomic/store atomic/ 5972 atomic/atomicrmw. atomicrmw-with-return-value. 5973 - s_waitcnt vscnt(0) 5974 must happen after 5975 any preceding 5976 global/generic 5977 store/store atomic/ 5978 atomicrmw-no-return-value. 5979 - s_waitcnt lgkmcnt(0) - s_waitcnt lgkmcnt(0) 5980 must happen after must happen after 5981 any preceding any preceding 5982 local/generic local/generic 5983 load/store/load load/store/load 5984 atomic/store atomic/store 5985 atomic/atomicrmw. atomic/atomicrmw. 5986 - Must happen before - Must happen before 5987 the following the following 5988 buffer_wbinvl1_vol. buffer_gl*_inv. 5989 - Ensures that the - Ensures that the 5990 preceding preceding 5991 global/local/generic global/local/generic 5992 load load 5993 atomic/atomicrmw atomic/atomicrmw 5994 with an equal or with an equal or 5995 wider sync scope wider sync scope 5996 and memory ordering and memory ordering 5997 stronger than stronger than 5998 unordered (this is unordered (this is 5999 termed the termed the 6000 acquire-fence-paired-atomic acquire-fence-paired-atomic 6001 ) has completed ) has completed 6002 before invalidating before invalidating 6003 the cache. This the caches. This 6004 satisfies the satisfies the 6005 requirements of requirements of 6006 acquire. acquire. 6007 - Ensures that all - Ensures that all 6008 previous memory previous memory 6009 operations have operations have 6010 completed before a completed before a 6011 following following 6012 global/local/generic global/local/generic 6013 store store 6014 atomic/atomicrmw atomic/atomicrmw 6015 with an equal or with an equal or 6016 wider sync scope wider sync scope 6017 and memory ordering and memory ordering 6018 stronger than stronger than 6019 unordered (this is unordered (this is 6020 termed the termed the 6021 release-fence-paired-atomic release-fence-paired-atomic 6022 ). This satisfies the ). This satisfies the 6023 requirements of requirements of 6024 release. release. 6025 6026 2. buffer_wbinvl1_vol 2. buffer_gl0_inv; 6027 buffer_gl1_inv 6028 6029 - Must happen before - Must happen before 6030 any following any following 6031 global/generic global/generic 6032 load/load load/load 6033 atomic/store/store atomic/store/store 6034 atomic/atomicrmw. atomic/atomicrmw. 6035 - Ensures that - Ensures that 6036 following loads following loads 6037 will not see stale will not see stale 6038 global data. This global data. This 6039 satisfies the satisfies the 6040 requirements of requirements of 6041 acquire. acquire. 6042 6043 **Sequential Consistent Atomic** 6044 ---------------------------------------------------------------------------------------------------------------------- 6045 load atomic seq_cst - singlethread - global *Same as corresponding *Same as corresponding 6046 - wavefront - local load atomic acquire, load atomic acquire, 6047 - generic except must generated except must generated 6048 all instructions even all instructions even 6049 for OpenCL.* for OpenCL.* 6050 load atomic seq_cst - workgroup - global 1. s_waitcnt lgkmcnt(0) 1. s_waitcnt lgkmcnt(0) & 6051 - generic vmcnt(0) & vscnt(0) 6052 6053 - If CU wavefront execution mode, omit vmcnt and 6054 vscnt. 6055 - Could be split into 6056 separate s_waitcnt 6057 vmcnt(0), s_waitcnt 6058 vscnt(0) and s_waitcnt 6059 lgkmcnt(0) to allow 6060 them to be 6061 independently moved 6062 according to the 6063 following rules. 6064 - Must - waitcnt lgkmcnt(0) must 6065 happen after happen after 6066 preceding preceding 6067 global/generic load local load 6068 atomic/store atomic/store 6069 atomic/atomicrmw atomic/atomicrmw 6070 with memory with memory 6071 ordering of seq_cst ordering of seq_cst 6072 and with equal or and with equal or 6073 wider sync scope. wider sync scope. 6074 (Note that seq_cst (Note that seq_cst 6075 fences have their fences have their 6076 own s_waitcnt own s_waitcnt 6077 lgkmcnt(0) and so do lgkmcnt(0) and so do 6078 not need to be not need to be 6079 considered.) considered.) 6080 - waitcnt vmcnt(0) 6081 Must happen after 6082 preceding 6083 global/generic load 6084 atomic/ 6085 atomicrmw-with-return-value 6086 with memory 6087 ordering of seq_cst 6088 and with equal or 6089 wider sync scope. 6090 (Note that seq_cst 6091 fences have their 6092 own s_waitcnt 6093 vmcnt(0) and so do 6094 not need to be 6095 considered.) 6096 - waitcnt vscnt(0) 6097 Must happen after 6098 preceding 6099 global/generic store 6100 atomic/ 6101 atomicrmw-no-return-value 6102 with memory 6103 ordering of seq_cst 6104 and with equal or 6105 wider sync scope. 6106 (Note that seq_cst 6107 fences have their 6108 own s_waitcnt 6109 vscnt(0) and so do 6110 not need to be 6111 considered.) 6112 - Ensures any - Ensures any 6113 preceding preceding 6114 sequential sequential 6115 consistent local consistent global/local 6116 memory instructions memory instructions 6117 have completed have completed 6118 before executing before executing 6119 this sequentially this sequentially 6120 consistent consistent 6121 instruction. This instruction. This 6122 prevents reordering prevents reordering 6123 a seq_cst store a seq_cst store 6124 followed by a followed by a 6125 seq_cst load. (Note seq_cst load. (Note 6126 that seq_cst is that seq_cst is 6127 stronger than stronger than 6128 acquire/release as acquire/release as 6129 the reordering of the reordering of 6130 load acquire load acquire 6131 followed by a store followed by a store 6132 release is release is 6133 prevented by the prevented by the 6134 waitcnt of waitcnt of 6135 the release, but the release, but 6136 there is nothing there is nothing 6137 preventing a store preventing a store 6138 release followed by release followed by 6139 load acquire from load acquire from 6140 competing out of competing out of 6141 order.) order.) 6142 6143 2. *Following 2. *Following 6144 instructions same as instructions same as 6145 corresponding load corresponding load 6146 atomic acquire, atomic acquire, 6147 except must generated except must generated 6148 all instructions even all instructions even 6149 for OpenCL.* for OpenCL.* 6150 load atomic seq_cst - workgroup - local *Same as corresponding 6151 load atomic acquire, 6152 except must generated 6153 all instructions even 6154 for OpenCL.* 6155 6156 1. s_waitcnt vmcnt(0) & vscnt(0) 6157 6158 - If CU wavefront execution mode, omit. 6159 - Could be split into 6160 separate s_waitcnt 6161 vmcnt(0) and s_waitcnt 6162 vscnt(0) to allow 6163 them to be 6164 independently moved 6165 according to the 6166 following rules. 6167 - waitcnt vmcnt(0) 6168 Must happen after 6169 preceding 6170 global/generic load 6171 atomic/ 6172 atomicrmw-with-return-value 6173 with memory 6174 ordering of seq_cst 6175 and with equal or 6176 wider sync scope. 6177 (Note that seq_cst 6178 fences have their 6179 own s_waitcnt 6180 vmcnt(0) and so do 6181 not need to be 6182 considered.) 6183 - waitcnt vscnt(0) 6184 Must happen after 6185 preceding 6186 global/generic store 6187 atomic/ 6188 atomicrmw-no-return-value 6189 with memory 6190 ordering of seq_cst 6191 and with equal or 6192 wider sync scope. 6193 (Note that seq_cst 6194 fences have their 6195 own s_waitcnt 6196 vscnt(0) and so do 6197 not need to be 6198 considered.) 6199 - Ensures any 6200 preceding 6201 sequential 6202 consistent global 6203 memory instructions 6204 have completed 6205 before executing 6206 this sequentially 6207 consistent 6208 instruction. This 6209 prevents reordering 6210 a seq_cst store 6211 followed by a 6212 seq_cst load. (Note 6213 that seq_cst is 6214 stronger than 6215 acquire/release as 6216 the reordering of 6217 load acquire 6218 followed by a store 6219 release is 6220 prevented by the 6221 waitcnt of 6222 the release, but 6223 there is nothing 6224 preventing a store 6225 release followed by 6226 load acquire from 6227 competing out of 6228 order.) 6229 6230 2. *Following 6231 instructions same as 6232 corresponding load 6233 atomic acquire, 6234 except must generated 6235 all instructions even 6236 for OpenCL.* 6237 6238 load atomic seq_cst - agent - global 1. s_waitcnt lgkmcnt(0) & 1. s_waitcnt lgkmcnt(0) & 6239 - system - generic vmcnt(0) vmcnt(0) & vscnt(0) 6240 6241 - Could be split into - Could be split into 6242 separate s_waitcnt separate s_waitcnt 6243 vmcnt(0) vmcnt(0), s_waitcnt 6244 and s_waitcnt vscnt(0) and s_waitcnt 6245 lgkmcnt(0) to allow lgkmcnt(0) to allow 6246 them to be them to be 6247 independently moved independently moved 6248 according to the according to the 6249 following rules. following rules. 6250 - waitcnt lgkmcnt(0) - waitcnt lgkmcnt(0) 6251 must happen after must happen after 6252 preceding preceding 6253 global/generic load local load 6254 atomic/store atomic/store 6255 atomic/atomicrmw atomic/atomicrmw 6256 with memory with memory 6257 ordering of seq_cst ordering of seq_cst 6258 and with equal or and with equal or 6259 wider sync scope. wider sync scope. 6260 (Note that seq_cst (Note that seq_cst 6261 fences have their fences have their 6262 own s_waitcnt own s_waitcnt 6263 lgkmcnt(0) and so do lgkmcnt(0) and so do 6264 not need to be not need to be 6265 considered.) considered.) 6266 - waitcnt vmcnt(0) - waitcnt vmcnt(0) 6267 must happen after must happen after 6268 preceding preceding 6269 global/generic load global/generic load 6270 atomic/store atomic/ 6271 atomic/atomicrmw atomicrmw-with-return-value 6272 with memory with memory 6273 ordering of seq_cst ordering of seq_cst 6274 and with equal or and with equal or 6275 wider sync scope. wider sync scope. 6276 (Note that seq_cst (Note that seq_cst 6277 fences have their fences have their 6278 own s_waitcnt own s_waitcnt 6279 vmcnt(0) and so do vmcnt(0) and so do 6280 not need to be not need to be 6281 considered.) considered.) 6282 - waitcnt vscnt(0) 6283 Must happen after 6284 preceding 6285 global/generic store 6286 atomic/ 6287 atomicrmw-no-return-value 6288 with memory 6289 ordering of seq_cst 6290 and with equal or 6291 wider sync scope. 6292 (Note that seq_cst 6293 fences have their 6294 own s_waitcnt 6295 vscnt(0) and so do 6296 not need to be 6297 considered.) 6298 - Ensures any - Ensures any 6299 preceding preceding 6300 sequential sequential 6301 consistent global consistent global 6302 memory instructions memory instructions 6303 have completed have completed 6304 before executing before executing 6305 this sequentially this sequentially 6306 consistent consistent 6307 instruction. This instruction. This 6308 prevents reordering prevents reordering 6309 a seq_cst store a seq_cst store 6310 followed by a followed by a 6311 seq_cst load. (Note seq_cst load. (Note 6312 that seq_cst is that seq_cst is 6313 stronger than stronger than 6314 acquire/release as acquire/release as 6315 the reordering of the reordering of 6316 load acquire load acquire 6317 followed by a store followed by a store 6318 release is release is 6319 prevented by the prevented by the 6320 waitcnt of waitcnt of 6321 the release, but the release, but 6322 there is nothing there is nothing 6323 preventing a store preventing a store 6324 release followed by release followed by 6325 load acquire from load acquire from 6326 competing out of competing out of 6327 order.) order.) 6328 6329 2. *Following 2. *Following 6330 instructions same as instructions same as 6331 corresponding load corresponding load 6332 atomic acquire, atomic acquire, 6333 except must generated except must generated 6334 all instructions even all instructions even 6335 for OpenCL.* for OpenCL.* 6336 store atomic seq_cst - singlethread - global *Same as corresponding *Same as corresponding 6337 - wavefront - local store atomic release, store atomic release, 6338 - workgroup - generic except must generated except must generated 6339 all instructions even all instructions even 6340 for OpenCL.* for OpenCL.* 6341 store atomic seq_cst - agent - global *Same as corresponding *Same as corresponding 6342 - system - generic store atomic release, store atomic release, 6343 except must generated except must generated 6344 all instructions even all instructions even 6345 for OpenCL.* for OpenCL.* 6346 atomicrmw seq_cst - singlethread - global *Same as corresponding *Same as corresponding 6347 - wavefront - local atomicrmw acq_rel, atomicrmw acq_rel, 6348 - workgroup - generic except must generated except must generated 6349 all instructions even all instructions even 6350 for OpenCL.* for OpenCL.* 6351 atomicrmw seq_cst - agent - global *Same as corresponding *Same as corresponding 6352 - system - generic atomicrmw acq_rel, atomicrmw acq_rel, 6353 except must generated except must generated 6354 all instructions even all instructions even 6355 for OpenCL.* for OpenCL.* 6356 fence seq_cst - singlethread *none* *Same as corresponding *Same as corresponding 6357 - wavefront fence acq_rel, fence acq_rel, 6358 - workgroup except must generated except must generated 6359 - agent all instructions even all instructions even 6360 - system for OpenCL.* for OpenCL.* 6361 ============ ============ ============== ========== =============================== ================================== 6362 6363The memory order also adds the single thread optimization constrains defined in 6364table 6365:ref:`amdgpu-amdhsa-memory-model-single-thread-optimization-constraints-gfx6-gfx10-table`. 6366 6367 .. table:: AMDHSA Memory Model Single Thread Optimization Constraints GFX6-GFX10 6368 :name: amdgpu-amdhsa-memory-model-single-thread-optimization-constraints-gfx6-gfx10-table 6369 6370 ============ ============================================================== 6371 LLVM Memory Optimization Constraints 6372 Ordering 6373 ============ ============================================================== 6374 unordered *none* 6375 monotonic *none* 6376 acquire - If a load atomic/atomicrmw then no following load/load 6377 atomic/store/ store atomic/atomicrmw/fence instruction can 6378 be moved before the acquire. 6379 - If a fence then same as load atomic, plus no preceding 6380 associated fence-paired-atomic can be moved after the fence. 6381 release - If a store atomic/atomicrmw then no preceding load/load 6382 atomic/store/ store atomic/atomicrmw/fence instruction can 6383 be moved after the release. 6384 - If a fence then same as store atomic, plus no following 6385 associated fence-paired-atomic can be moved before the 6386 fence. 6387 acq_rel Same constraints as both acquire and release. 6388 seq_cst - If a load atomic then same constraints as acquire, plus no 6389 preceding sequentially consistent load atomic/store 6390 atomic/atomicrmw/fence instruction can be moved after the 6391 seq_cst. 6392 - If a store atomic then the same constraints as release, plus 6393 no following sequentially consistent load atomic/store 6394 atomic/atomicrmw/fence instruction can be moved before the 6395 seq_cst. 6396 - If an atomicrmw/fence then same constraints as acq_rel. 6397 ============ ============================================================== 6398 6399Trap Handler ABI 6400~~~~~~~~~~~~~~~~ 6401 6402For code objects generated by AMDGPU backend for HSA [HSA]_ compatible runtimes 6403(such as ROCm [AMD-ROCm]_), the runtime installs a trap handler that supports 6404the ``s_trap`` instruction with the following usage: 6405 6406 .. table:: AMDGPU Trap Handler for AMDHSA OS 6407 :name: amdgpu-trap-handler-for-amdhsa-os-table 6408 6409 =================== =============== =============== ======================= 6410 Usage Code Sequence Trap Handler Description 6411 Inputs 6412 =================== =============== =============== ======================= 6413 reserved ``s_trap 0x00`` Reserved by hardware. 6414 ``debugtrap(arg)`` ``s_trap 0x01`` ``SGPR0-1``: Reserved for HSA 6415 ``queue_ptr`` ``debugtrap`` 6416 ``VGPR0``: intrinsic (not 6417 ``arg`` implemented). 6418 ``llvm.trap`` ``s_trap 0x02`` ``SGPR0-1``: Causes dispatch to be 6419 ``queue_ptr`` terminated and its 6420 associated queue put 6421 into the error state. 6422 ``llvm.debugtrap`` ``s_trap 0x03`` - If debugger not 6423 installed then 6424 behaves as a 6425 no-operation. The 6426 trap handler is 6427 entered and 6428 immediately returns 6429 to continue 6430 execution of the 6431 wavefront. 6432 - If the debugger is 6433 installed, causes 6434 the debug trap to be 6435 reported by the 6436 debugger and the 6437 wavefront is put in 6438 the halt state until 6439 resumed by the 6440 debugger. 6441 reserved ``s_trap 0x04`` Reserved. 6442 reserved ``s_trap 0x05`` Reserved. 6443 reserved ``s_trap 0x06`` Reserved. 6444 debugger breakpoint ``s_trap 0x07`` Reserved for debugger 6445 breakpoints. 6446 reserved ``s_trap 0x08`` Reserved. 6447 reserved ``s_trap 0xfe`` Reserved. 6448 reserved ``s_trap 0xff`` Reserved. 6449 =================== =============== =============== ======================= 6450 6451.. _amdgpu-amdhsa-function-call-convention: 6452 6453Call Convention 6454~~~~~~~~~~~~~~~ 6455 6456.. note:: 6457 6458 This section is currently incomplete and has inakkuracies. It is WIP that will 6459 be updated as information is determined. 6460 6461See :ref:`amdgpu-dwarf-address-space-identifier` for information on swizzled 6462addresses. Unswizzled addresses are normal linear addresses. 6463 6464.. _amdgpu-amdhsa-function-call-convention-kernel-functions: 6465 6466Kernel Functions 6467++++++++++++++++ 6468 6469This section describes the call convention ABI for the outer kernel function. 6470 6471See :ref:`amdgpu-amdhsa-initial-kernel-execution-state` for the kernel call 6472convention. 6473 6474The following is not part of the AMDGPU kernel calling convention but describes 6475how the AMDGPU implements function calls: 6476 64771. Clang decides the kernarg layout to match the *HSA Programmer's Language 6478 Reference* [HSA]_. 6479 6480 - All structs are passed directly. 6481 - Lambda values are passed *TBA*. 6482 6483 .. TODO:: 6484 6485 - Does this really follow HSA rules? Or are structs >16 bytes passed 6486 by-value struct? 6487 - What is ABI for lambda values? 6488 64894. The kernel performs certain setup in its prolog, as described in 6490 :ref:`amdgpu-amdhsa-kernel-prolog`. 6491 6492.. _amdgpu-amdhsa-function-call-convention-non-kernel-functions: 6493 6494Non-Kernel Functions 6495++++++++++++++++++++ 6496 6497This section describes the call convention ABI for functions other than the 6498outer kernel function. 6499 6500If a kernel has function calls then scratch is always allocated and used for 6501the call stack which grows from low address to high address using the swizzled 6502scratch address space. 6503 6504On entry to a function: 6505 65061. SGPR0-3 contain a V# with the following properties (see 6507 :ref:`amdgpu-amdhsa-kernel-prolog-private-segment-buffer`): 6508 6509 * Base address pointing to the beginning of the wavefront scratch backing 6510 memory. 6511 * Swizzled with dword element size and stride of wavefront size elements. 6512 65132. The FLAT_SCRATCH register pair is setup. See 6514 :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`. 65153. GFX6-8: M0 register set to the size of LDS in bytes. See 6516 :ref:`amdgpu-amdhsa-kernel-prolog-m0`. 65174. The EXEC register is set to the lanes active on entry to the function. 65185. MODE register: *TBD* 65196. VGPR0-31 and SGPR4-29 are used to pass function input arguments as described 6520 below. 65217. SGPR30-31 return address (RA). The code address that the function must 6522 return to when it completes. The value is undefined if the function is *no 6523 return*. 65248. SGPR32 is used for the stack pointer (SP). It is an unswizzled scratch 6525 offset relative to the beginning of the wavefront scratch backing memory. 6526 6527 The unswizzled SP can be used with buffer instructions as an unswizzled SGPR 6528 offset with the scratch V# in SGPR0-3 to access the stack in a swizzled 6529 manner. 6530 6531 The unswizzled SP value can be converted into the swizzled SP value by: 6532 6533 | swizzled SP = unswizzled SP / wavefront size 6534 6535 This may be used to obtain the private address space address of stack 6536 objects and to convert this address to a flat address by adding the flat 6537 scratch aperture base address. 6538 6539 The swizzled SP value is always 4 bytes aligned for the ``r600`` 6540 architecture and 16 byte aligned for the ``amdgcn`` architecture. 6541 6542 .. note:: 6543 6544 The ``amdgcn`` value is selected to avoid dynamic stack alignment for the 6545 OpenCL language which has the largest base type defined as 16 bytes. 6546 6547 On entry, the swizzled SP value is the address of the first function 6548 argument passed on the stack. Other stack passed arguments are positive 6549 offsets from the entry swizzled SP value. 6550 6551 The function may use positive offsets beyond the last stack passed argument 6552 for stack allocated local variables and register spill slots. If necessary, 6553 the function may align these to greater alignment than 16 bytes. After these 6554 the function may dynamically allocate space for such things as runtime sized 6555 ``alloca`` local allocations. 6556 6557 If the function calls another function, it will place any stack allocated 6558 arguments after the last local allocation and adjust SGPR32 to the address 6559 after the last local allocation. 6560 65619. All other registers are unspecified. 656210. Any necessary ``waitcnt`` has been performed to ensure memory is available 6563 to the function. 6564 6565On exit from a function: 6566 65671. VGPR0-31 and SGPR4-29 are used to pass function result arguments as 6568 described below. Any registers used are considered clobbered registers. 65692. The following registers are preserved and have the same value as on entry: 6570 6571 * FLAT_SCRATCH 6572 * EXEC 6573 * GFX6-8: M0 6574 * All SGPR registers except the clobbered registers of SGPR4-31. 6575 * VGPR40-47 6576 VGPR56-63 6577 VGPR72-79 6578 VGPR88-95 6579 VGPR104-111 6580 VGPR120-127 6581 VGPR136-143 6582 VGPR152-159 6583 VGPR168-175 6584 VGPR184-191 6585 VGPR200-207 6586 VGPR216-223 6587 VGPR232-239 6588 VGPR248-255 6589 6590 *Except the argument registers, the VGPR cloberred and the preserved 6591 registers are intermixed at regular intervals in order to 6592 get a better occupancy.* 6593 6594 For the AMDGPU backend, an inter-procedural register allocation (IPRA) 6595 optimization may mark some of clobbered SGPR and VGPR registers as 6596 preserved if it can be determined that the called function does not change 6597 their value. 6598 65992. The PC is set to the RA provided on entry. 66003. MODE register: *TBD*. 66014. All other registers are clobbered. 66025. Any necessary ``waitcnt`` has been performed to ensure memory accessed by 6603 function is available to the caller. 6604 6605.. TODO:: 6606 6607 - On gfx908 are all ACC registers clobbered? 6608 6609 - How are function results returned? The address of structured types is passed 6610 by reference, but what about other types? 6611 6612The function input arguments are made up of the formal arguments explicitly 6613declared by the source language function plus the implicit input arguments used 6614by the implementation. 6615 6616The source language input arguments are: 6617 66181. Any source language implicit ``this`` or ``self`` argument comes first as a 6619 pointer type. 66202. Followed by the function formal arguments in left to right source order. 6621 6622The source language result arguments are: 6623 66241. The function result argument. 6625 6626The source language input or result struct type arguments that are less than or 6627equal to 16 bytes, are decomposed recursively into their base type fields, and 6628each field is passed as if a separate argument. For input arguments, if the 6629called function requires the struct to be in memory, for example because its 6630address is taken, then the function body is responsible for allocating a stack 6631location and copying the field arguments into it. Clang terms this *direct 6632struct*. 6633 6634The source language input struct type arguments that are greater than 16 bytes, 6635are passed by reference. The caller is responsible for allocating a stack 6636location to make a copy of the struct value and pass the address as the input 6637argument. The called function is responsible to perform the dereference when 6638accessing the input argument. Clang terms this *by-value struct*. 6639 6640A source language result struct type argument that is greater than 16 bytes, is 6641returned by reference. The caller is responsible for allocating a stack location 6642to hold the result value and passes the address as the last input argument 6643(before the implicit input arguments). In this case there are no result 6644arguments. The called function is responsible to perform the dereference when 6645storing the result value. Clang terms this *structured return (sret)*. 6646 6647*TODO: correct the ``sret`` definition.* 6648 6649.. TODO:: 6650 6651 Is this definition correct? Or is ``sret`` only used if passing in registers, and 6652 pass as non-decomposed struct as stack argument? Or something else? Is the 6653 memory location in the caller stack frame, or a stack memory argument and so 6654 no address is passed as the caller can directly write to the argument stack 6655 location? But then the stack location is still live after return. If an 6656 argument stack location is it the first stack argument or the last one? 6657 6658Lambda argument types are treated as struct types with an implementation defined 6659set of fields. 6660 6661.. TODO:: 6662 6663 Need to specify the ABI for lambda types for AMDGPU. 6664 6665For AMDGPU backend all source language arguments (including the decomposed 6666struct type arguments) are passed in VGPRs unless marked ``inreg`` in which case 6667they are passed in SGPRs. 6668 6669The AMDGPU backend walks the function call graph from the leaves to determine 6670which implicit input arguments are used, propagating to each caller of the 6671function. The used implicit arguments are appended to the function arguments 6672after the source language arguments in the following order: 6673 6674.. TODO:: 6675 6676 Is recursion or external functions supported? 6677 66781. Work-Item ID (1 VGPR) 6679 6680 The X, Y and Z work-item ID are packed into a single VGRP with the following 6681 layout. Only fields actually used by the function are set. The other bits 6682 are undefined. 6683 6684 The values come from the initial kernel execution state. See 6685 :ref:`amdgpu-amdhsa-vgpr-register-set-up-order-table`. 6686 6687 .. table:: Work-item implicit argument layout 6688 :name: amdgpu-amdhsa-workitem-implicit-argument-layout-table 6689 6690 ======= ======= ============== 6691 Bits Size Field Name 6692 ======= ======= ============== 6693 9:0 10 bits X Work-Item ID 6694 19:10 10 bits Y Work-Item ID 6695 29:20 10 bits Z Work-Item ID 6696 31:30 2 bits Unused 6697 ======= ======= ============== 6698 66992. Dispatch Ptr (2 SGPRs) 6700 6701 The value comes from the initial kernel execution state. See 6702 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 6703 67043. Queue Ptr (2 SGPRs) 6705 6706 The value comes from the initial kernel execution state. See 6707 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 6708 67094. Kernarg Segment Ptr (2 SGPRs) 6710 6711 The value comes from the initial kernel execution state. See 6712 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 6713 67145. Dispatch id (2 SGPRs) 6715 6716 The value comes from the initial kernel execution state. See 6717 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 6718 67196. Work-Group ID X (1 SGPR) 6720 6721 The value comes from the initial kernel execution state. See 6722 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 6723 67247. Work-Group ID Y (1 SGPR) 6725 6726 The value comes from the initial kernel execution state. See 6727 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 6728 67298. Work-Group ID Z (1 SGPR) 6730 6731 The value comes from the initial kernel execution state. See 6732 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 6733 67349. Implicit Argument Ptr (2 SGPRs) 6735 6736 The value is computed by adding an offset to Kernarg Segment Ptr to get the 6737 global address space pointer to the first kernarg implicit argument. 6738 6739The input and result arguments are assigned in order in the following manner: 6740 6741.. note:: 6742 6743 There are likely some errors and omissions in the following description that 6744 need correction. 6745 6746 .. TODO:: 6747 6748 Check the clang source code to decipher how function arguments and return 6749 results are handled. Also see the AMDGPU specific values used. 6750 6751* VGPR arguments are assigned to consecutive VGPRs starting at VGPR0 up to 6752 VGPR31. 6753 6754 If there are more arguments than will fit in these registers, the remaining 6755 arguments are allocated on the stack in order on naturally aligned 6756 addresses. 6757 6758 .. TODO:: 6759 6760 How are overly aligned structures allocated on the stack? 6761 6762* SGPR arguments are assigned to consecutive SGPRs starting at SGPR0 up to 6763 SGPR29. 6764 6765 If there are more arguments than will fit in these registers, the remaining 6766 arguments are allocated on the stack in order on naturally aligned 6767 addresses. 6768 6769Note that decomposed struct type arguments may have some fields passed in 6770registers and some in memory. 6771 6772.. TODO:: 6773 6774 So, a struct which can pass some fields as decomposed register arguments, will 6775 pass the rest as decomposed stack elements? But an argument that will not start 6776 in registers will not be decomposed and will be passed as a non-decomposed 6777 stack value? 6778 6779The following is not part of the AMDGPU function calling convention but 6780describes how the AMDGPU implements function calls: 6781 67821. SGPR33 is used as a frame pointer (FP) if necessary. Like the SP it is an 6783 unswizzled scratch address. It is only needed if runtime sized ``alloca`` 6784 are used, or for the reasons defined in ``SIFrameLowering``. 67852. Runtime stack alignment is supported. SGPR34 is used as a base pointer (BP) 6786 to access the incoming stack arguments in the function. The BP is needed 6787 only when the function requires the runtime stack alignment. 6788 67893. Allocating SGPR arguments on the stack are not supported. 6790 67914. No CFI is currently generated. See 6792 :ref:`amdgpu-dwarf-call-frame-information`. 6793 6794 .. note:: 6795 6796 CFI will be generated that defines the CFA as the unswizzled address 6797 relative to the wave scratch base in the unswizzled private address space 6798 of the lowest address stack allocated local variable. 6799 6800 ``DW_AT_frame_base`` will be defined as the swizzled address in the 6801 swizzled private address space by dividing the CFA by the wavefront size 6802 (since CFA is always at least dword aligned which matches the scratch 6803 swizzle element size). 6804 6805 If no dynamic stack alignment was performed, the stack allocated arguments 6806 are accessed as negative offsets relative to ``DW_AT_frame_base``, and the 6807 local variables and register spill slots are accessed as positive offsets 6808 relative to ``DW_AT_frame_base``. 6809 68105. Function argument passing is implemented by copying the input physical 6811 registers to virtual registers on entry. The register allocator can spill if 6812 necessary. These are copied back to physical registers at call sites. The 6813 net effect is that each function call can have these values in entirely 6814 distinct locations. The IPRA can help avoid shuffling argument registers. 68156. Call sites are implemented by setting up the arguments at positive offsets 6816 from SP. Then SP is incremented to account for the known frame size before 6817 the call and decremented after the call. 6818 6819 .. note:: 6820 6821 The CFI will reflect the changed calculation needed to compute the CFA 6822 from SP. 6823 68247. 4 byte spill slots are used in the stack frame. One slot is allocated for an 6825 emergency spill slot. Buffer instructions are used for stack accesses and 6826 not the ``flat_scratch`` instruction. 6827 6828 .. TODO:: 6829 6830 Explain when the emergency spill slot is used. 6831 6832.. TODO:: 6833 6834 Possible broken issues: 6835 6836 - Stack arguments must be aligned to required alignment. 6837 - Stack is aligned to max(16, max formal argument alignment) 6838 - Direct argument < 64 bits should check register budget. 6839 - Register budget calculation should respect ``inreg`` for SGPR. 6840 - SGPR overflow is not handled. 6841 - struct with 1 member unpeeling is not checking size of member. 6842 - ``sret`` is after ``this`` pointer. 6843 - Caller is not implementing stack realignment: need an extra pointer. 6844 - Should say AMDGPU passes FP rather than SP. 6845 - Should CFI define CFA as address of locals or arguments. Difference is 6846 apparent when have implemented dynamic alignment. 6847 - If ``SCRATCH`` instruction could allow negative offsets, then can make FP be 6848 highest address of stack frame and use negative offset for locals. Would 6849 allow SP to be the same as FP and could support signal-handler-like as now 6850 have a real SP for the top of the stack. 6851 - How is ``sret`` passed on the stack? In argument stack area? Can it overlay 6852 arguments? 6853 6854AMDPAL 6855------ 6856 6857This section provides code conventions used when the target triple OS is 6858``amdpal`` (see :ref:`amdgpu-target-triples`) for passing runtime parameters 6859from the application/runtime to each invocation of a hardware shader. These 6860parameters include both generic, application-controlled parameters called 6861*user data* as well as system-generated parameters that are a product of the 6862draw or dispatch execution. 6863 6864User Data 6865~~~~~~~~~ 6866 6867Each hardware stage has a set of 32-bit *user data registers* which can be 6868written from a command buffer and then loaded into SGPRs when waves are launched 6869via a subsequent dispatch or draw operation. This is the way most arguments are 6870passed from the application/runtime to a hardware shader. 6871 6872Compute User Data 6873~~~~~~~~~~~~~~~~~ 6874 6875Compute shader user data mappings are simpler than graphics shaders and have a 6876fixed mapping. 6877 6878Note that there are always 10 available *user data entries* in registers - 6879entries beyond that limit must be fetched from memory (via the spill table 6880pointer) by the shader. 6881 6882 .. table:: PAL Compute Shader User Data Registers 6883 :name: pal-compute-user-data-registers 6884 6885 ============= ================================ 6886 User Register Description 6887 ============= ================================ 6888 0 Global Internal Table (32-bit pointer) 6889 1 Per-Shader Internal Table (32-bit pointer) 6890 2 - 11 Application-Controlled User Data (10 32-bit values) 6891 12 Spill Table (32-bit pointer) 6892 13 - 14 Thread Group Count (64-bit pointer) 6893 15 GDS Range 6894 ============= ================================ 6895 6896Graphics User Data 6897~~~~~~~~~~~~~~~~~~ 6898 6899Graphics pipelines support a much more flexible user data mapping: 6900 6901 .. table:: PAL Graphics Shader User Data Registers 6902 :name: pal-graphics-user-data-registers 6903 6904 ============= ================================ 6905 User Register Description 6906 ============= ================================ 6907 0 Global Internal Table (32-bit pointer) 6908 + Per-Shader Internal Table (32-bit pointer) 6909 + 1-15 Application Controlled User Data 6910 (1-15 Contiguous 32-bit Values in Registers) 6911 + Spill Table (32-bit pointer) 6912 + Draw Index (First Stage Only) 6913 + Vertex Offset (First Stage Only) 6914 + Instance Offset (First Stage Only) 6915 ============= ================================ 6916 6917 The placement of the global internal table remains fixed in the first *user 6918 data SGPR register*. Otherwise all parameters are optional, and can be mapped 6919 to any desired *user data SGPR register*, with the following restrictions: 6920 6921 * Draw Index, Vertex Offset, and Instance Offset can only be used by the first 6922 active hardware stage in a graphics pipeline (i.e. where the API vertex 6923 shader runs). 6924 6925 * Application-controlled user data must be mapped into a contiguous range of 6926 user data registers. 6927 6928 * The application-controlled user data range supports compaction remapping, so 6929 only *entries* that are actually consumed by the shader must be assigned to 6930 corresponding *registers*. Note that in order to support an efficient runtime 6931 implementation, the remapping must pack *registers* in the same order as 6932 *entries*, with unused *entries* removed. 6933 6934.. _pal_global_internal_table: 6935 6936Global Internal Table 6937~~~~~~~~~~~~~~~~~~~~~ 6938 6939The global internal table is a table of *shader resource descriptors* (SRDs) 6940that define how certain engine-wide, runtime-managed resources should be 6941accessed from a shader. The majority of these resources have HW-defined formats, 6942and it is up to the compiler to write/read data as required by the target 6943hardware. 6944 6945The following table illustrates the required format: 6946 6947 .. table:: PAL Global Internal Table 6948 :name: pal-git-table 6949 6950 ============= ================================ 6951 Offset Description 6952 ============= ================================ 6953 0-3 Graphics Scratch SRD 6954 4-7 Compute Scratch SRD 6955 8-11 ES/GS Ring Output SRD 6956 12-15 ES/GS Ring Input SRD 6957 16-19 GS/VS Ring Output #0 6958 20-23 GS/VS Ring Output #1 6959 24-27 GS/VS Ring Output #2 6960 28-31 GS/VS Ring Output #3 6961 32-35 GS/VS Ring Input SRD 6962 36-39 Tessellation Factor Buffer SRD 6963 40-43 Off-Chip LDS Buffer SRD 6964 44-47 Off-Chip Param Cache Buffer SRD 6965 48-51 Sample Position Buffer SRD 6966 52 vaRange::ShadowDescriptorTable High Bits 6967 ============= ================================ 6968 6969 The pointer to the global internal table passed to the shader as user data 6970 is a 32-bit pointer. The top 32 bits should be assumed to be the same as 6971 the top 32 bits of the pipeline, so the shader may use the program 6972 counter's top 32 bits. 6973 6974Unspecified OS 6975-------------- 6976 6977This section provides code conventions used when the target triple OS is 6978empty (see :ref:`amdgpu-target-triples`). 6979 6980Trap Handler ABI 6981~~~~~~~~~~~~~~~~ 6982 6983For code objects generated by AMDGPU backend for non-amdhsa OS, the runtime does 6984not install a trap handler. The ``llvm.trap`` and ``llvm.debugtrap`` 6985instructions are handled as follows: 6986 6987 .. table:: AMDGPU Trap Handler for Non-AMDHSA OS 6988 :name: amdgpu-trap-handler-for-non-amdhsa-os-table 6989 6990 =============== =============== =========================================== 6991 Usage Code Sequence Description 6992 =============== =============== =========================================== 6993 llvm.trap s_endpgm Causes wavefront to be terminated. 6994 llvm.debugtrap *none* Compiler warning given that there is no 6995 trap handler installed. 6996 =============== =============== =========================================== 6997 6998Source Languages 6999================ 7000 7001.. _amdgpu-opencl: 7002 7003OpenCL 7004------ 7005 7006When the language is OpenCL the following differences occur: 7007 70081. The OpenCL memory model is used (see :ref:`amdgpu-amdhsa-memory-model`). 70092. The AMDGPU backend appends additional arguments to the kernel's explicit 7010 arguments for the AMDHSA OS (see 7011 :ref:`opencl-kernel-implicit-arguments-appended-for-amdhsa-os-table`). 70123. Additional metadata is generated 7013 (see :ref:`amdgpu-amdhsa-code-object-metadata`). 7014 7015 .. table:: OpenCL kernel implicit arguments appended for AMDHSA OS 7016 :name: opencl-kernel-implicit-arguments-appended-for-amdhsa-os-table 7017 7018 ======== ==== ========= =========================================== 7019 Position Byte Byte Description 7020 Size Alignment 7021 ======== ==== ========= =========================================== 7022 1 8 8 OpenCL Global Offset X 7023 2 8 8 OpenCL Global Offset Y 7024 3 8 8 OpenCL Global Offset Z 7025 4 8 8 OpenCL address of printf buffer 7026 5 8 8 OpenCL address of virtual queue used by 7027 enqueue_kernel. 7028 6 8 8 OpenCL address of AqlWrap struct used by 7029 enqueue_kernel. 7030 7 8 8 Pointer argument used for Multi-gird 7031 synchronization. 7032 ======== ==== ========= =========================================== 7033 7034.. _amdgpu-hcc: 7035 7036HCC 7037--- 7038 7039When the language is HCC the following differences occur: 7040 70411. The HSA memory model is used (see :ref:`amdgpu-amdhsa-memory-model`). 7042 7043.. _amdgpu-assembler: 7044 7045Assembler 7046--------- 7047 7048AMDGPU backend has LLVM-MC based assembler which is currently in development. 7049It supports AMDGCN GFX6-GFX10. 7050 7051This section describes general syntax for instructions and operands. 7052 7053Instructions 7054~~~~~~~~~~~~ 7055 7056An instruction has the following :doc:`syntax<AMDGPUInstructionSyntax>`: 7057 7058 | ``<``\ *opcode*\ ``> <``\ *operand0*\ ``>, <``\ *operand1*\ ``>,... 7059 <``\ *modifier0*\ ``> <``\ *modifier1*\ ``>...`` 7060 7061:doc:`Operands<AMDGPUOperandSyntax>` are comma-separated while 7062:doc:`modifiers<AMDGPUModifierSyntax>` are space-separated. 7063 7064The order of operands and modifiers is fixed. 7065Most modifiers are optional and may be omitted. 7066 7067Links to detailed instruction syntax description may be found in the following 7068table. Note that features under development are not included 7069in this description. 7070 7071 =================================== ======================================= 7072 Core ISA ISA Extensions 7073 =================================== ======================================= 7074 :doc:`GFX7<AMDGPU/AMDGPUAsmGFX7>` \- 7075 :doc:`GFX8<AMDGPU/AMDGPUAsmGFX8>` \- 7076 :doc:`GFX9<AMDGPU/AMDGPUAsmGFX9>` :doc:`gfx900<AMDGPU/AMDGPUAsmGFX900>` 7077 7078 :doc:`gfx902<AMDGPU/AMDGPUAsmGFX900>` 7079 7080 :doc:`gfx904<AMDGPU/AMDGPUAsmGFX904>` 7081 7082 :doc:`gfx906<AMDGPU/AMDGPUAsmGFX906>` 7083 7084 :doc:`gfx908<AMDGPU/AMDGPUAsmGFX908>` 7085 7086 :doc:`gfx909<AMDGPU/AMDGPUAsmGFX900>` 7087 7088 :doc:`GFX10<AMDGPU/AMDGPUAsmGFX10>` :doc:`gfx1011<AMDGPU/AMDGPUAsmGFX1011>` 7089 7090 :doc:`gfx1012<AMDGPU/AMDGPUAsmGFX1011>` 7091 =================================== ======================================= 7092 7093For more information about instructions, their semantics and supported 7094combinations of operands, refer to one of instruction set architecture manuals 7095[AMD-GCN-GFX6]_, [AMD-GCN-GFX7]_, [AMD-GCN-GFX8]_, [AMD-GCN-GFX9]_ and 7096[AMD-GCN-GFX10]_. 7097 7098Operands 7099~~~~~~~~ 7100 7101Detailed description of operands may be found :doc:`here<AMDGPUOperandSyntax>`. 7102 7103Modifiers 7104~~~~~~~~~ 7105 7106Detailed description of modifiers may be found 7107:doc:`here<AMDGPUModifierSyntax>`. 7108 7109Instruction Examples 7110~~~~~~~~~~~~~~~~~~~~ 7111 7112DS 7113++ 7114 7115.. code-block:: nasm 7116 7117 ds_add_u32 v2, v4 offset:16 7118 ds_write_src2_b64 v2 offset0:4 offset1:8 7119 ds_cmpst_f32 v2, v4, v6 7120 ds_min_rtn_f64 v[8:9], v2, v[4:5] 7121 7122For full list of supported instructions, refer to "LDS/GDS instructions" in ISA 7123Manual. 7124 7125FLAT 7126++++ 7127 7128.. code-block:: nasm 7129 7130 flat_load_dword v1, v[3:4] 7131 flat_store_dwordx3 v[3:4], v[5:7] 7132 flat_atomic_swap v1, v[3:4], v5 glc 7133 flat_atomic_cmpswap v1, v[3:4], v[5:6] glc slc 7134 flat_atomic_fmax_x2 v[1:2], v[3:4], v[5:6] glc 7135 7136For full list of supported instructions, refer to "FLAT instructions" in ISA 7137Manual. 7138 7139MUBUF 7140+++++ 7141 7142.. code-block:: nasm 7143 7144 buffer_load_dword v1, off, s[4:7], s1 7145 buffer_store_dwordx4 v[1:4], v2, ttmp[4:7], s1 offen offset:4 glc tfe 7146 buffer_store_format_xy v[1:2], off, s[4:7], s1 7147 buffer_wbinvl1 7148 buffer_atomic_inc v1, v2, s[8:11], s4 idxen offset:4 slc 7149 7150For full list of supported instructions, refer to "MUBUF Instructions" in ISA 7151Manual. 7152 7153SMRD/SMEM 7154+++++++++ 7155 7156.. code-block:: nasm 7157 7158 s_load_dword s1, s[2:3], 0xfc 7159 s_load_dwordx8 s[8:15], s[2:3], s4 7160 s_load_dwordx16 s[88:103], s[2:3], s4 7161 s_dcache_inv_vol 7162 s_memtime s[4:5] 7163 7164For full list of supported instructions, refer to "Scalar Memory Operations" in 7165ISA Manual. 7166 7167SOP1 7168++++ 7169 7170.. code-block:: nasm 7171 7172 s_mov_b32 s1, s2 7173 s_mov_b64 s[0:1], 0x80000000 7174 s_cmov_b32 s1, 200 7175 s_wqm_b64 s[2:3], s[4:5] 7176 s_bcnt0_i32_b64 s1, s[2:3] 7177 s_swappc_b64 s[2:3], s[4:5] 7178 s_cbranch_join s[4:5] 7179 7180For full list of supported instructions, refer to "SOP1 Instructions" in ISA 7181Manual. 7182 7183SOP2 7184++++ 7185 7186.. code-block:: nasm 7187 7188 s_add_u32 s1, s2, s3 7189 s_and_b64 s[2:3], s[4:5], s[6:7] 7190 s_cselect_b32 s1, s2, s3 7191 s_andn2_b32 s2, s4, s6 7192 s_lshr_b64 s[2:3], s[4:5], s6 7193 s_ashr_i32 s2, s4, s6 7194 s_bfm_b64 s[2:3], s4, s6 7195 s_bfe_i64 s[2:3], s[4:5], s6 7196 s_cbranch_g_fork s[4:5], s[6:7] 7197 7198For full list of supported instructions, refer to "SOP2 Instructions" in ISA 7199Manual. 7200 7201SOPC 7202++++ 7203 7204.. code-block:: nasm 7205 7206 s_cmp_eq_i32 s1, s2 7207 s_bitcmp1_b32 s1, s2 7208 s_bitcmp0_b64 s[2:3], s4 7209 s_setvskip s3, s5 7210 7211For full list of supported instructions, refer to "SOPC Instructions" in ISA 7212Manual. 7213 7214SOPP 7215++++ 7216 7217.. code-block:: nasm 7218 7219 s_barrier 7220 s_nop 2 7221 s_endpgm 7222 s_waitcnt 0 ; Wait for all counters to be 0 7223 s_waitcnt vmcnt(0) & expcnt(0) & lgkmcnt(0) ; Equivalent to above 7224 s_waitcnt vmcnt(1) ; Wait for vmcnt counter to be 1. 7225 s_sethalt 9 7226 s_sleep 10 7227 s_sendmsg 0x1 7228 s_sendmsg sendmsg(MSG_INTERRUPT) 7229 s_trap 1 7230 7231For full list of supported instructions, refer to "SOPP Instructions" in ISA 7232Manual. 7233 7234Unless otherwise mentioned, little verification is performed on the operands 7235of SOPP Instructions, so it is up to the programmer to be familiar with the 7236range or acceptable values. 7237 7238VALU 7239++++ 7240 7241For vector ALU instruction opcodes (VOP1, VOP2, VOP3, VOPC, VOP_DPP, VOP_SDWA), 7242the assembler will automatically use optimal encoding based on its operands. To 7243force specific encoding, one can add a suffix to the opcode of the instruction: 7244 7245* _e32 for 32-bit VOP1/VOP2/VOPC 7246* _e64 for 64-bit VOP3 7247* _dpp for VOP_DPP 7248* _sdwa for VOP_SDWA 7249 7250VOP1/VOP2/VOP3/VOPC examples: 7251 7252.. code-block:: nasm 7253 7254 v_mov_b32 v1, v2 7255 v_mov_b32_e32 v1, v2 7256 v_nop 7257 v_cvt_f64_i32_e32 v[1:2], v2 7258 v_floor_f32_e32 v1, v2 7259 v_bfrev_b32_e32 v1, v2 7260 v_add_f32_e32 v1, v2, v3 7261 v_mul_i32_i24_e64 v1, v2, 3 7262 v_mul_i32_i24_e32 v1, -3, v3 7263 v_mul_i32_i24_e32 v1, -100, v3 7264 v_addc_u32 v1, s[0:1], v2, v3, s[2:3] 7265 v_max_f16_e32 v1, v2, v3 7266 7267VOP_DPP examples: 7268 7269.. code-block:: nasm 7270 7271 v_mov_b32 v0, v0 quad_perm:[0,2,1,1] 7272 v_sin_f32 v0, v0 row_shl:1 row_mask:0xa bank_mask:0x1 bound_ctrl:0 7273 v_mov_b32 v0, v0 wave_shl:1 7274 v_mov_b32 v0, v0 row_mirror 7275 v_mov_b32 v0, v0 row_bcast:31 7276 v_mov_b32 v0, v0 quad_perm:[1,3,0,1] row_mask:0xa bank_mask:0x1 bound_ctrl:0 7277 v_add_f32 v0, v0, |v0| row_shl:1 row_mask:0xa bank_mask:0x1 bound_ctrl:0 7278 v_max_f16 v1, v2, v3 row_shl:1 row_mask:0xa bank_mask:0x1 bound_ctrl:0 7279 7280VOP_SDWA examples: 7281 7282.. code-block:: nasm 7283 7284 v_mov_b32 v1, v2 dst_sel:BYTE_0 dst_unused:UNUSED_PRESERVE src0_sel:DWORD 7285 v_min_u32 v200, v200, v1 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:BYTE_1 src1_sel:DWORD 7286 v_sin_f32 v0, v0 dst_unused:UNUSED_PAD src0_sel:WORD_1 7287 v_fract_f32 v0, |v0| dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 7288 v_cmpx_le_u32 vcc, v1, v2 src0_sel:BYTE_2 src1_sel:WORD_0 7289 7290For full list of supported instructions, refer to "Vector ALU instructions". 7291 7292.. TODO:: 7293 7294 Remove once we switch to code object v3 by default. 7295 7296.. _amdgpu-amdhsa-assembler-predefined-symbols-v2: 7297 7298Code Object V2 Predefined Symbols (-mattr=-code-object-v3) 7299~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 7300 7301.. warning:: Code Object V2 is not the default code object version emitted by 7302 this version of LLVM. For a description of the predefined symbols available 7303 with the default configuration (Code Object V3) see 7304 :ref:`amdgpu-amdhsa-assembler-predefined-symbols-v3`. 7305 7306The AMDGPU assembler defines and updates some symbols automatically. These 7307symbols do not affect code generation. 7308 7309.option.machine_version_major 7310+++++++++++++++++++++++++++++ 7311 7312Set to the GFX major generation number of the target being assembled for. For 7313example, when assembling for a "GFX9" target this will be set to the integer 7314value "9". The possible GFX major generation numbers are presented in 7315:ref:`amdgpu-processors`. 7316 7317.option.machine_version_minor 7318+++++++++++++++++++++++++++++ 7319 7320Set to the GFX minor generation number of the target being assembled for. For 7321example, when assembling for a "GFX810" target this will be set to the integer 7322value "1". The possible GFX minor generation numbers are presented in 7323:ref:`amdgpu-processors`. 7324 7325.option.machine_version_stepping 7326++++++++++++++++++++++++++++++++ 7327 7328Set to the GFX stepping generation number of the target being assembled for. 7329For example, when assembling for a "GFX704" target this will be set to the 7330integer value "4". The possible GFX stepping generation numbers are presented 7331in :ref:`amdgpu-processors`. 7332 7333.kernel.vgpr_count 7334++++++++++++++++++ 7335 7336Set to zero each time a 7337:ref:`amdgpu-amdhsa-assembler-directive-amdgpu_hsa_kernel` directive is 7338encountered. At each instruction, if the current value of this symbol is less 7339than or equal to the maximum VPGR number explicitly referenced within that 7340instruction then the symbol value is updated to equal that VGPR number plus 7341one. 7342 7343.kernel.sgpr_count 7344++++++++++++++++++ 7345 7346Set to zero each time a 7347:ref:`amdgpu-amdhsa-assembler-directive-amdgpu_hsa_kernel` directive is 7348encountered. At each instruction, if the current value of this symbol is less 7349than or equal to the maximum VPGR number explicitly referenced within that 7350instruction then the symbol value is updated to equal that SGPR number plus 7351one. 7352 7353.. _amdgpu-amdhsa-assembler-directives-v2: 7354 7355Code Object V2 Directives (-mattr=-code-object-v3) 7356~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 7357 7358.. warning:: Code Object V2 is not the default code object version emitted by 7359 this version of LLVM. For a description of the directives supported with 7360 the default configuration (Code Object V3) see 7361 :ref:`amdgpu-amdhsa-assembler-directives-v3`. 7362 7363AMDGPU ABI defines auxiliary data in output code object. In assembly source, 7364one can specify them with assembler directives. 7365 7366.hsa_code_object_version major, minor 7367+++++++++++++++++++++++++++++++++++++ 7368 7369*major* and *minor* are integers that specify the version of the HSA code 7370object that will be generated by the assembler. 7371 7372.hsa_code_object_isa [major, minor, stepping, vendor, arch] 7373+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 7374 7375 7376*major*, *minor*, and *stepping* are all integers that describe the instruction 7377set architecture (ISA) version of the assembly program. 7378 7379*vendor* and *arch* are quoted strings. *vendor* should always be equal to 7380"AMD" and *arch* should always be equal to "AMDGPU". 7381 7382By default, the assembler will derive the ISA version, *vendor*, and *arch* 7383from the value of the -mcpu option that is passed to the assembler. 7384 7385.. _amdgpu-amdhsa-assembler-directive-amdgpu_hsa_kernel: 7386 7387.amdgpu_hsa_kernel (name) 7388+++++++++++++++++++++++++ 7389 7390This directives specifies that the symbol with given name is a kernel entry 7391point (label) and the object should contain corresponding symbol of type 7392STT_AMDGPU_HSA_KERNEL. 7393 7394.amd_kernel_code_t 7395++++++++++++++++++ 7396 7397This directive marks the beginning of a list of key / value pairs that are used 7398to specify the amd_kernel_code_t object that will be emitted by the assembler. 7399The list must be terminated by the *.end_amd_kernel_code_t* directive. For any 7400amd_kernel_code_t values that are unspecified a default value will be used. The 7401default value for all keys is 0, with the following exceptions: 7402 7403- *amd_code_version_major* defaults to 1. 7404- *amd_kernel_code_version_minor* defaults to 2. 7405- *amd_machine_kind* defaults to 1. 7406- *amd_machine_version_major*, *machine_version_minor*, and 7407 *amd_machine_version_stepping* are derived from the value of the -mcpu option 7408 that is passed to the assembler. 7409- *kernel_code_entry_byte_offset* defaults to 256. 7410- *wavefront_size* defaults 6 for all targets before GFX10. For GFX10 onwards 7411 defaults to 6 if target feature ``wavefrontsize64`` is enabled, otherwise 5. 7412 Note that wavefront size is specified as a power of two, so a value of **n** 7413 means a size of 2^ **n**. 7414- *call_convention* defaults to -1. 7415- *kernarg_segment_alignment*, *group_segment_alignment*, and 7416 *private_segment_alignment* default to 4. Note that alignments are specified 7417 as a power of 2, so a value of **n** means an alignment of 2^ **n**. 7418- *enable_wgp_mode* defaults to 1 if target feature ``cumode`` is disabled for 7419 GFX10 onwards. 7420- *enable_mem_ordered* defaults to 1 for GFX10 onwards. 7421 7422The *.amd_kernel_code_t* directive must be placed immediately after the 7423function label and before any instructions. 7424 7425For a full list of amd_kernel_code_t keys, refer to AMDGPU ABI document, 7426comments in lib/Target/AMDGPU/AmdKernelCodeT.h and test/CodeGen/AMDGPU/hsa.s. 7427 7428.. _amdgpu-amdhsa-assembler-example-v2: 7429 7430Code Object V2 Example Source Code (-mattr=-code-object-v3) 7431~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 7432 7433.. warning:: Code Object V2 is not the default code object version emitted by 7434 this version of LLVM. For a description of the directives supported with 7435 the default configuration (Code Object V3) see 7436 :ref:`amdgpu-amdhsa-assembler-example-v3`. 7437 7438Here is an example of a minimal assembly source file, defining one HSA kernel: 7439 7440.. code:: 7441 :number-lines: 7442 7443 .hsa_code_object_version 1,0 7444 .hsa_code_object_isa 7445 7446 .hsatext 7447 .globl hello_world 7448 .p2align 8 7449 .amdgpu_hsa_kernel hello_world 7450 7451 hello_world: 7452 7453 .amd_kernel_code_t 7454 enable_sgpr_kernarg_segment_ptr = 1 7455 is_ptr64 = 1 7456 compute_pgm_rsrc1_vgprs = 0 7457 compute_pgm_rsrc1_sgprs = 0 7458 compute_pgm_rsrc2_user_sgpr = 2 7459 compute_pgm_rsrc1_wgp_mode = 0 7460 compute_pgm_rsrc1_mem_ordered = 0 7461 compute_pgm_rsrc1_fwd_progress = 1 7462 .end_amd_kernel_code_t 7463 7464 s_load_dwordx2 s[0:1], s[0:1] 0x0 7465 v_mov_b32 v0, 3.14159 7466 s_waitcnt lgkmcnt(0) 7467 v_mov_b32 v1, s0 7468 v_mov_b32 v2, s1 7469 flat_store_dword v[1:2], v0 7470 s_endpgm 7471 .Lfunc_end0: 7472 .size hello_world, .Lfunc_end0-hello_world 7473 7474.. _amdgpu-amdhsa-assembler-predefined-symbols-v3: 7475 7476Code Object V3 Predefined Symbols (-mattr=+code-object-v3) 7477~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 7478 7479The AMDGPU assembler defines and updates some symbols automatically. These 7480symbols do not affect code generation. 7481 7482.amdgcn.gfx_generation_number 7483+++++++++++++++++++++++++++++ 7484 7485Set to the GFX major generation number of the target being assembled for. For 7486example, when assembling for a "GFX9" target this will be set to the integer 7487value "9". The possible GFX major generation numbers are presented in 7488:ref:`amdgpu-processors`. 7489 7490.amdgcn.gfx_generation_minor 7491++++++++++++++++++++++++++++ 7492 7493Set to the GFX minor generation number of the target being assembled for. For 7494example, when assembling for a "GFX810" target this will be set to the integer 7495value "1". The possible GFX minor generation numbers are presented in 7496:ref:`amdgpu-processors`. 7497 7498.amdgcn.gfx_generation_stepping 7499+++++++++++++++++++++++++++++++ 7500 7501Set to the GFX stepping generation number of the target being assembled for. 7502For example, when assembling for a "GFX704" target this will be set to the 7503integer value "4". The possible GFX stepping generation numbers are presented 7504in :ref:`amdgpu-processors`. 7505 7506.. _amdgpu-amdhsa-assembler-symbol-next_free_vgpr: 7507 7508.amdgcn.next_free_vgpr 7509++++++++++++++++++++++ 7510 7511Set to zero before assembly begins. At each instruction, if the current value 7512of this symbol is less than or equal to the maximum VGPR number explicitly 7513referenced within that instruction then the symbol value is updated to equal 7514that VGPR number plus one. 7515 7516May be used to set the `.amdhsa_next_free_vpgr` directive in 7517:ref:`amdhsa-kernel-directives-table`. 7518 7519May be set at any time, e.g. manually set to zero at the start of each kernel. 7520 7521.. _amdgpu-amdhsa-assembler-symbol-next_free_sgpr: 7522 7523.amdgcn.next_free_sgpr 7524++++++++++++++++++++++ 7525 7526Set to zero before assembly begins. At each instruction, if the current value 7527of this symbol is less than or equal the maximum SGPR number explicitly 7528referenced within that instruction then the symbol value is updated to equal 7529that SGPR number plus one. 7530 7531May be used to set the `.amdhsa_next_free_spgr` directive in 7532:ref:`amdhsa-kernel-directives-table`. 7533 7534May be set at any time, e.g. manually set to zero at the start of each kernel. 7535 7536.. _amdgpu-amdhsa-assembler-directives-v3: 7537 7538Code Object V3 Directives (-mattr=+code-object-v3) 7539~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 7540 7541Directives which begin with ``.amdgcn`` are valid for all ``amdgcn`` 7542architecture processors, and are not OS-specific. Directives which begin with 7543``.amdhsa`` are specific to ``amdgcn`` architecture processors when the 7544``amdhsa`` OS is specified. See :ref:`amdgpu-target-triples` and 7545:ref:`amdgpu-processors`. 7546 7547.amdgcn_target <target> 7548+++++++++++++++++++++++ 7549 7550Optional directive which declares the target supported by the containing 7551assembler source file. Valid values are described in 7552:ref:`amdgpu-amdhsa-code-object-target-identification`. Used by the assembler 7553to validate command-line options such as ``-triple``, ``-mcpu``, and those 7554which specify target features. 7555 7556.amdhsa_kernel <name> 7557+++++++++++++++++++++ 7558 7559Creates a correctly aligned AMDHSA kernel descriptor and a symbol, 7560``<name>.kd``, in the current location of the current section. Only valid when 7561the OS is ``amdhsa``. ``<name>`` must be a symbol that labels the first 7562instruction to execute, and does not need to be previously defined. 7563 7564Marks the beginning of a list of directives used to generate the bytes of a 7565kernel descriptor, as described in :ref:`amdgpu-amdhsa-kernel-descriptor`. 7566Directives which may appear in this list are described in 7567:ref:`amdhsa-kernel-directives-table`. Directives may appear in any order, must 7568be valid for the target being assembled for, and cannot be repeated. Directives 7569support the range of values specified by the field they reference in 7570:ref:`amdgpu-amdhsa-kernel-descriptor`. If a directive is not specified, it is 7571assumed to have its default value, unless it is marked as "Required", in which 7572case it is an error to omit the directive. This list of directives is 7573terminated by an ``.end_amdhsa_kernel`` directive. 7574 7575 .. table:: AMDHSA Kernel Assembler Directives 7576 :name: amdhsa-kernel-directives-table 7577 7578 ======================================================== =================== ============ =================== 7579 Directive Default Supported On Description 7580 ======================================================== =================== ============ =================== 7581 ``.amdhsa_group_segment_fixed_size`` 0 GFX6-GFX10 Controls GROUP_SEGMENT_FIXED_SIZE in 7582 :ref:`amdgpu-amdhsa-kernel-descriptor-gfx6-gfx10-table`. 7583 ``.amdhsa_private_segment_fixed_size`` 0 GFX6-GFX10 Controls PRIVATE_SEGMENT_FIXED_SIZE in 7584 :ref:`amdgpu-amdhsa-kernel-descriptor-gfx6-gfx10-table`. 7585 ``.amdhsa_user_sgpr_private_segment_buffer`` 0 GFX6-GFX10 Controls ENABLE_SGPR_PRIVATE_SEGMENT_BUFFER in 7586 :ref:`amdgpu-amdhsa-kernel-descriptor-gfx6-gfx10-table`. 7587 ``.amdhsa_user_sgpr_dispatch_ptr`` 0 GFX6-GFX10 Controls ENABLE_SGPR_DISPATCH_PTR in 7588 :ref:`amdgpu-amdhsa-kernel-descriptor-gfx6-gfx10-table`. 7589 ``.amdhsa_user_sgpr_queue_ptr`` 0 GFX6-GFX10 Controls ENABLE_SGPR_QUEUE_PTR in 7590 :ref:`amdgpu-amdhsa-kernel-descriptor-gfx6-gfx10-table`. 7591 ``.amdhsa_user_sgpr_kernarg_segment_ptr`` 0 GFX6-GFX10 Controls ENABLE_SGPR_KERNARG_SEGMENT_PTR in 7592 :ref:`amdgpu-amdhsa-kernel-descriptor-gfx6-gfx10-table`. 7593 ``.amdhsa_user_sgpr_dispatch_id`` 0 GFX6-GFX10 Controls ENABLE_SGPR_DISPATCH_ID in 7594 :ref:`amdgpu-amdhsa-kernel-descriptor-gfx6-gfx10-table`. 7595 ``.amdhsa_user_sgpr_flat_scratch_init`` 0 GFX6-GFX10 Controls ENABLE_SGPR_FLAT_SCRATCH_INIT in 7596 :ref:`amdgpu-amdhsa-kernel-descriptor-gfx6-gfx10-table`. 7597 ``.amdhsa_user_sgpr_private_segment_size`` 0 GFX6-GFX10 Controls ENABLE_SGPR_PRIVATE_SEGMENT_SIZE in 7598 :ref:`amdgpu-amdhsa-kernel-descriptor-gfx6-gfx10-table`. 7599 ``.amdhsa_wavefront_size32`` Target GFX10 Controls ENABLE_WAVEFRONT_SIZE32 in 7600 Feature :ref:`amdgpu-amdhsa-kernel-descriptor-gfx6-gfx10-table`. 7601 Specific 7602 (-wavefrontsize64) 7603 ``.amdhsa_system_sgpr_private_segment_wavefront_offset`` 0 GFX6-GFX10 Controls ENABLE_SGPR_PRIVATE_SEGMENT_WAVEFRONT_OFFSET in 7604 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 7605 ``.amdhsa_system_sgpr_workgroup_id_x`` 1 GFX6-GFX10 Controls ENABLE_SGPR_WORKGROUP_ID_X in 7606 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 7607 ``.amdhsa_system_sgpr_workgroup_id_y`` 0 GFX6-GFX10 Controls ENABLE_SGPR_WORKGROUP_ID_Y in 7608 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 7609 ``.amdhsa_system_sgpr_workgroup_id_z`` 0 GFX6-GFX10 Controls ENABLE_SGPR_WORKGROUP_ID_Z in 7610 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 7611 ``.amdhsa_system_sgpr_workgroup_info`` 0 GFX6-GFX10 Controls ENABLE_SGPR_WORKGROUP_INFO in 7612 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 7613 ``.amdhsa_system_vgpr_workitem_id`` 0 GFX6-GFX10 Controls ENABLE_VGPR_WORKITEM_ID in 7614 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 7615 Possible values are defined in 7616 :ref:`amdgpu-amdhsa-system-vgpr-work-item-id-enumeration-values-table`. 7617 ``.amdhsa_next_free_vgpr`` Required GFX6-GFX10 Maximum VGPR number explicitly referenced, plus one. 7618 Used to calculate GRANULATED_WORKITEM_VGPR_COUNT in 7619 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 7620 ``.amdhsa_next_free_sgpr`` Required GFX6-GFX10 Maximum SGPR number explicitly referenced, plus one. 7621 Used to calculate GRANULATED_WAVEFRONT_SGPR_COUNT in 7622 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 7623 ``.amdhsa_reserve_vcc`` 1 GFX6-GFX10 Whether the kernel may use the special VCC SGPR. 7624 Used to calculate GRANULATED_WAVEFRONT_SGPR_COUNT in 7625 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 7626 ``.amdhsa_reserve_flat_scratch`` 1 GFX7-GFX10 Whether the kernel may use flat instructions to access 7627 scratch memory. Used to calculate 7628 GRANULATED_WAVEFRONT_SGPR_COUNT in 7629 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 7630 ``.amdhsa_reserve_xnack_mask`` Target GFX8-GFX10 Whether the kernel may trigger XNACK replay. 7631 Feature Used to calculate GRANULATED_WAVEFRONT_SGPR_COUNT in 7632 Specific :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 7633 (+xnack) 7634 ``.amdhsa_float_round_mode_32`` 0 GFX6-GFX10 Controls FLOAT_ROUND_MODE_32 in 7635 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 7636 Possible values are defined in 7637 :ref:`amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table`. 7638 ``.amdhsa_float_round_mode_16_64`` 0 GFX6-GFX10 Controls FLOAT_ROUND_MODE_16_64 in 7639 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 7640 Possible values are defined in 7641 :ref:`amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table`. 7642 ``.amdhsa_float_denorm_mode_32`` 0 GFX6-GFX10 Controls FLOAT_DENORM_MODE_32 in 7643 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 7644 Possible values are defined in 7645 :ref:`amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table`. 7646 ``.amdhsa_float_denorm_mode_16_64`` 3 GFX6-GFX10 Controls FLOAT_DENORM_MODE_16_64 in 7647 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 7648 Possible values are defined in 7649 :ref:`amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table`. 7650 ``.amdhsa_dx10_clamp`` 1 GFX6-GFX10 Controls ENABLE_DX10_CLAMP in 7651 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 7652 ``.amdhsa_ieee_mode`` 1 GFX6-GFX10 Controls ENABLE_IEEE_MODE in 7653 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 7654 ``.amdhsa_fp16_overflow`` 0 GFX9-GFX10 Controls FP16_OVFL in 7655 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 7656 ``.amdhsa_workgroup_processor_mode`` Target GFX10 Controls ENABLE_WGP_MODE in 7657 Feature :ref:`amdgpu-amdhsa-kernel-descriptor-gfx6-gfx10-table`. 7658 Specific 7659 (-cumode) 7660 ``.amdhsa_memory_ordered`` 1 GFX10 Controls MEM_ORDERED in 7661 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 7662 ``.amdhsa_forward_progress`` 0 GFX10 Controls FWD_PROGRESS in 7663 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 7664 ``.amdhsa_exception_fp_ieee_invalid_op`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_INVALID_OPERATION in 7665 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 7666 ``.amdhsa_exception_fp_denorm_src`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_FP_DENORMAL_SOURCE in 7667 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 7668 ``.amdhsa_exception_fp_ieee_div_zero`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_DIVISION_BY_ZERO in 7669 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 7670 ``.amdhsa_exception_fp_ieee_overflow`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_OVERFLOW in 7671 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 7672 ``.amdhsa_exception_fp_ieee_underflow`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_UNDERFLOW in 7673 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 7674 ``.amdhsa_exception_fp_ieee_inexact`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_INEXACT in 7675 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 7676 ``.amdhsa_exception_int_div_zero`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_INT_DIVIDE_BY_ZERO in 7677 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 7678 ======================================================== =================== ============ =================== 7679 7680.amdgpu_metadata 7681++++++++++++++++ 7682 7683Optional directive which declares the contents of the ``NT_AMDGPU_METADATA`` 7684note record (see :ref:`amdgpu-elf-note-records-table-v3`). 7685 7686The contents must be in the [YAML]_ markup format, with the same structure and 7687semantics described in :ref:`amdgpu-amdhsa-code-object-metadata-v3`. 7688 7689This directive is terminated by an ``.end_amdgpu_metadata`` directive. 7690 7691.. _amdgpu-amdhsa-assembler-example-v3: 7692 7693Code Object V3 Example Source Code (-mattr=+code-object-v3) 7694~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 7695 7696Here is an example of a minimal assembly source file, defining one HSA kernel: 7697 7698.. code:: 7699 :number-lines: 7700 7701 .amdgcn_target "amdgcn-amd-amdhsa--gfx900+xnack" // optional 7702 7703 .text 7704 .globl hello_world 7705 .p2align 8 7706 .type hello_world,@function 7707 hello_world: 7708 s_load_dwordx2 s[0:1], s[0:1] 0x0 7709 v_mov_b32 v0, 3.14159 7710 s_waitcnt lgkmcnt(0) 7711 v_mov_b32 v1, s0 7712 v_mov_b32 v2, s1 7713 flat_store_dword v[1:2], v0 7714 s_endpgm 7715 .Lfunc_end0: 7716 .size hello_world, .Lfunc_end0-hello_world 7717 7718 .rodata 7719 .p2align 6 7720 .amdhsa_kernel hello_world 7721 .amdhsa_user_sgpr_kernarg_segment_ptr 1 7722 .amdhsa_next_free_vgpr .amdgcn.next_free_vgpr 7723 .amdhsa_next_free_sgpr .amdgcn.next_free_sgpr 7724 .end_amdhsa_kernel 7725 7726 .amdgpu_metadata 7727 --- 7728 amdhsa.version: 7729 - 1 7730 - 0 7731 amdhsa.kernels: 7732 - .name: hello_world 7733 .symbol: hello_world.kd 7734 .kernarg_segment_size: 48 7735 .group_segment_fixed_size: 0 7736 .private_segment_fixed_size: 0 7737 .kernarg_segment_align: 4 7738 .wavefront_size: 64 7739 .sgpr_count: 2 7740 .vgpr_count: 3 7741 .max_flat_workgroup_size: 256 7742 ... 7743 .end_amdgpu_metadata 7744 7745If an assembly source file contains multiple kernels and/or functions, the 7746:ref:`amdgpu-amdhsa-assembler-symbol-next_free_vgpr` and 7747:ref:`amdgpu-amdhsa-assembler-symbol-next_free_sgpr` symbols may be reset using 7748the ``.set <symbol>, <expression>`` directive. For example, in the case of two 7749kernels, where ``function1`` is only called from ``kernel1`` it is sufficient 7750to group the function with the kernel that calls it and reset the symbols 7751between the two connected components: 7752 7753.. code:: 7754 :number-lines: 7755 7756 .amdgcn_target "amdgcn-amd-amdhsa--gfx900+xnack" // optional 7757 7758 // gpr tracking symbols are implicitly set to zero 7759 7760 .text 7761 .globl kern0 7762 .p2align 8 7763 .type kern0,@function 7764 kern0: 7765 // ... 7766 s_endpgm 7767 .Lkern0_end: 7768 .size kern0, .Lkern0_end-kern0 7769 7770 .rodata 7771 .p2align 6 7772 .amdhsa_kernel kern0 7773 // ... 7774 .amdhsa_next_free_vgpr .amdgcn.next_free_vgpr 7775 .amdhsa_next_free_sgpr .amdgcn.next_free_sgpr 7776 .end_amdhsa_kernel 7777 7778 // reset symbols to begin tracking usage in func1 and kern1 7779 .set .amdgcn.next_free_vgpr, 0 7780 .set .amdgcn.next_free_sgpr, 0 7781 7782 .text 7783 .hidden func1 7784 .global func1 7785 .p2align 2 7786 .type func1,@function 7787 func1: 7788 // ... 7789 s_setpc_b64 s[30:31] 7790 .Lfunc1_end: 7791 .size func1, .Lfunc1_end-func1 7792 7793 .globl kern1 7794 .p2align 8 7795 .type kern1,@function 7796 kern1: 7797 // ... 7798 s_getpc_b64 s[4:5] 7799 s_add_u32 s4, s4, func1@rel32@lo+4 7800 s_addc_u32 s5, s5, func1@rel32@lo+4 7801 s_swappc_b64 s[30:31], s[4:5] 7802 // ... 7803 s_endpgm 7804 .Lkern1_end: 7805 .size kern1, .Lkern1_end-kern1 7806 7807 .rodata 7808 .p2align 6 7809 .amdhsa_kernel kern1 7810 // ... 7811 .amdhsa_next_free_vgpr .amdgcn.next_free_vgpr 7812 .amdhsa_next_free_sgpr .amdgcn.next_free_sgpr 7813 .end_amdhsa_kernel 7814 7815These symbols cannot identify connected components in order to automatically 7816track the usage for each kernel. However, in some cases careful organization of 7817the kernels and functions in the source file means there is minimal additional 7818effort required to accurately calculate GPR usage. 7819 7820Additional Documentation 7821======================== 7822 7823.. [AMD-GCN-GFX6] `AMD Southern Islands Series ISA <http://developer.amd.com/wordpress/media/2012/12/AMD_Southern_Islands_Instruction_Set_Architecture.pdf>`__ 7824.. [AMD-GCN-GFX7] `AMD Sea Islands Series ISA <http://developer.amd.com/wordpress/media/2013/07/AMD_Sea_Islands_Instruction_Set_Architecture.pdf>`_ 7825.. [AMD-GCN-GFX8] `AMD GCN3 Instruction Set Architecture <http://amd-dev.wpengine.netdna-cdn.com/wordpress/media/2013/12/AMD_GCN3_Instruction_Set_Architecture_rev1.1.pdf>`__ 7826.. [AMD-GCN-GFX9] `AMD "Vega" Instruction Set Architecture <http://developer.amd.com/wordpress/media/2013/12/Vega_Shader_ISA_28July2017.pdf>`__ 7827.. [AMD-GCN-GFX10] `AMD "RDNA 1.0" Instruction Set Architecture <https://gpuopen.com/wp-content/uploads/2019/08/RDNA_Shader_ISA_5August2019.pdf>`__ 7828.. [AMD-RADEON-HD-2000-3000] `AMD R6xx shader ISA <http://developer.amd.com/wordpress/media/2012/10/R600_Instruction_Set_Architecture.pdf>`__ 7829.. [AMD-RADEON-HD-4000] `AMD R7xx shader ISA <http://developer.amd.com/wordpress/media/2012/10/R700-Family_Instruction_Set_Architecture.pdf>`__ 7830.. [AMD-RADEON-HD-5000] `AMD Evergreen shader ISA <http://developer.amd.com/wordpress/media/2012/10/AMD_Evergreen-Family_Instruction_Set_Architecture.pdf>`__ 7831.. [AMD-RADEON-HD-6000] `AMD Cayman/Trinity shader ISA <http://developer.amd.com/wordpress/media/2012/10/AMD_HD_6900_Series_Instruction_Set_Architecture.pdf>`__ 7832.. [AMD-ROCm] `AMD ROCm Platform <https://rocm-documentation.readthedocs.io>`__ 7833.. [AMD-ROCm-github] `ROCm github <http://github.com/RadeonOpenCompute>`__ 7834.. [CLANG-ATTR] `Attributes in Clang <https://clang.llvm.org/docs/AttributeReference.html>`__ 7835.. [DWARF] `DWARF Debugging Information Format <http://dwarfstd.org/>`__ 7836.. [ELF] `Executable and Linkable Format (ELF) <http://www.sco.com/developers/gabi/>`__ 7837.. [HRF] `Heterogeneous-race-free Memory Models <http://benedictgaster.org/wp-content/uploads/2014/01/asplos269-FINAL.pdf>`__ 7838.. [HSA] `Heterogeneous System Architecture (HSA) Foundation <http://www.hsafoundation.com/>`__ 7839.. [MsgPack] `Message Pack <http://www.msgpack.org/>`__ 7840.. [OpenCL] `The OpenCL Specification Version 2.0 <http://www.khronos.org/registry/cl/specs/opencl-2.0.pdf>`__ 7841.. [SEMVER] `Semantic Versioning <https://semver.org/>`__ 7842.. [YAML] `YAML Ain't Markup Language (YAML™) Version 1.2 <http://www.yaml.org/spec/1.2/spec.html>`__ 7843