1============================= 2User Guide for AMDGPU Backend 3============================= 4 5.. contents:: 6 :local: 7 8.. toctree:: 9 :hidden: 10 11 AMDGPU/AMDGPUAsmGFX7 12 AMDGPU/AMDGPUAsmGFX8 13 AMDGPU/AMDGPUAsmGFX9 14 AMDGPU/AMDGPUAsmGFX900 15 AMDGPU/AMDGPUAsmGFX904 16 AMDGPU/AMDGPUAsmGFX906 17 AMDGPU/AMDGPUAsmGFX908 18 AMDGPU/AMDGPUAsmGFX10 19 AMDGPU/AMDGPUAsmGFX1011 20 AMDGPUModifierSyntax 21 AMDGPUOperandSyntax 22 AMDGPUInstructionSyntax 23 AMDGPUInstructionNotation 24 AMDGPUDwarfExtensionsForHeterogeneousDebugging 25 26Introduction 27============ 28 29The AMDGPU backend provides ISA code generation for AMD GPUs, starting with the 30R600 family up until the current GCN families. It lives in the 31``llvm/lib/Target/AMDGPU`` directory. 32 33LLVM 34==== 35 36.. _amdgpu-target-triples: 37 38Target Triples 39-------------- 40 41Use the ``clang -target <Architecture>-<Vendor>-<OS>-<Environment>`` option to 42specify the target triple: 43 44 .. table:: AMDGPU Architectures 45 :name: amdgpu-architecture-table 46 47 ============ ============================================================== 48 Architecture Description 49 ============ ============================================================== 50 ``r600`` AMD GPUs HD2XXX-HD6XXX for graphics and compute shaders. 51 ``amdgcn`` AMD GPUs GCN GFX6 onwards for graphics and compute shaders. 52 ============ ============================================================== 53 54 .. table:: AMDGPU Vendors 55 :name: amdgpu-vendor-table 56 57 ============ ============================================================== 58 Vendor Description 59 ============ ============================================================== 60 ``amd`` Can be used for all AMD GPU usage. 61 ``mesa3d`` Can be used if the OS is ``mesa3d``. 62 ============ ============================================================== 63 64 .. table:: AMDGPU Operating Systems 65 :name: amdgpu-os-table 66 67 ============== ============================================================ 68 OS Description 69 ============== ============================================================ 70 *<empty>* Defaults to the *unknown* OS. 71 ``amdhsa`` Compute kernels executed on HSA [HSA]_ compatible runtimes 72 such as AMD's ROCm [AMD-ROCm]_. 73 ``amdpal`` Graphic shaders and compute kernels executed on AMD PAL 74 runtime. 75 ``mesa3d`` Graphic shaders and compute kernels executed on Mesa 3D 76 runtime. 77 ============== ============================================================ 78 79 .. table:: AMDGPU Environments 80 :name: amdgpu-environment-table 81 82 ============ ============================================================== 83 Environment Description 84 ============ ============================================================== 85 *<empty>* Default. 86 ============ ============================================================== 87 88.. _amdgpu-processors: 89 90Processors 91---------- 92 93Use the ``clang -mcpu <Processor>`` option to specify the AMDGPU processor. The 94names from both the *Processor* and *Alternative Processor* can be used. 95 96 .. table:: AMDGPU Processors 97 :name: amdgpu-processor-table 98 99 =========== =============== ============ ===== ============================= ======= ====================== 100 Processor Alternative Target dGPU/ Target ROCm Example 101 Processor Triple APU Features Support Products 102 Architecture Supported 103 [Default] 104 =========== =============== ============ ===== ============================= ======= ====================== 105 **Radeon HD 2000/3000 Series (R600)** [AMD-RADEON-HD-2000-3000]_ 106 ----------------------------------------------------------------------------------------------------------- 107 ``r600`` ``r600`` dGPU 108 ``r630`` ``r600`` dGPU 109 ``rs880`` ``r600`` dGPU 110 ``rv670`` ``r600`` dGPU 111 **Radeon HD 4000 Series (R700)** [AMD-RADEON-HD-4000]_ 112 ----------------------------------------------------------------------------------------------------------- 113 ``rv710`` ``r600`` dGPU 114 ``rv730`` ``r600`` dGPU 115 ``rv770`` ``r600`` dGPU 116 **Radeon HD 5000 Series (Evergreen)** [AMD-RADEON-HD-5000]_ 117 ----------------------------------------------------------------------------------------------------------- 118 ``cedar`` ``r600`` dGPU 119 ``cypress`` ``r600`` dGPU 120 ``juniper`` ``r600`` dGPU 121 ``redwood`` ``r600`` dGPU 122 ``sumo`` ``r600`` dGPU 123 **Radeon HD 6000 Series (Northern Islands)** [AMD-RADEON-HD-6000]_ 124 ----------------------------------------------------------------------------------------------------------- 125 ``barts`` ``r600`` dGPU 126 ``caicos`` ``r600`` dGPU 127 ``cayman`` ``r600`` dGPU 128 ``turks`` ``r600`` dGPU 129 **GCN GFX6 (Southern Islands (SI))** [AMD-GCN-GFX6]_ 130 ----------------------------------------------------------------------------------------------------------- 131 ``gfx600`` - ``tahiti`` ``amdgcn`` dGPU 132 ``gfx601`` - ``pitcairn`` ``amdgcn`` dGPU 133 - ``verde`` 134 ``gfx602`` - ``hainan`` ``amdgcn`` dGPU 135 - ``oland`` 136 **GCN GFX7 (Sea Islands (CI))** [AMD-GCN-GFX7]_ 137 ----------------------------------------------------------------------------------------------------------- 138 ``gfx700`` - ``kaveri`` ``amdgcn`` APU - A6-7000 139 - A6 Pro-7050B 140 - A8-7100 141 - A8 Pro-7150B 142 - A10-7300 143 - A10 Pro-7350B 144 - FX-7500 145 - A8-7200P 146 - A10-7400P 147 - FX-7600P 148 ``gfx701`` - ``hawaii`` ``amdgcn`` dGPU ROCm - FirePro W8100 149 - FirePro W9100 150 - FirePro S9150 151 - FirePro S9170 152 ``gfx702`` ``amdgcn`` dGPU ROCm - Radeon R9 290 153 - Radeon R9 290x 154 - Radeon R390 155 - Radeon R390x 156 ``gfx703`` - ``kabini`` ``amdgcn`` APU - E1-2100 157 - ``mullins`` - E1-2200 158 - E1-2500 159 - E2-3000 160 - E2-3800 161 - A4-5000 162 - A4-5100 163 - A6-5200 164 - A4 Pro-3340B 165 ``gfx704`` - ``bonaire`` ``amdgcn`` dGPU - Radeon HD 7790 166 - Radeon HD 8770 167 - R7 260 168 - R7 260X 169 ``gfx705`` ``amdgcn`` APU *TBA* 170 171 .. TODO:: 172 173 Add product 174 names. 175 176 **GCN GFX8 (Volcanic Islands (VI))** [AMD-GCN-GFX8]_ 177 ----------------------------------------------------------------------------------------------------------- 178 ``gfx801`` - ``carrizo`` ``amdgcn`` APU - xnack - A6-8500P 179 [on] - Pro A6-8500B 180 - A8-8600P 181 - Pro A8-8600B 182 - FX-8800P 183 - Pro A12-8800B 184 \ ``amdgcn`` APU - xnack ROCm - A10-8700P 185 [on] - Pro A10-8700B 186 - A10-8780P 187 \ ``amdgcn`` APU - xnack - A10-9600P 188 [on] - A10-9630P 189 - A12-9700P 190 - A12-9730P 191 - FX-9800P 192 - FX-9830P 193 \ ``amdgcn`` APU - xnack - E2-9010 194 [on] - A6-9210 195 - A9-9410 196 ``gfx802`` - ``iceland`` ``amdgcn`` dGPU - xnack ROCm - Radeon R285 197 - ``tonga`` [off] - Radeon R9 380 198 - Radeon R9 385 199 ``gfx803`` - ``fiji`` ``amdgcn`` dGPU - xnack ROCm - Radeon R9 Nano 200 [off] - Radeon R9 Fury 201 - Radeon R9 FuryX 202 - Radeon Pro Duo 203 - FirePro S9300x2 204 - Radeon Instinct MI8 205 \ - ``polaris10`` ``amdgcn`` dGPU - xnack ROCm - Radeon RX 470 206 [off] - Radeon RX 480 207 - Radeon Instinct MI6 208 \ - ``polaris11`` ``amdgcn`` dGPU - xnack ROCm - Radeon RX 460 209 [off] 210 ``gfx805`` - ``tongapro`` ``amdgcn`` dGPU - xnack ROCm - FirePro S7150 211 [off] - FirePro S7100 212 - FirePro W7100 213 - Mobile FirePro 214 M7170 215 ``gfx810`` - ``stoney`` ``amdgcn`` APU - xnack *TBA* 216 [on] 217 .. TODO:: 218 219 Add product 220 names. 221 222 **GCN GFX9** [AMD-GCN-GFX9]_ 223 ----------------------------------------------------------------------------------------------------------- 224 ``gfx900`` ``amdgcn`` dGPU - xnack ROCm - Radeon Vega 225 [off] Frontier Edition 226 - Radeon RX Vega 56 227 - Radeon RX Vega 64 228 - Radeon RX Vega 64 229 Liquid 230 - Radeon Instinct MI25 231 ``gfx902`` ``amdgcn`` APU - xnack - Ryzen 3 2200G 232 [on] - Ryzen 5 2400G 233 ``gfx904`` ``amdgcn`` dGPU - xnack *TBA* 234 [off] 235 .. TODO:: 236 237 Add product 238 names. 239 240 ``gfx906`` ``amdgcn`` dGPU - xnack - Radeon Instinct MI50 241 [off] - Radeon Instinct MI60 242 - sram-ecc - Radeon VII 243 [off] - Radeon Pro VII 244 ``gfx908`` ``amdgcn`` dGPU - xnack *TBA* 245 [off] 246 - sram-ecc 247 [on] 248 .. TODO:: 249 250 Add product 251 names. 252 253 ``gfx909`` ``amdgcn`` APU - xnack *TBA* 254 [off] 255 .. TODO:: 256 257 Add product 258 names. 259 260 **GCN GFX10** [AMD-GCN-GFX10]_ 261 ----------------------------------------------------------------------------------------------------------- 262 ``gfx1010`` ``amdgcn`` dGPU - xnack - Radeon RX 5700 263 [off] - Radeon RX 5700 XT 264 - wavefrontsize64 - Radeon Pro 5600 XT 265 [off] - Radeon Pro 5600M 266 - cumode 267 [off] 268 ``gfx1011`` ``amdgcn`` dGPU - xnack *TBA* 269 [off] 270 - wavefrontsize64 271 [off] 272 - cumode 273 [off] 274 .. TODO:: 275 276 Add product 277 names. 278 279 ``gfx1012`` ``amdgcn`` dGPU - xnack - Radeon RX 5500 280 [off] - Radeon RX 5500 XT 281 - wavefrontsize64 282 [off] 283 - cumode 284 [off] 285 ``gfx1030`` ``amdgcn`` dGPU - wavefrontsize64 *TBA* 286 [off] 287 - cumode 288 [off] 289 .. TODO:: 290 291 Add product 292 names. 293 294 ``gfx1031`` ``amdgcn`` dGPU - wavefrontsize64 *TBA* 295 [off] 296 - cumode 297 [off] 298 .. TODO:: 299 300 Add product 301 names. 302 303 ``gfx1032`` ``amdgcn`` dGPU - wavefrontsize64 *TBA* 304 [off] 305 - cumode 306 [off] 307 .. TODO:: 308 309 Add product 310 names. 311 312 =========== =============== ============ ===== ============================= ======= ====================== 313 314.. _amdgpu-target-features: 315 316Target Features 317--------------- 318 319Target features control how code is generated to support certain 320processor specific features. Not all target features are supported by 321all processors. The runtime must ensure that the features supported by 322the device used to execute the code match the features enabled when 323generating the code. A mismatch of features may result in incorrect 324execution, or a reduction in performance. 325 326The target features supported by each processor, and the default value 327used if not specified explicitly, is listed in 328:ref:`amdgpu-processor-table`. 329 330Use the ``clang -m[no-]<TargetFeature>`` option to specify the AMDGPU 331target features. 332 333For example: 334 335``-mxnack`` 336 Enable the ``xnack`` feature. 337``-mno-xnack`` 338 Disable the ``xnack`` feature. 339 340 .. table:: AMDGPU Target Features 341 :name: amdgpu-target-feature-table 342 343 ====================== ================================================== 344 Target Feature Description 345 ====================== ================================================== 346 -m[no-]xnack Enable/disable generating code that has 347 memory clauses that are compatible with 348 having XNACK replay enabled. 349 350 This is used for demand paging and page 351 migration. If XNACK replay is enabled in 352 the device, then if a page fault occurs 353 the code may execute incorrectly if the 354 ``xnack`` feature is not enabled. Executing 355 code that has the feature enabled on a 356 device that does not have XNACK replay 357 enabled will execute correctly but may 358 be less performant than code with the 359 feature disabled. 360 361 -m[no-]sram-ecc Enable/disable generating code that assumes SRAM 362 ECC is enabled/disabled. 363 364 -m[no-]wavefrontsize64 Control the default wavefront size used when 365 generating code for kernels. When disabled 366 native wavefront size 32 is used, when enabled 367 wavefront size 64 is used. 368 369 -m[no-]cumode Control the default wavefront execution mode used 370 when generating code for kernels. When disabled 371 native WGP wavefront execution mode is used, 372 when enabled CU wavefront execution mode is used 373 (see :ref:`amdgpu-amdhsa-memory-model`). 374 ====================== ================================================== 375 376.. _amdgpu-address-spaces: 377 378Address Spaces 379-------------- 380 381The AMDGPU architecture supports a number of memory address spaces. The address 382space names use the OpenCL standard names, with some additions. 383 384The AMDGPU address spaces correspond to target architecture specific LLVM 385address space numbers used in LLVM IR. 386 387The AMDGPU address spaces are described in 388:ref:`amdgpu-address-spaces-table`. Only 64-bit process address spaces are 389supported for the ``amdgcn`` target. 390 391 .. table:: AMDGPU Address Spaces 392 :name: amdgpu-address-spaces-table 393 394 ================================= =============== =========== ================ ======= ============================ 395 .. 64-Bit Process Address Space 396 --------------------------------- --------------- ----------- ---------------- ------------------------------------ 397 Address Space Name LLVM IR Address HSA Segment Hardware Address NULL Value 398 Space Number Name Name Size 399 ================================= =============== =========== ================ ======= ============================ 400 Generic 0 flat flat 64 0x0000000000000000 401 Global 1 global global 64 0x0000000000000000 402 Region 2 N/A GDS 32 *not implemented for AMDHSA* 403 Local 3 group LDS 32 0xFFFFFFFF 404 Constant 4 constant *same as global* 64 0x0000000000000000 405 Private 5 private scratch 32 0xFFFFFFFF 406 Constant 32-bit 6 *TODO* 0x00000000 407 Buffer Fat Pointer (experimental) 7 *TODO* 408 ================================= =============== =========== ================ ======= ============================ 409 410**Generic** 411 The generic address space uses the hardware flat address support available in 412 GFX7-GFX10. This uses two fixed ranges of virtual addresses (the private and 413 local apertures), that are outside the range of addressable global memory, to 414 map from a flat address to a private or local address. 415 416 FLAT instructions can take a flat address and access global, private 417 (scratch), and group (LDS) memory depending on if the address is within one 418 of the aperture ranges. Flat access to scratch requires hardware aperture 419 setup and setup in the kernel prologue (see 420 :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`). Flat access to LDS requires 421 hardware aperture setup and M0 (GFX7-GFX8) register setup (see 422 :ref:`amdgpu-amdhsa-kernel-prolog-m0`). 423 424 To convert between a private or group address space address (termed a segment 425 address) and a flat address the base address of the corresponding aperture 426 can be used. For GFX7-GFX8 these are available in the 427 :ref:`amdgpu-amdhsa-hsa-aql-queue` the address of which can be obtained with 428 Queue Ptr SGPR (see :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). For 429 GFX9-GFX10 the aperture base addresses are directly available as inline 430 constant registers ``SRC_SHARED_BASE/LIMIT`` and ``SRC_PRIVATE_BASE/LIMIT``. 431 In 64-bit address mode the aperture sizes are 2^32 bytes and the base is 432 aligned to 2^32 which makes it easier to convert from flat to segment or 433 segment to flat. 434 435 A global address space address has the same value when used as a flat address 436 so no conversion is needed. 437 438**Global and Constant** 439 The global and constant address spaces both use global virtual addresses, 440 which are the same virtual address space used by the CPU. However, some 441 virtual addresses may only be accessible to the CPU, some only accessible 442 by the GPU, and some by both. 443 444 Using the constant address space indicates that the data will not change 445 during the execution of the kernel. This allows scalar read instructions to 446 be used. The vector and scalar L1 caches are invalidated of volatile data 447 before each kernel dispatch execution to allow constant memory to change 448 values between kernel dispatches. 449 450**Region** 451 The region address space uses the hardware Global Data Store (GDS). All 452 wavefronts executing on the same device will access the same memory for any 453 given region address. However, the same region address accessed by wavefronts 454 executing on different devices will access different memory. It is higher 455 performance than global memory. It is allocated by the runtime. The data 456 store (DS) instructions can be used to access it. 457 458**Local** 459 The local address space uses the hardware Local Data Store (LDS) which is 460 automatically allocated when the hardware creates the wavefronts of a 461 work-group, and freed when all the wavefronts of a work-group have 462 terminated. All wavefronts belonging to the same work-group will access the 463 same memory for any given local address. However, the same local address 464 accessed by wavefronts belonging to different work-groups will access 465 different memory. It is higher performance than global memory. The data store 466 (DS) instructions can be used to access it. 467 468**Private** 469 The private address space uses the hardware scratch memory support which 470 automatically allocates memory when it creates a wavefront and frees it when 471 a wavefronts terminates. The memory accessed by a lane of a wavefront for any 472 given private address will be different to the memory accessed by another lane 473 of the same or different wavefront for the same private address. 474 475 If a kernel dispatch uses scratch, then the hardware allocates memory from a 476 pool of backing memory allocated by the runtime for each wavefront. The lanes 477 of the wavefront access this using dword (4 byte) interleaving. The mapping 478 used from private address to backing memory address is: 479 480 ``wavefront-scratch-base + 481 ((private-address / 4) * wavefront-size * 4) + 482 (wavefront-lane-id * 4) + (private-address % 4)`` 483 484 If each lane of a wavefront accesses the same private address, the 485 interleaving results in adjacent dwords being accessed and hence requires 486 fewer cache lines to be fetched. 487 488 There are different ways that the wavefront scratch base address is 489 determined by a wavefront (see 490 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 491 492 Scratch memory can be accessed in an interleaved manner using buffer 493 instructions with the scratch buffer descriptor and per wavefront scratch 494 offset, by the scratch instructions, or by flat instructions. Multi-dword 495 access is not supported except by flat and scratch instructions in 496 GFX9-GFX10. 497 498**Constant 32-bit** 499 *TODO* 500 501**Buffer Fat Pointer** 502 The buffer fat pointer is an experimental address space that is currently 503 unsupported in the backend. It exposes a non-integral pointer that is in 504 the future intended to support the modelling of 128-bit buffer descriptors 505 plus a 32-bit offset into the buffer (in total encapsulating a 160-bit 506 *pointer*), allowing normal LLVM load/store/atomic operations to be used to 507 model the buffer descriptors used heavily in graphics workloads targeting 508 the backend. 509 510.. _amdgpu-memory-scopes: 511 512Memory Scopes 513------------- 514 515This section provides LLVM memory synchronization scopes supported by the AMDGPU 516backend memory model when the target triple OS is ``amdhsa`` (see 517:ref:`amdgpu-amdhsa-memory-model` and :ref:`amdgpu-target-triples`). 518 519The memory model supported is based on the HSA memory model [HSA]_ which is 520based in turn on HRF-indirect with scope inclusion [HRF]_. The happens-before 521relation is transitive over the synchronizes-with relation independent of scope 522and synchronizes-with allows the memory scope instances to be inclusive (see 523table :ref:`amdgpu-amdhsa-llvm-sync-scopes-table`). 524 525This is different to the OpenCL [OpenCL]_ memory model which does not have scope 526inclusion and requires the memory scopes to exactly match. However, this 527is conservatively correct for OpenCL. 528 529 .. table:: AMDHSA LLVM Sync Scopes 530 :name: amdgpu-amdhsa-llvm-sync-scopes-table 531 532 ======================= =================================================== 533 LLVM Sync Scope Description 534 ======================= =================================================== 535 *none* The default: ``system``. 536 537 Synchronizes with, and participates in modification 538 and seq_cst total orderings with, other operations 539 (except image operations) for all address spaces 540 (except private, or generic that accesses private) 541 provided the other operation's sync scope is: 542 543 - ``system``. 544 - ``agent`` and executed by a thread on the same 545 agent. 546 - ``workgroup`` and executed by a thread in the 547 same work-group. 548 - ``wavefront`` and executed by a thread in the 549 same wavefront. 550 551 ``agent`` Synchronizes with, and participates in modification 552 and seq_cst total orderings with, other operations 553 (except image operations) for all address spaces 554 (except private, or generic that accesses private) 555 provided the other operation's sync scope is: 556 557 - ``system`` or ``agent`` and executed by a thread 558 on the same agent. 559 - ``workgroup`` and executed by a thread in the 560 same work-group. 561 - ``wavefront`` and executed by a thread in the 562 same wavefront. 563 564 ``workgroup`` Synchronizes with, and participates in modification 565 and seq_cst total orderings with, other operations 566 (except image operations) for all address spaces 567 (except private, or generic that accesses private) 568 provided the other operation's sync scope is: 569 570 - ``system``, ``agent`` or ``workgroup`` and 571 executed by a thread in the same work-group. 572 - ``wavefront`` and executed by a thread in the 573 same wavefront. 574 575 ``wavefront`` Synchronizes with, and participates in modification 576 and seq_cst total orderings with, other operations 577 (except image operations) for all address spaces 578 (except private, or generic that accesses private) 579 provided the other operation's sync scope is: 580 581 - ``system``, ``agent``, ``workgroup`` or 582 ``wavefront`` and executed by a thread in the 583 same wavefront. 584 585 ``singlethread`` Only synchronizes with and participates in 586 modification and seq_cst total orderings with, 587 other operations (except image operations) running 588 in the same thread for all address spaces (for 589 example, in signal handlers). 590 591 ``one-as`` Same as ``system`` but only synchronizes with other 592 operations within the same address space. 593 594 ``agent-one-as`` Same as ``agent`` but only synchronizes with other 595 operations within the same address space. 596 597 ``workgroup-one-as`` Same as ``workgroup`` but only synchronizes with 598 other operations within the same address space. 599 600 ``wavefront-one-as`` Same as ``wavefront`` but only synchronizes with 601 other operations within the same address space. 602 603 ``singlethread-one-as`` Same as ``singlethread`` but only synchronizes with 604 other operations within the same address space. 605 ======================= =================================================== 606 607LLVM IR Intrinsics 608------------------ 609 610The AMDGPU backend implements the following LLVM IR intrinsics. 611 612*This section is WIP.* 613 614.. TODO:: 615 616 List AMDGPU intrinsics. 617 618LLVM IR Attributes 619------------------ 620 621The AMDGPU backend supports the following LLVM IR attributes. 622 623 .. table:: AMDGPU LLVM IR Attributes 624 :name: amdgpu-llvm-ir-attributes-table 625 626 ======================================= ========================================================== 627 LLVM Attribute Description 628 ======================================= ========================================================== 629 "amdgpu-flat-work-group-size"="min,max" Specify the minimum and maximum flat work group sizes that 630 will be specified when the kernel is dispatched. Generated 631 by the ``amdgpu_flat_work_group_size`` CLANG attribute [CLANG-ATTR]_. 632 "amdgpu-implicitarg-num-bytes"="n" Number of kernel argument bytes to add to the kernel 633 argument block size for the implicit arguments. This 634 varies by OS and language (for OpenCL see 635 :ref:`opencl-kernel-implicit-arguments-appended-for-amdhsa-os-table`). 636 "amdgpu-num-sgpr"="n" Specifies the number of SGPRs to use. Generated by 637 the ``amdgpu_num_sgpr`` CLANG attribute [CLANG-ATTR]_. 638 "amdgpu-num-vgpr"="n" Specifies the number of VGPRs to use. Generated by the 639 ``amdgpu_num_vgpr`` CLANG attribute [CLANG-ATTR]_. 640 "amdgpu-waves-per-eu"="m,n" Specify the minimum and maximum number of waves per 641 execution unit. Generated by the ``amdgpu_waves_per_eu`` 642 CLANG attribute [CLANG-ATTR]_. 643 "amdgpu-ieee" true/false. Specify whether the function expects the IEEE field of the 644 mode register to be set on entry. Overrides the default for 645 the calling convention. 646 "amdgpu-dx10-clamp" true/false. Specify whether the function expects the DX10_CLAMP field of 647 the mode register to be set on entry. Overrides the default 648 for the calling convention. 649 ======================================= ========================================================== 650 651.. _amdgpu-elf-code-object: 652 653ELF Code Object 654=============== 655 656The AMDGPU backend generates a standard ELF [ELF]_ relocatable code object that 657can be linked by ``lld`` to produce a standard ELF shared code object which can 658be loaded and executed on an AMDGPU target. 659 660.. _amdgpu-elf-header: 661 662Header 663------ 664 665The AMDGPU backend uses the following ELF header: 666 667 .. table:: AMDGPU ELF Header 668 :name: amdgpu-elf-header-table 669 670 ========================== =============================== 671 Field Value 672 ========================== =============================== 673 ``e_ident[EI_CLASS]`` ``ELFCLASS64`` 674 ``e_ident[EI_DATA]`` ``ELFDATA2LSB`` 675 ``e_ident[EI_OSABI]`` - ``ELFOSABI_NONE`` 676 - ``ELFOSABI_AMDGPU_HSA`` 677 - ``ELFOSABI_AMDGPU_PAL`` 678 - ``ELFOSABI_AMDGPU_MESA3D`` 679 ``e_ident[EI_ABIVERSION]`` - ``ELFABIVERSION_AMDGPU_HSA`` 680 - ``ELFABIVERSION_AMDGPU_PAL`` 681 - ``ELFABIVERSION_AMDGPU_MESA3D`` 682 ``e_type`` - ``ET_REL`` 683 - ``ET_DYN`` 684 ``e_machine`` ``EM_AMDGPU`` 685 ``e_entry`` 0 686 ``e_flags`` See :ref:`amdgpu-elf-header-e_flags-table` 687 ========================== =============================== 688 689.. 690 691 .. table:: AMDGPU ELF Header Enumeration Values 692 :name: amdgpu-elf-header-enumeration-values-table 693 694 =============================== ===== 695 Name Value 696 =============================== ===== 697 ``EM_AMDGPU`` 224 698 ``ELFOSABI_NONE`` 0 699 ``ELFOSABI_AMDGPU_HSA`` 64 700 ``ELFOSABI_AMDGPU_PAL`` 65 701 ``ELFOSABI_AMDGPU_MESA3D`` 66 702 ``ELFABIVERSION_AMDGPU_HSA`` 1 703 ``ELFABIVERSION_AMDGPU_PAL`` 0 704 ``ELFABIVERSION_AMDGPU_MESA3D`` 0 705 =============================== ===== 706 707``e_ident[EI_CLASS]`` 708 The ELF class is: 709 710 * ``ELFCLASS32`` for ``r600`` architecture. 711 712 * ``ELFCLASS64`` for ``amdgcn`` architecture which only supports 64-bit 713 process address space applications. 714 715``e_ident[EI_DATA]`` 716 All AMDGPU targets use ``ELFDATA2LSB`` for little-endian byte ordering. 717 718``e_ident[EI_OSABI]`` 719 One of the following AMDGPU target architecture specific OS ABIs 720 (see :ref:`amdgpu-os-table`): 721 722 * ``ELFOSABI_NONE`` for *unknown* OS. 723 724 * ``ELFOSABI_AMDGPU_HSA`` for ``amdhsa`` OS. 725 726 * ``ELFOSABI_AMDGPU_PAL`` for ``amdpal`` OS. 727 728 * ``ELFOSABI_AMDGPU_MESA3D`` for ``mesa3D`` OS. 729 730``e_ident[EI_ABIVERSION]`` 731 The ABI version of the AMDGPU target architecture specific OS ABI to which the code 732 object conforms: 733 734 * ``ELFABIVERSION_AMDGPU_HSA`` is used to specify the version of AMD HSA 735 runtime ABI. 736 737 * ``ELFABIVERSION_AMDGPU_PAL`` is used to specify the version of AMD PAL 738 runtime ABI. 739 740 * ``ELFABIVERSION_AMDGPU_MESA3D`` is used to specify the version of AMD MESA 741 3D runtime ABI. 742 743``e_type`` 744 Can be one of the following values: 745 746 747 ``ET_REL`` 748 The type produced by the AMDGPU backend compiler as it is relocatable code 749 object. 750 751 ``ET_DYN`` 752 The type produced by the linker as it is a shared code object. 753 754 The AMD HSA runtime loader requires a ``ET_DYN`` code object. 755 756``e_machine`` 757 The value ``EM_AMDGPU`` is used for the machine for all processors supported 758 by the ``r600`` and ``amdgcn`` architectures (see 759 :ref:`amdgpu-processor-table`). The specific processor is specified in the 760 ``EF_AMDGPU_MACH`` bit field of the ``e_flags`` (see 761 :ref:`amdgpu-elf-header-e_flags-table`). 762 763``e_entry`` 764 The entry point is 0 as the entry points for individual kernels must be 765 selected in order to invoke them through AQL packets. 766 767``e_flags`` 768 The AMDGPU backend uses the following ELF header flags: 769 770 .. table:: AMDGPU ELF Header ``e_flags`` 771 :name: amdgpu-elf-header-e_flags-table 772 773 ================================= ========== ============================= 774 Name Value Description 775 ================================= ========== ============================= 776 **AMDGPU Processor Flag** See :ref:`amdgpu-processor-table`. 777 -------------------------------------------- ----------------------------- 778 ``EF_AMDGPU_MACH`` 0x000000ff AMDGPU processor selection 779 mask for 780 ``EF_AMDGPU_MACH_xxx`` values 781 defined in 782 :ref:`amdgpu-ef-amdgpu-mach-table`. 783 ``EF_AMDGPU_XNACK`` 0x00000100 Indicates if the ``xnack`` 784 target feature is 785 enabled for all code 786 contained in the code object. 787 If the processor 788 does not support the 789 ``xnack`` target 790 feature then must 791 be 0. 792 See 793 :ref:`amdgpu-target-features`. 794 ``EF_AMDGPU_SRAM_ECC`` 0x00000200 Indicates if the ``sram-ecc`` 795 target feature is 796 enabled for all code 797 contained in the code object. 798 If the processor 799 does not support the 800 ``sram-ecc`` target 801 feature then must 802 be 0. 803 See 804 :ref:`amdgpu-target-features`. 805 ================================= ========== ============================= 806 807 .. table:: AMDGPU ``EF_AMDGPU_MACH`` Values 808 :name: amdgpu-ef-amdgpu-mach-table 809 810 ==================================== ========== ============================= 811 Name Value Description (see 812 :ref:`amdgpu-processor-table`) 813 ==================================== ========== ============================= 814 ``EF_AMDGPU_MACH_NONE`` 0x000 *not specified* 815 ``EF_AMDGPU_MACH_R600_R600`` 0x001 ``r600`` 816 ``EF_AMDGPU_MACH_R600_R630`` 0x002 ``r630`` 817 ``EF_AMDGPU_MACH_R600_RS880`` 0x003 ``rs880`` 818 ``EF_AMDGPU_MACH_R600_RV670`` 0x004 ``rv670`` 819 ``EF_AMDGPU_MACH_R600_RV710`` 0x005 ``rv710`` 820 ``EF_AMDGPU_MACH_R600_RV730`` 0x006 ``rv730`` 821 ``EF_AMDGPU_MACH_R600_RV770`` 0x007 ``rv770`` 822 ``EF_AMDGPU_MACH_R600_CEDAR`` 0x008 ``cedar`` 823 ``EF_AMDGPU_MACH_R600_CYPRESS`` 0x009 ``cypress`` 824 ``EF_AMDGPU_MACH_R600_JUNIPER`` 0x00a ``juniper`` 825 ``EF_AMDGPU_MACH_R600_REDWOOD`` 0x00b ``redwood`` 826 ``EF_AMDGPU_MACH_R600_SUMO`` 0x00c ``sumo`` 827 ``EF_AMDGPU_MACH_R600_BARTS`` 0x00d ``barts`` 828 ``EF_AMDGPU_MACH_R600_CAICOS`` 0x00e ``caicos`` 829 ``EF_AMDGPU_MACH_R600_CAYMAN`` 0x00f ``cayman`` 830 ``EF_AMDGPU_MACH_R600_TURKS`` 0x010 ``turks`` 831 *reserved* 0x011 - Reserved for ``r600`` 832 0x01f architecture processors. 833 ``EF_AMDGPU_MACH_AMDGCN_GFX600`` 0x020 ``gfx600`` 834 ``EF_AMDGPU_MACH_AMDGCN_GFX601`` 0x021 ``gfx601`` 835 ``EF_AMDGPU_MACH_AMDGCN_GFX700`` 0x022 ``gfx700`` 836 ``EF_AMDGPU_MACH_AMDGCN_GFX701`` 0x023 ``gfx701`` 837 ``EF_AMDGPU_MACH_AMDGCN_GFX702`` 0x024 ``gfx702`` 838 ``EF_AMDGPU_MACH_AMDGCN_GFX703`` 0x025 ``gfx703`` 839 ``EF_AMDGPU_MACH_AMDGCN_GFX704`` 0x026 ``gfx704`` 840 *reserved* 0x027 Reserved. 841 ``EF_AMDGPU_MACH_AMDGCN_GFX801`` 0x028 ``gfx801`` 842 ``EF_AMDGPU_MACH_AMDGCN_GFX802`` 0x029 ``gfx802`` 843 ``EF_AMDGPU_MACH_AMDGCN_GFX803`` 0x02a ``gfx803`` 844 ``EF_AMDGPU_MACH_AMDGCN_GFX810`` 0x02b ``gfx810`` 845 ``EF_AMDGPU_MACH_AMDGCN_GFX900`` 0x02c ``gfx900`` 846 ``EF_AMDGPU_MACH_AMDGCN_GFX902`` 0x02d ``gfx902`` 847 ``EF_AMDGPU_MACH_AMDGCN_GFX904`` 0x02e ``gfx904`` 848 ``EF_AMDGPU_MACH_AMDGCN_GFX906`` 0x02f ``gfx906`` 849 ``EF_AMDGPU_MACH_AMDGCN_GFX908`` 0x030 ``gfx908`` 850 ``EF_AMDGPU_MACH_AMDGCN_GFX909`` 0x031 ``gfx909`` 851 *reserved* 0x032 Reserved. 852 ``EF_AMDGPU_MACH_AMDGCN_GFX1010`` 0x033 ``gfx1010`` 853 ``EF_AMDGPU_MACH_AMDGCN_GFX1011`` 0x034 ``gfx1011`` 854 ``EF_AMDGPU_MACH_AMDGCN_GFX1012`` 0x035 ``gfx1012`` 855 ``EF_AMDGPU_MACH_AMDGCN_GFX1030`` 0x036 ``gfx1030`` 856 ``EF_AMDGPU_MACH_AMDGCN_GFX1031`` 0x037 ``gfx1031`` 857 ``EF_AMDGPU_MACH_AMDGCN_GFX1032`` 0x038 ``gfx1032`` 858 *reserved* 0x039 Reserved. 859 ``EF_AMDGPU_MACH_AMDGCN_GFX602`` 0x03a ``gfx602`` 860 ``EF_AMDGPU_MACH_AMDGCN_GFX705`` 0x03b ``gfx705`` 861 ``EF_AMDGPU_MACH_AMDGCN_GFX805`` 0x03c ``gfx805`` 862 ==================================== ========== ============================= 863 864Sections 865-------- 866 867An AMDGPU target ELF code object has the standard ELF sections which include: 868 869 .. table:: AMDGPU ELF Sections 870 :name: amdgpu-elf-sections-table 871 872 ================== ================ ================================= 873 Name Type Attributes 874 ================== ================ ================================= 875 ``.bss`` ``SHT_NOBITS`` ``SHF_ALLOC`` + ``SHF_WRITE`` 876 ``.data`` ``SHT_PROGBITS`` ``SHF_ALLOC`` + ``SHF_WRITE`` 877 ``.debug_``\ *\** ``SHT_PROGBITS`` *none* 878 ``.dynamic`` ``SHT_DYNAMIC`` ``SHF_ALLOC`` 879 ``.dynstr`` ``SHT_PROGBITS`` ``SHF_ALLOC`` 880 ``.dynsym`` ``SHT_PROGBITS`` ``SHF_ALLOC`` 881 ``.got`` ``SHT_PROGBITS`` ``SHF_ALLOC`` + ``SHF_WRITE`` 882 ``.hash`` ``SHT_HASH`` ``SHF_ALLOC`` 883 ``.note`` ``SHT_NOTE`` *none* 884 ``.rela``\ *name* ``SHT_RELA`` *none* 885 ``.rela.dyn`` ``SHT_RELA`` *none* 886 ``.rodata`` ``SHT_PROGBITS`` ``SHF_ALLOC`` 887 ``.shstrtab`` ``SHT_STRTAB`` *none* 888 ``.strtab`` ``SHT_STRTAB`` *none* 889 ``.symtab`` ``SHT_SYMTAB`` *none* 890 ``.text`` ``SHT_PROGBITS`` ``SHF_ALLOC`` + ``SHF_EXECINSTR`` 891 ================== ================ ================================= 892 893These sections have their standard meanings (see [ELF]_) and are only generated 894if needed. 895 896``.debug``\ *\** 897 The standard DWARF sections. See :ref:`amdgpu-dwarf-debug-information` for 898 information on the DWARF produced by the AMDGPU backend. 899 900``.dynamic``, ``.dynstr``, ``.dynsym``, ``.hash`` 901 The standard sections used by a dynamic loader. 902 903``.note`` 904 See :ref:`amdgpu-note-records` for the note records supported by the AMDGPU 905 backend. 906 907``.rela``\ *name*, ``.rela.dyn`` 908 For relocatable code objects, *name* is the name of the section that the 909 relocation records apply. For example, ``.rela.text`` is the section name for 910 relocation records associated with the ``.text`` section. 911 912 For linked shared code objects, ``.rela.dyn`` contains all the relocation 913 records from each of the relocatable code object's ``.rela``\ *name* sections. 914 915 See :ref:`amdgpu-relocation-records` for the relocation records supported by 916 the AMDGPU backend. 917 918``.text`` 919 The executable machine code for the kernels and functions they call. Generated 920 as position independent code. See :ref:`amdgpu-code-conventions` for 921 information on conventions used in the isa generation. 922 923.. _amdgpu-note-records: 924 925Note Records 926------------ 927 928The AMDGPU backend code object contains ELF note records in the ``.note`` 929section. The set of generated notes and their semantics depend on the code 930object version; see :ref:`amdgpu-note-records-v2` and 931:ref:`amdgpu-note-records-v3`. 932 933As required by ``ELFCLASS32`` and ``ELFCLASS64``, minimal zero-byte padding 934must be generated after the ``name`` field to ensure the ``desc`` field is 4 935byte aligned. In addition, minimal zero-byte padding must be generated to 936ensure the ``desc`` field size is a multiple of 4 bytes. The ``sh_addralign`` 937field of the ``.note`` section must be at least 4 to indicate at least 8 byte 938alignment. 939 940.. _amdgpu-note-records-v2: 941 942Code Object V2 Note Records (--amdhsa-code-object-version=2) 943~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 944 945.. warning:: Code Object V2 is not the default code object version emitted by 946 this version of LLVM. For a description of the notes generated with the 947 default configuration (Code Object V3) see :ref:`amdgpu-note-records-v3`. 948 949The AMDGPU backend code object uses the following ELF note record in the 950``.note`` section when compiling for Code Object V2 951(--amdhsa-code-object-version=2). 952 953Additional note records may be present, but any which are not documented here 954are deprecated and should not be used. 955 956 .. table:: AMDGPU Code Object V2 ELF Note Records 957 :name: amdgpu-elf-note-records-table-v2 958 959 ===== ============================== ====================================== 960 Name Type Description 961 ===== ============================== ====================================== 962 "AMD" ``NT_AMD_AMDGPU_HSA_METADATA`` <metadata null terminated string> 963 ===== ============================== ====================================== 964 965.. 966 967 .. table:: AMDGPU Code Object V2 ELF Note Record Enumeration Values 968 :name: amdgpu-elf-note-record-enumeration-values-table-v2 969 970 ============================== ===== 971 Name Value 972 ============================== ===== 973 *reserved* 0-9 974 ``NT_AMD_AMDGPU_HSA_METADATA`` 10 975 *reserved* 11 976 ============================== ===== 977 978``NT_AMD_AMDGPU_HSA_METADATA`` 979 Specifies extensible metadata associated with the code objects executed on HSA 980 [HSA]_ compatible runtimes such as AMD's ROCm [AMD-ROCm]_. It is required when 981 the target triple OS is ``amdhsa`` (see :ref:`amdgpu-target-triples`). See 982 :ref:`amdgpu-amdhsa-code-object-metadata-v2` for the syntax of the code 983 object metadata string. 984 985.. _amdgpu-note-records-v3: 986 987Code Object V3 Note Records (--amdhsa-code-object-version=3) 988~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 989 990The AMDGPU backend code object uses the following ELF note record in the 991``.note`` section when compiling for Code Object V3 992(--amdhsa-code-object-version=3). 993 994Additional note records may be present, but any which are not documented here 995are deprecated and should not be used. 996 997 .. table:: AMDGPU Code Object V3 ELF Note Records 998 :name: amdgpu-elf-note-records-table-v3 999 1000 ======== ============================== ====================================== 1001 Name Type Description 1002 ======== ============================== ====================================== 1003 "AMDGPU" ``NT_AMDGPU_METADATA`` Metadata in Message Pack [MsgPack]_ 1004 binary format. 1005 ======== ============================== ====================================== 1006 1007.. 1008 1009 .. table:: AMDGPU Code Object V3 ELF Note Record Enumeration Values 1010 :name: amdgpu-elf-note-record-enumeration-values-table-v3 1011 1012 ============================== ===== 1013 Name Value 1014 ============================== ===== 1015 *reserved* 0-31 1016 ``NT_AMDGPU_METADATA`` 32 1017 ============================== ===== 1018 1019``NT_AMDGPU_METADATA`` 1020 Specifies extensible metadata associated with an AMDGPU code 1021 object. It is encoded as a map in the Message Pack [MsgPack]_ binary 1022 data format. See :ref:`amdgpu-amdhsa-code-object-metadata-v3` for the 1023 map keys defined for the ``amdhsa`` OS. 1024 1025.. _amdgpu-symbols: 1026 1027Symbols 1028------- 1029 1030Symbols include the following: 1031 1032 .. table:: AMDGPU ELF Symbols 1033 :name: amdgpu-elf-symbols-table 1034 1035 ===================== ================== ================ ================== 1036 Name Type Section Description 1037 ===================== ================== ================ ================== 1038 *link-name* ``STT_OBJECT`` - ``.data`` Global variable 1039 - ``.rodata`` 1040 - ``.bss`` 1041 *link-name*\ ``.kd`` ``STT_OBJECT`` - ``.rodata`` Kernel descriptor 1042 *link-name* ``STT_FUNC`` - ``.text`` Kernel entry point 1043 *link-name* ``STT_OBJECT`` - SHN_AMDGPU_LDS Global variable in LDS 1044 ===================== ================== ================ ================== 1045 1046Global variable 1047 Global variables both used and defined by the compilation unit. 1048 1049 If the symbol is defined in the compilation unit then it is allocated in the 1050 appropriate section according to if it has initialized data or is readonly. 1051 1052 If the symbol is external then its section is ``STN_UNDEF`` and the loader 1053 will resolve relocations using the definition provided by another code object 1054 or explicitly defined by the runtime. 1055 1056 If the symbol resides in local/group memory (LDS) then its section is the 1057 special processor specific section name ``SHN_AMDGPU_LDS``, and the 1058 ``st_value`` field describes alignment requirements as it does for common 1059 symbols. 1060 1061 .. TODO:: 1062 1063 Add description of linked shared object symbols. Seems undefined symbols 1064 are marked as STT_NOTYPE. 1065 1066Kernel descriptor 1067 Every HSA kernel has an associated kernel descriptor. It is the address of the 1068 kernel descriptor that is used in the AQL dispatch packet used to invoke the 1069 kernel, not the kernel entry point. The layout of the HSA kernel descriptor is 1070 defined in :ref:`amdgpu-amdhsa-kernel-descriptor`. 1071 1072Kernel entry point 1073 Every HSA kernel also has a symbol for its machine code entry point. 1074 1075.. _amdgpu-relocation-records: 1076 1077Relocation Records 1078------------------ 1079 1080AMDGPU backend generates ``Elf64_Rela`` relocation records. Supported 1081relocatable fields are: 1082 1083``word32`` 1084 This specifies a 32-bit field occupying 4 bytes with arbitrary byte 1085 alignment. These values use the same byte order as other word values in the 1086 AMDGPU architecture. 1087 1088``word64`` 1089 This specifies a 64-bit field occupying 8 bytes with arbitrary byte 1090 alignment. These values use the same byte order as other word values in the 1091 AMDGPU architecture. 1092 1093Following notations are used for specifying relocation calculations: 1094 1095**A** 1096 Represents the addend used to compute the value of the relocatable field. 1097 1098**G** 1099 Represents the offset into the global offset table at which the relocation 1100 entry's symbol will reside during execution. 1101 1102**GOT** 1103 Represents the address of the global offset table. 1104 1105**P** 1106 Represents the place (section offset for ``et_rel`` or address for ``et_dyn``) 1107 of the storage unit being relocated (computed using ``r_offset``). 1108 1109**S** 1110 Represents the value of the symbol whose index resides in the relocation 1111 entry. Relocations not using this must specify a symbol index of 1112 ``STN_UNDEF``. 1113 1114**B** 1115 Represents the base address of a loaded executable or shared object which is 1116 the difference between the ELF address and the actual load address. 1117 Relocations using this are only valid in executable or shared objects. 1118 1119The following relocation types are supported: 1120 1121 .. table:: AMDGPU ELF Relocation Records 1122 :name: amdgpu-elf-relocation-records-table 1123 1124 ========================== ======= ===== ========== ============================== 1125 Relocation Type Kind Value Field Calculation 1126 ========================== ======= ===== ========== ============================== 1127 ``R_AMDGPU_NONE`` 0 *none* *none* 1128 ``R_AMDGPU_ABS32_LO`` Static, 1 ``word32`` (S + A) & 0xFFFFFFFF 1129 Dynamic 1130 ``R_AMDGPU_ABS32_HI`` Static, 2 ``word32`` (S + A) >> 32 1131 Dynamic 1132 ``R_AMDGPU_ABS64`` Static, 3 ``word64`` S + A 1133 Dynamic 1134 ``R_AMDGPU_REL32`` Static 4 ``word32`` S + A - P 1135 ``R_AMDGPU_REL64`` Static 5 ``word64`` S + A - P 1136 ``R_AMDGPU_ABS32`` Static, 6 ``word32`` S + A 1137 Dynamic 1138 ``R_AMDGPU_GOTPCREL`` Static 7 ``word32`` G + GOT + A - P 1139 ``R_AMDGPU_GOTPCREL32_LO`` Static 8 ``word32`` (G + GOT + A - P) & 0xFFFFFFFF 1140 ``R_AMDGPU_GOTPCREL32_HI`` Static 9 ``word32`` (G + GOT + A - P) >> 32 1141 ``R_AMDGPU_REL32_LO`` Static 10 ``word32`` (S + A - P) & 0xFFFFFFFF 1142 ``R_AMDGPU_REL32_HI`` Static 11 ``word32`` (S + A - P) >> 32 1143 *reserved* 12 1144 ``R_AMDGPU_RELATIVE64`` Dynamic 13 ``word64`` B + A 1145 ========================== ======= ===== ========== ============================== 1146 1147``R_AMDGPU_ABS32_LO`` and ``R_AMDGPU_ABS32_HI`` are only supported by 1148the ``mesa3d`` OS, which does not support ``R_AMDGPU_ABS64``. 1149 1150There is no current OS loader support for 32-bit programs and so 1151``R_AMDGPU_ABS32`` is not used. 1152 1153.. _amdgpu-loaded-code-object-path-uniform-resource-identifier: 1154 1155Loaded Code Object Path Uniform Resource Identifier (URI) 1156--------------------------------------------------------- 1157 1158The AMD GPU code object loader represents the path of the ELF shared object from 1159which the code object was loaded as a textual Unifom Resource Identifier (URI). 1160Note that the code object is the in memory loaded relocated form of the ELF 1161shared object. Multiple code objects may be loaded at different memory 1162addresses in the same process from the same ELF shared object. 1163 1164The loaded code object path URI syntax is defined by the following BNF syntax: 1165 1166.. code:: 1167 1168 code_object_uri ::== file_uri | memory_uri 1169 file_uri ::== "file://" file_path [ range_specifier ] 1170 memory_uri ::== "memory://" process_id range_specifier 1171 range_specifier ::== [ "#" | "?" ] "offset=" number "&" "size=" number 1172 file_path ::== URI_ENCODED_OS_FILE_PATH 1173 process_id ::== DECIMAL_NUMBER 1174 number ::== HEX_NUMBER | DECIMAL_NUMBER | OCTAL_NUMBER 1175 1176**number** 1177 Is a C integral literal where hexadecimal values are prefixed by "0x" or "0X", 1178 and octal values by "0". 1179 1180**file_path** 1181 Is the file's path specified as a URI encoded UTF-8 string. In URI encoding, 1182 every character that is not in the regular expression ``[a-zA-Z0-9/_.~-]`` is 1183 encoded as two uppercase hexadecimal digits proceeded by "%". Directories in 1184 the path are separated by "/". 1185 1186**offset** 1187 Is a 0-based byte offset to the start of the code object. For a file URI, it 1188 is from the start of the file specified by the ``file_path``, and if omitted 1189 defaults to 0. For a memory URI, it is the memory address and is required. 1190 1191**size** 1192 Is the number of bytes in the code object. For a file URI, if omitted it 1193 defaults to the size of the file. It is required for a memory URI. 1194 1195**process_id** 1196 Is the identity of the process owning the memory. For Linux it is the C 1197 unsigned integral decimal literal for the process ID (PID). 1198 1199For example: 1200 1201.. code:: 1202 1203 file:///dir1/dir2/file1 1204 file:///dir3/dir4/file2#offset=0x2000&size=3000 1205 memory://1234#offset=0x20000&size=3000 1206 1207.. _amdgpu-dwarf-debug-information: 1208 1209DWARF Debug Information 1210======================= 1211 1212.. warning:: 1213 1214 This section describes **provisional support** for AMDGPU DWARF [DWARF]_ that 1215 is not currently fully implemented and is subject to change. 1216 1217AMDGPU generates DWARF [DWARF]_ debugging information ELF sections (see 1218:ref:`amdgpu-elf-code-object`) which contain information that maps the code 1219object executable code and data to the source language constructs. It can be 1220used by tools such as debuggers and profilers. It uses features defined in 1221:doc:`AMDGPUDwarfExtensionsForHeterogeneousDebugging` that are made available in 1222DWARF Version 4 and DWARF Version 5 as an LLVM vendor extension. 1223 1224This section defines the AMDGPU target architecture specific DWARF mappings. 1225 1226.. _amdgpu-dwarf-register-identifier: 1227 1228Register Identifier 1229------------------- 1230 1231This section defines the AMDGPU target architecture register numbers used in 1232DWARF operation expressions (see DWARF Version 5 section 2.5 and 1233:ref:`amdgpu-dwarf-operation-expressions`) and Call Frame Information 1234instructions (see DWARF Version 5 section 6.4 and 1235:ref:`amdgpu-dwarf-call-frame-information`). 1236 1237A single code object can contain code for kernels that have different wavefront 1238sizes. The vector registers and some scalar registers are based on the wavefront 1239size. AMDGPU defines distinct DWARF registers for each wavefront size. This 1240simplifies the consumer of the DWARF so that each register has a fixed size, 1241rather than being dynamic according to the wavefront size mode. Similarly, 1242distinct DWARF registers are defined for those registers that vary in size 1243according to the process address size. This allows a consumer to treat a 1244specific AMDGPU processor as a single architecture regardless of how it is 1245configured at run time. The compiler explicitly specifies the DWARF registers 1246that match the mode in which the code it is generating will be executed. 1247 1248DWARF registers are encoded as numbers, which are mapped to architecture 1249registers. The mapping for AMDGPU is defined in 1250:ref:`amdgpu-dwarf-register-mapping-table`. All AMDGPU targets use the same 1251mapping. 1252 1253.. table:: AMDGPU DWARF Register Mapping 1254 :name: amdgpu-dwarf-register-mapping-table 1255 1256 ============== ================= ======== ================================== 1257 DWARF Register AMDGPU Register Bit Size Description 1258 ============== ================= ======== ================================== 1259 0 PC_32 32 Program Counter (PC) when 1260 executing in a 32-bit process 1261 address space. Used in the CFI to 1262 describe the PC of the calling 1263 frame. 1264 1 EXEC_MASK_32 32 Execution Mask Register when 1265 executing in wavefront 32 mode. 1266 2-15 *Reserved* *Reserved for highly accessed 1267 registers using DWARF shortcut.* 1268 16 PC_64 64 Program Counter (PC) when 1269 executing in a 64-bit process 1270 address space. Used in the CFI to 1271 describe the PC of the calling 1272 frame. 1273 17 EXEC_MASK_64 64 Execution Mask Register when 1274 executing in wavefront 64 mode. 1275 18-31 *Reserved* *Reserved for highly accessed 1276 registers using DWARF shortcut.* 1277 32-95 SGPR0-SGPR63 32 Scalar General Purpose 1278 Registers. 1279 96-127 *Reserved* *Reserved for frequently accessed 1280 registers using DWARF 1-byte ULEB.* 1281 128 STATUS 32 Status Register. 1282 129-511 *Reserved* *Reserved for future Scalar 1283 Architectural Registers.* 1284 512 VCC_32 32 Vector Condition Code Register 1285 when executing in wavefront 32 1286 mode. 1287 513-1023 *Reserved* *Reserved for future Vector 1288 Architectural Registers when 1289 executing in wavefront 32 mode.* 1290 768 VCC_64 64 Vector Condition Code Register 1291 when executing in wavefront 64 1292 mode. 1293 769-1023 *Reserved* *Reserved for future Vector 1294 Architectural Registers when 1295 executing in wavefront 64 mode.* 1296 1024-1087 *Reserved* *Reserved for padding.* 1297 1088-1129 SGPR64-SGPR105 32 Scalar General Purpose Registers. 1298 1130-1535 *Reserved* *Reserved for future Scalar 1299 General Purpose Registers.* 1300 1536-1791 VGPR0-VGPR255 32*32 Vector General Purpose Registers 1301 when executing in wavefront 32 1302 mode. 1303 1792-2047 *Reserved* *Reserved for future Vector 1304 General Purpose Registers when 1305 executing in wavefront 32 mode.* 1306 2048-2303 AGPR0-AGPR255 32*32 Vector Accumulation Registers 1307 when executing in wavefront 32 1308 mode. 1309 2304-2559 *Reserved* *Reserved for future Vector 1310 Accumulation Registers when 1311 executing in wavefront 32 mode.* 1312 2560-2815 VGPR0-VGPR255 64*32 Vector General Purpose Registers 1313 when executing in wavefront 64 1314 mode. 1315 2816-3071 *Reserved* *Reserved for future Vector 1316 General Purpose Registers when 1317 executing in wavefront 64 mode.* 1318 3072-3327 AGPR0-AGPR255 64*32 Vector Accumulation Registers 1319 when executing in wavefront 64 1320 mode. 1321 3328-3583 *Reserved* *Reserved for future Vector 1322 Accumulation Registers when 1323 executing in wavefront 64 mode.* 1324 ============== ================= ======== ================================== 1325 1326The vector registers are represented as the full size for the wavefront. They 1327are organized as consecutive dwords (32-bits), one per lane, with the dword at 1328the least significant bit position corresponding to lane 0 and so forth. DWARF 1329location expressions involving the ``DW_OP_LLVM_offset`` and 1330``DW_OP_LLVM_push_lane`` operations are used to select the part of the vector 1331register corresponding to the lane that is executing the current thread of 1332execution in languages that are implemented using a SIMD or SIMT execution 1333model. 1334 1335If the wavefront size is 32 lanes then the wavefront 32 mode register 1336definitions are used. If the wavefront size is 64 lanes then the wavefront 64 1337mode register definitions are used. Some AMDGPU targets support executing in 1338both wavefront 32 and wavefront 64 mode. The register definitions corresponding 1339to the wavefront mode of the generated code will be used. 1340 1341If code is generated to execute in a 32-bit process address space, then the 134232-bit process address space register definitions are used. If code is generated 1343to execute in a 64-bit process address space, then the 64-bit process address 1344space register definitions are used. The ``amdgcn`` target only supports the 134564-bit process address space. 1346 1347.. _amdgpu-dwarf-address-class-identifier: 1348 1349Address Class Identifier 1350------------------------ 1351 1352The DWARF address class represents the source language memory space. See DWARF 1353Version 5 section 2.12 which is updated by the *DWARF Extensions For 1354Heterogeneous Debugging* section :ref:`amdgpu-dwarf-segment_addresses`. 1355 1356The DWARF address class mapping used for AMDGPU is defined in 1357:ref:`amdgpu-dwarf-address-class-mapping-table`. 1358 1359.. table:: AMDGPU DWARF Address Class Mapping 1360 :name: amdgpu-dwarf-address-class-mapping-table 1361 1362 ========================= ====== ================= 1363 DWARF AMDGPU 1364 -------------------------------- ----------------- 1365 Address Class Name Value Address Space 1366 ========================= ====== ================= 1367 ``DW_ADDR_none`` 0x0000 Generic (Flat) 1368 ``DW_ADDR_LLVM_global`` 0x0001 Global 1369 ``DW_ADDR_LLVM_constant`` 0x0002 Global 1370 ``DW_ADDR_LLVM_group`` 0x0003 Local (group/LDS) 1371 ``DW_ADDR_LLVM_private`` 0x0004 Private (Scratch) 1372 ``DW_ADDR_AMDGPU_region`` 0x8000 Region (GDS) 1373 ========================= ====== ================= 1374 1375The DWARF address class values defined in the *DWARF Extensions For 1376Heterogeneous Debugging* section :ref:`amdgpu-dwarf-segment_addresses` are used. 1377 1378In addition, ``DW_ADDR_AMDGPU_region`` is encoded as a vendor extension. This is 1379available for use for the AMD extension for access to the hardware GDS memory 1380which is scratchpad memory allocated per device. 1381 1382For AMDGPU if no ``DW_AT_address_class`` attribute is present, then the default 1383address class of ``DW_ADDR_none`` is used. 1384 1385See :ref:`amdgpu-dwarf-address-space-identifier` for information on the AMDGPU 1386mapping of DWARF address classes to DWARF address spaces, including address size 1387and NULL value. 1388 1389.. _amdgpu-dwarf-address-space-identifier: 1390 1391Address Space Identifier 1392------------------------ 1393 1394DWARF address spaces correspond to target architecture specific linear 1395addressable memory areas. See DWARF Version 5 section 2.12 and *DWARF Extensions 1396For Heterogeneous Debugging* section :ref:`amdgpu-dwarf-segment_addresses`. 1397 1398The DWARF address space mapping used for AMDGPU is defined in 1399:ref:`amdgpu-dwarf-address-space-mapping-table`. 1400 1401.. table:: AMDGPU DWARF Address Space Mapping 1402 :name: amdgpu-dwarf-address-space-mapping-table 1403 1404 ======================================= ===== ======= ======== ================= ======================= 1405 DWARF AMDGPU Notes 1406 --------------------------------------- ----- ---------------- ----------------- ----------------------- 1407 Address Space Name Value Address Bit Size Address Space 1408 --------------------------------------- ----- ------- -------- ----------------- ----------------------- 1409 .. 64-bit 32-bit 1410 process process 1411 address address 1412 space space 1413 ======================================= ===== ======= ======== ================= ======================= 1414 ``DW_ASPACE_none`` 0x00 8 4 Global *default address space* 1415 ``DW_ASPACE_AMDGPU_generic`` 0x01 8 4 Generic (Flat) 1416 ``DW_ASPACE_AMDGPU_region`` 0x02 4 4 Region (GDS) 1417 ``DW_ASPACE_AMDGPU_local`` 0x03 4 4 Local (group/LDS) 1418 *Reserved* 0x04 1419 ``DW_ASPACE_AMDGPU_private_lane`` 0x05 4 4 Private (Scratch) *focused lane* 1420 ``DW_ASPACE_AMDGPU_private_wave`` 0x06 4 4 Private (Scratch) *unswizzled wavefront* 1421 ======================================= ===== ======= ======== ================= ======================= 1422 1423See :ref:`amdgpu-address-spaces` for information on the AMDGPU address spaces 1424including address size and NULL value. 1425 1426The ``DW_ASPACE_none`` address space is the default target architecture address 1427space used in DWARF operations that do not specify an address space. It 1428therefore has to map to the global address space so that the ``DW_OP_addr*`` and 1429related operations can refer to addresses in the program code. 1430 1431The ``DW_ASPACE_AMDGPU_generic`` address space allows location expressions to 1432specify the flat address space. If the address corresponds to an address in the 1433local address space, then it corresponds to the wavefront that is executing the 1434focused thread of execution. If the address corresponds to an address in the 1435private address space, then it corresponds to the lane that is executing the 1436focused thread of execution for languages that are implemented using a SIMD or 1437SIMT execution model. 1438 1439.. note:: 1440 1441 CUDA-like languages such as HIP that do not have address spaces in the 1442 language type system, but do allow variables to be allocated in different 1443 address spaces, need to explicitly specify the ``DW_ASPACE_AMDGPU_generic`` 1444 address space in the DWARF expression operations as the default address space 1445 is the global address space. 1446 1447The ``DW_ASPACE_AMDGPU_local`` address space allows location expressions to 1448specify the local address space corresponding to the wavefront that is executing 1449the focused thread of execution. 1450 1451The ``DW_ASPACE_AMDGPU_private_lane`` address space allows location expressions 1452to specify the private address space corresponding to the lane that is executing 1453the focused thread of execution for languages that are implemented using a SIMD 1454or SIMT execution model. 1455 1456The ``DW_ASPACE_AMDGPU_private_wave`` address space allows location expressions 1457to specify the unswizzled private address space corresponding to the wavefront 1458that is executing the focused thread of execution. The wavefront view of private 1459memory is the per wavefront unswizzled backing memory layout defined in 1460:ref:`amdgpu-address-spaces`, such that address 0 corresponds to the first 1461location for the backing memory of the wavefront (namely the address is not 1462offset by ``wavefront-scratch-base``). The following formula can be used to 1463convert from a ``DW_ASPACE_AMDGPU_private_lane`` address to a 1464``DW_ASPACE_AMDGPU_private_wave`` address: 1465 1466:: 1467 1468 private-address-wavefront = 1469 ((private-address-lane / 4) * wavefront-size * 4) + 1470 (wavefront-lane-id * 4) + (private-address-lane % 4) 1471 1472If the ``DW_ASPACE_AMDGPU_private_lane`` address is dword aligned, and the start 1473of the dwords for each lane starting with lane 0 is required, then this 1474simplifies to: 1475 1476:: 1477 1478 private-address-wavefront = 1479 private-address-lane * wavefront-size 1480 1481A compiler can use the ``DW_ASPACE_AMDGPU_private_wave`` address space to read a 1482complete spilled vector register back into a complete vector register in the 1483CFI. The frame pointer can be a private lane address which is dword aligned, 1484which can be shifted to multiply by the wavefront size, and then used to form a 1485private wavefront address that gives a location for a contiguous set of dwords, 1486one per lane, where the vector register dwords are spilled. The compiler knows 1487the wavefront size since it generates the code. Note that the type of the 1488address may have to be converted as the size of a 1489``DW_ASPACE_AMDGPU_private_lane`` address may be smaller than the size of a 1490``DW_ASPACE_AMDGPU_private_wave`` address. 1491 1492.. _amdgpu-dwarf-lane-identifier: 1493 1494Lane identifier 1495--------------- 1496 1497DWARF lane identifies specify a target architecture lane position for hardware 1498that executes in a SIMD or SIMT manner, and on which a source language maps its 1499threads of execution onto those lanes. The DWARF lane identifier is pushed by 1500the ``DW_OP_LLVM_push_lane`` DWARF expression operation. See DWARF Version 5 1501section 2.5 which is updated by *DWARF Extensions For Heterogeneous Debugging* 1502section :ref:`amdgpu-dwarf-operation-expressions`. 1503 1504For AMDGPU, the lane identifier corresponds to the hardware lane ID of a 1505wavefront. It is numbered from 0 to the wavefront size minus 1. 1506 1507Operation Expressions 1508--------------------- 1509 1510DWARF expressions are used to compute program values and the locations of 1511program objects. See DWARF Version 5 section 2.5 and 1512:ref:`amdgpu-dwarf-operation-expressions`. 1513 1514DWARF location descriptions describe how to access storage which includes memory 1515and registers. When accessing storage on AMDGPU, bytes are ordered with least 1516significant bytes first, and bits are ordered within bytes with least 1517significant bits first. 1518 1519For AMDGPU CFI expressions, ``DW_OP_LLVM_select_bit_piece`` is used to describe 1520unwinding vector registers that are spilled under the execution mask to memory: 1521the zero-single location description is the vector register, and the one-single 1522location description is the spilled memory location description. The 1523``DW_OP_LLVM_form_aspace_address`` is used to specify the address space of the 1524memory location description. 1525 1526In AMDGPU expressions, ``DW_OP_LLVM_select_bit_piece`` is used by the 1527``DW_AT_LLVM_lane_pc`` attribute expression where divergent control flow is 1528controlled by the execution mask. An undefined location description together 1529with ``DW_OP_LLVM_extend`` is used to indicate the lane was not active on entry 1530to the subprogram. See :ref:`amdgpu-dwarf-dw-at-llvm-lane-pc` for an example. 1531 1532Debugger Information Entry Attributes 1533------------------------------------- 1534 1535This section describes how certain debugger information entry attributes are 1536used by AMDGPU. See the sections in DWARF Version 5 section 2 which are updated 1537by *DWARF Extensions For Heterogeneous Debugging* section 1538:ref:`amdgpu-dwarf-debugging-information-entry-attributes`. 1539 1540.. _amdgpu-dwarf-dw-at-llvm-lane-pc: 1541 1542``DW_AT_LLVM_lane_pc`` 1543~~~~~~~~~~~~~~~~~~~~~~ 1544 1545For AMDGPU, the ``DW_AT_LLVM_lane_pc`` attribute is used to specify the program 1546location of the separate lanes of a SIMT thread. 1547 1548If the lane is an active lane then this will be the same as the current program 1549location. 1550 1551If the lane is inactive, but was active on entry to the subprogram, then this is 1552the program location in the subprogram at which execution of the lane is 1553conceptual positioned. 1554 1555If the lane was not active on entry to the subprogram, then this will be the 1556undefined location. A client debugger can check if the lane is part of a valid 1557work-group by checking that the lane is in the range of the associated 1558work-group within the grid, accounting for partial work-groups. If it is not, 1559then the debugger can omit any information for the lane. Otherwise, the debugger 1560may repeatedly unwind the stack and inspect the ``DW_AT_LLVM_lane_pc`` of the 1561calling subprogram until it finds a non-undefined location. Conceptually the 1562lane only has the call frames that it has a non-undefined 1563``DW_AT_LLVM_lane_pc``. 1564 1565The following example illustrates how the AMDGPU backend can generate a DWARF 1566location list expression for the nested ``IF/THEN/ELSE`` structures of the 1567following subprogram pseudo code for a target with 64 lanes per wavefront. 1568 1569.. code:: 1570 :number-lines: 1571 1572 SUBPROGRAM X 1573 BEGIN 1574 a; 1575 IF (c1) THEN 1576 b; 1577 IF (c2) THEN 1578 c; 1579 ELSE 1580 d; 1581 ENDIF 1582 e; 1583 ELSE 1584 f; 1585 ENDIF 1586 g; 1587 END 1588 1589The AMDGPU backend may generate the following pseudo LLVM MIR to manipulate the 1590execution mask (``EXEC``) to linearize the control flow. The condition is 1591evaluated to make a mask of the lanes for which the condition evaluates to true. 1592First the ``THEN`` region is executed by setting the ``EXEC`` mask to the 1593logical ``AND`` of the current ``EXEC`` mask with the condition mask. Then the 1594``ELSE`` region is executed by negating the ``EXEC`` mask and logical ``AND`` of 1595the saved ``EXEC`` mask at the start of the region. After the ``IF/THEN/ELSE`` 1596region the ``EXEC`` mask is restored to the value it had at the beginning of the 1597region. This is shown below. Other approaches are possible, but the basic 1598concept is the same. 1599 1600.. code:: 1601 :number-lines: 1602 1603 $lex_start: 1604 a; 1605 %1 = EXEC 1606 %2 = c1 1607 $lex_1_start: 1608 EXEC = %1 & %2 1609 $if_1_then: 1610 b; 1611 %3 = EXEC 1612 %4 = c2 1613 $lex_1_1_start: 1614 EXEC = %3 & %4 1615 $lex_1_1_then: 1616 c; 1617 EXEC = ~EXEC & %3 1618 $lex_1_1_else: 1619 d; 1620 EXEC = %3 1621 $lex_1_1_end: 1622 e; 1623 EXEC = ~EXEC & %1 1624 $lex_1_else: 1625 f; 1626 EXEC = %1 1627 $lex_1_end: 1628 g; 1629 $lex_end: 1630 1631To create the DWARF location list expression that defines the location 1632description of a vector of lane program locations, the LLVM MIR ``DBG_VALUE`` 1633pseudo instruction can be used to annotate the linearized control flow. This can 1634be done by defining an artificial variable for the lane PC. The DWARF location 1635list expression created for it is used as the value of the 1636``DW_AT_LLVM_lane_pc`` attribute on the subprogram's debugger information entry. 1637 1638A DWARF procedure is defined for each well nested structured control flow region 1639which provides the conceptual lane program location for a lane if it is not 1640active (namely it is divergent). The DWARF operation expression for each region 1641conceptually inherits the value of the immediately enclosing region and modifies 1642it according to the semantics of the region. 1643 1644For an ``IF/THEN/ELSE`` region the divergent program location is at the start of 1645the region for the ``THEN`` region since it is executed first. For the ``ELSE`` 1646region the divergent program location is at the end of the ``IF/THEN/ELSE`` 1647region since the ``THEN`` region has completed. 1648 1649The lane PC artificial variable is assigned at each region transition. It uses 1650the immediately enclosing region's DWARF procedure to compute the program 1651location for each lane assuming they are divergent, and then modifies the result 1652by inserting the current program location for each lane that the ``EXEC`` mask 1653indicates is active. 1654 1655By having separate DWARF procedures for each region, they can be reused to 1656define the value for any nested region. This reduces the total size of the DWARF 1657operation expressions. 1658 1659The following provides an example using pseudo LLVM MIR. 1660 1661.. code:: 1662 :number-lines: 1663 1664 $lex_start: 1665 DEFINE_DWARF %__uint_64 = DW_TAG_base_type[ 1666 DW_AT_name = "__uint64"; 1667 DW_AT_byte_size = 8; 1668 DW_AT_encoding = DW_ATE_unsigned; 1669 ]; 1670 DEFINE_DWARF %__active_lane_pc = DW_TAG_dwarf_procedure[ 1671 DW_AT_name = "__active_lane_pc"; 1672 DW_AT_location = [ 1673 DW_OP_regx PC; 1674 DW_OP_LLVM_extend 64, 64; 1675 DW_OP_regval_type EXEC, %uint_64; 1676 DW_OP_LLVM_select_bit_piece 64, 64; 1677 ]; 1678 ]; 1679 DEFINE_DWARF %__divergent_lane_pc = DW_TAG_dwarf_procedure[ 1680 DW_AT_name = "__divergent_lane_pc"; 1681 DW_AT_location = [ 1682 DW_OP_LLVM_undefined; 1683 DW_OP_LLVM_extend 64, 64; 1684 ]; 1685 ]; 1686 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 1687 DW_OP_call_ref %__divergent_lane_pc; 1688 DW_OP_call_ref %__active_lane_pc; 1689 ]; 1690 a; 1691 %1 = EXEC; 1692 DBG_VALUE %1, $noreg, %__lex_1_save_exec; 1693 %2 = c1; 1694 $lex_1_start: 1695 EXEC = %1 & %2; 1696 $lex_1_then: 1697 DEFINE_DWARF %__divergent_lane_pc_1_then = DW_TAG_dwarf_procedure[ 1698 DW_AT_name = "__divergent_lane_pc_1_then"; 1699 DW_AT_location = DIExpression[ 1700 DW_OP_call_ref %__divergent_lane_pc; 1701 DW_OP_addrx &lex_1_start; 1702 DW_OP_stack_value; 1703 DW_OP_LLVM_extend 64, 64; 1704 DW_OP_call_ref %__lex_1_save_exec; 1705 DW_OP_deref_type 64, %__uint_64; 1706 DW_OP_LLVM_select_bit_piece 64, 64; 1707 ]; 1708 ]; 1709 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 1710 DW_OP_call_ref %__divergent_lane_pc_1_then; 1711 DW_OP_call_ref %__active_lane_pc; 1712 ]; 1713 b; 1714 %3 = EXEC; 1715 DBG_VALUE %3, %__lex_1_1_save_exec; 1716 %4 = c2; 1717 $lex_1_1_start: 1718 EXEC = %3 & %4; 1719 $lex_1_1_then: 1720 DEFINE_DWARF %__divergent_lane_pc_1_1_then = DW_TAG_dwarf_procedure[ 1721 DW_AT_name = "__divergent_lane_pc_1_1_then"; 1722 DW_AT_location = DIExpression[ 1723 DW_OP_call_ref %__divergent_lane_pc_1_then; 1724 DW_OP_addrx &lex_1_1_start; 1725 DW_OP_stack_value; 1726 DW_OP_LLVM_extend 64, 64; 1727 DW_OP_call_ref %__lex_1_1_save_exec; 1728 DW_OP_deref_type 64, %__uint_64; 1729 DW_OP_LLVM_select_bit_piece 64, 64; 1730 ]; 1731 ]; 1732 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 1733 DW_OP_call_ref %__divergent_lane_pc_1_1_then; 1734 DW_OP_call_ref %__active_lane_pc; 1735 ]; 1736 c; 1737 EXEC = ~EXEC & %3; 1738 $lex_1_1_else: 1739 DEFINE_DWARF %__divergent_lane_pc_1_1_else = DW_TAG_dwarf_procedure[ 1740 DW_AT_name = "__divergent_lane_pc_1_1_else"; 1741 DW_AT_location = DIExpression[ 1742 DW_OP_call_ref %__divergent_lane_pc_1_then; 1743 DW_OP_addrx &lex_1_1_end; 1744 DW_OP_stack_value; 1745 DW_OP_LLVM_extend 64, 64; 1746 DW_OP_call_ref %__lex_1_1_save_exec; 1747 DW_OP_deref_type 64, %__uint_64; 1748 DW_OP_LLVM_select_bit_piece 64, 64; 1749 ]; 1750 ]; 1751 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 1752 DW_OP_call_ref %__divergent_lane_pc_1_1_else; 1753 DW_OP_call_ref %__active_lane_pc; 1754 ]; 1755 d; 1756 EXEC = %3; 1757 $lex_1_1_end: 1758 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 1759 DW_OP_call_ref %__divergent_lane_pc; 1760 DW_OP_call_ref %__active_lane_pc; 1761 ]; 1762 e; 1763 EXEC = ~EXEC & %1; 1764 $lex_1_else: 1765 DEFINE_DWARF %__divergent_lane_pc_1_else = DW_TAG_dwarf_procedure[ 1766 DW_AT_name = "__divergent_lane_pc_1_else"; 1767 DW_AT_location = DIExpression[ 1768 DW_OP_call_ref %__divergent_lane_pc; 1769 DW_OP_addrx &lex_1_end; 1770 DW_OP_stack_value; 1771 DW_OP_LLVM_extend 64, 64; 1772 DW_OP_call_ref %__lex_1_save_exec; 1773 DW_OP_deref_type 64, %__uint_64; 1774 DW_OP_LLVM_select_bit_piece 64, 64; 1775 ]; 1776 ]; 1777 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 1778 DW_OP_call_ref %__divergent_lane_pc_1_else; 1779 DW_OP_call_ref %__active_lane_pc; 1780 ]; 1781 f; 1782 EXEC = %1; 1783 $lex_1_end: 1784 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc DIExpression[ 1785 DW_OP_call_ref %__divergent_lane_pc; 1786 DW_OP_call_ref %__active_lane_pc; 1787 ]; 1788 g; 1789 $lex_end: 1790 1791The DWARF procedure ``%__active_lane_pc`` is used to update the lane pc elements 1792that are active, with the current program location. 1793 1794Artificial variables %__lex_1_save_exec and %__lex_1_1_save_exec are created for 1795the execution masks saved on entry to a region. Using the ``DBG_VALUE`` pseudo 1796instruction, location list entries will be created that describe where the 1797artificial variables are allocated at any given program location. The compiler 1798may allocate them to registers or spill them to memory. 1799 1800The DWARF procedures for each region use the values of the saved execution mask 1801artificial variables to only update the lanes that are active on entry to the 1802region. All other lanes retain the value of the enclosing region where they were 1803last active. If they were not active on entry to the subprogram, then will have 1804the undefined location description. 1805 1806Other structured control flow regions can be handled similarly. For example, 1807loops would set the divergent program location for the region at the end of the 1808loop. Any lanes active will be in the loop, and any lanes not active must have 1809exited the loop. 1810 1811An ``IF/THEN/ELSEIF/ELSEIF/...`` region can be treated as a nest of 1812``IF/THEN/ELSE`` regions. 1813 1814The DWARF procedures can use the active lane artificial variable described in 1815:ref:`amdgpu-dwarf-amdgpu-dw-at-llvm-active-lane` rather than the actual 1816``EXEC`` mask in order to support whole or quad wavefront mode. 1817 1818.. _amdgpu-dwarf-amdgpu-dw-at-llvm-active-lane: 1819 1820``DW_AT_LLVM_active_lane`` 1821~~~~~~~~~~~~~~~~~~~~~~~~~~ 1822 1823The ``DW_AT_LLVM_active_lane`` attribute on a subprogram debugger information 1824entry is used to specify the lanes that are conceptually active for a SIMT 1825thread. 1826 1827The execution mask may be modified to implement whole or quad wavefront mode 1828operations. For example, all lanes may need to temporarily be made active to 1829execute a whole wavefront operation. Such regions would save the ``EXEC`` mask, 1830update it to enable the necessary lanes, perform the operations, and then 1831restore the ``EXEC`` mask from the saved value. While executing the whole 1832wavefront region, the conceptual execution mask is the saved value, not the 1833``EXEC`` value. 1834 1835This is handled by defining an artificial variable for the active lane mask. The 1836active lane mask artificial variable would be the actual ``EXEC`` mask for 1837normal regions, and the saved execution mask for regions where the mask is 1838temporarily updated. The location list expression created for this artificial 1839variable is used to define the value of the ``DW_AT_LLVM_active_lane`` 1840attribute. 1841 1842``DW_AT_LLVM_augmentation`` 1843~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1844 1845For AMDGPU, the ``DW_AT_LLVM_augmentation`` attribute of a compilation unit 1846debugger information entry has the following value for the augmentation string: 1847 1848:: 1849 1850 [amdgpu:v0.0] 1851 1852The "vX.Y" specifies the major X and minor Y version number of the AMDGPU 1853extensions used in the DWARF of the compilation unit. The version number 1854conforms to [SEMVER]_. 1855 1856Call Frame Information 1857---------------------- 1858 1859DWARF Call Frame Information (CFI) describes how a consumer can virtually 1860*unwind* call frames in a running process or core dump. See DWARF Version 5 1861section 6.4 and :ref:`amdgpu-dwarf-call-frame-information`. 1862 1863For AMDGPU, the Common Information Entry (CIE) fields have the following values: 1864 18651. ``augmentation`` string contains the following null-terminated UTF-8 string: 1866 1867 :: 1868 1869 [amd:v0.0] 1870 1871 The ``vX.Y`` specifies the major X and minor Y version number of the AMDGPU 1872 extensions used in this CIE or to the FDEs that use it. The version number 1873 conforms to [SEMVER]_. 1874 18752. ``address_size`` for the ``Global`` address space is defined in 1876 :ref:`amdgpu-dwarf-address-space-identifier`. 1877 18783. ``segment_selector_size`` is 0 as AMDGPU does not use a segment selector. 1879 18804. ``code_alignment_factor`` is 4 bytes. 1881 1882 .. TODO:: 1883 1884 Add to :ref:`amdgpu-processor-table` table. 1885 18865. ``data_alignment_factor`` is 4 bytes. 1887 1888 .. TODO:: 1889 1890 Add to :ref:`amdgpu-processor-table` table. 1891 18926. ``return_address_register`` is ``PC_32`` for 32-bit processes and ``PC_64`` 1893 for 64-bit processes defined in :ref:`amdgpu-dwarf-register-identifier`. 1894 18957. ``initial_instructions`` Since a subprogram X with fewer registers can be 1896 called from subprogram Y that has more allocated, X will not change any of 1897 the extra registers as it cannot access them. Therefore, the default rule 1898 for all columns is ``same value``. 1899 1900For AMDGPU the register number follows the numbering defined in 1901:ref:`amdgpu-dwarf-register-identifier`. 1902 1903For AMDGPU the instructions are variable size. A consumer can subtract 1 from 1904the return address to get the address of a byte within the call site 1905instructions. See DWARF Version 5 section 6.4.4. 1906 1907Accelerated Access 1908------------------ 1909 1910See DWARF Version 5 section 6.1. 1911 1912Lookup By Name Section Header 1913~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1914 1915See DWARF Version 5 section 6.1.1.4.1 and :ref:`amdgpu-dwarf-lookup-by-name`. 1916 1917For AMDGPU the lookup by name section header table: 1918 1919``augmentation_string_size`` (uword) 1920 1921 Set to the length of the ``augmentation_string`` value which is always a 1922 multiple of 4. 1923 1924``augmentation_string`` (sequence of UTF-8 characters) 1925 1926 Contains the following UTF-8 string null padded to a multiple of 4 bytes: 1927 1928 :: 1929 1930 [amdgpu:v0.0] 1931 1932 The "vX.Y" specifies the major X and minor Y version number of the AMDGPU 1933 extensions used in the DWARF of this index. The version number conforms to 1934 [SEMVER]_. 1935 1936 .. note:: 1937 1938 This is different to the DWARF Version 5 definition that requires the first 1939 4 characters to be the vendor ID. But this is consistent with the other 1940 augmentation strings and does allow multiple vendor contributions. However, 1941 backwards compatibility may be more desirable. 1942 1943Lookup By Address Section Header 1944~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1945 1946See DWARF Version 5 section 6.1.2. 1947 1948For AMDGPU the lookup by address section header table: 1949 1950``address_size`` (ubyte) 1951 1952 Match the address size for the ``Global`` address space defined in 1953 :ref:`amdgpu-dwarf-address-space-identifier`. 1954 1955``segment_selector_size`` (ubyte) 1956 1957 AMDGPU does not use a segment selector so this is 0. The entries in the 1958 ``.debug_aranges`` do not have a segment selector. 1959 1960Line Number Information 1961----------------------- 1962 1963See DWARF Version 5 section 6.2 and :ref:`amdgpu-dwarf-line-number-information`. 1964 1965AMDGPU does not use the ``isa`` state machine registers and always sets it to 0. 1966The instruction set must be obtained from the ELF file header ``e_flags`` field 1967in the ``EF_AMDGPU_MACH`` bit position (see :ref:`ELF Header 1968<amdgpu-elf-header>`). See DWARF Version 5 section 6.2.2. 1969 1970.. TODO:: 1971 1972 Should the ``isa`` state machine register be used to indicate if the code is 1973 in wavefront32 or wavefront64 mode? Or used to specify the architecture ISA? 1974 1975For AMDGPU the line number program header fields have the following values (see 1976DWARF Version 5 section 6.2.4): 1977 1978``address_size`` (ubyte) 1979 Matches the address size for the ``Global`` address space defined in 1980 :ref:`amdgpu-dwarf-address-space-identifier`. 1981 1982``segment_selector_size`` (ubyte) 1983 AMDGPU does not use a segment selector so this is 0. 1984 1985``minimum_instruction_length`` (ubyte) 1986 For GFX9-GFX10 this is 4. 1987 1988``maximum_operations_per_instruction`` (ubyte) 1989 For GFX9-GFX10 this is 1. 1990 1991Source text for online-compiled programs (for example, those compiled by the 1992OpenCL language runtime) may be embedded into the DWARF Version 5 line table. 1993See DWARF Version 5 section 6.2.4.1 which is updated by *DWARF Extensions For 1994Heterogeneous Debugging* section :ref:`DW_LNCT_LLVM_source 1995<amdgpu-dwarf-line-number-information-dw-lnct-llvm-source>`. 1996 1997The Clang option used to control source embedding in AMDGPU is defined in 1998:ref:`amdgpu-clang-debug-options-table`. 1999 2000 .. table:: AMDGPU Clang Debug Options 2001 :name: amdgpu-clang-debug-options-table 2002 2003 ==================== ================================================== 2004 Debug Flag Description 2005 ==================== ================================================== 2006 -g[no-]embed-source Enable/disable embedding source text in DWARF 2007 debug sections. Useful for environments where 2008 source cannot be written to disk, such as 2009 when performing online compilation. 2010 ==================== ================================================== 2011 2012For example: 2013 2014``-gembed-source`` 2015 Enable the embedded source. 2016 2017``-gno-embed-source`` 2018 Disable the embedded source. 2019 202032-Bit and 64-Bit DWARF Formats 2021------------------------------- 2022 2023See DWARF Version 5 section 7.4 and 2024:ref:`amdgpu-dwarf-32-bit-and-64-bit-dwarf-formats`. 2025 2026For AMDGPU: 2027 2028* For the ``amdgcn`` target architecture only the 64-bit process address space 2029 is supported. 2030 2031* The producer can generate either 32-bit or 64-bit DWARF format. LLVM generates 2032 the 32-bit DWARF format. 2033 2034Unit Headers 2035------------ 2036 2037For AMDGPU the following values apply for each of the unit headers described in 2038DWARF Version 5 sections 7.5.1.1, 7.5.1.2, and 7.5.1.3: 2039 2040``address_size`` (ubyte) 2041 Matches the address size for the ``Global`` address space defined in 2042 :ref:`amdgpu-dwarf-address-space-identifier`. 2043 2044.. _amdgpu-code-conventions: 2045 2046Code Conventions 2047================ 2048 2049This section provides code conventions used for each supported target triple OS 2050(see :ref:`amdgpu-target-triples`). 2051 2052AMDHSA 2053------ 2054 2055This section provides code conventions used when the target triple OS is 2056``amdhsa`` (see :ref:`amdgpu-target-triples`). 2057 2058.. _amdgpu-amdhsa-code-object-target-identification: 2059 2060Code Object Target Identification 2061~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2062 2063The AMDHSA OS uses the following syntax to specify the code object 2064target as a single string: 2065 2066 ``<Architecture>-<Vendor>-<OS>-<Environment>-<Processor><Target Features>`` 2067 2068Where: 2069 2070 - ``<Architecture>``, ``<Vendor>``, ``<OS>`` and ``<Environment>`` 2071 are the same as the *Target Triple* (see 2072 :ref:`amdgpu-target-triples`). 2073 2074 - ``<Processor>`` is the same as the *Processor* (see 2075 :ref:`amdgpu-processors`). 2076 2077 - ``<Target Features>`` is a list of the enabled *Target Features* 2078 (see :ref:`amdgpu-target-features`), each prefixed by a plus, that 2079 apply to *Processor*. The list must be in the same order as listed 2080 in the table :ref:`amdgpu-target-feature-table`. Note that *Target 2081 Features* must be included in the list if they are enabled even if 2082 that is the default for *Processor*. 2083 2084For example: 2085 2086 ``"amdgcn-amd-amdhsa--gfx902+xnack"`` 2087 2088.. _amdgpu-amdhsa-code-object-metadata: 2089 2090Code Object Metadata 2091~~~~~~~~~~~~~~~~~~~~ 2092 2093The code object metadata specifies extensible metadata associated with the code 2094objects executed on HSA [HSA]_ compatible runtimes such as AMD's ROCm 2095[AMD-ROCm]_. The encoding and semantics of this metadata depends on the code 2096object version; see :ref:`amdgpu-amdhsa-code-object-metadata-v2` and 2097:ref:`amdgpu-amdhsa-code-object-metadata-v3`. 2098 2099Code object metadata is specified in a note record (see 2100:ref:`amdgpu-note-records`) and is required when the target triple OS is 2101``amdhsa`` (see :ref:`amdgpu-target-triples`). It must contain the minimum 2102information necessary to support the ROCM kernel queries. For example, the 2103segment sizes needed in a dispatch packet. In addition, a high-level language 2104runtime may require other information to be included. For example, the AMD 2105OpenCL runtime records kernel argument information. 2106 2107.. _amdgpu-amdhsa-code-object-metadata-v2: 2108 2109Code Object V2 Metadata (--amdhsa-code-object-version=2) 2110++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 2111 2112.. warning:: Code Object V2 is not the default code object version emitted by 2113 this version of LLVM. For a description of the metadata generated with the 2114 default configuration (Code Object V3) see 2115 :ref:`amdgpu-amdhsa-code-object-metadata-v3`. 2116 2117Code object V2 metadata is specified by the ``NT_AMD_AMDGPU_METADATA`` note 2118record (see :ref:`amdgpu-note-records-v2`). 2119 2120The metadata is specified as a YAML formatted string (see [YAML]_ and 2121:doc:`YamlIO`). 2122 2123.. TODO:: 2124 2125 Is the string null terminated? It probably should not if YAML allows it to 2126 contain null characters, otherwise it should be. 2127 2128The metadata is represented as a single YAML document comprised of the mapping 2129defined in table :ref:`amdgpu-amdhsa-code-object-metadata-map-table-v2` and 2130referenced tables. 2131 2132For boolean values, the string values of ``false`` and ``true`` are used for 2133false and true respectively. 2134 2135Additional information can be added to the mappings. To avoid conflicts, any 2136non-AMD key names should be prefixed by "*vendor-name*.". 2137 2138 .. table:: AMDHSA Code Object V2 Metadata Map 2139 :name: amdgpu-amdhsa-code-object-metadata-map-table-v2 2140 2141 ========== ============== ========= ======================================= 2142 String Key Value Type Required? Description 2143 ========== ============== ========= ======================================= 2144 "Version" sequence of Required - The first integer is the major 2145 2 integers version. Currently 1. 2146 - The second integer is the minor 2147 version. Currently 0. 2148 "Printf" sequence of Each string is encoded information 2149 strings about a printf function call. The 2150 encoded information is organized as 2151 fields separated by colon (':'): 2152 2153 ``ID:N:S[0]:S[1]:...:S[N-1]:FormatString`` 2154 2155 where: 2156 2157 ``ID`` 2158 A 32-bit integer as a unique id for 2159 each printf function call 2160 2161 ``N`` 2162 A 32-bit integer equal to the number 2163 of arguments of printf function call 2164 minus 1 2165 2166 ``S[i]`` (where i = 0, 1, ... , N-1) 2167 32-bit integers for the size in bytes 2168 of the i-th FormatString argument of 2169 the printf function call 2170 2171 FormatString 2172 The format string passed to the 2173 printf function call. 2174 "Kernels" sequence of Required Sequence of the mappings for each 2175 mapping kernel in the code object. See 2176 :ref:`amdgpu-amdhsa-code-object-kernel-metadata-map-table-v2` 2177 for the definition of the mapping. 2178 ========== ============== ========= ======================================= 2179 2180.. 2181 2182 .. table:: AMDHSA Code Object V2 Kernel Metadata Map 2183 :name: amdgpu-amdhsa-code-object-kernel-metadata-map-table-v2 2184 2185 ================= ============== ========= ================================ 2186 String Key Value Type Required? Description 2187 ================= ============== ========= ================================ 2188 "Name" string Required Source name of the kernel. 2189 "SymbolName" string Required Name of the kernel 2190 descriptor ELF symbol. 2191 "Language" string Source language of the kernel. 2192 Values include: 2193 2194 - "OpenCL C" 2195 - "OpenCL C++" 2196 - "HCC" 2197 - "OpenMP" 2198 2199 "LanguageVersion" sequence of - The first integer is the major 2200 2 integers version. 2201 - The second integer is the 2202 minor version. 2203 "Attrs" mapping Mapping of kernel attributes. 2204 See 2205 :ref:`amdgpu-amdhsa-code-object-kernel-attribute-metadata-map-table-v2` 2206 for the mapping definition. 2207 "Args" sequence of Sequence of mappings of the 2208 mapping kernel arguments. See 2209 :ref:`amdgpu-amdhsa-code-object-kernel-argument-metadata-map-table-v2` 2210 for the definition of the mapping. 2211 "CodeProps" mapping Mapping of properties related to 2212 the kernel code. See 2213 :ref:`amdgpu-amdhsa-code-object-kernel-code-properties-metadata-map-table-v2` 2214 for the mapping definition. 2215 ================= ============== ========= ================================ 2216 2217.. 2218 2219 .. table:: AMDHSA Code Object V2 Kernel Attribute Metadata Map 2220 :name: amdgpu-amdhsa-code-object-kernel-attribute-metadata-map-table-v2 2221 2222 =================== ============== ========= ============================== 2223 String Key Value Type Required? Description 2224 =================== ============== ========= ============================== 2225 "ReqdWorkGroupSize" sequence of If not 0, 0, 0 then all values 2226 3 integers must be >=1 and the dispatch 2227 work-group size X, Y, Z must 2228 correspond to the specified 2229 values. Defaults to 0, 0, 0. 2230 2231 Corresponds to the OpenCL 2232 ``reqd_work_group_size`` 2233 attribute. 2234 "WorkGroupSizeHint" sequence of The dispatch work-group size 2235 3 integers X, Y, Z is likely to be the 2236 specified values. 2237 2238 Corresponds to the OpenCL 2239 ``work_group_size_hint`` 2240 attribute. 2241 "VecTypeHint" string The name of a scalar or vector 2242 type. 2243 2244 Corresponds to the OpenCL 2245 ``vec_type_hint`` attribute. 2246 2247 "RuntimeHandle" string The external symbol name 2248 associated with a kernel. 2249 OpenCL runtime allocates a 2250 global buffer for the symbol 2251 and saves the kernel's address 2252 to it, which is used for 2253 device side enqueueing. Only 2254 available for device side 2255 enqueued kernels. 2256 =================== ============== ========= ============================== 2257 2258.. 2259 2260 .. table:: AMDHSA Code Object V2 Kernel Argument Metadata Map 2261 :name: amdgpu-amdhsa-code-object-kernel-argument-metadata-map-table-v2 2262 2263 ================= ============== ========= ================================ 2264 String Key Value Type Required? Description 2265 ================= ============== ========= ================================ 2266 "Name" string Kernel argument name. 2267 "TypeName" string Kernel argument type name. 2268 "Size" integer Required Kernel argument size in bytes. 2269 "Align" integer Required Kernel argument alignment in 2270 bytes. Must be a power of two. 2271 "ValueKind" string Required Kernel argument kind that 2272 specifies how to set up the 2273 corresponding argument. 2274 Values include: 2275 2276 "ByValue" 2277 The argument is copied 2278 directly into the kernarg. 2279 2280 "GlobalBuffer" 2281 A global address space pointer 2282 to the buffer data is passed 2283 in the kernarg. 2284 2285 "DynamicSharedPointer" 2286 A group address space pointer 2287 to dynamically allocated LDS 2288 is passed in the kernarg. 2289 2290 "Sampler" 2291 A global address space 2292 pointer to a S# is passed in 2293 the kernarg. 2294 2295 "Image" 2296 A global address space 2297 pointer to a T# is passed in 2298 the kernarg. 2299 2300 "Pipe" 2301 A global address space pointer 2302 to an OpenCL pipe is passed in 2303 the kernarg. 2304 2305 "Queue" 2306 A global address space pointer 2307 to an OpenCL device enqueue 2308 queue is passed in the 2309 kernarg. 2310 2311 "HiddenGlobalOffsetX" 2312 The OpenCL grid dispatch 2313 global offset for the X 2314 dimension is passed in the 2315 kernarg. 2316 2317 "HiddenGlobalOffsetY" 2318 The OpenCL grid dispatch 2319 global offset for the Y 2320 dimension is passed in the 2321 kernarg. 2322 2323 "HiddenGlobalOffsetZ" 2324 The OpenCL grid dispatch 2325 global offset for the Z 2326 dimension is passed in the 2327 kernarg. 2328 2329 "HiddenNone" 2330 An argument that is not used 2331 by the kernel. Space needs to 2332 be left for it, but it does 2333 not need to be set up. 2334 2335 "HiddenPrintfBuffer" 2336 A global address space pointer 2337 to the runtime printf buffer 2338 is passed in kernarg. 2339 2340 "HiddenHostcallBuffer" 2341 A global address space pointer 2342 to the runtime hostcall buffer 2343 is passed in kernarg. 2344 2345 "HiddenDefaultQueue" 2346 A global address space pointer 2347 to the OpenCL device enqueue 2348 queue that should be used by 2349 the kernel by default is 2350 passed in the kernarg. 2351 2352 "HiddenCompletionAction" 2353 A global address space pointer 2354 to help link enqueued kernels into 2355 the ancestor tree for determining 2356 when the parent kernel has finished. 2357 2358 "HiddenMultiGridSyncArg" 2359 A global address space pointer for 2360 multi-grid synchronization is 2361 passed in the kernarg. 2362 2363 "ValueType" string Unused and deprecated. This should no longer 2364 be emitted, but is accepted for compatibility. 2365 2366 2367 "PointeeAlign" integer Alignment in bytes of pointee 2368 type for pointer type kernel 2369 argument. Must be a power 2370 of 2. Only present if 2371 "ValueKind" is 2372 "DynamicSharedPointer". 2373 "AddrSpaceQual" string Kernel argument address space 2374 qualifier. Only present if 2375 "ValueKind" is "GlobalBuffer" or 2376 "DynamicSharedPointer". Values 2377 are: 2378 2379 - "Private" 2380 - "Global" 2381 - "Constant" 2382 - "Local" 2383 - "Generic" 2384 - "Region" 2385 2386 .. TODO:: 2387 2388 Is GlobalBuffer only Global 2389 or Constant? Is 2390 DynamicSharedPointer always 2391 Local? Can HCC allow Generic? 2392 How can Private or Region 2393 ever happen? 2394 2395 "AccQual" string Kernel argument access 2396 qualifier. Only present if 2397 "ValueKind" is "Image" or 2398 "Pipe". Values 2399 are: 2400 2401 - "ReadOnly" 2402 - "WriteOnly" 2403 - "ReadWrite" 2404 2405 .. TODO:: 2406 2407 Does this apply to 2408 GlobalBuffer? 2409 2410 "ActualAccQual" string The actual memory accesses 2411 performed by the kernel on the 2412 kernel argument. Only present if 2413 "ValueKind" is "GlobalBuffer", 2414 "Image", or "Pipe". This may be 2415 more restrictive than indicated 2416 by "AccQual" to reflect what the 2417 kernel actual does. If not 2418 present then the runtime must 2419 assume what is implied by 2420 "AccQual" and "IsConst". Values 2421 are: 2422 2423 - "ReadOnly" 2424 - "WriteOnly" 2425 - "ReadWrite" 2426 2427 "IsConst" boolean Indicates if the kernel argument 2428 is const qualified. Only present 2429 if "ValueKind" is 2430 "GlobalBuffer". 2431 2432 "IsRestrict" boolean Indicates if the kernel argument 2433 is restrict qualified. Only 2434 present if "ValueKind" is 2435 "GlobalBuffer". 2436 2437 "IsVolatile" boolean Indicates if the kernel argument 2438 is volatile qualified. Only 2439 present if "ValueKind" is 2440 "GlobalBuffer". 2441 2442 "IsPipe" boolean Indicates if the kernel argument 2443 is pipe qualified. Only present 2444 if "ValueKind" is "Pipe". 2445 2446 .. TODO:: 2447 2448 Can GlobalBuffer be pipe 2449 qualified? 2450 2451 ================= ============== ========= ================================ 2452 2453.. 2454 2455 .. table:: AMDHSA Code Object V2 Kernel Code Properties Metadata Map 2456 :name: amdgpu-amdhsa-code-object-kernel-code-properties-metadata-map-table-v2 2457 2458 ============================ ============== ========= ===================== 2459 String Key Value Type Required? Description 2460 ============================ ============== ========= ===================== 2461 "KernargSegmentSize" integer Required The size in bytes of 2462 the kernarg segment 2463 that holds the values 2464 of the arguments to 2465 the kernel. 2466 "GroupSegmentFixedSize" integer Required The amount of group 2467 segment memory 2468 required by a 2469 work-group in 2470 bytes. This does not 2471 include any 2472 dynamically allocated 2473 group segment memory 2474 that may be added 2475 when the kernel is 2476 dispatched. 2477 "PrivateSegmentFixedSize" integer Required The amount of fixed 2478 private address space 2479 memory required for a 2480 work-item in 2481 bytes. If the kernel 2482 uses a dynamic call 2483 stack then additional 2484 space must be added 2485 to this value for the 2486 call stack. 2487 "KernargSegmentAlign" integer Required The maximum byte 2488 alignment of 2489 arguments in the 2490 kernarg segment. Must 2491 be a power of 2. 2492 "WavefrontSize" integer Required Wavefront size. Must 2493 be a power of 2. 2494 "NumSGPRs" integer Required Number of scalar 2495 registers used by a 2496 wavefront for 2497 GFX6-GFX10. This 2498 includes the special 2499 SGPRs for VCC, Flat 2500 Scratch (GFX7-GFX10) 2501 and XNACK (for 2502 GFX8-GFX10). It does 2503 not include the 16 2504 SGPR added if a trap 2505 handler is 2506 enabled. It is not 2507 rounded up to the 2508 allocation 2509 granularity. 2510 "NumVGPRs" integer Required Number of vector 2511 registers used by 2512 each work-item for 2513 GFX6-GFX10 2514 "MaxFlatWorkGroupSize" integer Required Maximum flat 2515 work-group size 2516 supported by the 2517 kernel in work-items. 2518 Must be >=1 and 2519 consistent with 2520 ReqdWorkGroupSize if 2521 not 0, 0, 0. 2522 "NumSpilledSGPRs" integer Number of stores from 2523 a scalar register to 2524 a register allocator 2525 created spill 2526 location. 2527 "NumSpilledVGPRs" integer Number of stores from 2528 a vector register to 2529 a register allocator 2530 created spill 2531 location. 2532 ============================ ============== ========= ===================== 2533 2534.. _amdgpu-amdhsa-code-object-metadata-v3: 2535 2536Code Object V3 Metadata (--amdhsa-code-object-version=3) 2537++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 2538 2539Code object V3 metadata is specified by the ``NT_AMDGPU_METADATA`` note record 2540(see :ref:`amdgpu-note-records-v3`). 2541 2542The metadata is represented as Message Pack formatted binary data (see 2543[MsgPack]_). The top level is a Message Pack map that includes the 2544keys defined in table 2545:ref:`amdgpu-amdhsa-code-object-metadata-map-table-v3` and referenced 2546tables. 2547 2548Additional information can be added to the maps. To avoid conflicts, 2549any key names should be prefixed by "*vendor-name*." where 2550``vendor-name`` can be the name of the vendor and specific vendor 2551tool that generates the information. The prefix is abbreviated to 2552simply "." when it appears within a map that has been added by the 2553same *vendor-name*. 2554 2555 .. table:: AMDHSA Code Object V3 Metadata Map 2556 :name: amdgpu-amdhsa-code-object-metadata-map-table-v3 2557 2558 ================= ============== ========= ======================================= 2559 String Key Value Type Required? Description 2560 ================= ============== ========= ======================================= 2561 "amdhsa.version" sequence of Required - The first integer is the major 2562 2 integers version. Currently 1. 2563 - The second integer is the minor 2564 version. Currently 0. 2565 "amdhsa.printf" sequence of Each string is encoded information 2566 strings about a printf function call. The 2567 encoded information is organized as 2568 fields separated by colon (':'): 2569 2570 ``ID:N:S[0]:S[1]:...:S[N-1]:FormatString`` 2571 2572 where: 2573 2574 ``ID`` 2575 A 32-bit integer as a unique id for 2576 each printf function call 2577 2578 ``N`` 2579 A 32-bit integer equal to the number 2580 of arguments of printf function call 2581 minus 1 2582 2583 ``S[i]`` (where i = 0, 1, ... , N-1) 2584 32-bit integers for the size in bytes 2585 of the i-th FormatString argument of 2586 the printf function call 2587 2588 FormatString 2589 The format string passed to the 2590 printf function call. 2591 "amdhsa.kernels" sequence of Required Sequence of the maps for each 2592 map kernel in the code object. See 2593 :ref:`amdgpu-amdhsa-code-object-kernel-metadata-map-table-v3` 2594 for the definition of the keys included 2595 in that map. 2596 ================= ============== ========= ======================================= 2597 2598.. 2599 2600 .. table:: AMDHSA Code Object V3 Kernel Metadata Map 2601 :name: amdgpu-amdhsa-code-object-kernel-metadata-map-table-v3 2602 2603 =================================== ============== ========= ================================ 2604 String Key Value Type Required? Description 2605 =================================== ============== ========= ================================ 2606 ".name" string Required Source name of the kernel. 2607 ".symbol" string Required Name of the kernel 2608 descriptor ELF symbol. 2609 ".language" string Source language of the kernel. 2610 Values include: 2611 2612 - "OpenCL C" 2613 - "OpenCL C++" 2614 - "HCC" 2615 - "HIP" 2616 - "OpenMP" 2617 - "Assembler" 2618 2619 ".language_version" sequence of - The first integer is the major 2620 2 integers version. 2621 - The second integer is the 2622 minor version. 2623 ".args" sequence of Sequence of maps of the 2624 map kernel arguments. See 2625 :ref:`amdgpu-amdhsa-code-object-kernel-argument-metadata-map-table-v3` 2626 for the definition of the keys 2627 included in that map. 2628 ".reqd_workgroup_size" sequence of If not 0, 0, 0 then all values 2629 3 integers must be >=1 and the dispatch 2630 work-group size X, Y, Z must 2631 correspond to the specified 2632 values. Defaults to 0, 0, 0. 2633 2634 Corresponds to the OpenCL 2635 ``reqd_work_group_size`` 2636 attribute. 2637 ".workgroup_size_hint" sequence of The dispatch work-group size 2638 3 integers X, Y, Z is likely to be the 2639 specified values. 2640 2641 Corresponds to the OpenCL 2642 ``work_group_size_hint`` 2643 attribute. 2644 ".vec_type_hint" string The name of a scalar or vector 2645 type. 2646 2647 Corresponds to the OpenCL 2648 ``vec_type_hint`` attribute. 2649 2650 ".device_enqueue_symbol" string The external symbol name 2651 associated with a kernel. 2652 OpenCL runtime allocates a 2653 global buffer for the symbol 2654 and saves the kernel's address 2655 to it, which is used for 2656 device side enqueueing. Only 2657 available for device side 2658 enqueued kernels. 2659 ".kernarg_segment_size" integer Required The size in bytes of 2660 the kernarg segment 2661 that holds the values 2662 of the arguments to 2663 the kernel. 2664 ".group_segment_fixed_size" integer Required The amount of group 2665 segment memory 2666 required by a 2667 work-group in 2668 bytes. This does not 2669 include any 2670 dynamically allocated 2671 group segment memory 2672 that may be added 2673 when the kernel is 2674 dispatched. 2675 ".private_segment_fixed_size" integer Required The amount of fixed 2676 private address space 2677 memory required for a 2678 work-item in 2679 bytes. If the kernel 2680 uses a dynamic call 2681 stack then additional 2682 space must be added 2683 to this value for the 2684 call stack. 2685 ".kernarg_segment_align" integer Required The maximum byte 2686 alignment of 2687 arguments in the 2688 kernarg segment. Must 2689 be a power of 2. 2690 ".wavefront_size" integer Required Wavefront size. Must 2691 be a power of 2. 2692 ".sgpr_count" integer Required Number of scalar 2693 registers required by a 2694 wavefront for 2695 GFX6-GFX9. A register 2696 is required if it is 2697 used explicitly, or 2698 if a higher numbered 2699 register is used 2700 explicitly. This 2701 includes the special 2702 SGPRs for VCC, Flat 2703 Scratch (GFX7-GFX9) 2704 and XNACK (for 2705 GFX8-GFX9). It does 2706 not include the 16 2707 SGPR added if a trap 2708 handler is 2709 enabled. It is not 2710 rounded up to the 2711 allocation 2712 granularity. 2713 ".vgpr_count" integer Required Number of vector 2714 registers required by 2715 each work-item for 2716 GFX6-GFX9. A register 2717 is required if it is 2718 used explicitly, or 2719 if a higher numbered 2720 register is used 2721 explicitly. 2722 ".max_flat_workgroup_size" integer Required Maximum flat 2723 work-group size 2724 supported by the 2725 kernel in work-items. 2726 Must be >=1 and 2727 consistent with 2728 ReqdWorkGroupSize if 2729 not 0, 0, 0. 2730 ".sgpr_spill_count" integer Number of stores from 2731 a scalar register to 2732 a register allocator 2733 created spill 2734 location. 2735 ".vgpr_spill_count" integer Number of stores from 2736 a vector register to 2737 a register allocator 2738 created spill 2739 location. 2740 =================================== ============== ========= ================================ 2741 2742.. 2743 2744 .. table:: AMDHSA Code Object V3 Kernel Argument Metadata Map 2745 :name: amdgpu-amdhsa-code-object-kernel-argument-metadata-map-table-v3 2746 2747 ====================== ============== ========= ================================ 2748 String Key Value Type Required? Description 2749 ====================== ============== ========= ================================ 2750 ".name" string Kernel argument name. 2751 ".type_name" string Kernel argument type name. 2752 ".size" integer Required Kernel argument size in bytes. 2753 ".offset" integer Required Kernel argument offset in 2754 bytes. The offset must be a 2755 multiple of the alignment 2756 required by the argument. 2757 ".value_kind" string Required Kernel argument kind that 2758 specifies how to set up the 2759 corresponding argument. 2760 Values include: 2761 2762 "by_value" 2763 The argument is copied 2764 directly into the kernarg. 2765 2766 "global_buffer" 2767 A global address space pointer 2768 to the buffer data is passed 2769 in the kernarg. 2770 2771 "dynamic_shared_pointer" 2772 A group address space pointer 2773 to dynamically allocated LDS 2774 is passed in the kernarg. 2775 2776 "sampler" 2777 A global address space 2778 pointer to a S# is passed in 2779 the kernarg. 2780 2781 "image" 2782 A global address space 2783 pointer to a T# is passed in 2784 the kernarg. 2785 2786 "pipe" 2787 A global address space pointer 2788 to an OpenCL pipe is passed in 2789 the kernarg. 2790 2791 "queue" 2792 A global address space pointer 2793 to an OpenCL device enqueue 2794 queue is passed in the 2795 kernarg. 2796 2797 "hidden_global_offset_x" 2798 The OpenCL grid dispatch 2799 global offset for the X 2800 dimension is passed in the 2801 kernarg. 2802 2803 "hidden_global_offset_y" 2804 The OpenCL grid dispatch 2805 global offset for the Y 2806 dimension is passed in the 2807 kernarg. 2808 2809 "hidden_global_offset_z" 2810 The OpenCL grid dispatch 2811 global offset for the Z 2812 dimension is passed in the 2813 kernarg. 2814 2815 "hidden_none" 2816 An argument that is not used 2817 by the kernel. Space needs to 2818 be left for it, but it does 2819 not need to be set up. 2820 2821 "hidden_printf_buffer" 2822 A global address space pointer 2823 to the runtime printf buffer 2824 is passed in kernarg. 2825 2826 "hidden_hostcall_buffer" 2827 A global address space pointer 2828 to the runtime hostcall buffer 2829 is passed in kernarg. 2830 2831 "hidden_default_queue" 2832 A global address space pointer 2833 to the OpenCL device enqueue 2834 queue that should be used by 2835 the kernel by default is 2836 passed in the kernarg. 2837 2838 "hidden_completion_action" 2839 A global address space pointer 2840 to help link enqueued kernels into 2841 the ancestor tree for determining 2842 when the parent kernel has finished. 2843 2844 "hidden_multigrid_sync_arg" 2845 A global address space pointer for 2846 multi-grid synchronization is 2847 passed in the kernarg. 2848 2849 ".value_type" string Unused and deprecated. This should no longer 2850 be emitted, but is accepted for compatibility. 2851 2852 ".pointee_align" integer Alignment in bytes of pointee 2853 type for pointer type kernel 2854 argument. Must be a power 2855 of 2. Only present if 2856 ".value_kind" is 2857 "dynamic_shared_pointer". 2858 ".address_space" string Kernel argument address space 2859 qualifier. Only present if 2860 ".value_kind" is "global_buffer" or 2861 "dynamic_shared_pointer". Values 2862 are: 2863 2864 - "private" 2865 - "global" 2866 - "constant" 2867 - "local" 2868 - "generic" 2869 - "region" 2870 2871 .. TODO:: 2872 2873 Is "global_buffer" only "global" 2874 or "constant"? Is 2875 "dynamic_shared_pointer" always 2876 "local"? Can HCC allow "generic"? 2877 How can "private" or "region" 2878 ever happen? 2879 2880 ".access" string Kernel argument access 2881 qualifier. Only present if 2882 ".value_kind" is "image" or 2883 "pipe". Values 2884 are: 2885 2886 - "read_only" 2887 - "write_only" 2888 - "read_write" 2889 2890 .. TODO:: 2891 2892 Does this apply to 2893 "global_buffer"? 2894 2895 ".actual_access" string The actual memory accesses 2896 performed by the kernel on the 2897 kernel argument. Only present if 2898 ".value_kind" is "global_buffer", 2899 "image", or "pipe". This may be 2900 more restrictive than indicated 2901 by ".access" to reflect what the 2902 kernel actual does. If not 2903 present then the runtime must 2904 assume what is implied by 2905 ".access" and ".is_const" . Values 2906 are: 2907 2908 - "read_only" 2909 - "write_only" 2910 - "read_write" 2911 2912 ".is_const" boolean Indicates if the kernel argument 2913 is const qualified. Only present 2914 if ".value_kind" is 2915 "global_buffer". 2916 2917 ".is_restrict" boolean Indicates if the kernel argument 2918 is restrict qualified. Only 2919 present if ".value_kind" is 2920 "global_buffer". 2921 2922 ".is_volatile" boolean Indicates if the kernel argument 2923 is volatile qualified. Only 2924 present if ".value_kind" is 2925 "global_buffer". 2926 2927 ".is_pipe" boolean Indicates if the kernel argument 2928 is pipe qualified. Only present 2929 if ".value_kind" is "pipe". 2930 2931 .. TODO:: 2932 2933 Can "global_buffer" be pipe 2934 qualified? 2935 2936 ====================== ============== ========= ================================ 2937 2938.. 2939 2940Kernel Dispatch 2941~~~~~~~~~~~~~~~ 2942 2943The HSA architected queuing language (AQL) defines a user space memory interface 2944that can be used to control the dispatch of kernels, in an agent independent 2945way. An agent can have zero or more AQL queues created for it using the ROCm 2946runtime, in which AQL packets (all of which are 64 bytes) can be placed. See the 2947*HSA Platform System Architecture Specification* [HSA]_ for the AQL queue 2948mechanics and packet layouts. 2949 2950The packet processor of a kernel agent is responsible for detecting and 2951dispatching HSA kernels from the AQL queues associated with it. For AMD GPUs the 2952packet processor is implemented by the hardware command processor (CP), 2953asynchronous dispatch controller (ADC) and shader processor input controller 2954(SPI). 2955 2956The ROCm runtime can be used to allocate an AQL queue object. It uses the kernel 2957mode driver to initialize and register the AQL queue with CP. 2958 2959To dispatch a kernel the following actions are performed. This can occur in the 2960CPU host program, or from an HSA kernel executing on a GPU. 2961 29621. A pointer to an AQL queue for the kernel agent on which the kernel is to be 2963 executed is obtained. 29642. A pointer to the kernel descriptor (see 2965 :ref:`amdgpu-amdhsa-kernel-descriptor`) of the kernel to execute is obtained. 2966 It must be for a kernel that is contained in a code object that that was 2967 loaded by the ROCm runtime on the kernel agent with which the AQL queue is 2968 associated. 29693. Space is allocated for the kernel arguments using the ROCm runtime allocator 2970 for a memory region with the kernarg property for the kernel agent that will 2971 execute the kernel. It must be at least 16-byte aligned. 29724. Kernel argument values are assigned to the kernel argument memory 2973 allocation. The layout is defined in the *HSA Programmer's Language 2974 Reference* [HSA]_. For AMDGPU the kernel execution directly accesses the 2975 kernel argument memory in the same way constant memory is accessed. (Note 2976 that the HSA specification allows an implementation to copy the kernel 2977 argument contents to another location that is accessed by the kernel.) 29785. An AQL kernel dispatch packet is created on the AQL queue. The ROCm runtime 2979 api uses 64-bit atomic operations to reserve space in the AQL queue for the 2980 packet. The packet must be set up, and the final write must use an atomic 2981 store release to set the packet kind to ensure the packet contents are 2982 visible to the kernel agent. AQL defines a doorbell signal mechanism to 2983 notify the kernel agent that the AQL queue has been updated. These rules, and 2984 the layout of the AQL queue and kernel dispatch packet is defined in the *HSA 2985 System Architecture Specification* [HSA]_. 29866. A kernel dispatch packet includes information about the actual dispatch, 2987 such as grid and work-group size, together with information from the code 2988 object about the kernel, such as segment sizes. The ROCm runtime queries on 2989 the kernel symbol can be used to obtain the code object values which are 2990 recorded in the :ref:`amdgpu-amdhsa-code-object-metadata`. 29917. CP executes micro-code and is responsible for detecting and setting up the 2992 GPU to execute the wavefronts of a kernel dispatch. 29938. CP ensures that when the a wavefront starts executing the kernel machine 2994 code, the scalar general purpose registers (SGPR) and vector general purpose 2995 registers (VGPR) are set up as required by the machine code. The required 2996 setup is defined in the :ref:`amdgpu-amdhsa-kernel-descriptor`. The initial 2997 register state is defined in 2998 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`. 29999. The prolog of the kernel machine code (see 3000 :ref:`amdgpu-amdhsa-kernel-prolog`) sets up the machine state as necessary 3001 before continuing executing the machine code that corresponds to the kernel. 300210. When the kernel dispatch has completed execution, CP signals the completion 3003 signal specified in the kernel dispatch packet if not 0. 3004 3005.. _amdgpu-amdhsa-memory-spaces: 3006 3007Memory Spaces 3008~~~~~~~~~~~~~ 3009 3010The memory space properties are: 3011 3012 .. table:: AMDHSA Memory Spaces 3013 :name: amdgpu-amdhsa-memory-spaces-table 3014 3015 ================= =========== ======== ======= ================== 3016 Memory Space Name HSA Segment Hardware Address NULL Value 3017 Name Name Size 3018 ================= =========== ======== ======= ================== 3019 Private private scratch 32 0x00000000 3020 Local group LDS 32 0xFFFFFFFF 3021 Global global global 64 0x0000000000000000 3022 Constant constant *same as 64 0x0000000000000000 3023 global* 3024 Generic flat flat 64 0x0000000000000000 3025 Region N/A GDS 32 *not implemented 3026 for AMDHSA* 3027 ================= =========== ======== ======= ================== 3028 3029The global and constant memory spaces both use global virtual addresses, which 3030are the same virtual address space used by the CPU. However, some virtual 3031addresses may only be accessible to the CPU, some only accessible by the GPU, 3032and some by both. 3033 3034Using the constant memory space indicates that the data will not change during 3035the execution of the kernel. This allows scalar read instructions to be 3036used. The vector and scalar L1 caches are invalidated of volatile data before 3037each kernel dispatch execution to allow constant memory to change values between 3038kernel dispatches. 3039 3040The local memory space uses the hardware Local Data Store (LDS) which is 3041automatically allocated when the hardware creates work-groups of wavefronts, and 3042freed when all the wavefronts of a work-group have terminated. The data store 3043(DS) instructions can be used to access it. 3044 3045The private memory space uses the hardware scratch memory support. If the kernel 3046uses scratch, then the hardware allocates memory that is accessed using 3047wavefront lane dword (4 byte) interleaving. The mapping used from private 3048address to physical address is: 3049 3050 ``wavefront-scratch-base + 3051 (private-address * wavefront-size * 4) + 3052 (wavefront-lane-id * 4)`` 3053 3054There are different ways that the wavefront scratch base address is determined 3055by a wavefront (see :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). This 3056memory can be accessed in an interleaved manner using buffer instruction with 3057the scratch buffer descriptor and per wavefront scratch offset, by the scratch 3058instructions, or by flat instructions. If each lane of a wavefront accesses the 3059same private address, the interleaving results in adjacent dwords being accessed 3060and hence requires fewer cache lines to be fetched. Multi-dword access is not 3061supported except by flat and scratch instructions in GFX9-GFX10. 3062 3063The generic address space uses the hardware flat address support available in 3064GFX7-GFX10. This uses two fixed ranges of virtual addresses (the private and 3065local apertures), that are outside the range of addressible global memory, to 3066map from a flat address to a private or local address. 3067 3068FLAT instructions can take a flat address and access global, private (scratch) 3069and group (LDS) memory depending in if the address is within one of the 3070aperture ranges. Flat access to scratch requires hardware aperture setup and 3071setup in the kernel prologue (see 3072:ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`). Flat access to LDS requires 3073hardware aperture setup and M0 (GFX7-GFX8) register setup (see 3074:ref:`amdgpu-amdhsa-kernel-prolog-m0`). 3075 3076To convert between a segment address and a flat address the base address of the 3077apertures address can be used. For GFX7-GFX8 these are available in the 3078:ref:`amdgpu-amdhsa-hsa-aql-queue` the address of which can be obtained with 3079Queue Ptr SGPR (see :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). For 3080GFX9-GFX10 the aperture base addresses are directly available as inline constant 3081registers ``SRC_SHARED_BASE/LIMIT`` and ``SRC_PRIVATE_BASE/LIMIT``. In 64 bit 3082address mode the aperture sizes are 2^32 bytes and the base is aligned to 2^32 3083which makes it easier to convert from flat to segment or segment to flat. 3084 3085Image and Samplers 3086~~~~~~~~~~~~~~~~~~ 3087 3088Image and sample handles created by the ROCm runtime are 64-bit addresses of a 3089hardware 32-byte V# and 48 byte S# object respectively. In order to support the 3090HSA ``query_sampler`` operations two extra dwords are used to store the HSA BRIG 3091enumeration values for the queries that are not trivially deducible from the S# 3092representation. 3093 3094HSA Signals 3095~~~~~~~~~~~ 3096 3097HSA signal handles created by the ROCm runtime are 64-bit addresses of a 3098structure allocated in memory accessible from both the CPU and GPU. The 3099structure is defined by the ROCm runtime and subject to change between releases 3100(see [AMD-ROCm-github]_). 3101 3102.. _amdgpu-amdhsa-hsa-aql-queue: 3103 3104HSA AQL Queue 3105~~~~~~~~~~~~~ 3106 3107The HSA AQL queue structure is defined by the ROCm runtime and subject to change 3108between releases (see [AMD-ROCm-github]_). For some processors it contains 3109fields needed to implement certain language features such as the flat address 3110aperture bases. It also contains fields used by CP such as managing the 3111allocation of scratch memory. 3112 3113.. _amdgpu-amdhsa-kernel-descriptor: 3114 3115Kernel Descriptor 3116~~~~~~~~~~~~~~~~~ 3117 3118A kernel descriptor consists of the information needed by CP to initiate the 3119execution of a kernel, including the entry point address of the machine code 3120that implements the kernel. 3121 3122Kernel Descriptor for GFX6-GFX10 3123++++++++++++++++++++++++++++++++ 3124 3125CP microcode requires the Kernel descriptor to be allocated on 64-byte 3126alignment. 3127 3128 .. table:: Kernel Descriptor for GFX6-GFX10 3129 :name: amdgpu-amdhsa-kernel-descriptor-gfx6-gfx10-table 3130 3131 ======= ======= =============================== ============================ 3132 Bits Size Field Name Description 3133 ======= ======= =============================== ============================ 3134 31:0 4 bytes GROUP_SEGMENT_FIXED_SIZE The amount of fixed local 3135 address space memory 3136 required for a work-group 3137 in bytes. This does not 3138 include any dynamically 3139 allocated local address 3140 space memory that may be 3141 added when the kernel is 3142 dispatched. 3143 63:32 4 bytes PRIVATE_SEGMENT_FIXED_SIZE The amount of fixed 3144 private address space 3145 memory required for a 3146 work-item in bytes. If 3147 is_dynamic_callstack is 1 3148 then additional space must 3149 be added to this value for 3150 the call stack. 3151 127:64 8 bytes Reserved, must be 0. 3152 191:128 8 bytes KERNEL_CODE_ENTRY_BYTE_OFFSET Byte offset (possibly 3153 negative) from base 3154 address of kernel 3155 descriptor to kernel's 3156 entry point instruction 3157 which must be 256 byte 3158 aligned. 3159 351:272 20 Reserved, must be 0. 3160 bytes 3161 383:352 4 bytes COMPUTE_PGM_RSRC3 GFX6-9 3162 Reserved, must be 0. 3163 GFX10 3164 Compute Shader (CS) 3165 program settings used by 3166 CP to set up 3167 ``COMPUTE_PGM_RSRC3`` 3168 configuration 3169 register. See 3170 :ref:`amdgpu-amdhsa-compute_pgm_rsrc3-gfx10-table`. 3171 415:384 4 bytes COMPUTE_PGM_RSRC1 Compute Shader (CS) 3172 program settings used by 3173 CP to set up 3174 ``COMPUTE_PGM_RSRC1`` 3175 configuration 3176 register. See 3177 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 3178 447:416 4 bytes COMPUTE_PGM_RSRC2 Compute Shader (CS) 3179 program settings used by 3180 CP to set up 3181 ``COMPUTE_PGM_RSRC2`` 3182 configuration 3183 register. See 3184 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 3185 448 1 bit ENABLE_SGPR_PRIVATE_SEGMENT Enable the setup of the 3186 _BUFFER SGPR user data registers 3187 (see 3188 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 3189 3190 The total number of SGPR 3191 user data registers 3192 requested must not exceed 3193 16 and match value in 3194 ``compute_pgm_rsrc2.user_sgpr.user_sgpr_count``. 3195 Any requests beyond 16 3196 will be ignored. 3197 449 1 bit ENABLE_SGPR_DISPATCH_PTR *see above* 3198 450 1 bit ENABLE_SGPR_QUEUE_PTR *see above* 3199 451 1 bit ENABLE_SGPR_KERNARG_SEGMENT_PTR *see above* 3200 452 1 bit ENABLE_SGPR_DISPATCH_ID *see above* 3201 453 1 bit ENABLE_SGPR_FLAT_SCRATCH_INIT *see above* 3202 454 1 bit ENABLE_SGPR_PRIVATE_SEGMENT *see above* 3203 _SIZE 3204 457:455 3 bits Reserved, must be 0. 3205 458 1 bit ENABLE_WAVEFRONT_SIZE32 GFX6-9 3206 Reserved, must be 0. 3207 GFX10 3208 - If 0 execute in 3209 wavefront size 64 mode. 3210 - If 1 execute in 3211 native wavefront size 3212 32 mode. 3213 463:459 5 bits Reserved, must be 0. 3214 511:464 6 bytes Reserved, must be 0. 3215 512 **Total size 64 bytes.** 3216 ======= ==================================================================== 3217 3218.. 3219 3220 .. table:: compute_pgm_rsrc1 for GFX6-GFX10 3221 :name: amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table 3222 3223 ======= ======= =============================== =========================================================================== 3224 Bits Size Field Name Description 3225 ======= ======= =============================== =========================================================================== 3226 5:0 6 bits GRANULATED_WORKITEM_VGPR_COUNT Number of vector register 3227 blocks used by each work-item; 3228 granularity is device 3229 specific: 3230 3231 GFX6-GFX9 3232 - vgprs_used 0..256 3233 - max(0, ceil(vgprs_used / 4) - 1) 3234 GFX10 (wavefront size 64) 3235 - max_vgpr 1..256 3236 - max(0, ceil(vgprs_used / 4) - 1) 3237 GFX10 (wavefront size 32) 3238 - max_vgpr 1..256 3239 - max(0, ceil(vgprs_used / 8) - 1) 3240 3241 Where vgprs_used is defined 3242 as the highest VGPR number 3243 explicitly referenced plus 3244 one. 3245 3246 Used by CP to set up 3247 ``COMPUTE_PGM_RSRC1.VGPRS``. 3248 3249 The 3250 :ref:`amdgpu-assembler` 3251 calculates this 3252 automatically for the 3253 selected processor from 3254 values provided to the 3255 `.amdhsa_kernel` directive 3256 by the 3257 `.amdhsa_next_free_vgpr` 3258 nested directive (see 3259 :ref:`amdhsa-kernel-directives-table`). 3260 9:6 4 bits GRANULATED_WAVEFRONT_SGPR_COUNT Number of scalar register 3261 blocks used by a wavefront; 3262 granularity is device 3263 specific: 3264 3265 GFX6-GFX8 3266 - sgprs_used 0..112 3267 - max(0, ceil(sgprs_used / 8) - 1) 3268 GFX9 3269 - sgprs_used 0..112 3270 - 2 * max(0, ceil(sgprs_used / 16) - 1) 3271 GFX10 3272 Reserved, must be 0. 3273 (128 SGPRs always 3274 allocated.) 3275 3276 Where sgprs_used is 3277 defined as the highest 3278 SGPR number explicitly 3279 referenced plus one, plus 3280 a target specific number 3281 of additional special 3282 SGPRs for VCC, 3283 FLAT_SCRATCH (GFX7+) and 3284 XNACK_MASK (GFX8+), and 3285 any additional 3286 target specific 3287 limitations. It does not 3288 include the 16 SGPRs added 3289 if a trap handler is 3290 enabled. 3291 3292 The target specific 3293 limitations and special 3294 SGPR layout are defined in 3295 the hardware 3296 documentation, which can 3297 be found in the 3298 :ref:`amdgpu-processors` 3299 table. 3300 3301 Used by CP to set up 3302 ``COMPUTE_PGM_RSRC1.SGPRS``. 3303 3304 The 3305 :ref:`amdgpu-assembler` 3306 calculates this 3307 automatically for the 3308 selected processor from 3309 values provided to the 3310 `.amdhsa_kernel` directive 3311 by the 3312 `.amdhsa_next_free_sgpr` 3313 and `.amdhsa_reserve_*` 3314 nested directives (see 3315 :ref:`amdhsa-kernel-directives-table`). 3316 11:10 2 bits PRIORITY Must be 0. 3317 3318 Start executing wavefront 3319 at the specified priority. 3320 3321 CP is responsible for 3322 filling in 3323 ``COMPUTE_PGM_RSRC1.PRIORITY``. 3324 13:12 2 bits FLOAT_ROUND_MODE_32 Wavefront starts execution 3325 with specified rounding 3326 mode for single (32 3327 bit) floating point 3328 precision floating point 3329 operations. 3330 3331 Floating point rounding 3332 mode values are defined in 3333 :ref:`amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table`. 3334 3335 Used by CP to set up 3336 ``COMPUTE_PGM_RSRC1.FLOAT_MODE``. 3337 15:14 2 bits FLOAT_ROUND_MODE_16_64 Wavefront starts execution 3338 with specified rounding 3339 denorm mode for half/double (16 3340 and 64-bit) floating point 3341 precision floating point 3342 operations. 3343 3344 Floating point rounding 3345 mode values are defined in 3346 :ref:`amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table`. 3347 3348 Used by CP to set up 3349 ``COMPUTE_PGM_RSRC1.FLOAT_MODE``. 3350 17:16 2 bits FLOAT_DENORM_MODE_32 Wavefront starts execution 3351 with specified denorm mode 3352 for single (32 3353 bit) floating point 3354 precision floating point 3355 operations. 3356 3357 Floating point denorm mode 3358 values are defined in 3359 :ref:`amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table`. 3360 3361 Used by CP to set up 3362 ``COMPUTE_PGM_RSRC1.FLOAT_MODE``. 3363 19:18 2 bits FLOAT_DENORM_MODE_16_64 Wavefront starts execution 3364 with specified denorm mode 3365 for half/double (16 3366 and 64-bit) floating point 3367 precision floating point 3368 operations. 3369 3370 Floating point denorm mode 3371 values are defined in 3372 :ref:`amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table`. 3373 3374 Used by CP to set up 3375 ``COMPUTE_PGM_RSRC1.FLOAT_MODE``. 3376 20 1 bit PRIV Must be 0. 3377 3378 Start executing wavefront 3379 in privilege trap handler 3380 mode. 3381 3382 CP is responsible for 3383 filling in 3384 ``COMPUTE_PGM_RSRC1.PRIV``. 3385 21 1 bit ENABLE_DX10_CLAMP Wavefront starts execution 3386 with DX10 clamp mode 3387 enabled. Used by the vector 3388 ALU to force DX10 style 3389 treatment of NaN's (when 3390 set, clamp NaN to zero, 3391 otherwise pass NaN 3392 through). 3393 3394 Used by CP to set up 3395 ``COMPUTE_PGM_RSRC1.DX10_CLAMP``. 3396 22 1 bit DEBUG_MODE Must be 0. 3397 3398 Start executing wavefront 3399 in single step mode. 3400 3401 CP is responsible for 3402 filling in 3403 ``COMPUTE_PGM_RSRC1.DEBUG_MODE``. 3404 23 1 bit ENABLE_IEEE_MODE Wavefront starts execution 3405 with IEEE mode 3406 enabled. Floating point 3407 opcodes that support 3408 exception flag gathering 3409 will quiet and propagate 3410 signaling-NaN inputs per 3411 IEEE 754-2008. Min_dx10 and 3412 max_dx10 become IEEE 3413 754-2008 compliant due to 3414 signaling-NaN propagation 3415 and quieting. 3416 3417 Used by CP to set up 3418 ``COMPUTE_PGM_RSRC1.IEEE_MODE``. 3419 24 1 bit BULKY Must be 0. 3420 3421 Only one work-group allowed 3422 to execute on a compute 3423 unit. 3424 3425 CP is responsible for 3426 filling in 3427 ``COMPUTE_PGM_RSRC1.BULKY``. 3428 25 1 bit CDBG_USER Must be 0. 3429 3430 Flag that can be used to 3431 control debugging code. 3432 3433 CP is responsible for 3434 filling in 3435 ``COMPUTE_PGM_RSRC1.CDBG_USER``. 3436 26 1 bit FP16_OVFL GFX6-GFX8 3437 Reserved, must be 0. 3438 GFX9-GFX10 3439 Wavefront starts execution 3440 with specified fp16 overflow 3441 mode. 3442 3443 - If 0, fp16 overflow generates 3444 +/-INF values. 3445 - If 1, fp16 overflow that is the 3446 result of an +/-INF input value 3447 or divide by 0 produces a +/-INF, 3448 otherwise clamps computed 3449 overflow to +/-MAX_FP16 as 3450 appropriate. 3451 3452 Used by CP to set up 3453 ``COMPUTE_PGM_RSRC1.FP16_OVFL``. 3454 28:27 2 bits Reserved, must be 0. 3455 29 1 bit WGP_MODE GFX6-GFX9 3456 Reserved, must be 0. 3457 GFX10 3458 - If 0 execute work-groups in 3459 CU wavefront execution mode. 3460 - If 1 execute work-groups on 3461 in WGP wavefront execution mode. 3462 3463 See :ref:`amdgpu-amdhsa-memory-model`. 3464 3465 Used by CP to set up 3466 ``COMPUTE_PGM_RSRC1.WGP_MODE``. 3467 30 1 bit MEM_ORDERED GFX6-9 3468 Reserved, must be 0. 3469 GFX10 3470 Controls the behavior of the 3471 waitcnt's vmcnt and vscnt 3472 counters. 3473 3474 - If 0 vmcnt reports completion 3475 of load and atomic with return 3476 out of order with sample 3477 instructions, and the vscnt 3478 reports the completion of 3479 store and atomic without 3480 return in order. 3481 - If 1 vmcnt reports completion 3482 of load, atomic with return 3483 and sample instructions in 3484 order, and the vscnt reports 3485 the completion of store and 3486 atomic without return in order. 3487 3488 Used by CP to set up 3489 ``COMPUTE_PGM_RSRC1.MEM_ORDERED``. 3490 31 1 bit FWD_PROGRESS GFX6-9 3491 Reserved, must be 0. 3492 GFX10 3493 - If 0 execute SIMD wavefronts 3494 using oldest first policy. 3495 - If 1 execute SIMD wavefronts to 3496 ensure wavefronts will make some 3497 forward progress. 3498 3499 Used by CP to set up 3500 ``COMPUTE_PGM_RSRC1.FWD_PROGRESS``. 3501 32 **Total size 4 bytes** 3502 ======= =================================================================================================================== 3503 3504.. 3505 3506 .. table:: compute_pgm_rsrc2 for GFX6-GFX10 3507 :name: amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table 3508 3509 ======= ======= =============================== =========================================================================== 3510 Bits Size Field Name Description 3511 ======= ======= =============================== =========================================================================== 3512 0 1 bit ENABLE_SGPR_PRIVATE_SEGMENT Enable the setup of the 3513 _WAVEFRONT_OFFSET SGPR wavefront scratch offset 3514 system register (see 3515 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 3516 3517 Used by CP to set up 3518 ``COMPUTE_PGM_RSRC2.SCRATCH_EN``. 3519 5:1 5 bits USER_SGPR_COUNT The total number of SGPR 3520 user data registers 3521 requested. This number must 3522 match the number of user 3523 data registers enabled. 3524 3525 Used by CP to set up 3526 ``COMPUTE_PGM_RSRC2.USER_SGPR``. 3527 6 1 bit ENABLE_TRAP_HANDLER Must be 0. 3528 3529 This bit represents 3530 ``COMPUTE_PGM_RSRC2.TRAP_PRESENT``, 3531 which is set by the CP if 3532 the runtime has installed a 3533 trap handler. 3534 7 1 bit ENABLE_SGPR_WORKGROUP_ID_X Enable the setup of the 3535 system SGPR register for 3536 the work-group id in the X 3537 dimension (see 3538 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 3539 3540 Used by CP to set up 3541 ``COMPUTE_PGM_RSRC2.TGID_X_EN``. 3542 8 1 bit ENABLE_SGPR_WORKGROUP_ID_Y Enable the setup of the 3543 system SGPR register for 3544 the work-group id in the Y 3545 dimension (see 3546 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 3547 3548 Used by CP to set up 3549 ``COMPUTE_PGM_RSRC2.TGID_Y_EN``. 3550 9 1 bit ENABLE_SGPR_WORKGROUP_ID_Z Enable the setup of the 3551 system SGPR register for 3552 the work-group id in the Z 3553 dimension (see 3554 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 3555 3556 Used by CP to set up 3557 ``COMPUTE_PGM_RSRC2.TGID_Z_EN``. 3558 10 1 bit ENABLE_SGPR_WORKGROUP_INFO Enable the setup of the 3559 system SGPR register for 3560 work-group information (see 3561 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 3562 3563 Used by CP to set up 3564 ``COMPUTE_PGM_RSRC2.TGID_SIZE_EN``. 3565 12:11 2 bits ENABLE_VGPR_WORKITEM_ID Enable the setup of the 3566 VGPR system registers used 3567 for the work-item ID. 3568 :ref:`amdgpu-amdhsa-system-vgpr-work-item-id-enumeration-values-table` 3569 defines the values. 3570 3571 Used by CP to set up 3572 ``COMPUTE_PGM_RSRC2.TIDIG_CMP_CNT``. 3573 13 1 bit ENABLE_EXCEPTION_ADDRESS_WATCH Must be 0. 3574 3575 Wavefront starts execution 3576 with address watch 3577 exceptions enabled which 3578 are generated when L1 has 3579 witnessed a thread access 3580 an *address of 3581 interest*. 3582 3583 CP is responsible for 3584 filling in the address 3585 watch bit in 3586 ``COMPUTE_PGM_RSRC2.EXCP_EN_MSB`` 3587 according to what the 3588 runtime requests. 3589 14 1 bit ENABLE_EXCEPTION_MEMORY Must be 0. 3590 3591 Wavefront starts execution 3592 with memory violation 3593 exceptions exceptions 3594 enabled which are generated 3595 when a memory violation has 3596 occurred for this wavefront from 3597 L1 or LDS 3598 (write-to-read-only-memory, 3599 mis-aligned atomic, LDS 3600 address out of range, 3601 illegal address, etc.). 3602 3603 CP sets the memory 3604 violation bit in 3605 ``COMPUTE_PGM_RSRC2.EXCP_EN_MSB`` 3606 according to what the 3607 runtime requests. 3608 23:15 9 bits GRANULATED_LDS_SIZE Must be 0. 3609 3610 CP uses the rounded value 3611 from the dispatch packet, 3612 not this value, as the 3613 dispatch may contain 3614 dynamically allocated group 3615 segment memory. CP writes 3616 directly to 3617 ``COMPUTE_PGM_RSRC2.LDS_SIZE``. 3618 3619 Amount of group segment 3620 (LDS) to allocate for each 3621 work-group. Granularity is 3622 device specific: 3623 3624 GFX6: 3625 roundup(lds-size / (64 * 4)) 3626 GFX7-GFX10: 3627 roundup(lds-size / (128 * 4)) 3628 3629 24 1 bit ENABLE_EXCEPTION_IEEE_754_FP Wavefront starts execution 3630 _INVALID_OPERATION with specified exceptions 3631 enabled. 3632 3633 Used by CP to set up 3634 ``COMPUTE_PGM_RSRC2.EXCP_EN`` 3635 (set from bits 0..6). 3636 3637 IEEE 754 FP Invalid 3638 Operation 3639 25 1 bit ENABLE_EXCEPTION_FP_DENORMAL FP Denormal one or more 3640 _SOURCE input operands is a 3641 denormal number 3642 26 1 bit ENABLE_EXCEPTION_IEEE_754_FP IEEE 754 FP Division by 3643 _DIVISION_BY_ZERO Zero 3644 27 1 bit ENABLE_EXCEPTION_IEEE_754_FP IEEE 754 FP FP Overflow 3645 _OVERFLOW 3646 28 1 bit ENABLE_EXCEPTION_IEEE_754_FP IEEE 754 FP Underflow 3647 _UNDERFLOW 3648 29 1 bit ENABLE_EXCEPTION_IEEE_754_FP IEEE 754 FP Inexact 3649 _INEXACT 3650 30 1 bit ENABLE_EXCEPTION_INT_DIVIDE_BY Integer Division by Zero 3651 _ZERO (rcp_iflag_f32 instruction 3652 only) 3653 31 1 bit Reserved, must be 0. 3654 32 **Total size 4 bytes.** 3655 ======= =================================================================================================================== 3656 3657.. 3658 3659 .. table:: compute_pgm_rsrc3 for GFX10 3660 :name: amdgpu-amdhsa-compute_pgm_rsrc3-gfx10-table 3661 3662 ======= ======= =============================== =========================================================================== 3663 Bits Size Field Name Description 3664 ======= ======= =============================== =========================================================================== 3665 3:0 4 bits SHARED_VGPR_COUNT Number of shared VGPRs for wavefront size 64. Granularity 8. Value 0-120. 3666 compute_pgm_rsrc1.vgprs + shared_vgpr_cnt cannot exceed 64. 3667 31:4 28 Reserved, must be 0. 3668 bits 3669 32 **Total size 4 bytes.** 3670 ======= =================================================================================================================== 3671 3672.. 3673 3674 .. table:: Floating Point Rounding Mode Enumeration Values 3675 :name: amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table 3676 3677 ====================================== ===== ============================== 3678 Enumeration Name Value Description 3679 ====================================== ===== ============================== 3680 FLOAT_ROUND_MODE_NEAR_EVEN 0 Round Ties To Even 3681 FLOAT_ROUND_MODE_PLUS_INFINITY 1 Round Toward +infinity 3682 FLOAT_ROUND_MODE_MINUS_INFINITY 2 Round Toward -infinity 3683 FLOAT_ROUND_MODE_ZERO 3 Round Toward 0 3684 ====================================== ===== ============================== 3685 3686.. 3687 3688 .. table:: Floating Point Denorm Mode Enumeration Values 3689 :name: amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table 3690 3691 ====================================== ===== ============================== 3692 Enumeration Name Value Description 3693 ====================================== ===== ============================== 3694 FLOAT_DENORM_MODE_FLUSH_SRC_DST 0 Flush Source and Destination 3695 Denorms 3696 FLOAT_DENORM_MODE_FLUSH_DST 1 Flush Output Denorms 3697 FLOAT_DENORM_MODE_FLUSH_SRC 2 Flush Source Denorms 3698 FLOAT_DENORM_MODE_FLUSH_NONE 3 No Flush 3699 ====================================== ===== ============================== 3700 3701.. 3702 3703 .. table:: System VGPR Work-Item ID Enumeration Values 3704 :name: amdgpu-amdhsa-system-vgpr-work-item-id-enumeration-values-table 3705 3706 ======================================== ===== ============================ 3707 Enumeration Name Value Description 3708 ======================================== ===== ============================ 3709 SYSTEM_VGPR_WORKITEM_ID_X 0 Set work-item X dimension 3710 ID. 3711 SYSTEM_VGPR_WORKITEM_ID_X_Y 1 Set work-item X and Y 3712 dimensions ID. 3713 SYSTEM_VGPR_WORKITEM_ID_X_Y_Z 2 Set work-item X, Y and Z 3714 dimensions ID. 3715 SYSTEM_VGPR_WORKITEM_ID_UNDEFINED 3 Undefined. 3716 ======================================== ===== ============================ 3717 3718.. _amdgpu-amdhsa-initial-kernel-execution-state: 3719 3720Initial Kernel Execution State 3721~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 3722 3723This section defines the register state that will be set up by the packet 3724processor prior to the start of execution of every wavefront. This is limited by 3725the constraints of the hardware controllers of CP/ADC/SPI. 3726 3727The order of the SGPR registers is defined, but the compiler can specify which 3728ones are actually setup in the kernel descriptor using the ``enable_sgpr_*`` bit 3729fields (see :ref:`amdgpu-amdhsa-kernel-descriptor`). The register numbers used 3730for enabled registers are dense starting at SGPR0: the first enabled register is 3731SGPR0, the next enabled register is SGPR1 etc.; disabled registers do not have 3732an SGPR number. 3733 3734The initial SGPRs comprise up to 16 User SRGPs that are set by CP and apply to 3735all wavefronts of the grid. It is possible to specify more than 16 User SGPRs 3736using the ``enable_sgpr_*`` bit fields, in which case only the first 16 are 3737actually initialized. These are then immediately followed by the System SGPRs 3738that are set up by ADC/SPI and can have different values for each wavefront of 3739the grid dispatch. 3740 3741SGPR register initial state is defined in 3742:ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 3743 3744 .. table:: SGPR Register Set Up Order 3745 :name: amdgpu-amdhsa-sgpr-register-set-up-order-table 3746 3747 ========== ========================== ====== ============================== 3748 SGPR Order Name Number Description 3749 (kernel descriptor enable of 3750 field) SGPRs 3751 ========== ========================== ====== ============================== 3752 First Private Segment Buffer 4 V# that can be used, together 3753 (enable_sgpr_private with Scratch Wavefront Offset 3754 _segment_buffer) as an offset, to access the 3755 private memory space using a 3756 segment address. 3757 3758 CP uses the value provided by 3759 the runtime. 3760 then Dispatch Ptr 2 64-bit address of AQL dispatch 3761 (enable_sgpr_dispatch_ptr) packet for kernel dispatch 3762 actually executing. 3763 then Queue Ptr 2 64-bit address of amd_queue_t 3764 (enable_sgpr_queue_ptr) object for AQL queue on which 3765 the dispatch packet was 3766 queued. 3767 then Kernarg Segment Ptr 2 64-bit address of Kernarg 3768 (enable_sgpr_kernarg segment. This is directly 3769 _segment_ptr) copied from the 3770 kernarg_address in the kernel 3771 dispatch packet. 3772 3773 Having CP load it once avoids 3774 loading it at the beginning of 3775 every wavefront. 3776 then Dispatch Id 2 64-bit Dispatch ID of the 3777 (enable_sgpr_dispatch_id) dispatch packet being 3778 executed. 3779 then Flat Scratch Init 2 This is 2 SGPRs: 3780 (enable_sgpr_flat_scratch 3781 _init) GFX6 3782 Not supported. 3783 GFX7-GFX8 3784 The first SGPR is a 32-bit 3785 byte offset from 3786 ``SH_HIDDEN_PRIVATE_BASE_VIMID`` 3787 to per SPI base of memory 3788 for scratch for the queue 3789 executing the kernel 3790 dispatch. CP obtains this 3791 from the runtime. (The 3792 Scratch Segment Buffer base 3793 address is 3794 ``SH_HIDDEN_PRIVATE_BASE_VIMID`` 3795 plus this offset.) The value 3796 of Scratch Wavefront Offset must 3797 be added to this offset by 3798 the kernel machine code, 3799 right shifted by 8, and 3800 moved to the FLAT_SCRATCH_HI 3801 SGPR register. 3802 FLAT_SCRATCH_HI corresponds 3803 to SGPRn-4 on GFX7, and 3804 SGPRn-6 on GFX8 (where SGPRn 3805 is the highest numbered SGPR 3806 allocated to the wavefront). 3807 FLAT_SCRATCH_HI is 3808 multiplied by 256 (as it is 3809 in units of 256 bytes) and 3810 added to 3811 ``SH_HIDDEN_PRIVATE_BASE_VIMID`` 3812 to calculate the per wavefront 3813 FLAT SCRATCH BASE in flat 3814 memory instructions that 3815 access the scratch 3816 aperture. 3817 3818 The second SGPR is 32-bit 3819 byte size of a single 3820 work-item's scratch memory 3821 usage. CP obtains this from 3822 the runtime, and it is 3823 always a multiple of DWORD. 3824 CP checks that the value in 3825 the kernel dispatch packet 3826 Private Segment Byte Size is 3827 not larger and requests the 3828 runtime to increase the 3829 queue's scratch size if 3830 necessary. The kernel code 3831 must move it to 3832 FLAT_SCRATCH_LO which is 3833 SGPRn-3 on GFX7 and SGPRn-5 3834 on GFX8. FLAT_SCRATCH_LO is 3835 used as the FLAT SCRATCH 3836 SIZE in flat memory 3837 instructions. Having CP load 3838 it once avoids loading it at 3839 the beginning of every 3840 wavefront. 3841 GFX9-GFX10 3842 This is the 3843 64-bit base address of the 3844 per SPI scratch backing 3845 memory managed by SPI for 3846 the queue executing the 3847 kernel dispatch. CP obtains 3848 this from the runtime (and 3849 divides it if there are 3850 multiple Shader Arrays each 3851 with its own SPI). The value 3852 of Scratch Wavefront Offset must 3853 be added by the kernel 3854 machine code and the result 3855 moved to the FLAT_SCRATCH 3856 SGPR which is SGPRn-6 and 3857 SGPRn-5. It is used as the 3858 FLAT SCRATCH BASE in flat 3859 memory instructions. 3860 then Private Segment Size 1 The 32-bit byte size of a 3861 (enable_sgpr_private single 3862 work-item's 3863 scratch_segment_size) memory 3864 allocation. This is the 3865 value from the kernel 3866 dispatch packet Private 3867 Segment Byte Size rounded up 3868 by CP to a multiple of 3869 DWORD. 3870 3871 Having CP load it once avoids 3872 loading it at the beginning of 3873 every wavefront. 3874 3875 This is not used for 3876 GFX7-GFX8 since it is the same 3877 value as the second SGPR of 3878 Flat Scratch Init. However, it 3879 may be needed for GFX9-GFX10 which 3880 changes the meaning of the 3881 Flat Scratch Init value. 3882 then Grid Work-Group Count X 1 32-bit count of the number of 3883 (enable_sgpr_grid work-groups in the X dimension 3884 _workgroup_count_X) for the grid being 3885 executed. Computed from the 3886 fields in the kernel dispatch 3887 packet as ((grid_size.x + 3888 workgroup_size.x - 1) / 3889 workgroup_size.x). 3890 then Grid Work-Group Count Y 1 32-bit count of the number of 3891 (enable_sgpr_grid work-groups in the Y dimension 3892 _workgroup_count_Y && for the grid being 3893 less than 16 previous executed. Computed from the 3894 SGPRs) fields in the kernel dispatch 3895 packet as ((grid_size.y + 3896 workgroup_size.y - 1) / 3897 workgroupSize.y). 3898 3899 Only initialized if <16 3900 previous SGPRs initialized. 3901 then Grid Work-Group Count Z 1 32-bit count of the number of 3902 (enable_sgpr_grid work-groups in the Z dimension 3903 _workgroup_count_Z && for the grid being 3904 less than 16 previous executed. Computed from the 3905 SGPRs) fields in the kernel dispatch 3906 packet as ((grid_size.z + 3907 workgroup_size.z - 1) / 3908 workgroupSize.z). 3909 3910 Only initialized if <16 3911 previous SGPRs initialized. 3912 then Work-Group Id X 1 32-bit work-group id in X 3913 (enable_sgpr_workgroup_id dimension of grid for 3914 _X) wavefront. 3915 then Work-Group Id Y 1 32-bit work-group id in Y 3916 (enable_sgpr_workgroup_id dimension of grid for 3917 _Y) wavefront. 3918 then Work-Group Id Z 1 32-bit work-group id in Z 3919 (enable_sgpr_workgroup_id dimension of grid for 3920 _Z) wavefront. 3921 then Work-Group Info 1 {first_wavefront, 14'b0000, 3922 (enable_sgpr_workgroup ordered_append_term[10:0], 3923 _info) threadgroup_size_in_wavefronts[5:0]} 3924 then Scratch Wavefront Offset 1 32-bit byte offset from base 3925 (enable_sgpr_private of scratch base of queue 3926 _segment_wavefront_offset) executing the kernel 3927 dispatch. Must be used as an 3928 offset with Private 3929 segment address when using 3930 Scratch Segment Buffer. It 3931 must be used to set up FLAT 3932 SCRATCH for flat addressing 3933 (see 3934 :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`). 3935 ========== ========================== ====== ============================== 3936 3937The order of the VGPR registers is defined, but the compiler can specify which 3938ones are actually setup in the kernel descriptor using the ``enable_vgpr*`` bit 3939fields (see :ref:`amdgpu-amdhsa-kernel-descriptor`). The register numbers used 3940for enabled registers are dense starting at VGPR0: the first enabled register is 3941VGPR0, the next enabled register is VGPR1 etc.; disabled registers do not have a 3942VGPR number. 3943 3944VGPR register initial state is defined in 3945:ref:`amdgpu-amdhsa-vgpr-register-set-up-order-table`. 3946 3947 .. table:: VGPR Register Set Up Order 3948 :name: amdgpu-amdhsa-vgpr-register-set-up-order-table 3949 3950 ========== ========================== ====== ============================== 3951 VGPR Order Name Number Description 3952 (kernel descriptor enable of 3953 field) VGPRs 3954 ========== ========================== ====== ============================== 3955 First Work-Item Id X 1 32-bit work-item id in X 3956 (Always initialized) dimension of work-group for 3957 wavefront lane. 3958 then Work-Item Id Y 1 32-bit work-item id in Y 3959 (enable_vgpr_workitem_id dimension of work-group for 3960 > 0) wavefront lane. 3961 then Work-Item Id Z 1 32-bit work-item id in Z 3962 (enable_vgpr_workitem_id dimension of work-group for 3963 > 1) wavefront lane. 3964 ========== ========================== ====== ============================== 3965 3966The setting of registers is done by GPU CP/ADC/SPI hardware as follows: 3967 39681. SGPRs before the Work-Group Ids are set by CP using the 16 User Data 3969 registers. 39702. Work-group Id registers X, Y, Z are set by ADC which supports any 3971 combination including none. 39723. Scratch Wavefront Offset is set by SPI in a per wavefront basis which is why 3973 its value cannot be included with the flat scratch init value which is per 3974 queue. 39754. The VGPRs are set by SPI which only supports specifying either (X), (X, Y) 3976 or (X, Y, Z). 3977 3978Flat Scratch register pair are adjacent SGPRs so they can be moved as a 64-bit 3979value to the hardware required SGPRn-3 and SGPRn-4 respectively. 3980 3981The global segment can be accessed either using buffer instructions (GFX6 which 3982has V# 64-bit address support), flat instructions (GFX7-GFX10), or global 3983instructions (GFX9-GFX10). 3984 3985If buffer operations are used, then the compiler can generate a V# with the 3986following properties: 3987 3988* base address of 0 3989* no swizzle 3990* ATC: 1 if IOMMU present (such as APU) 3991* ptr64: 1 3992* MTYPE set to support memory coherence that matches the runtime (such as CC for 3993 APU and NC for dGPU). 3994 3995.. _amdgpu-amdhsa-kernel-prolog: 3996 3997Kernel Prolog 3998~~~~~~~~~~~~~ 3999 4000The compiler performs initialization in the kernel prologue depending on the 4001target and information about things like stack usage in the kernel and called 4002functions. Some of this initialization requires the compiler to request certain 4003User and System SGPRs be present in the 4004:ref:`amdgpu-amdhsa-initial-kernel-execution-state` via the 4005:ref:`amdgpu-amdhsa-kernel-descriptor`. 4006 4007.. _amdgpu-amdhsa-kernel-prolog-cfi: 4008 4009CFI 4010+++ 4011 40121. The CFI return address is undefined. 4013 40142. The CFI CFA is defined using an expression which evaluates to a location 4015 description that comprises one memory location description for the 4016 ``DW_ASPACE_AMDGPU_private_lane`` address space address ``0``. 4017 4018.. _amdgpu-amdhsa-kernel-prolog-m0: 4019 4020M0 4021++ 4022 4023GFX6-GFX8 4024 The M0 register must be initialized with a value at least the total LDS size 4025 if the kernel may access LDS via DS or flat operations. Total LDS size is 4026 available in dispatch packet. For M0, it is also possible to use maximum 4027 possible value of LDS for given target (0x7FFF for GFX6 and 0xFFFF for 4028 GFX7-GFX8). 4029GFX9-GFX10 4030 The M0 register is not used for range checking LDS accesses and so does not 4031 need to be initialized in the prolog. 4032 4033.. _amdgpu-amdhsa-kernel-prolog-stack-pointer: 4034 4035Stack Pointer 4036+++++++++++++ 4037 4038If the kernel has function calls it must set up the ABI stack pointer described 4039in :ref:`amdgpu-amdhsa-function-call-convention-non-kernel-functions` by setting 4040SGPR32 to the unswizzled scratch offset of the address past the last local 4041allocation. 4042 4043.. _amdgpu-amdhsa-kernel-prolog-frame-pointer: 4044 4045Frame Pointer 4046+++++++++++++ 4047 4048If the kernel needs a frame pointer for the reasons defined in 4049``SIFrameLowering`` then SGPR33 is used and is always set to ``0`` in the 4050kernel prolog. If a frame pointer is not required then all uses of the frame 4051pointer are replaced with immediate ``0`` offsets. 4052 4053.. _amdgpu-amdhsa-kernel-prolog-flat-scratch: 4054 4055Flat Scratch 4056++++++++++++ 4057 4058If the kernel or any function it calls may use flat operations to access 4059scratch memory, the prolog code must set up the FLAT_SCRATCH register pair 4060(FLAT_SCRATCH_LO/FLAT_SCRATCH_HI which are in SGPRn-4/SGPRn-3). Initialization 4061uses Flat Scratch Init and Scratch Wavefront Offset SGPR registers (see 4062:ref:`amdgpu-amdhsa-initial-kernel-execution-state`): 4063 4064GFX6 4065 Flat scratch is not supported. 4066 4067GFX7-GFX8 4068 4069 1. The low word of Flat Scratch Init is 32-bit byte offset from 4070 ``SH_HIDDEN_PRIVATE_BASE_VIMID`` to the base of scratch backing memory 4071 being managed by SPI for the queue executing the kernel dispatch. This is 4072 the same value used in the Scratch Segment Buffer V# base address. The 4073 prolog must add the value of Scratch Wavefront Offset to get the 4074 wavefront's byte scratch backing memory offset from 4075 ``SH_HIDDEN_PRIVATE_BASE_VIMID``. Since FLAT_SCRATCH_LO is in units of 256 4076 bytes, the offset must be right shifted by 8 before moving into 4077 FLAT_SCRATCH_LO. 4078 2. The second word of Flat Scratch Init is 32-bit byte size of a single 4079 work-items scratch memory usage. This is directly loaded from the kernel 4080 dispatch packet Private Segment Byte Size and rounded up to a multiple of 4081 DWORD. Having CP load it once avoids loading it at the beginning of every 4082 wavefront. The prolog must move it to FLAT_SCRATCH_LO for use as FLAT 4083 SCRATCH SIZE. 4084 4085GFX9-GFX10 4086 The Flat Scratch Init is the 64-bit address of the base of scratch backing 4087 memory being managed by SPI for the queue executing the kernel dispatch. The 4088 prolog must add the value of Scratch Wavefront Offset and moved to the 4089 FLAT_SCRATCH pair for use as the flat scratch base in flat memory 4090 instructions. 4091 4092.. _amdgpu-amdhsa-kernel-prolog-private-segment-buffer: 4093 4094Private Segment Buffer 4095++++++++++++++++++++++ 4096 4097A set of four SGPRs beginning at a four-aligned SGPR index are always selected 4098to serve as the scratch V# for the kernel as follows: 4099 4100 - If it is known during instruction selection that there is stack usage, 4101 SGPR0-3 is reserved for use as the scratch V#. Stack usage is assumed if 4102 optimizations are disabled (``-O0``), if stack objects already exist (for 4103 locals, etc.), or if there are any function calls. 4104 4105 - Otherwise, four high numbered SGPRs beginning at a four-aligned SGPR index 4106 are reserved for the tentative scratch V#. These will be used if it is 4107 determined that spilling is needed. 4108 4109 - If no use is made of the tentative scratch V#, then it is unreserved, 4110 and the register count is determined ignoring it. 4111 - If use is made of the tentative scratch V#, then its register numbers 4112 are shifted to the first four-aligned SGPR index after the highest one 4113 allocated by the register allocator, and all uses are updated. The 4114 register count includes them in the shifted location. 4115 - In either case, if the processor has the SGPR allocation bug, the 4116 tentative allocation is not shifted or unreserved in order to ensure 4117 the register count is higher to workaround the bug. 4118 4119 .. note:: 4120 4121 This approach of using a tentative scratch V# and shifting the register 4122 numbers if used avoids having to perform register allocation a second 4123 time if the tentative V# is eliminated. This is more efficient and 4124 avoids the problem that the second register allocation may perform 4125 spilling which will fail as there is no longer a scratch V#. 4126 4127When the kernel prolog code is being emitted it is known whether the scratch V# 4128described above is actually used. If it is, the prolog code must set it up by 4129copying the Private Segment Buffer to the scratch V# registers and then adding 4130the Private Segment Wavefront Offset to the queue base address in the V#. The 4131result is a V# with a base address pointing to the beginning of the wavefront 4132scratch backing memory. 4133 4134The Private Segment Buffer is always requested, but the Private Segment 4135Wavefront Offset is only requested if it is used (see 4136:ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4137 4138.. _amdgpu-amdhsa-memory-model: 4139 4140Memory Model 4141~~~~~~~~~~~~ 4142 4143This section describes the mapping of LLVM memory model onto AMDGPU machine code 4144(see :ref:`memmodel`). 4145 4146The AMDGPU backend supports the memory synchronization scopes specified in 4147:ref:`amdgpu-memory-scopes`. 4148 4149The code sequences used to implement the memory model are defined in table 4150:ref:`amdgpu-amdhsa-memory-model-code-sequences-gfx6-gfx10-table`. 4151 4152The sequences specify the order of instructions that a single thread must 4153execute. The ``s_waitcnt`` and ``buffer_wbinvl1_vol`` are defined with respect 4154to other memory instructions executed by the same thread. This allows them to be 4155moved earlier or later which can allow them to be combined with other instances 4156of the same instruction, or hoisted/sunk out of loops to improve 4157performance. Only the instructions related to the memory model are given; 4158additional ``s_waitcnt`` instructions are required to ensure registers are 4159defined before being used. These may be able to be combined with the memory 4160model ``s_waitcnt`` instructions as described above. 4161 4162The AMDGPU backend supports the following memory models: 4163 4164 HSA Memory Model [HSA]_ 4165 The HSA memory model uses a single happens-before relation for all address 4166 spaces (see :ref:`amdgpu-address-spaces`). 4167 OpenCL Memory Model [OpenCL]_ 4168 The OpenCL memory model which has separate happens-before relations for the 4169 global and local address spaces. Only a fence specifying both global and 4170 local address space, and seq_cst instructions join the relationships. Since 4171 the LLVM ``memfence`` instruction does not allow an address space to be 4172 specified the OpenCL fence has to conservatively assume both local and 4173 global address space was specified. However, optimizations can often be 4174 done to eliminate the additional ``s_waitcnt`` instructions when there are 4175 no intervening memory instructions which access the corresponding address 4176 space. The code sequences in the table indicate what can be omitted for the 4177 OpenCL memory. The target triple environment is used to determine if the 4178 source language is OpenCL (see :ref:`amdgpu-opencl`). 4179 4180``ds/flat_load/store/atomic`` instructions to local memory are termed LDS 4181operations. 4182 4183``buffer/global/flat_load/store/atomic`` instructions to global memory are 4184termed vector memory operations. 4185 4186For GFX6-GFX9: 4187 4188* Each agent has multiple shader arrays (SA). 4189* Each SA has multiple compute units (CU). 4190* Each CU has multiple SIMDs that execute wavefronts. 4191* The wavefronts for a single work-group are executed in the same CU but may be 4192 executed by different SIMDs. 4193* Each CU has a single LDS memory shared by the wavefronts of the work-groups 4194 executing on it. 4195* All LDS operations of a CU are performed as wavefront wide operations in a 4196 global order and involve no caching. Completion is reported to a wavefront in 4197 execution order. 4198* The LDS memory has multiple request queues shared by the SIMDs of a 4199 CU. Therefore, the LDS operations performed by different wavefronts of a 4200 work-group can be reordered relative to each other, which can result in 4201 reordering the visibility of vector memory operations with respect to LDS 4202 operations of other wavefronts in the same work-group. A ``s_waitcnt 4203 lgkmcnt(0)`` is required to ensure synchronization between LDS operations and 4204 vector memory operations between wavefronts of a work-group, but not between 4205 operations performed by the same wavefront. 4206* The vector memory operations are performed as wavefront wide operations and 4207 completion is reported to a wavefront in execution order. The exception is 4208 that for GFX7-GFX9 ``flat_load/store/atomic`` instructions can report out of 4209 vector memory order if they access LDS memory, and out of LDS operation order 4210 if they access global memory. 4211* The vector memory operations access a single vector L1 cache shared by all 4212 SIMDs a CU. Therefore, no special action is required for coherence between the 4213 lanes of a single wavefront, or for coherence between wavefronts in the same 4214 work-group. A ``buffer_wbinvl1_vol`` is required for coherence between 4215 wavefronts executing in different work-groups as they may be executing on 4216 different CUs. 4217* The scalar memory operations access a scalar L1 cache shared by all wavefronts 4218 on a group of CUs. The scalar and vector L1 caches are not coherent. However, 4219 scalar operations are used in a restricted way so do not impact the memory 4220 model. See :ref:`amdgpu-amdhsa-memory-spaces`. 4221* The vector and scalar memory operations use an L2 cache shared by all CUs on 4222 the same agent. 4223* The L2 cache has independent channels to service disjoint ranges of virtual 4224 addresses. 4225* Each CU has a separate request queue per channel. Therefore, the vector and 4226 scalar memory operations performed by wavefronts executing in different 4227 work-groups (which may be executing on different CUs) of an agent can be 4228 reordered relative to each other. A ``s_waitcnt vmcnt(0)`` is required to 4229 ensure synchronization between vector memory operations of different CUs. It 4230 ensures a previous vector memory operation has completed before executing a 4231 subsequent vector memory or LDS operation and so can be used to meet the 4232 requirements of acquire and release. 4233* The L2 cache can be kept coherent with other agents on some targets, or ranges 4234 of virtual addresses can be set up to bypass it to ensure system coherence. 4235 4236For GFX10: 4237 4238* Each agent has multiple shader arrays (SA). 4239* Each SA has multiple work-group processors (WGP). 4240* Each WGP has multiple compute units (CU). 4241* Each CU has multiple SIMDs that execute wavefronts. 4242* The wavefronts for a single work-group are executed in the same 4243 WGP. In CU wavefront execution mode the wavefronts may be executed by 4244 different SIMDs in the same CU. In WGP wavefront execution mode the 4245 wavefronts may be executed by different SIMDs in different CUs in the same 4246 WGP. 4247* Each WGP has a single LDS memory shared by the wavefronts of the work-groups 4248 executing on it. 4249* All LDS operations of a WGP are performed as wavefront wide operations in a 4250 global order and involve no caching. Completion is reported to a wavefront in 4251 execution order. 4252* The LDS memory has multiple request queues shared by the SIMDs of a 4253 WGP. Therefore, the LDS operations performed by different wavefronts of a 4254 work-group can be reordered relative to each other, which can result in 4255 reordering the visibility of vector memory operations with respect to LDS 4256 operations of other wavefronts in the same work-group. A ``s_waitcnt 4257 lgkmcnt(0)`` is required to ensure synchronization between LDS operations and 4258 vector memory operations between wavefronts of a work-group, but not between 4259 operations performed by the same wavefront. 4260* The vector memory operations are performed as wavefront wide operations. 4261 Completion of load/store/sample operations are reported to a wavefront in 4262 execution order of other load/store/sample operations performed by that 4263 wavefront. 4264* The vector memory operations access a vector L0 cache. There is a single L0 4265 cache per CU. Each SIMD of a CU accesses the same L0 cache. Therefore, no 4266 special action is required for coherence between the lanes of a single 4267 wavefront. However, a ``BUFFER_GL0_INV`` is required for coherence between 4268 wavefronts executing in the same work-group as they may be executing on SIMDs 4269 of different CUs that access different L0s. A ``BUFFER_GL0_INV`` is also 4270 required for coherence between wavefronts executing in different work-groups 4271 as they may be executing on different WGPs. 4272* The scalar memory operations access a scalar L0 cache shared by all wavefronts 4273 on a WGP. The scalar and vector L0 caches are not coherent. However, scalar 4274 operations are used in a restricted way so do not impact the memory model. See 4275 :ref:`amdgpu-amdhsa-memory-spaces`. 4276* The vector and scalar memory L0 caches use an L1 cache shared by all WGPs on 4277 the same SA. Therefore, no special action is required for coherence between 4278 the wavefronts of a single work-group. However, a ``BUFFER_GL1_INV`` is 4279 required for coherence between wavefronts executing in different work-groups 4280 as they may be executing on different SAs that access different L1s. 4281* The L1 caches have independent quadrants to service disjoint ranges of virtual 4282 addresses. 4283* Each L0 cache has a separate request queue per L1 quadrant. Therefore, the 4284 vector and scalar memory operations performed by different wavefronts, whether 4285 executing in the same or different work-groups (which may be executing on 4286 different CUs accessing different L0s), can be reordered relative to each 4287 other. A ``s_waitcnt vmcnt(0) & vscnt(0)`` is required to ensure 4288 synchronization between vector memory operations of different wavefronts. It 4289 ensures a previous vector memory operation has completed before executing a 4290 subsequent vector memory or LDS operation and so can be used to meet the 4291 requirements of acquire, release and sequential consistency. 4292* The L1 caches use an L2 cache shared by all SAs on the same agent. 4293* The L2 cache has independent channels to service disjoint ranges of virtual 4294 addresses. 4295* Each L1 quadrant of a single SA accesses a different L2 channel. Each L1 4296 quadrant has a separate request queue per L2 channel. Therefore, the vector 4297 and scalar memory operations performed by wavefronts executing in different 4298 work-groups (which may be executing on different SAs) of an agent can be 4299 reordered relative to each other. A ``s_waitcnt vmcnt(0) & vscnt(0)`` is 4300 required to ensure synchronization between vector memory operations of 4301 different SAs. It ensures a previous vector memory operation has completed 4302 before executing a subsequent vector memory and so can be used to meet the 4303 requirements of acquire, release and sequential consistency. 4304* The L2 cache can be kept coherent with other agents on some targets, or ranges 4305 of virtual addresses can be set up to bypass it to ensure system coherence. 4306 4307Private address space uses ``buffer_load/store`` using the scratch V# 4308(GFX6-GFX8), or ``scratch_load/store`` (GFX9-GFX10). Since only a single thread 4309is accessing the memory, atomic memory orderings are not meaningful, and all 4310accesses are treated as non-atomic. 4311 4312Constant address space uses ``buffer/global_load`` instructions (or equivalent 4313scalar memory instructions). Since the constant address space contents do not 4314change during the execution of a kernel dispatch it is not legal to perform 4315stores, and atomic memory orderings are not meaningful, and all access are 4316treated as non-atomic. 4317 4318A memory synchronization scope wider than work-group is not meaningful for the 4319group (LDS) address space and is treated as work-group. 4320 4321The memory model does not support the region address space which is treated as 4322non-atomic. 4323 4324Acquire memory ordering is not meaningful on store atomic instructions and is 4325treated as non-atomic. 4326 4327Release memory ordering is not meaningful on load atomic instructions and is 4328treated a non-atomic. 4329 4330Acquire-release memory ordering is not meaningful on load or store atomic 4331instructions and is treated as acquire and release respectively. 4332 4333AMDGPU backend only uses scalar memory operations to access memory that is 4334proven to not change during the execution of the kernel dispatch. This includes 4335constant address space and global address space for program scope const 4336variables. Therefore, the kernel machine code does not have to maintain the 4337scalar L1 cache to ensure it is coherent with the vector L1 cache. The scalar 4338and vector L1 caches are invalidated between kernel dispatches by CP since 4339constant address space data may change between kernel dispatch executions. See 4340:ref:`amdgpu-amdhsa-memory-spaces`. 4341 4342The one exception is if scalar writes are used to spill SGPR registers. In this 4343case the AMDGPU backend ensures the memory location used to spill is never 4344accessed by vector memory operations at the same time. If scalar writes are used 4345then a ``s_dcache_wb`` is inserted before the ``s_endpgm`` and before a function 4346return since the locations may be used for vector memory instructions by a 4347future wavefront that uses the same scratch area, or a function call that 4348creates a frame at the same address, respectively. There is no need for a 4349``s_dcache_inv`` as all scalar writes are write-before-read in the same thread. 4350 4351For GFX6-GFX9, scratch backing memory (which is used for the private address 4352space) is accessed with MTYPE NC_NV (non-coherent non-volatile). Since the 4353private address space is only accessed by a single thread, and is always 4354write-before-read, there is never a need to invalidate these entries from the L1 4355cache. Hence all cache invalidates are done as ``*_vol`` to only invalidate the 4356volatile cache lines. 4357 4358For GFX10, scratch backing memory (which is used for the private address space) 4359is accessed with MTYPE NC (non-coherent). Since the private address space is 4360only accessed by a single thread, and is always write-before-read, there is 4361never a need to invalidate these entries from the L0 or L1 caches. 4362 4363For GFX10, wavefronts are executed in native mode with in-order reporting of 4364loads and sample instructions. In this mode vmcnt reports completion of load, 4365atomic with return and sample instructions in order, and the vscnt reports the 4366completion of store and atomic without return in order. See ``MEM_ORDERED`` 4367field in :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 4368 4369In GFX10, wavefronts can be executed in WGP or CU wavefront execution mode: 4370 4371* In WGP wavefront execution mode the wavefronts of a work-group are executed 4372 on the SIMDs of both CUs of the WGP. Therefore, explicit management of the per 4373 CU L0 caches is required for work-group synchronization. Also accesses to L1 4374 at work-group scope need to be explicitly ordered as the accesses from 4375 different CUs are not ordered. 4376* In CU wavefront execution mode the wavefronts of a work-group are executed on 4377 the SIMDs of a single CU of the WGP. Therefore, all global memory access by 4378 the work-group access the same L0 which in turn ensures L1 accesses are 4379 ordered and so do not require explicit management of the caches for 4380 work-group synchronization. 4381 4382See ``WGP_MODE`` field in 4383:ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table` and 4384:ref:`amdgpu-target-features`. 4385 4386On dGPU the kernarg backing memory is accessed as UC (uncached) to avoid needing 4387to invalidate the L2 cache. For GFX6-GFX9, this also causes it to be treated as 4388non-volatile and so is not invalidated by ``*_vol``. On APU it is accessed as CC 4389(cache coherent) and so the L2 cache will be coherent with the CPU and other 4390agents. 4391 4392 .. table:: AMDHSA Memory Model Code Sequences GFX6-GFX10 4393 :name: amdgpu-amdhsa-memory-model-code-sequences-gfx6-gfx10-table 4394 4395 ============ ============ ============== ========== =============================== ================================== 4396 LLVM Instr LLVM Memory LLVM Memory AMDGPU AMDGPU Machine Code AMDGPU Machine Code 4397 Ordering Sync Scope Address GFX6-9 GFX10 4398 Space 4399 ============ ============ ============== ========== =============================== ================================== 4400 **Non-Atomic** 4401 ---------------------------------------------------------------------------------------------------------------------- 4402 load *none* *none* - global - !volatile & !nontemporal - !volatile & !nontemporal 4403 - generic 4404 - private 1. buffer/global/flat_load 1. buffer/global/flat_load 4405 - constant 4406 - volatile & !nontemporal - volatile & !nontemporal 4407 4408 1. buffer/global/flat_load 1. buffer/global/flat_load 4409 glc=1 glc=1 dlc=1 4410 4411 - nontemporal - nontemporal 4412 4413 1. buffer/global/flat_load 1. buffer/global/flat_load 4414 glc=1 slc=1 slc=1 4415 4416 load *none* *none* - local 1. ds_load 1. ds_load 4417 store *none* *none* - global - !nontemporal - !nontemporal 4418 - generic 4419 - private 1. buffer/global/flat_store 1. buffer/global/flat_store 4420 - constant 4421 - nontemporal - nontemporal 4422 4423 1. buffer/global/flat_store 1. buffer/global/flat_store 4424 glc=1 slc=1 slc=1 4425 4426 store *none* *none* - local 1. ds_store 1. ds_store 4427 **Unordered Atomic** 4428 ---------------------------------------------------------------------------------------------------------------------- 4429 load atomic unordered *any* *any* *Same as non-atomic*. *Same as non-atomic*. 4430 store atomic unordered *any* *any* *Same as non-atomic*. *Same as non-atomic*. 4431 atomicrmw unordered *any* *any* *Same as monotonic *Same as monotonic 4432 atomic*. atomic*. 4433 **Monotonic Atomic** 4434 ---------------------------------------------------------------------------------------------------------------------- 4435 load atomic monotonic - singlethread - global 1. buffer/global/flat_load 1. buffer/global/flat_load 4436 - wavefront - generic 4437 load atomic monotonic - workgroup - global 1. buffer/global/flat_load 1. buffer/global/flat_load 4438 - generic glc=1 4439 4440 - If CU wavefront execution mode, omit glc=1. 4441 4442 load atomic monotonic - singlethread - local 1. ds_load 1. ds_load 4443 - wavefront 4444 - workgroup 4445 load atomic monotonic - agent - global 1. buffer/global/flat_load 1. buffer/global/flat_load 4446 - system - generic glc=1 glc=1 dlc=1 4447 store atomic monotonic - singlethread - global 1. buffer/global/flat_store 1. buffer/global/flat_store 4448 - wavefront - generic 4449 - workgroup 4450 - agent 4451 - system 4452 store atomic monotonic - singlethread - local 1. ds_store 1. ds_store 4453 - wavefront 4454 - workgroup 4455 atomicrmw monotonic - singlethread - global 1. buffer/global/flat_atomic 1. buffer/global/flat_atomic 4456 - wavefront - generic 4457 - workgroup 4458 - agent 4459 - system 4460 atomicrmw monotonic - singlethread - local 1. ds_atomic 1. ds_atomic 4461 - wavefront 4462 - workgroup 4463 **Acquire Atomic** 4464 ---------------------------------------------------------------------------------------------------------------------- 4465 load atomic acquire - singlethread - global 1. buffer/global/ds/flat_load 1. buffer/global/ds/flat_load 4466 - wavefront - local 4467 - generic 4468 load atomic acquire - workgroup - global 1. buffer/global/flat_load 1. buffer/global_load glc=1 4469 4470 - If CU wavefront execution mode, omit glc=1. 4471 4472 2. s_waitcnt vmcnt(0) 4473 4474 - If CU wavefront execution mode, omit. 4475 - Must happen before 4476 the following buffer_gl0_inv 4477 and before any following 4478 global/generic 4479 load/load 4480 atomic/store/store 4481 atomic/atomicrmw. 4482 4483 3. buffer_gl0_inv 4484 4485 - If CU wavefront execution mode, omit. 4486 - Ensures that 4487 following 4488 loads will not see 4489 stale data. 4490 4491 load atomic acquire - workgroup - local 1. ds_load 1. ds_load 4492 2. s_waitcnt lgkmcnt(0) 2. s_waitcnt lgkmcnt(0) 4493 4494 - If OpenCL, omit. - If OpenCL, omit. 4495 - Must happen before - Must happen before 4496 any following the following buffer_gl0_inv 4497 global/generic and before any following 4498 load/load global/generic load/load 4499 atomic/store/store atomic/store/store 4500 atomic/atomicrmw. atomic/atomicrmw. 4501 - Ensures any - Ensures any 4502 following global following global 4503 data read is no data read is no 4504 older than the load older than the load 4505 atomic value being atomic value being 4506 acquired. acquired. 4507 4508 3. buffer_gl0_inv 4509 4510 - If CU wavefront execution mode, omit. 4511 - If OpenCL, omit. 4512 - Ensures that 4513 following 4514 loads will not see 4515 stale data. 4516 4517 load atomic acquire - workgroup - generic 1. flat_load 1. flat_load glc=1 4518 4519 - If CU wavefront execution mode, omit glc=1. 4520 4521 2. s_waitcnt lgkmcnt(0) 2. s_waitcnt lgkmcnt(0) & 4522 vmcnt(0) 4523 4524 - If CU wavefront execution mode, omit vmcnt. 4525 - If OpenCL, omit. - If OpenCL, omit 4526 lgkmcnt(0). 4527 - Must happen before - Must happen before 4528 any following the following 4529 global/generic buffer_gl0_inv and any 4530 load/load following global/generic 4531 atomic/store/store load/load 4532 atomic/atomicrmw. atomic/store/store 4533 atomic/atomicrmw. 4534 - Ensures any - Ensures any 4535 following global following global 4536 data read is no data read is no 4537 older than the load older than the load 4538 atomic value being atomic value being 4539 acquired. acquired. 4540 4541 3. buffer_gl0_inv 4542 4543 - If CU wavefront execution mode, omit. 4544 - Ensures that 4545 following 4546 loads will not see 4547 stale data. 4548 4549 load atomic acquire - agent - global 1. buffer/global/flat_load 1. buffer/global_load 4550 - system glc=1 glc=1 dlc=1 4551 2. s_waitcnt vmcnt(0) 2. s_waitcnt vmcnt(0) 4552 4553 - Must happen before - Must happen before 4554 following following 4555 buffer_wbinvl1_vol. buffer_gl*_inv. 4556 - Ensures the load - Ensures the load 4557 has completed has completed 4558 before invalidating before invalidating 4559 the cache. the caches. 4560 4561 3. buffer_wbinvl1_vol 3. buffer_gl0_inv; 4562 buffer_gl1_inv 4563 4564 - Must happen before - Must happen before 4565 any following any following 4566 global/generic global/generic 4567 load/load load/load 4568 atomic/atomicrmw. atomic/atomicrmw. 4569 - Ensures that - Ensures that 4570 following following 4571 loads will not see loads will not see 4572 stale global data. stale global data. 4573 4574 load atomic acquire - agent - generic 1. flat_load glc=1 1. flat_load glc=1 dlc=1 4575 - system 2. s_waitcnt vmcnt(0) & 2. s_waitcnt vmcnt(0) & 4576 lgkmcnt(0) lgkmcnt(0) 4577 4578 - If OpenCL omit - If OpenCL omit 4579 lgkmcnt(0). lgkmcnt(0). 4580 - Must happen before - Must happen before 4581 following following 4582 buffer_wbinvl1_vol. buffer_gl*_invl. 4583 - Ensures the flat_load - Ensures the flat_load 4584 has completed has completed 4585 before invalidating before invalidating 4586 the cache. the caches. 4587 4588 3. buffer_wbinvl1_vol 3. buffer_gl0_inv; 4589 buffer_gl1_inv 4590 4591 - Must happen before - Must happen before 4592 any following any following 4593 global/generic global/generic 4594 load/load load/load 4595 atomic/atomicrmw. atomic/atomicrmw. 4596 - Ensures that - Ensures that 4597 following loads following loads 4598 will not see stale will not see stale 4599 global data. global data. 4600 4601 atomicrmw acquire - singlethread - global 1. buffer/global/ds/flat_atomic 1. buffer/global/ds/flat_atomic 4602 - wavefront - local 4603 - generic 4604 atomicrmw acquire - workgroup - global 1. buffer/global/flat_atomic 1. buffer/global_atomic 4605 2. s_waitcnt vm/vscnt(0) 4606 4607 - If CU wavefront execution mode, omit. 4608 - Use vmcnt if atomic with 4609 return and vscnt if atomic 4610 with no-return. 4611 - Must happen before 4612 the following buffer_gl0_inv 4613 and before any following 4614 global/generic 4615 load/load 4616 atomic/store/store 4617 atomic/atomicrmw. 4618 4619 3. buffer_gl0_inv 4620 4621 - If CU wavefront execution mode, omit. 4622 - Ensures that 4623 following 4624 loads will not see 4625 stale data. 4626 4627 atomicrmw acquire - workgroup - local 1. ds_atomic 1. ds_atomic 4628 2. waitcnt lgkmcnt(0) 2. waitcnt lgkmcnt(0) 4629 4630 - If OpenCL, omit. - If OpenCL, omit. 4631 - Must happen before - Must happen before 4632 any following the following 4633 global/generic buffer_gl0_inv. 4634 load/load 4635 atomic/store/store 4636 atomic/atomicrmw. 4637 - Ensures any - Ensures any 4638 following global following global 4639 data read is no data read is no 4640 older than the older than the 4641 atomicrmw value atomicrmw value 4642 being acquired. being acquired. 4643 4644 3. buffer_gl0_inv 4645 4646 - If OpenCL omit. 4647 - Ensures that 4648 following 4649 loads will not see 4650 stale data. 4651 4652 atomicrmw acquire - workgroup - generic 1. flat_atomic 1. flat_atomic 4653 2. waitcnt lgkmcnt(0) 2. waitcnt lgkmcnt(0) & 4654 vm/vscnt(0) 4655 4656 - If CU wavefront execution mode, omit vm/vscnt. 4657 - If OpenCL, omit. - If OpenCL, omit 4658 waitcnt lgkmcnt(0).. 4659 - Use vmcnt if atomic with 4660 return and vscnt if atomic 4661 with no-return. 4662 waitcnt lgkmcnt(0). 4663 - Must happen before - Must happen before 4664 any following the following 4665 global/generic buffer_gl0_inv. 4666 load/load 4667 atomic/store/store 4668 atomic/atomicrmw. 4669 - Ensures any - Ensures any 4670 following global following global 4671 data read is no data read is no 4672 older than the older than the 4673 atomicrmw value atomicrmw value 4674 being acquired. being acquired. 4675 4676 3. buffer_gl0_inv 4677 4678 - If CU wavefront execution mode, omit. 4679 - Ensures that 4680 following 4681 loads will not see 4682 stale data. 4683 4684 atomicrmw acquire - agent - global 1. buffer/global/flat_atomic 1. buffer/global_atomic 4685 - system 2. s_waitcnt vmcnt(0) 2. s_waitcnt vm/vscnt(0) 4686 4687 - Use vmcnt if atomic with 4688 return and vscnt if atomic 4689 with no-return. 4690 waitcnt lgkmcnt(0). 4691 - Must happen before - Must happen before 4692 following following 4693 buffer_wbinvl1_vol. buffer_gl*_inv. 4694 - Ensures the - Ensures the 4695 atomicrmw has atomicrmw has 4696 completed before completed before 4697 invalidating the invalidating the 4698 cache. caches. 4699 4700 3. buffer_wbinvl1_vol 3. buffer_gl0_inv; 4701 buffer_gl1_inv 4702 4703 - Must happen before - Must happen before 4704 any following any following 4705 global/generic global/generic 4706 load/load load/load 4707 atomic/atomicrmw. atomic/atomicrmw. 4708 - Ensures that - Ensures that 4709 following loads following loads 4710 will not see stale will not see stale 4711 global data. global data. 4712 4713 atomicrmw acquire - agent - generic 1. flat_atomic 1. flat_atomic 4714 - system 2. s_waitcnt vmcnt(0) & 2. s_waitcnt vm/vscnt(0) & 4715 lgkmcnt(0) lgkmcnt(0) 4716 4717 - If OpenCL, omit - If OpenCL, omit 4718 lgkmcnt(0). lgkmcnt(0). 4719 - Use vmcnt if atomic with 4720 return and vscnt if atomic 4721 with no-return. 4722 - Must happen before - Must happen before 4723 following following 4724 buffer_wbinvl1_vol. buffer_gl*_inv. 4725 - Ensures the - Ensures the 4726 atomicrmw has atomicrmw has 4727 completed before completed before 4728 invalidating the invalidating the 4729 cache. caches. 4730 4731 3. buffer_wbinvl1_vol 3. buffer_gl0_inv; 4732 buffer_gl1_inv 4733 4734 - Must happen before - Must happen before 4735 any following any following 4736 global/generic global/generic 4737 load/load load/load 4738 atomic/atomicrmw. atomic/atomicrmw. 4739 - Ensures that - Ensures that 4740 following loads following loads 4741 will not see stale will not see stale 4742 global data. global data. 4743 4744 fence acquire - singlethread *none* *none* *none* 4745 - wavefront 4746 fence acquire - workgroup *none* 1. s_waitcnt lgkmcnt(0) 1. s_waitcnt lgkmcnt(0) & 4747 vmcnt(0) & vscnt(0) 4748 4749 - If CU wavefront execution mode, omit vmcnt and 4750 vscnt. 4751 - If OpenCL and - If OpenCL and 4752 address space is address space is 4753 not generic, omit. not generic, omit 4754 lgkmcnt(0). 4755 - If OpenCL and 4756 address space is 4757 local, omit 4758 vmcnt(0) and vscnt(0). 4759 - However, since LLVM - However, since LLVM 4760 currently has no currently has no 4761 address space on address space on 4762 the fence need to the fence need to 4763 conservatively conservatively 4764 always generate. If always generate. If 4765 fence had an fence had an 4766 address space then address space then 4767 set to address set to address 4768 space of OpenCL space of OpenCL 4769 fence flag, or to fence flag, or to 4770 generic if both generic if both 4771 local and global local and global 4772 flags are flags are 4773 specified. specified. 4774 - Must happen after 4775 any preceding 4776 local/generic load 4777 atomic/atomicrmw 4778 with an equal or 4779 wider sync scope 4780 and memory ordering 4781 stronger than 4782 unordered (this is 4783 termed the 4784 fence-paired-atomic). 4785 - Must happen before 4786 any following 4787 global/generic 4788 load/load 4789 atomic/store/store 4790 atomic/atomicrmw. 4791 - Ensures any 4792 following global 4793 data read is no 4794 older than the 4795 value read by the 4796 fence-paired-atomic. 4797 - Could be split into 4798 separate s_waitcnt 4799 vmcnt(0), s_waitcnt 4800 vscnt(0) and s_waitcnt 4801 lgkmcnt(0) to allow 4802 them to be 4803 independently moved 4804 according to the 4805 following rules. 4806 - s_waitcnt vmcnt(0) 4807 must happen after 4808 any preceding 4809 global/generic load 4810 atomic/ 4811 atomicrmw-with-return-value 4812 with an equal or 4813 wider sync scope 4814 and memory ordering 4815 stronger than 4816 unordered (this is 4817 termed the 4818 fence-paired-atomic). 4819 - s_waitcnt vscnt(0) 4820 must happen after 4821 any preceding 4822 global/generic 4823 atomicrmw-no-return-value 4824 with an equal or 4825 wider sync scope 4826 and memory ordering 4827 stronger than 4828 unordered (this is 4829 termed the 4830 fence-paired-atomic). 4831 - s_waitcnt lgkmcnt(0) 4832 must happen after 4833 any preceding 4834 local/generic load 4835 atomic/atomicrmw 4836 with an equal or 4837 wider sync scope 4838 and memory ordering 4839 stronger than 4840 unordered (this is 4841 termed the 4842 fence-paired-atomic). 4843 - Must happen before 4844 the following 4845 buffer_gl0_inv. 4846 - Ensures that the 4847 fence-paired atomic 4848 has completed 4849 before invalidating 4850 the 4851 cache. Therefore 4852 any following 4853 locations read must 4854 be no older than 4855 the value read by 4856 the 4857 fence-paired-atomic. 4858 4859 3. buffer_gl0_inv 4860 4861 - If CU wavefront execution mode, omit. 4862 - Ensures that 4863 following 4864 loads will not see 4865 stale data. 4866 4867 fence acquire - agent *none* 1. s_waitcnt lgkmcnt(0) & 1. s_waitcnt lgkmcnt(0) & 4868 - system vmcnt(0) vmcnt(0) & vscnt(0) 4869 4870 - If OpenCL and - If OpenCL and 4871 address space is address space is 4872 not generic, omit not generic, omit 4873 lgkmcnt(0). lgkmcnt(0). 4874 - If OpenCL and 4875 address space is 4876 local, omit 4877 vmcnt(0) and vscnt(0). 4878 - However, since LLVM - However, since LLVM 4879 currently has no currently has no 4880 address space on address space on 4881 the fence need to the fence need to 4882 conservatively conservatively 4883 always generate always generate 4884 (see comment for (see comment for 4885 previous fence). previous fence). 4886 - Could be split into 4887 separate s_waitcnt 4888 vmcnt(0) and 4889 s_waitcnt 4890 lgkmcnt(0) to allow 4891 them to be 4892 independently moved 4893 according to the 4894 following rules. 4895 - s_waitcnt vmcnt(0) 4896 must happen after 4897 any preceding 4898 global/generic load 4899 atomic/atomicrmw 4900 with an equal or 4901 wider sync scope 4902 and memory ordering 4903 stronger than 4904 unordered (this is 4905 termed the 4906 fence-paired-atomic). 4907 - s_waitcnt lgkmcnt(0) 4908 must happen after 4909 any preceding 4910 local/generic load 4911 atomic/atomicrmw 4912 with an equal or 4913 wider sync scope 4914 and memory ordering 4915 stronger than 4916 unordered (this is 4917 termed the 4918 fence-paired-atomic). 4919 - Must happen before 4920 the following 4921 buffer_wbinvl1_vol. 4922 - Ensures that the 4923 fence-paired atomic 4924 has completed 4925 before invalidating 4926 the 4927 cache. Therefore 4928 any following 4929 locations read must 4930 be no older than 4931 the value read by 4932 the 4933 fence-paired-atomic. 4934 - Could be split into 4935 separate s_waitcnt 4936 vmcnt(0), s_waitcnt 4937 vscnt(0) and s_waitcnt 4938 lgkmcnt(0) to allow 4939 them to be 4940 independently moved 4941 according to the 4942 following rules. 4943 - s_waitcnt vmcnt(0) 4944 must happen after 4945 any preceding 4946 global/generic load 4947 atomic/ 4948 atomicrmw-with-return-value 4949 with an equal or 4950 wider sync scope 4951 and memory ordering 4952 stronger than 4953 unordered (this is 4954 termed the 4955 fence-paired-atomic). 4956 - s_waitcnt vscnt(0) 4957 must happen after 4958 any preceding 4959 global/generic 4960 atomicrmw-no-return-value 4961 with an equal or 4962 wider sync scope 4963 and memory ordering 4964 stronger than 4965 unordered (this is 4966 termed the 4967 fence-paired-atomic). 4968 - s_waitcnt lgkmcnt(0) 4969 must happen after 4970 any preceding 4971 local/generic load 4972 atomic/atomicrmw 4973 with an equal or 4974 wider sync scope 4975 and memory ordering 4976 stronger than 4977 unordered (this is 4978 termed the 4979 fence-paired-atomic). 4980 - Must happen before 4981 the following 4982 buffer_gl*_inv. 4983 - Ensures that the 4984 fence-paired atomic 4985 has completed 4986 before invalidating 4987 the 4988 caches. Therefore 4989 any following 4990 locations read must 4991 be no older than 4992 the value read by 4993 the 4994 fence-paired-atomic. 4995 4996 2. buffer_wbinvl1_vol 2. buffer_gl0_inv; 4997 buffer_gl1_inv 4998 4999 - Must happen before any - Must happen before any 5000 following global/generic following global/generic 5001 load/load load/load 5002 atomic/store/store atomic/store/store 5003 atomic/atomicrmw. atomic/atomicrmw. 5004 - Ensures that - Ensures that 5005 following loads following loads 5006 will not see stale will not see stale 5007 global data. global data. 5008 5009 **Release Atomic** 5010 ---------------------------------------------------------------------------------------------------------------------- 5011 store atomic release - singlethread - global 1. buffer/global/ds/flat_store 1. buffer/global/ds/flat_store 5012 - wavefront - local 5013 - generic 5014 store atomic release - workgroup - global 1. s_waitcnt lgkmcnt(0) 1. s_waitcnt lgkmcnt(0) & 5015 vmcnt(0) & vscnt(0) 5016 5017 - If CU wavefront execution mode, omit vmcnt and 5018 vscnt. 5019 - If OpenCL, omit. - If OpenCL, omit 5020 lgkmcnt(0). 5021 - Must happen after 5022 any preceding 5023 local/generic 5024 load/store/load 5025 atomic/store 5026 atomic/atomicrmw. 5027 - Could be split into 5028 separate s_waitcnt 5029 vmcnt(0), s_waitcnt 5030 vscnt(0) and s_waitcnt 5031 lgkmcnt(0) to allow 5032 them to be 5033 independently moved 5034 according to the 5035 following rules. 5036 - s_waitcnt vmcnt(0) 5037 must happen after 5038 any preceding 5039 global/generic load/load 5040 atomic/ 5041 atomicrmw-with-return-value. 5042 - s_waitcnt vscnt(0) 5043 must happen after 5044 any preceding 5045 global/generic 5046 store/store 5047 atomic/ 5048 atomicrmw-no-return-value. 5049 - s_waitcnt lgkmcnt(0) 5050 must happen after 5051 any preceding 5052 local/generic 5053 load/store/load 5054 atomic/store 5055 atomic/atomicrmw. 5056 - Must happen before - Must happen before 5057 the following the following 5058 store. store. 5059 - Ensures that all - Ensures that all 5060 memory operations memory operations 5061 to local have have 5062 completed before completed before 5063 performing the performing the 5064 store that is being store that is being 5065 released. released. 5066 5067 2. buffer/global/flat_store 2. buffer/global_store 5068 store atomic release - workgroup - local 1. waitcnt vmcnt(0) & vscnt(0) 5069 5070 - If CU wavefront execution mode, omit. 5071 - If OpenCL, omit. 5072 - Could be split into 5073 separate s_waitcnt 5074 vmcnt(0) and s_waitcnt 5075 vscnt(0) to allow 5076 them to be 5077 independently moved 5078 according to the 5079 following rules. 5080 - s_waitcnt vmcnt(0) 5081 must happen after 5082 any preceding 5083 global/generic load/load 5084 atomic/ 5085 atomicrmw-with-return-value. 5086 - s_waitcnt vscnt(0) 5087 must happen after 5088 any preceding 5089 global/generic 5090 store/store atomic/ 5091 atomicrmw-no-return-value. 5092 - Must happen before 5093 the following 5094 store. 5095 - Ensures that all 5096 global memory 5097 operations have 5098 completed before 5099 performing the 5100 store that is being 5101 released. 5102 5103 1. ds_store 2. ds_store 5104 store atomic release - workgroup - generic 1. s_waitcnt lgkmcnt(0) 1. s_waitcnt lgkmcnt(0) & 5105 vmcnt(0) & vscnt(0) 5106 5107 - If CU wavefront execution mode, omit vmcnt and 5108 vscnt. 5109 - If OpenCL, omit. - If OpenCL, omit 5110 lgkmcnt(0). 5111 - Must happen after 5112 any preceding 5113 local/generic 5114 load/store/load 5115 atomic/store 5116 atomic/atomicrmw. 5117 - Could be split into 5118 separate s_waitcnt 5119 vmcnt(0), s_waitcnt 5120 vscnt(0) and s_waitcnt 5121 lgkmcnt(0) to allow 5122 them to be 5123 independently moved 5124 according to the 5125 following rules. 5126 - s_waitcnt vmcnt(0) 5127 must happen after 5128 any preceding 5129 global/generic load/load 5130 atomic/ 5131 atomicrmw-with-return-value. 5132 - s_waitcnt vscnt(0) 5133 must happen after 5134 any preceding 5135 global/generic 5136 store/store 5137 atomic/ 5138 atomicrmw-no-return-value. 5139 - s_waitcnt lgkmcnt(0) 5140 must happen after 5141 any preceding 5142 local/generic load/store/load 5143 atomic/store atomic/atomicrmw. 5144 - Must happen before - Must happen before 5145 the following the following 5146 store. store. 5147 - Ensures that all - Ensures that all 5148 memory operations memory operations 5149 to local have have 5150 completed before completed before 5151 performing the performing the 5152 store that is being store that is being 5153 released. released. 5154 5155 2. flat_store 2. flat_store 5156 store atomic release - agent - global 1. s_waitcnt lgkmcnt(0) & 1. s_waitcnt lgkmcnt(0) & 5157 - system - generic vmcnt(0) vmcnt(0) & vscnt(0) 5158 5159 - If OpenCL, omit - If OpenCL, omit 5160 lgkmcnt(0). lgkmcnt(0). 5161 - Could be split into - Could be split into 5162 separate s_waitcnt separate s_waitcnt 5163 vmcnt(0) and vmcnt(0), s_waitcnt vscnt(0) 5164 s_waitcnt and s_waitcnt 5165 lgkmcnt(0) to allow lgkmcnt(0) to allow 5166 them to be them to be 5167 independently moved independently moved 5168 according to the according to the 5169 following rules. following rules. 5170 - s_waitcnt vmcnt(0) - s_waitcnt vmcnt(0) 5171 must happen after must happen after 5172 any preceding any preceding 5173 global/generic global/generic 5174 load/store/load load/load 5175 atomic/store atomic/ 5176 atomic/atomicrmw. atomicrmw-with-return-value. 5177 - s_waitcnt vscnt(0) 5178 must happen after 5179 any preceding 5180 global/generic 5181 store/store atomic/ 5182 atomicrmw-no-return-value. 5183 - s_waitcnt lgkmcnt(0) - s_waitcnt lgkmcnt(0) 5184 must happen after must happen after 5185 any preceding any preceding 5186 local/generic local/generic 5187 load/store/load load/store/load 5188 atomic/store atomic/store 5189 atomic/atomicrmw. atomic/atomicrmw. 5190 - Must happen before - Must happen before 5191 the following the following 5192 store. store. 5193 - Ensures that all - Ensures that all 5194 memory operations memory operations 5195 to memory have to memory have 5196 completed before completed before 5197 performing the performing the 5198 store that is being store that is being 5199 released. released. 5200 5201 2. buffer/global/ds/flat_store 2. buffer/global/ds/flat_store 5202 atomicrmw release - singlethread - global 1. buffer/global/ds/flat_atomic 1. buffer/global/ds/flat_atomic 5203 - wavefront - local 5204 - generic 5205 atomicrmw release - workgroup - global 1. s_waitcnt lgkmcnt(0) 1. s_waitcnt lgkmcnt(0) & 5206 vmcnt(0) & vscnt(0) 5207 5208 - If CU wavefront execution mode, omit vmcnt and 5209 vscnt. 5210 - If OpenCL, omit. 5211 5212 - Must happen after 5213 any preceding 5214 local/generic 5215 load/store/load 5216 atomic/store 5217 atomic/atomicrmw. 5218 - Could be split into 5219 separate s_waitcnt 5220 vmcnt(0), s_waitcnt 5221 vscnt(0) and s_waitcnt 5222 lgkmcnt(0) to allow 5223 them to be 5224 independently moved 5225 according to the 5226 following rules. 5227 - s_waitcnt vmcnt(0) 5228 must happen after 5229 any preceding 5230 global/generic load/load 5231 atomic/ 5232 atomicrmw-with-return-value. 5233 - s_waitcnt vscnt(0) 5234 must happen after 5235 any preceding 5236 global/generic 5237 store/store 5238 atomic/ 5239 atomicrmw-no-return-value. 5240 - s_waitcnt lgkmcnt(0) 5241 must happen after 5242 any preceding 5243 local/generic 5244 load/store/load 5245 atomic/store 5246 atomic/atomicrmw. 5247 - Must happen before - Must happen before 5248 the following the following 5249 atomicrmw. atomicrmw. 5250 - Ensures that all - Ensures that all 5251 memory operations memory operations 5252 to local have have 5253 completed before completed before 5254 performing the performing the 5255 atomicrmw that is atomicrmw that is 5256 being released. being released. 5257 5258 2. buffer/global/flat_atomic 2. buffer/global_atomic 5259 atomicrmw release - workgroup - local 1. waitcnt vmcnt(0) & vscnt(0) 5260 5261 - If CU wavefront execution mode, omit. 5262 - If OpenCL, omit. 5263 - Could be split into 5264 separate s_waitcnt 5265 vmcnt(0) and s_waitcnt 5266 vscnt(0) to allow 5267 them to be 5268 independently moved 5269 according to the 5270 following rules. 5271 - s_waitcnt vmcnt(0) 5272 must happen after 5273 any preceding 5274 global/generic load/load 5275 atomic/ 5276 atomicrmw-with-return-value. 5277 - s_waitcnt vscnt(0) 5278 must happen after 5279 any preceding 5280 global/generic 5281 store/store atomic/ 5282 atomicrmw-no-return-value. 5283 - Must happen before 5284 the following 5285 store. 5286 - Ensures that all 5287 global memory 5288 operations have 5289 completed before 5290 performing the 5291 store that is being 5292 released. 5293 5294 1. ds_atomic 2. ds_atomic 5295 atomicrmw release - workgroup - generic 1. s_waitcnt lgkmcnt(0) 1. s_waitcnt lgkmcnt(0) & 5296 vmcnt(0) & vscnt(0) 5297 5298 - If CU wavefront execution mode, omit vmcnt and 5299 vscnt. 5300 - If OpenCL, omit. - If OpenCL, omit 5301 waitcnt lgkmcnt(0). 5302 - Must happen after 5303 any preceding 5304 local/generic 5305 load/store/load 5306 atomic/store 5307 atomic/atomicrmw. 5308 - Could be split into 5309 separate s_waitcnt 5310 vmcnt(0), s_waitcnt 5311 vscnt(0) and s_waitcnt 5312 lgkmcnt(0) to allow 5313 them to be 5314 independently moved 5315 according to the 5316 following rules. 5317 - s_waitcnt vmcnt(0) 5318 must happen after 5319 any preceding 5320 global/generic load/load 5321 atomic/ 5322 atomicrmw-with-return-value. 5323 - s_waitcnt vscnt(0) 5324 must happen after 5325 any preceding 5326 global/generic 5327 store/store 5328 atomic/ 5329 atomicrmw-no-return-value. 5330 - s_waitcnt lgkmcnt(0) 5331 must happen after 5332 any preceding 5333 local/generic load/store/load 5334 atomic/store atomic/atomicrmw. 5335 - Must happen before - Must happen before 5336 the following the following 5337 atomicrmw. atomicrmw. 5338 - Ensures that all - Ensures that all 5339 memory operations memory operations 5340 to local have have 5341 completed before completed before 5342 performing the performing the 5343 atomicrmw that is atomicrmw that is 5344 being released. being released. 5345 5346 2. flat_atomic 2. flat_atomic 5347 atomicrmw release - agent - global 1. s_waitcnt lgkmcnt(0) & 1. s_waitcnt lkkmcnt(0) & 5348 - system - generic vmcnt(0) vmcnt(0) & vscnt(0) 5349 5350 - If OpenCL, omit - If OpenCL, omit 5351 lgkmcnt(0). lgkmcnt(0). 5352 - Could be split into - Could be split into 5353 separate s_waitcnt separate s_waitcnt 5354 vmcnt(0) and vmcnt(0), s_waitcnt 5355 s_waitcnt vscnt(0) and s_waitcnt 5356 lgkmcnt(0) to allow lgkmcnt(0) to allow 5357 them to be them to be 5358 independently moved independently moved 5359 according to the according to the 5360 following rules. following rules. 5361 - s_waitcnt vmcnt(0) - s_waitcnt vmcnt(0) 5362 must happen after must happen after 5363 any preceding any preceding 5364 global/generic global/generic 5365 load/store/load load/load atomic/ 5366 atomic/store atomicrmw-with-return-value. 5367 atomic/atomicrmw. 5368 - s_waitcnt vscnt(0) 5369 must happen after 5370 any preceding 5371 global/generic 5372 store/store atomic/ 5373 atomicrmw-no-return-value. 5374 - s_waitcnt lgkmcnt(0) - s_waitcnt lgkmcnt(0) 5375 must happen after must happen after 5376 any preceding any preceding 5377 local/generic local/generic 5378 load/store/load load/store/load 5379 atomic/store atomic/store 5380 atomic/atomicrmw. atomic/atomicrmw. 5381 - Must happen before - Must happen before 5382 the following the following 5383 atomicrmw. atomicrmw. 5384 - Ensures that all - Ensures that all 5385 memory operations memory operations 5386 to global and local to global and local 5387 have completed have completed 5388 before performing before performing 5389 the atomicrmw that the atomicrmw that 5390 is being released. is being released. 5391 5392 2. buffer/global/ds/flat_atomic 2. buffer/global/ds/flat_atomic 5393 fence release - singlethread *none* *none* *none* 5394 - wavefront 5395 fence release - workgroup *none* 1. s_waitcnt lgkmcnt(0) 1. s_waitcnt lgkmcnt(0) & 5396 vmcnt(0) & vscnt(0) 5397 5398 - If CU wavefront execution mode, omit vmcnt and 5399 vscnt. 5400 - If OpenCL and - If OpenCL and 5401 address space is address space is 5402 not generic, omit. not generic, omit 5403 lgkmcnt(0). 5404 - If OpenCL and 5405 address space is 5406 local, omit 5407 vmcnt(0) and vscnt(0). 5408 - However, since LLVM - However, since LLVM 5409 currently has no currently has no 5410 address space on address space on 5411 the fence need to the fence need to 5412 conservatively conservatively 5413 always generate. If always generate. If 5414 fence had an fence had an 5415 address space then address space then 5416 set to address set to address 5417 space of OpenCL space of OpenCL 5418 fence flag, or to fence flag, or to 5419 generic if both generic if both 5420 local and global local and global 5421 flags are flags are 5422 specified. specified. 5423 - Must happen after 5424 any preceding 5425 local/generic 5426 load/load 5427 atomic/store/store 5428 atomic/atomicrmw. 5429 - Could be split into 5430 separate s_waitcnt 5431 vmcnt(0), s_waitcnt 5432 vscnt(0) and s_waitcnt 5433 lgkmcnt(0) to allow 5434 them to be 5435 independently moved 5436 according to the 5437 following rules. 5438 - s_waitcnt vmcnt(0) 5439 must happen after 5440 any preceding 5441 global/generic 5442 load/load 5443 atomic/ 5444 atomicrmw-with-return-value. 5445 - s_waitcnt vscnt(0) 5446 must happen after 5447 any preceding 5448 global/generic 5449 store/store atomic/ 5450 atomicrmw-no-return-value. 5451 - s_waitcnt lgkmcnt(0) 5452 must happen after 5453 any preceding 5454 local/generic 5455 load/store/load 5456 atomic/store atomic/ 5457 atomicrmw. 5458 - Must happen before - Must happen before 5459 any following store any following store 5460 atomic/atomicrmw atomic/atomicrmw 5461 with an equal or with an equal or 5462 wider sync scope wider sync scope 5463 and memory ordering and memory ordering 5464 stronger than stronger than 5465 unordered (this is unordered (this is 5466 termed the termed the 5467 fence-paired-atomic). fence-paired-atomic). 5468 - Ensures that all - Ensures that all 5469 memory operations memory operations 5470 to local have have 5471 completed before completed before 5472 performing the performing the 5473 following following 5474 fence-paired-atomic. fence-paired-atomic. 5475 5476 fence release - agent *none* 1. s_waitcnt lgkmcnt(0) & 1. s_waitcnt lgkmcnt(0) & 5477 - system vmcnt(0) vmcnt(0) & vscnt(0) 5478 5479 - If OpenCL and - If OpenCL and 5480 address space is address space is 5481 not generic, omit not generic, omit 5482 lgkmcnt(0). lgkmcnt(0). 5483 - If OpenCL and - If OpenCL and 5484 address space is address space is 5485 local, omit local, omit 5486 vmcnt(0). vmcnt(0) and vscnt(0). 5487 - However, since LLVM - However, since LLVM 5488 currently has no currently has no 5489 address space on address space on 5490 the fence need to the fence need to 5491 conservatively conservatively 5492 always generate. If always generate. If 5493 fence had an fence had an 5494 address space then address space then 5495 set to address set to address 5496 space of OpenCL space of OpenCL 5497 fence flag, or to fence flag, or to 5498 generic if both generic if both 5499 local and global local and global 5500 flags are flags are 5501 specified. specified. 5502 - Could be split into - Could be split into 5503 separate s_waitcnt separate s_waitcnt 5504 vmcnt(0) and vmcnt(0), s_waitcnt 5505 s_waitcnt vscnt(0) and s_waitcnt 5506 lgkmcnt(0) to allow lgkmcnt(0) to allow 5507 them to be them to be 5508 independently moved independently moved 5509 according to the according to the 5510 following rules. following rules. 5511 - s_waitcnt vmcnt(0) - s_waitcnt vmcnt(0) 5512 must happen after must happen after 5513 any preceding any preceding 5514 global/generic global/generic 5515 load/store/load load/load atomic/ 5516 atomic/store atomicrmw-with-return-value. 5517 atomic/atomicrmw. 5518 - s_waitcnt vscnt(0) 5519 must happen after 5520 any preceding 5521 global/generic 5522 store/store atomic/ 5523 atomicrmw-no-return-value. 5524 - s_waitcnt lgkmcnt(0) - s_waitcnt lgkmcnt(0) 5525 must happen after must happen after 5526 any preceding any preceding 5527 local/generic local/generic 5528 load/store/load load/store/load 5529 atomic/store atomic/store 5530 atomic/atomicrmw. atomic/atomicrmw. 5531 - Must happen before - Must happen before 5532 any following store any following store 5533 atomic/atomicrmw atomic/atomicrmw 5534 with an equal or with an equal or 5535 wider sync scope wider sync scope 5536 and memory ordering and memory ordering 5537 stronger than stronger than 5538 unordered (this is unordered (this is 5539 termed the termed the 5540 fence-paired-atomic). fence-paired-atomic). 5541 - Ensures that all - Ensures that all 5542 memory operations memory operations 5543 have have 5544 completed before completed before 5545 performing the performing the 5546 following following 5547 fence-paired-atomic. fence-paired-atomic. 5548 5549 **Acquire-Release Atomic** 5550 ---------------------------------------------------------------------------------------------------------------------- 5551 atomicrmw acq_rel - singlethread - global 1. buffer/global/ds/flat_atomic 1. buffer/global/ds/flat_atomic 5552 - wavefront - local 5553 - generic 5554 atomicrmw acq_rel - workgroup - global 1. s_waitcnt lgkmcnt(0) 1. s_waitcnt lgkmcnt(0) & 5555 vmcnt(0) & vscnt(0) 5556 5557 - If CU wavefront execution mode, omit vmcnt and 5558 vscnt. 5559 - If OpenCL, omit. - If OpenCL, omit 5560 s_waitcnt lgkmcnt(0). 5561 - Must happen after - Must happen after 5562 any preceding any preceding 5563 local/generic local/generic 5564 load/store/load load/store/load 5565 atomic/store atomic/store 5566 atomic/atomicrmw. atomic/atomicrmw. 5567 - Could be split into 5568 separate s_waitcnt 5569 vmcnt(0), s_waitcnt 5570 vscnt(0) and s_waitcnt 5571 lgkmcnt(0) to allow 5572 them to be 5573 independently moved 5574 according to the 5575 following rules. 5576 - s_waitcnt vmcnt(0) 5577 must happen after 5578 any preceding 5579 global/generic load/load 5580 atomic/ 5581 atomicrmw-with-return-value. 5582 - s_waitcnt vscnt(0) 5583 must happen after 5584 any preceding 5585 global/generic 5586 store/store 5587 atomic/ 5588 atomicrmw-no-return-value. 5589 - s_waitcnt lgkmcnt(0) 5590 must happen after 5591 any preceding 5592 local/generic load/store/load 5593 atomic/store atomic/atomicrmw. 5594 - Must happen before - Must happen before 5595 the following the following 5596 atomicrmw. atomicrmw. 5597 - Ensures that all - Ensures that all 5598 memory operations memory operations 5599 to local have have 5600 completed before completed before 5601 performing the performing the 5602 atomicrmw that is atomicrmw that is 5603 being released. being released. 5604 5605 2. buffer/global/flat_atomic 2. buffer/global_atomic 5606 3. s_waitcnt vm/vscnt(0) 5607 5608 - If CU wavefront execution mode, omit vm/vscnt. 5609 - Use vmcnt if atomic with 5610 return and vscnt if atomic 5611 with no-return. 5612 waitcnt lgkmcnt(0). 5613 - Must happen before 5614 the following 5615 buffer_gl0_inv. 5616 - Ensures any 5617 following global 5618 data read is no 5619 older than the 5620 atomicrmw value 5621 being acquired. 5622 5623 4. buffer_gl0_inv 5624 5625 - If CU wavefront execution mode, omit. 5626 - Ensures that 5627 following 5628 loads will not see 5629 stale data. 5630 5631 atomicrmw acq_rel - workgroup - local 1. waitcnt vmcnt(0) & vscnt(0) 5632 5633 - If CU wavefront execution mode, omit. 5634 - If OpenCL, omit. 5635 - Could be split into 5636 separate s_waitcnt 5637 vmcnt(0) and s_waitcnt 5638 vscnt(0) to allow 5639 them to be 5640 independently moved 5641 according to the 5642 following rules. 5643 - s_waitcnt vmcnt(0) 5644 must happen after 5645 any preceding 5646 global/generic load/load 5647 atomic/ 5648 atomicrmw-with-return-value. 5649 - s_waitcnt vscnt(0) 5650 must happen after 5651 any preceding 5652 global/generic 5653 store/store atomic/ 5654 atomicrmw-no-return-value. 5655 - Must happen before 5656 the following 5657 store. 5658 - Ensures that all 5659 global memory 5660 operations have 5661 completed before 5662 performing the 5663 store that is being 5664 released. 5665 5666 1. ds_atomic 2. ds_atomic 5667 2. s_waitcnt lgkmcnt(0) 3. s_waitcnt lgkmcnt(0) 5668 5669 - If OpenCL, omit. - If OpenCL, omit. 5670 - Must happen before - Must happen before 5671 any following the following 5672 global/generic buffer_gl0_inv. 5673 load/load 5674 atomic/store/store 5675 atomic/atomicrmw. 5676 - Ensures any - Ensures any 5677 following global following global 5678 data read is no data read is no 5679 older than the load older than the load 5680 atomic value being atomic value being 5681 acquired. acquired. 5682 5683 4. buffer_gl0_inv 5684 5685 - If CU wavefront execution mode, omit. 5686 - If OpenCL omit. 5687 - Ensures that 5688 following 5689 loads will not see 5690 stale data. 5691 5692 atomicrmw acq_rel - workgroup - generic 1. s_waitcnt lgkmcnt(0) 1. s_waitcnt lgkmcnt(0) & 5693 vmcnt(0) & vscnt(0) 5694 5695 - If CU wavefront execution mode, omit vmcnt and 5696 vscnt. 5697 - If OpenCL, omit. - If OpenCL, omit 5698 waitcnt lgkmcnt(0). 5699 - Must happen after 5700 any preceding 5701 local/generic 5702 load/store/load 5703 atomic/store 5704 atomic/atomicrmw. 5705 - Could be split into 5706 separate s_waitcnt 5707 vmcnt(0), s_waitcnt 5708 vscnt(0) and s_waitcnt 5709 lgkmcnt(0) to allow 5710 them to be 5711 independently moved 5712 according to the 5713 following rules. 5714 - s_waitcnt vmcnt(0) 5715 must happen after 5716 any preceding 5717 global/generic load/load 5718 atomic/ 5719 atomicrmw-with-return-value. 5720 - s_waitcnt vscnt(0) 5721 must happen after 5722 any preceding 5723 global/generic 5724 store/store 5725 atomic/ 5726 atomicrmw-no-return-value. 5727 - s_waitcnt lgkmcnt(0) 5728 must happen after 5729 any preceding 5730 local/generic load/store/load 5731 atomic/store atomic/atomicrmw. 5732 - Must happen before - Must happen before 5733 the following the following 5734 atomicrmw. atomicrmw. 5735 - Ensures that all - Ensures that all 5736 memory operations memory operations 5737 to local have have 5738 completed before completed before 5739 performing the performing the 5740 atomicrmw that is atomicrmw that is 5741 being released. being released. 5742 5743 2. flat_atomic 2. flat_atomic 5744 3. s_waitcnt lgkmcnt(0) 3. s_waitcnt lgkmcnt(0) & 5745 vm/vscnt(0) 5746 5747 - If CU wavefront execution mode, omit vm/vscnt. 5748 - If OpenCL, omit. - If OpenCL, omit 5749 waitcnt lgkmcnt(0). 5750 - Must happen before - Must happen before 5751 any following the following 5752 global/generic buffer_gl0_inv. 5753 load/load 5754 atomic/store/store 5755 atomic/atomicrmw. 5756 - Ensures any - Ensures any 5757 following global following global 5758 data read is no data read is no 5759 older than the load older than the load 5760 atomic value being atomic value being 5761 acquired. acquired. 5762 5763 3. buffer_gl0_inv 5764 5765 - If CU wavefront execution mode, omit. 5766 - Ensures that 5767 following 5768 loads will not see 5769 stale data. 5770 5771 atomicrmw acq_rel - agent - global 1. s_waitcnt lgkmcnt(0) & 1. s_waitcnt lgkmcnt(0) & 5772 - system vmcnt(0) vmcnt(0) & vscnt(0) 5773 5774 - If OpenCL, omit - If OpenCL, omit 5775 lgkmcnt(0). lgkmcnt(0). 5776 - Could be split into - Could be split into 5777 separate s_waitcnt separate s_waitcnt 5778 vmcnt(0) and vmcnt(0), s_waitcnt 5779 s_waitcnt vscnt(0) and s_waitcnt 5780 lgkmcnt(0) to allow lgkmcnt(0) to allow 5781 them to be them to be 5782 independently moved independently moved 5783 according to the according to the 5784 following rules. following rules. 5785 - s_waitcnt vmcnt(0) - s_waitcnt vmcnt(0) 5786 must happen after must happen after 5787 any preceding any preceding 5788 global/generic global/generic 5789 load/store/load load/load atomic/ 5790 atomic/store atomicrmw-with-return-value. 5791 atomic/atomicrmw. 5792 - s_waitcnt vscnt(0) 5793 must happen after 5794 any preceding 5795 global/generic 5796 store/store atomic/ 5797 atomicrmw-no-return-value. 5798 - s_waitcnt lgkmcnt(0) - s_waitcnt lgkmcnt(0) 5799 must happen after must happen after 5800 any preceding any preceding 5801 local/generic local/generic 5802 load/store/load load/store/load 5803 atomic/store atomic/store 5804 atomic/atomicrmw. atomic/atomicrmw. 5805 - Must happen before - Must happen before 5806 the following the following 5807 atomicrmw. atomicrmw. 5808 - Ensures that all - Ensures that all 5809 memory operations memory operations 5810 to global have to global have 5811 completed before completed before 5812 performing the performing the 5813 atomicrmw that is atomicrmw that is 5814 being released. being released. 5815 5816 2. buffer/global/flat_atomic 2. buffer/global_atomic 5817 3. s_waitcnt vmcnt(0) 3. s_waitcnt vm/vscnt(0) 5818 5819 - Use vmcnt if atomic with 5820 return and vscnt if atomic 5821 with no-return. 5822 waitcnt lgkmcnt(0). 5823 - Must happen before - Must happen before 5824 following following 5825 buffer_wbinvl1_vol. buffer_gl*_inv. 5826 - Ensures the - Ensures the 5827 atomicrmw has atomicrmw has 5828 completed before completed before 5829 invalidating the invalidating the 5830 cache. caches. 5831 5832 4. buffer_wbinvl1_vol 4. buffer_gl0_inv; 5833 buffer_gl1_inv 5834 5835 - Must happen before - Must happen before 5836 any following any following 5837 global/generic global/generic 5838 load/load load/load 5839 atomic/atomicrmw. atomic/atomicrmw. 5840 - Ensures that - Ensures that 5841 following loads following loads 5842 will not see stale will not see stale 5843 global data. global data. 5844 5845 atomicrmw acq_rel - agent - generic 1. s_waitcnt lgkmcnt(0) & 1. s_waitcnt lgkmcnt(0) & 5846 - system vmcnt(0) vmcnt(0) & vscnt(0) 5847 5848 - If OpenCL, omit - If OpenCL, omit 5849 lgkmcnt(0). lgkmcnt(0). 5850 - Could be split into - Could be split into 5851 separate s_waitcnt separate s_waitcnt 5852 vmcnt(0) and vmcnt(0), s_waitcnt 5853 s_waitcnt vscnt(0) and s_waitcnt 5854 lgkmcnt(0) to allow lgkmcnt(0) to allow 5855 them to be them to be 5856 independently moved independently moved 5857 according to the according to the 5858 following rules. following rules. 5859 - s_waitcnt vmcnt(0) - s_waitcnt vmcnt(0) 5860 must happen after must happen after 5861 any preceding any preceding 5862 global/generic global/generic 5863 load/store/load load/load atomic 5864 atomic/store atomicrmw-with-return-value. 5865 atomic/atomicrmw. 5866 - s_waitcnt vscnt(0) 5867 must happen after 5868 any preceding 5869 global/generic 5870 store/store atomic/ 5871 atomicrmw-no-return-value. 5872 - s_waitcnt lgkmcnt(0) - s_waitcnt lgkmcnt(0) 5873 must happen after must happen after 5874 any preceding any preceding 5875 local/generic local/generic 5876 load/store/load load/store/load 5877 atomic/store atomic/store 5878 atomic/atomicrmw. atomic/atomicrmw. 5879 - Must happen before - Must happen before 5880 the following the following 5881 atomicrmw. atomicrmw. 5882 - Ensures that all - Ensures that all 5883 memory operations memory operations 5884 to global have have 5885 completed before completed before 5886 performing the performing the 5887 atomicrmw that is atomicrmw that is 5888 being released. being released. 5889 5890 2. flat_atomic 2. flat_atomic 5891 3. s_waitcnt vmcnt(0) & 3. s_waitcnt vm/vscnt(0) & 5892 lgkmcnt(0) lgkmcnt(0) 5893 5894 - If OpenCL, omit - If OpenCL, omit 5895 lgkmcnt(0). lgkmcnt(0). 5896 - Use vmcnt if atomic with 5897 return and vscnt if atomic 5898 with no-return. 5899 - Must happen before - Must happen before 5900 following following 5901 buffer_wbinvl1_vol. buffer_gl*_inv. 5902 - Ensures the - Ensures the 5903 atomicrmw has atomicrmw has 5904 completed before completed before 5905 invalidating the invalidating the 5906 cache. caches. 5907 5908 4. buffer_wbinvl1_vol 4. buffer_gl0_inv; 5909 buffer_gl1_inv 5910 5911 - Must happen before - Must happen before 5912 any following any following 5913 global/generic global/generic 5914 load/load load/load 5915 atomic/atomicrmw. atomic/atomicrmw. 5916 - Ensures that - Ensures that 5917 following loads following loads 5918 will not see stale will not see stale 5919 global data. global data. 5920 5921 fence acq_rel - singlethread *none* *none* *none* 5922 - wavefront 5923 fence acq_rel - workgroup *none* 1. s_waitcnt lgkmcnt(0) 1. s_waitcnt lgkmcnt(0) & 5924 vmcnt(0) & vscnt(0) 5925 5926 - If CU wavefront execution mode, omit vmcnt and 5927 vscnt. 5928 - If OpenCL and - If OpenCL and 5929 address space is address space is 5930 not generic, omit. not generic, omit 5931 lgkmcnt(0). 5932 - If OpenCL and 5933 address space is 5934 local, omit 5935 vmcnt(0) and vscnt(0). 5936 - However, - However, 5937 since LLVM since LLVM 5938 currently has no currently has no 5939 address space on address space on 5940 the fence need to the fence need to 5941 conservatively conservatively 5942 always generate always generate 5943 (see comment for (see comment for 5944 previous fence). previous fence). 5945 - Must happen after 5946 any preceding 5947 local/generic 5948 load/load 5949 atomic/store/store 5950 atomic/atomicrmw. 5951 - Could be split into 5952 separate s_waitcnt 5953 vmcnt(0), s_waitcnt 5954 vscnt(0) and s_waitcnt 5955 lgkmcnt(0) to allow 5956 them to be 5957 independently moved 5958 according to the 5959 following rules. 5960 - s_waitcnt vmcnt(0) 5961 must happen after 5962 any preceding 5963 global/generic 5964 load/load 5965 atomic/ 5966 atomicrmw-with-return-value. 5967 - s_waitcnt vscnt(0) 5968 must happen after 5969 any preceding 5970 global/generic 5971 store/store atomic/ 5972 atomicrmw-no-return-value. 5973 - s_waitcnt lgkmcnt(0) 5974 must happen after 5975 any preceding 5976 local/generic 5977 load/store/load 5978 atomic/store atomic/ 5979 atomicrmw. 5980 - Must happen before - Must happen before 5981 any following any following 5982 global/generic global/generic 5983 load/load load/load 5984 atomic/store/store atomic/store/store 5985 atomic/atomicrmw. atomic/atomicrmw. 5986 - Ensures that all - Ensures that all 5987 memory operations memory operations 5988 to local have have 5989 completed before completed before 5990 performing any performing any 5991 following global following global 5992 memory operations. memory operations. 5993 - Ensures that the - Ensures that the 5994 preceding preceding 5995 local/generic load local/generic load 5996 atomic/atomicrmw atomic/atomicrmw 5997 with an equal or with an equal or 5998 wider sync scope wider sync scope 5999 and memory ordering and memory ordering 6000 stronger than stronger than 6001 unordered (this is unordered (this is 6002 termed the termed the 6003 acquire-fence-paired-atomic acquire-fence-paired-atomic 6004 ) has completed ) has completed 6005 before following before following 6006 global memory global memory 6007 operations. This operations. This 6008 satisfies the satisfies the 6009 requirements of requirements of 6010 acquire. acquire. 6011 - Ensures that all - Ensures that all 6012 previous memory previous memory 6013 operations have operations have 6014 completed before a completed before a 6015 following following 6016 local/generic store local/generic store 6017 atomic/atomicrmw atomic/atomicrmw 6018 with an equal or with an equal or 6019 wider sync scope wider sync scope 6020 and memory ordering and memory ordering 6021 stronger than stronger than 6022 unordered (this is unordered (this is 6023 termed the termed the 6024 release-fence-paired-atomic release-fence-paired-atomic 6025 ). This satisfies the ). This satisfies the 6026 requirements of requirements of 6027 release. release. 6028 - Must happen before 6029 the following 6030 buffer_gl0_inv. 6031 - Ensures that the 6032 acquire-fence-paired 6033 atomic has completed 6034 before invalidating 6035 the 6036 cache. Therefore 6037 any following 6038 locations read must 6039 be no older than 6040 the value read by 6041 the 6042 acquire-fence-paired-atomic. 6043 6044 3. buffer_gl0_inv 6045 6046 - If CU wavefront execution mode, omit. 6047 - Ensures that 6048 following 6049 loads will not see 6050 stale data. 6051 6052 fence acq_rel - agent *none* 1. s_waitcnt lgkmcnt(0) & 1. s_waitcnt lgkmcnt(0) & 6053 - system vmcnt(0) vmcnt(0) & vscnt(0) 6054 6055 - If OpenCL and - If OpenCL and 6056 address space is address space is 6057 not generic, omit not generic, omit 6058 lgkmcnt(0). lgkmcnt(0). 6059 - If OpenCL and 6060 address space is 6061 local, omit 6062 vmcnt(0) and vscnt(0). 6063 - However, since LLVM - However, since LLVM 6064 currently has no currently has no 6065 address space on address space on 6066 the fence need to the fence need to 6067 conservatively conservatively 6068 always generate always generate 6069 (see comment for (see comment for 6070 previous fence). previous fence). 6071 - Could be split into - Could be split into 6072 separate s_waitcnt separate s_waitcnt 6073 vmcnt(0) and vmcnt(0), s_waitcnt 6074 s_waitcnt vscnt(0) and s_waitcnt 6075 lgkmcnt(0) to allow lgkmcnt(0) to allow 6076 them to be them to be 6077 independently moved independently moved 6078 according to the according to the 6079 following rules. following rules. 6080 - s_waitcnt vmcnt(0) - s_waitcnt vmcnt(0) 6081 must happen after must happen after 6082 any preceding any preceding 6083 global/generic global/generic 6084 load/store/load load/load 6085 atomic/store atomic/ 6086 atomic/atomicrmw. atomicrmw-with-return-value. 6087 - s_waitcnt vscnt(0) 6088 must happen after 6089 any preceding 6090 global/generic 6091 store/store atomic/ 6092 atomicrmw-no-return-value. 6093 - s_waitcnt lgkmcnt(0) - s_waitcnt lgkmcnt(0) 6094 must happen after must happen after 6095 any preceding any preceding 6096 local/generic local/generic 6097 load/store/load load/store/load 6098 atomic/store atomic/store 6099 atomic/atomicrmw. atomic/atomicrmw. 6100 - Must happen before - Must happen before 6101 the following the following 6102 buffer_wbinvl1_vol. buffer_gl*_inv. 6103 - Ensures that the - Ensures that the 6104 preceding preceding 6105 global/local/generic global/local/generic 6106 load load 6107 atomic/atomicrmw atomic/atomicrmw 6108 with an equal or with an equal or 6109 wider sync scope wider sync scope 6110 and memory ordering and memory ordering 6111 stronger than stronger than 6112 unordered (this is unordered (this is 6113 termed the termed the 6114 acquire-fence-paired-atomic acquire-fence-paired-atomic 6115 ) has completed ) has completed 6116 before invalidating before invalidating 6117 the cache. This the caches. This 6118 satisfies the satisfies the 6119 requirements of requirements of 6120 acquire. acquire. 6121 - Ensures that all - Ensures that all 6122 previous memory previous memory 6123 operations have operations have 6124 completed before a completed before a 6125 following following 6126 global/local/generic global/local/generic 6127 store store 6128 atomic/atomicrmw atomic/atomicrmw 6129 with an equal or with an equal or 6130 wider sync scope wider sync scope 6131 and memory ordering and memory ordering 6132 stronger than stronger than 6133 unordered (this is unordered (this is 6134 termed the termed the 6135 release-fence-paired-atomic release-fence-paired-atomic 6136 ). This satisfies the ). This satisfies the 6137 requirements of requirements of 6138 release. release. 6139 6140 2. buffer_wbinvl1_vol 2. buffer_gl0_inv; 6141 buffer_gl1_inv 6142 6143 - Must happen before - Must happen before 6144 any following any following 6145 global/generic global/generic 6146 load/load load/load 6147 atomic/store/store atomic/store/store 6148 atomic/atomicrmw. atomic/atomicrmw. 6149 - Ensures that - Ensures that 6150 following loads following loads 6151 will not see stale will not see stale 6152 global data. This global data. This 6153 satisfies the satisfies the 6154 requirements of requirements of 6155 acquire. acquire. 6156 6157 **Sequential Consistent Atomic** 6158 ---------------------------------------------------------------------------------------------------------------------- 6159 load atomic seq_cst - singlethread - global *Same as corresponding *Same as corresponding 6160 - wavefront - local load atomic acquire, load atomic acquire, 6161 - generic except must generated except must generated 6162 all instructions even all instructions even 6163 for OpenCL.* for OpenCL.* 6164 load atomic seq_cst - workgroup - global 1. s_waitcnt lgkmcnt(0) 1. s_waitcnt lgkmcnt(0) & 6165 - generic vmcnt(0) & vscnt(0) 6166 6167 - If CU wavefront execution mode, omit vmcnt and 6168 vscnt. 6169 - Could be split into 6170 separate s_waitcnt 6171 vmcnt(0), s_waitcnt 6172 vscnt(0) and s_waitcnt 6173 lgkmcnt(0) to allow 6174 them to be 6175 independently moved 6176 according to the 6177 following rules. 6178 - Must - waitcnt lgkmcnt(0) must 6179 happen after happen after 6180 preceding preceding 6181 global/generic load local load 6182 atomic/store atomic/store 6183 atomic/atomicrmw atomic/atomicrmw 6184 with memory with memory 6185 ordering of seq_cst ordering of seq_cst 6186 and with equal or and with equal or 6187 wider sync scope. wider sync scope. 6188 (Note that seq_cst (Note that seq_cst 6189 fences have their fences have their 6190 own s_waitcnt own s_waitcnt 6191 lgkmcnt(0) and so do lgkmcnt(0) and so do 6192 not need to be not need to be 6193 considered.) considered.) 6194 - waitcnt vmcnt(0) 6195 Must happen after 6196 preceding 6197 global/generic load 6198 atomic/ 6199 atomicrmw-with-return-value 6200 with memory 6201 ordering of seq_cst 6202 and with equal or 6203 wider sync scope. 6204 (Note that seq_cst 6205 fences have their 6206 own s_waitcnt 6207 vmcnt(0) and so do 6208 not need to be 6209 considered.) 6210 - waitcnt vscnt(0) 6211 Must happen after 6212 preceding 6213 global/generic store 6214 atomic/ 6215 atomicrmw-no-return-value 6216 with memory 6217 ordering of seq_cst 6218 and with equal or 6219 wider sync scope. 6220 (Note that seq_cst 6221 fences have their 6222 own s_waitcnt 6223 vscnt(0) and so do 6224 not need to be 6225 considered.) 6226 - Ensures any - Ensures any 6227 preceding preceding 6228 sequential sequential 6229 consistent local consistent global/local 6230 memory instructions memory instructions 6231 have completed have completed 6232 before executing before executing 6233 this sequentially this sequentially 6234 consistent consistent 6235 instruction. This instruction. This 6236 prevents reordering prevents reordering 6237 a seq_cst store a seq_cst store 6238 followed by a followed by a 6239 seq_cst load. (Note seq_cst load. (Note 6240 that seq_cst is that seq_cst is 6241 stronger than stronger than 6242 acquire/release as acquire/release as 6243 the reordering of the reordering of 6244 load acquire load acquire 6245 followed by a store followed by a store 6246 release is release is 6247 prevented by the prevented by the 6248 waitcnt of waitcnt of 6249 the release, but the release, but 6250 there is nothing there is nothing 6251 preventing a store preventing a store 6252 release followed by release followed by 6253 load acquire from load acquire from 6254 competing out of competing out of 6255 order.) order.) 6256 6257 2. *Following 2. *Following 6258 instructions same as instructions same as 6259 corresponding load corresponding load 6260 atomic acquire, atomic acquire, 6261 except must generated except must generated 6262 all instructions even all instructions even 6263 for OpenCL.* for OpenCL.* 6264 load atomic seq_cst - workgroup - local *Same as corresponding 6265 load atomic acquire, 6266 except must generated 6267 all instructions even 6268 for OpenCL.* 6269 6270 1. s_waitcnt vmcnt(0) & vscnt(0) 6271 6272 - If CU wavefront execution mode, omit. 6273 - Could be split into 6274 separate s_waitcnt 6275 vmcnt(0) and s_waitcnt 6276 vscnt(0) to allow 6277 them to be 6278 independently moved 6279 according to the 6280 following rules. 6281 - waitcnt vmcnt(0) 6282 Must happen after 6283 preceding 6284 global/generic load 6285 atomic/ 6286 atomicrmw-with-return-value 6287 with memory 6288 ordering of seq_cst 6289 and with equal or 6290 wider sync scope. 6291 (Note that seq_cst 6292 fences have their 6293 own s_waitcnt 6294 vmcnt(0) and so do 6295 not need to be 6296 considered.) 6297 - waitcnt vscnt(0) 6298 Must happen after 6299 preceding 6300 global/generic store 6301 atomic/ 6302 atomicrmw-no-return-value 6303 with memory 6304 ordering of seq_cst 6305 and with equal or 6306 wider sync scope. 6307 (Note that seq_cst 6308 fences have their 6309 own s_waitcnt 6310 vscnt(0) and so do 6311 not need to be 6312 considered.) 6313 - Ensures any 6314 preceding 6315 sequential 6316 consistent global 6317 memory instructions 6318 have completed 6319 before executing 6320 this sequentially 6321 consistent 6322 instruction. This 6323 prevents reordering 6324 a seq_cst store 6325 followed by a 6326 seq_cst load. (Note 6327 that seq_cst is 6328 stronger than 6329 acquire/release as 6330 the reordering of 6331 load acquire 6332 followed by a store 6333 release is 6334 prevented by the 6335 waitcnt of 6336 the release, but 6337 there is nothing 6338 preventing a store 6339 release followed by 6340 load acquire from 6341 competing out of 6342 order.) 6343 6344 2. *Following 6345 instructions same as 6346 corresponding load 6347 atomic acquire, 6348 except must generated 6349 all instructions even 6350 for OpenCL.* 6351 6352 load atomic seq_cst - agent - global 1. s_waitcnt lgkmcnt(0) & 1. s_waitcnt lgkmcnt(0) & 6353 - system - generic vmcnt(0) vmcnt(0) & vscnt(0) 6354 6355 - Could be split into - Could be split into 6356 separate s_waitcnt separate s_waitcnt 6357 vmcnt(0) vmcnt(0), s_waitcnt 6358 and s_waitcnt vscnt(0) and s_waitcnt 6359 lgkmcnt(0) to allow lgkmcnt(0) to allow 6360 them to be them to be 6361 independently moved independently moved 6362 according to the according to the 6363 following rules. following rules. 6364 - waitcnt lgkmcnt(0) - waitcnt lgkmcnt(0) 6365 must happen after must happen after 6366 preceding preceding 6367 global/generic load local load 6368 atomic/store atomic/store 6369 atomic/atomicrmw atomic/atomicrmw 6370 with memory with memory 6371 ordering of seq_cst ordering of seq_cst 6372 and with equal or and with equal or 6373 wider sync scope. wider sync scope. 6374 (Note that seq_cst (Note that seq_cst 6375 fences have their fences have their 6376 own s_waitcnt own s_waitcnt 6377 lgkmcnt(0) and so do lgkmcnt(0) and so do 6378 not need to be not need to be 6379 considered.) considered.) 6380 - waitcnt vmcnt(0) - waitcnt vmcnt(0) 6381 must happen after must happen after 6382 preceding preceding 6383 global/generic load global/generic load 6384 atomic/store atomic/ 6385 atomic/atomicrmw atomicrmw-with-return-value 6386 with memory with memory 6387 ordering of seq_cst ordering of seq_cst 6388 and with equal or and with equal or 6389 wider sync scope. wider sync scope. 6390 (Note that seq_cst (Note that seq_cst 6391 fences have their fences have their 6392 own s_waitcnt own s_waitcnt 6393 vmcnt(0) and so do vmcnt(0) and so do 6394 not need to be not need to be 6395 considered.) considered.) 6396 - waitcnt vscnt(0) 6397 Must happen after 6398 preceding 6399 global/generic store 6400 atomic/ 6401 atomicrmw-no-return-value 6402 with memory 6403 ordering of seq_cst 6404 and with equal or 6405 wider sync scope. 6406 (Note that seq_cst 6407 fences have their 6408 own s_waitcnt 6409 vscnt(0) and so do 6410 not need to be 6411 considered.) 6412 - Ensures any - Ensures any 6413 preceding preceding 6414 sequential sequential 6415 consistent global consistent global 6416 memory instructions memory instructions 6417 have completed have completed 6418 before executing before executing 6419 this sequentially this sequentially 6420 consistent consistent 6421 instruction. This instruction. This 6422 prevents reordering prevents reordering 6423 a seq_cst store a seq_cst store 6424 followed by a followed by a 6425 seq_cst load. (Note seq_cst load. (Note 6426 that seq_cst is that seq_cst is 6427 stronger than stronger than 6428 acquire/release as acquire/release as 6429 the reordering of the reordering of 6430 load acquire load acquire 6431 followed by a store followed by a store 6432 release is release is 6433 prevented by the prevented by the 6434 waitcnt of waitcnt of 6435 the release, but the release, but 6436 there is nothing there is nothing 6437 preventing a store preventing a store 6438 release followed by release followed by 6439 load acquire from load acquire from 6440 competing out of competing out of 6441 order.) order.) 6442 6443 2. *Following 2. *Following 6444 instructions same as instructions same as 6445 corresponding load corresponding load 6446 atomic acquire, atomic acquire, 6447 except must generated except must generated 6448 all instructions even all instructions even 6449 for OpenCL.* for OpenCL.* 6450 store atomic seq_cst - singlethread - global *Same as corresponding *Same as corresponding 6451 - wavefront - local store atomic release, store atomic release, 6452 - workgroup - generic except must generated except must generated 6453 all instructions even all instructions even 6454 for OpenCL.* for OpenCL.* 6455 store atomic seq_cst - agent - global *Same as corresponding *Same as corresponding 6456 - system - generic store atomic release, store atomic release, 6457 except must generated except must generated 6458 all instructions even all instructions even 6459 for OpenCL.* for OpenCL.* 6460 atomicrmw seq_cst - singlethread - global *Same as corresponding *Same as corresponding 6461 - wavefront - local atomicrmw acq_rel, atomicrmw acq_rel, 6462 - workgroup - generic except must generated except must generated 6463 all instructions even all instructions even 6464 for OpenCL.* for OpenCL.* 6465 atomicrmw seq_cst - agent - global *Same as corresponding *Same as corresponding 6466 - system - generic atomicrmw acq_rel, atomicrmw acq_rel, 6467 except must generated except must generated 6468 all instructions even all instructions even 6469 for OpenCL.* for OpenCL.* 6470 fence seq_cst - singlethread *none* *Same as corresponding *Same as corresponding 6471 - wavefront fence acq_rel, fence acq_rel, 6472 - workgroup except must generated except must generated 6473 - agent all instructions even all instructions even 6474 - system for OpenCL.* for OpenCL.* 6475 ============ ============ ============== ========== =============================== ================================== 6476 6477The memory order also adds the single thread optimization constrains defined in 6478table 6479:ref:`amdgpu-amdhsa-memory-model-single-thread-optimization-constraints-gfx6-gfx10-table`. 6480 6481 .. table:: AMDHSA Memory Model Single Thread Optimization Constraints GFX6-GFX10 6482 :name: amdgpu-amdhsa-memory-model-single-thread-optimization-constraints-gfx6-gfx10-table 6483 6484 ============ ============================================================== 6485 LLVM Memory Optimization Constraints 6486 Ordering 6487 ============ ============================================================== 6488 unordered *none* 6489 monotonic *none* 6490 acquire - If a load atomic/atomicrmw then no following load/load 6491 atomic/store/ store atomic/atomicrmw/fence instruction can 6492 be moved before the acquire. 6493 - If a fence then same as load atomic, plus no preceding 6494 associated fence-paired-atomic can be moved after the fence. 6495 release - If a store atomic/atomicrmw then no preceding load/load 6496 atomic/store/ store atomic/atomicrmw/fence instruction can 6497 be moved after the release. 6498 - If a fence then same as store atomic, plus no following 6499 associated fence-paired-atomic can be moved before the 6500 fence. 6501 acq_rel Same constraints as both acquire and release. 6502 seq_cst - If a load atomic then same constraints as acquire, plus no 6503 preceding sequentially consistent load atomic/store 6504 atomic/atomicrmw/fence instruction can be moved after the 6505 seq_cst. 6506 - If a store atomic then the same constraints as release, plus 6507 no following sequentially consistent load atomic/store 6508 atomic/atomicrmw/fence instruction can be moved before the 6509 seq_cst. 6510 - If an atomicrmw/fence then same constraints as acq_rel. 6511 ============ ============================================================== 6512 6513Trap Handler ABI 6514~~~~~~~~~~~~~~~~ 6515 6516For code objects generated by AMDGPU backend for HSA [HSA]_ compatible runtimes 6517(such as ROCm [AMD-ROCm]_), the runtime installs a trap handler that supports 6518the ``s_trap`` instruction with the following usage: 6519 6520 .. table:: AMDGPU Trap Handler for AMDHSA OS 6521 :name: amdgpu-trap-handler-for-amdhsa-os-table 6522 6523 =================== =============== =============== ======================= 6524 Usage Code Sequence Trap Handler Description 6525 Inputs 6526 =================== =============== =============== ======================= 6527 reserved ``s_trap 0x00`` Reserved by hardware. 6528 ``debugtrap(arg)`` ``s_trap 0x01`` ``SGPR0-1``: Reserved for HSA 6529 ``queue_ptr`` ``debugtrap`` 6530 ``VGPR0``: intrinsic (not 6531 ``arg`` implemented). 6532 ``llvm.trap`` ``s_trap 0x02`` ``SGPR0-1``: Causes dispatch to be 6533 ``queue_ptr`` terminated and its 6534 associated queue put 6535 into the error state. 6536 ``llvm.debugtrap`` ``s_trap 0x03`` - If debugger not 6537 installed then 6538 behaves as a 6539 no-operation. The 6540 trap handler is 6541 entered and 6542 immediately returns 6543 to continue 6544 execution of the 6545 wavefront. 6546 - If the debugger is 6547 installed, causes 6548 the debug trap to be 6549 reported by the 6550 debugger and the 6551 wavefront is put in 6552 the halt state until 6553 resumed by the 6554 debugger. 6555 reserved ``s_trap 0x04`` Reserved. 6556 reserved ``s_trap 0x05`` Reserved. 6557 reserved ``s_trap 0x06`` Reserved. 6558 debugger breakpoint ``s_trap 0x07`` Reserved for debugger 6559 breakpoints. 6560 reserved ``s_trap 0x08`` Reserved. 6561 reserved ``s_trap 0xfe`` Reserved. 6562 reserved ``s_trap 0xff`` Reserved. 6563 =================== =============== =============== ======================= 6564 6565.. _amdgpu-amdhsa-function-call-convention: 6566 6567Call Convention 6568~~~~~~~~~~~~~~~ 6569 6570.. note:: 6571 6572 This section is currently incomplete and has inaccuracies. It is WIP that will 6573 be updated as information is determined. 6574 6575See :ref:`amdgpu-dwarf-address-space-identifier` for information on swizzled 6576addresses. Unswizzled addresses are normal linear addresses. 6577 6578.. _amdgpu-amdhsa-function-call-convention-kernel-functions: 6579 6580Kernel Functions 6581++++++++++++++++ 6582 6583This section describes the call convention ABI for the outer kernel function. 6584 6585See :ref:`amdgpu-amdhsa-initial-kernel-execution-state` for the kernel call 6586convention. 6587 6588The following is not part of the AMDGPU kernel calling convention but describes 6589how the AMDGPU implements function calls: 6590 65911. Clang decides the kernarg layout to match the *HSA Programmer's Language 6592 Reference* [HSA]_. 6593 6594 - All structs are passed directly. 6595 - Lambda values are passed *TBA*. 6596 6597 .. TODO:: 6598 6599 - Does this really follow HSA rules? Or are structs >16 bytes passed 6600 by-value struct? 6601 - What is ABI for lambda values? 6602 66034. The kernel performs certain setup in its prolog, as described in 6604 :ref:`amdgpu-amdhsa-kernel-prolog`. 6605 6606.. _amdgpu-amdhsa-function-call-convention-non-kernel-functions: 6607 6608Non-Kernel Functions 6609++++++++++++++++++++ 6610 6611This section describes the call convention ABI for functions other than the 6612outer kernel function. 6613 6614If a kernel has function calls then scratch is always allocated and used for 6615the call stack which grows from low address to high address using the swizzled 6616scratch address space. 6617 6618On entry to a function: 6619 66201. SGPR0-3 contain a V# with the following properties (see 6621 :ref:`amdgpu-amdhsa-kernel-prolog-private-segment-buffer`): 6622 6623 * Base address pointing to the beginning of the wavefront scratch backing 6624 memory. 6625 * Swizzled with dword element size and stride of wavefront size elements. 6626 66272. The FLAT_SCRATCH register pair is setup. See 6628 :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`. 66293. GFX6-8: M0 register set to the size of LDS in bytes. See 6630 :ref:`amdgpu-amdhsa-kernel-prolog-m0`. 66314. The EXEC register is set to the lanes active on entry to the function. 66325. MODE register: *TBD* 66336. VGPR0-31 and SGPR4-29 are used to pass function input arguments as described 6634 below. 66357. SGPR30-31 return address (RA). The code address that the function must 6636 return to when it completes. The value is undefined if the function is *no 6637 return*. 66388. SGPR32 is used for the stack pointer (SP). It is an unswizzled scratch 6639 offset relative to the beginning of the wavefront scratch backing memory. 6640 6641 The unswizzled SP can be used with buffer instructions as an unswizzled SGPR 6642 offset with the scratch V# in SGPR0-3 to access the stack in a swizzled 6643 manner. 6644 6645 The unswizzled SP value can be converted into the swizzled SP value by: 6646 6647 | swizzled SP = unswizzled SP / wavefront size 6648 6649 This may be used to obtain the private address space address of stack 6650 objects and to convert this address to a flat address by adding the flat 6651 scratch aperture base address. 6652 6653 The swizzled SP value is always 4 bytes aligned for the ``r600`` 6654 architecture and 16 byte aligned for the ``amdgcn`` architecture. 6655 6656 .. note:: 6657 6658 The ``amdgcn`` value is selected to avoid dynamic stack alignment for the 6659 OpenCL language which has the largest base type defined as 16 bytes. 6660 6661 On entry, the swizzled SP value is the address of the first function 6662 argument passed on the stack. Other stack passed arguments are positive 6663 offsets from the entry swizzled SP value. 6664 6665 The function may use positive offsets beyond the last stack passed argument 6666 for stack allocated local variables and register spill slots. If necessary, 6667 the function may align these to greater alignment than 16 bytes. After these 6668 the function may dynamically allocate space for such things as runtime sized 6669 ``alloca`` local allocations. 6670 6671 If the function calls another function, it will place any stack allocated 6672 arguments after the last local allocation and adjust SGPR32 to the address 6673 after the last local allocation. 6674 66759. All other registers are unspecified. 667610. Any necessary ``waitcnt`` has been performed to ensure memory is available 6677 to the function. 6678 6679On exit from a function: 6680 66811. VGPR0-31 and SGPR4-29 are used to pass function result arguments as 6682 described below. Any registers used are considered clobbered registers. 66832. The following registers are preserved and have the same value as on entry: 6684 6685 * FLAT_SCRATCH 6686 * EXEC 6687 * GFX6-8: M0 6688 * All SGPR registers except the clobbered registers of SGPR4-31. 6689 * VGPR40-47 6690 VGPR56-63 6691 VGPR72-79 6692 VGPR88-95 6693 VGPR104-111 6694 VGPR120-127 6695 VGPR136-143 6696 VGPR152-159 6697 VGPR168-175 6698 VGPR184-191 6699 VGPR200-207 6700 VGPR216-223 6701 VGPR232-239 6702 VGPR248-255 6703 6704 *Except the argument registers, the VGPR clobbered and the preserved 6705 registers are intermixed at regular intervals in order to 6706 get a better occupancy.* 6707 6708 For the AMDGPU backend, an inter-procedural register allocation (IPRA) 6709 optimization may mark some of clobbered SGPR and VGPR registers as 6710 preserved if it can be determined that the called function does not change 6711 their value. 6712 67132. The PC is set to the RA provided on entry. 67143. MODE register: *TBD*. 67154. All other registers are clobbered. 67165. Any necessary ``waitcnt`` has been performed to ensure memory accessed by 6717 function is available to the caller. 6718 6719.. TODO:: 6720 6721 - On gfx908 are all ACC registers clobbered? 6722 6723 - How are function results returned? The address of structured types is passed 6724 by reference, but what about other types? 6725 6726The function input arguments are made up of the formal arguments explicitly 6727declared by the source language function plus the implicit input arguments used 6728by the implementation. 6729 6730The source language input arguments are: 6731 67321. Any source language implicit ``this`` or ``self`` argument comes first as a 6733 pointer type. 67342. Followed by the function formal arguments in left to right source order. 6735 6736The source language result arguments are: 6737 67381. The function result argument. 6739 6740The source language input or result struct type arguments that are less than or 6741equal to 16 bytes, are decomposed recursively into their base type fields, and 6742each field is passed as if a separate argument. For input arguments, if the 6743called function requires the struct to be in memory, for example because its 6744address is taken, then the function body is responsible for allocating a stack 6745location and copying the field arguments into it. Clang terms this *direct 6746struct*. 6747 6748The source language input struct type arguments that are greater than 16 bytes, 6749are passed by reference. The caller is responsible for allocating a stack 6750location to make a copy of the struct value and pass the address as the input 6751argument. The called function is responsible to perform the dereference when 6752accessing the input argument. Clang terms this *by-value struct*. 6753 6754A source language result struct type argument that is greater than 16 bytes, is 6755returned by reference. The caller is responsible for allocating a stack location 6756to hold the result value and passes the address as the last input argument 6757(before the implicit input arguments). In this case there are no result 6758arguments. The called function is responsible to perform the dereference when 6759storing the result value. Clang terms this *structured return (sret)*. 6760 6761*TODO: correct the ``sret`` definition.* 6762 6763.. TODO:: 6764 6765 Is this definition correct? Or is ``sret`` only used if passing in registers, and 6766 pass as non-decomposed struct as stack argument? Or something else? Is the 6767 memory location in the caller stack frame, or a stack memory argument and so 6768 no address is passed as the caller can directly write to the argument stack 6769 location? But then the stack location is still live after return. If an 6770 argument stack location is it the first stack argument or the last one? 6771 6772Lambda argument types are treated as struct types with an implementation defined 6773set of fields. 6774 6775.. TODO:: 6776 6777 Need to specify the ABI for lambda types for AMDGPU. 6778 6779For AMDGPU backend all source language arguments (including the decomposed 6780struct type arguments) are passed in VGPRs unless marked ``inreg`` in which case 6781they are passed in SGPRs. 6782 6783The AMDGPU backend walks the function call graph from the leaves to determine 6784which implicit input arguments are used, propagating to each caller of the 6785function. The used implicit arguments are appended to the function arguments 6786after the source language arguments in the following order: 6787 6788.. TODO:: 6789 6790 Is recursion or external functions supported? 6791 67921. Work-Item ID (1 VGPR) 6793 6794 The X, Y and Z work-item ID are packed into a single VGRP with the following 6795 layout. Only fields actually used by the function are set. The other bits 6796 are undefined. 6797 6798 The values come from the initial kernel execution state. See 6799 :ref:`amdgpu-amdhsa-vgpr-register-set-up-order-table`. 6800 6801 .. table:: Work-item implicit argument layout 6802 :name: amdgpu-amdhsa-workitem-implicit-argument-layout-table 6803 6804 ======= ======= ============== 6805 Bits Size Field Name 6806 ======= ======= ============== 6807 9:0 10 bits X Work-Item ID 6808 19:10 10 bits Y Work-Item ID 6809 29:20 10 bits Z Work-Item ID 6810 31:30 2 bits Unused 6811 ======= ======= ============== 6812 68132. Dispatch Ptr (2 SGPRs) 6814 6815 The value comes from the initial kernel execution state. See 6816 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 6817 68183. Queue Ptr (2 SGPRs) 6819 6820 The value comes from the initial kernel execution state. See 6821 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 6822 68234. Kernarg Segment Ptr (2 SGPRs) 6824 6825 The value comes from the initial kernel execution state. See 6826 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 6827 68285. Dispatch id (2 SGPRs) 6829 6830 The value comes from the initial kernel execution state. See 6831 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 6832 68336. Work-Group ID X (1 SGPR) 6834 6835 The value comes from the initial kernel execution state. See 6836 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 6837 68387. Work-Group ID Y (1 SGPR) 6839 6840 The value comes from the initial kernel execution state. See 6841 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 6842 68438. Work-Group ID Z (1 SGPR) 6844 6845 The value comes from the initial kernel execution state. See 6846 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 6847 68489. Implicit Argument Ptr (2 SGPRs) 6849 6850 The value is computed by adding an offset to Kernarg Segment Ptr to get the 6851 global address space pointer to the first kernarg implicit argument. 6852 6853The input and result arguments are assigned in order in the following manner: 6854 6855.. note:: 6856 6857 There are likely some errors and omissions in the following description that 6858 need correction. 6859 6860 .. TODO:: 6861 6862 Check the clang source code to decipher how function arguments and return 6863 results are handled. Also see the AMDGPU specific values used. 6864 6865* VGPR arguments are assigned to consecutive VGPRs starting at VGPR0 up to 6866 VGPR31. 6867 6868 If there are more arguments than will fit in these registers, the remaining 6869 arguments are allocated on the stack in order on naturally aligned 6870 addresses. 6871 6872 .. TODO:: 6873 6874 How are overly aligned structures allocated on the stack? 6875 6876* SGPR arguments are assigned to consecutive SGPRs starting at SGPR0 up to 6877 SGPR29. 6878 6879 If there are more arguments than will fit in these registers, the remaining 6880 arguments are allocated on the stack in order on naturally aligned 6881 addresses. 6882 6883Note that decomposed struct type arguments may have some fields passed in 6884registers and some in memory. 6885 6886.. TODO:: 6887 6888 So, a struct which can pass some fields as decomposed register arguments, will 6889 pass the rest as decomposed stack elements? But an argument that will not start 6890 in registers will not be decomposed and will be passed as a non-decomposed 6891 stack value? 6892 6893The following is not part of the AMDGPU function calling convention but 6894describes how the AMDGPU implements function calls: 6895 68961. SGPR33 is used as a frame pointer (FP) if necessary. Like the SP it is an 6897 unswizzled scratch address. It is only needed if runtime sized ``alloca`` 6898 are used, or for the reasons defined in ``SIFrameLowering``. 68992. Runtime stack alignment is supported. SGPR34 is used as a base pointer (BP) 6900 to access the incoming stack arguments in the function. The BP is needed 6901 only when the function requires the runtime stack alignment. 6902 69033. Allocating SGPR arguments on the stack are not supported. 6904 69054. No CFI is currently generated. See 6906 :ref:`amdgpu-dwarf-call-frame-information`. 6907 6908 .. note:: 6909 6910 CFI will be generated that defines the CFA as the unswizzled address 6911 relative to the wave scratch base in the unswizzled private address space 6912 of the lowest address stack allocated local variable. 6913 6914 ``DW_AT_frame_base`` will be defined as the swizzled address in the 6915 swizzled private address space by dividing the CFA by the wavefront size 6916 (since CFA is always at least dword aligned which matches the scratch 6917 swizzle element size). 6918 6919 If no dynamic stack alignment was performed, the stack allocated arguments 6920 are accessed as negative offsets relative to ``DW_AT_frame_base``, and the 6921 local variables and register spill slots are accessed as positive offsets 6922 relative to ``DW_AT_frame_base``. 6923 69245. Function argument passing is implemented by copying the input physical 6925 registers to virtual registers on entry. The register allocator can spill if 6926 necessary. These are copied back to physical registers at call sites. The 6927 net effect is that each function call can have these values in entirely 6928 distinct locations. The IPRA can help avoid shuffling argument registers. 69296. Call sites are implemented by setting up the arguments at positive offsets 6930 from SP. Then SP is incremented to account for the known frame size before 6931 the call and decremented after the call. 6932 6933 .. note:: 6934 6935 The CFI will reflect the changed calculation needed to compute the CFA 6936 from SP. 6937 69387. 4 byte spill slots are used in the stack frame. One slot is allocated for an 6939 emergency spill slot. Buffer instructions are used for stack accesses and 6940 not the ``flat_scratch`` instruction. 6941 6942 .. TODO:: 6943 6944 Explain when the emergency spill slot is used. 6945 6946.. TODO:: 6947 6948 Possible broken issues: 6949 6950 - Stack arguments must be aligned to required alignment. 6951 - Stack is aligned to max(16, max formal argument alignment) 6952 - Direct argument < 64 bits should check register budget. 6953 - Register budget calculation should respect ``inreg`` for SGPR. 6954 - SGPR overflow is not handled. 6955 - struct with 1 member unpeeling is not checking size of member. 6956 - ``sret`` is after ``this`` pointer. 6957 - Caller is not implementing stack realignment: need an extra pointer. 6958 - Should say AMDGPU passes FP rather than SP. 6959 - Should CFI define CFA as address of locals or arguments. Difference is 6960 apparent when have implemented dynamic alignment. 6961 - If ``SCRATCH`` instruction could allow negative offsets, then can make FP be 6962 highest address of stack frame and use negative offset for locals. Would 6963 allow SP to be the same as FP and could support signal-handler-like as now 6964 have a real SP for the top of the stack. 6965 - How is ``sret`` passed on the stack? In argument stack area? Can it overlay 6966 arguments? 6967 6968AMDPAL 6969------ 6970 6971This section provides code conventions used when the target triple OS is 6972``amdpal`` (see :ref:`amdgpu-target-triples`) for passing runtime parameters 6973from the application/runtime to each invocation of a hardware shader. These 6974parameters include both generic, application-controlled parameters called 6975*user data* as well as system-generated parameters that are a product of the 6976draw or dispatch execution. 6977 6978User Data 6979~~~~~~~~~ 6980 6981Each hardware stage has a set of 32-bit *user data registers* which can be 6982written from a command buffer and then loaded into SGPRs when waves are launched 6983via a subsequent dispatch or draw operation. This is the way most arguments are 6984passed from the application/runtime to a hardware shader. 6985 6986Compute User Data 6987~~~~~~~~~~~~~~~~~ 6988 6989Compute shader user data mappings are simpler than graphics shaders and have a 6990fixed mapping. 6991 6992Note that there are always 10 available *user data entries* in registers - 6993entries beyond that limit must be fetched from memory (via the spill table 6994pointer) by the shader. 6995 6996 .. table:: PAL Compute Shader User Data Registers 6997 :name: pal-compute-user-data-registers 6998 6999 ============= ================================ 7000 User Register Description 7001 ============= ================================ 7002 0 Global Internal Table (32-bit pointer) 7003 1 Per-Shader Internal Table (32-bit pointer) 7004 2 - 11 Application-Controlled User Data (10 32-bit values) 7005 12 Spill Table (32-bit pointer) 7006 13 - 14 Thread Group Count (64-bit pointer) 7007 15 GDS Range 7008 ============= ================================ 7009 7010Graphics User Data 7011~~~~~~~~~~~~~~~~~~ 7012 7013Graphics pipelines support a much more flexible user data mapping: 7014 7015 .. table:: PAL Graphics Shader User Data Registers 7016 :name: pal-graphics-user-data-registers 7017 7018 ============= ================================ 7019 User Register Description 7020 ============= ================================ 7021 0 Global Internal Table (32-bit pointer) 7022 + Per-Shader Internal Table (32-bit pointer) 7023 + 1-15 Application Controlled User Data 7024 (1-15 Contiguous 32-bit Values in Registers) 7025 + Spill Table (32-bit pointer) 7026 + Draw Index (First Stage Only) 7027 + Vertex Offset (First Stage Only) 7028 + Instance Offset (First Stage Only) 7029 ============= ================================ 7030 7031 The placement of the global internal table remains fixed in the first *user 7032 data SGPR register*. Otherwise all parameters are optional, and can be mapped 7033 to any desired *user data SGPR register*, with the following restrictions: 7034 7035 * Draw Index, Vertex Offset, and Instance Offset can only be used by the first 7036 active hardware stage in a graphics pipeline (i.e. where the API vertex 7037 shader runs). 7038 7039 * Application-controlled user data must be mapped into a contiguous range of 7040 user data registers. 7041 7042 * The application-controlled user data range supports compaction remapping, so 7043 only *entries* that are actually consumed by the shader must be assigned to 7044 corresponding *registers*. Note that in order to support an efficient runtime 7045 implementation, the remapping must pack *registers* in the same order as 7046 *entries*, with unused *entries* removed. 7047 7048.. _pal_global_internal_table: 7049 7050Global Internal Table 7051~~~~~~~~~~~~~~~~~~~~~ 7052 7053The global internal table is a table of *shader resource descriptors* (SRDs) 7054that define how certain engine-wide, runtime-managed resources should be 7055accessed from a shader. The majority of these resources have HW-defined formats, 7056and it is up to the compiler to write/read data as required by the target 7057hardware. 7058 7059The following table illustrates the required format: 7060 7061 .. table:: PAL Global Internal Table 7062 :name: pal-git-table 7063 7064 ============= ================================ 7065 Offset Description 7066 ============= ================================ 7067 0-3 Graphics Scratch SRD 7068 4-7 Compute Scratch SRD 7069 8-11 ES/GS Ring Output SRD 7070 12-15 ES/GS Ring Input SRD 7071 16-19 GS/VS Ring Output #0 7072 20-23 GS/VS Ring Output #1 7073 24-27 GS/VS Ring Output #2 7074 28-31 GS/VS Ring Output #3 7075 32-35 GS/VS Ring Input SRD 7076 36-39 Tessellation Factor Buffer SRD 7077 40-43 Off-Chip LDS Buffer SRD 7078 44-47 Off-Chip Param Cache Buffer SRD 7079 48-51 Sample Position Buffer SRD 7080 52 vaRange::ShadowDescriptorTable High Bits 7081 ============= ================================ 7082 7083 The pointer to the global internal table passed to the shader as user data 7084 is a 32-bit pointer. The top 32 bits should be assumed to be the same as 7085 the top 32 bits of the pipeline, so the shader may use the program 7086 counter's top 32 bits. 7087 7088Unspecified OS 7089-------------- 7090 7091This section provides code conventions used when the target triple OS is 7092empty (see :ref:`amdgpu-target-triples`). 7093 7094Trap Handler ABI 7095~~~~~~~~~~~~~~~~ 7096 7097For code objects generated by AMDGPU backend for non-amdhsa OS, the runtime does 7098not install a trap handler. The ``llvm.trap`` and ``llvm.debugtrap`` 7099instructions are handled as follows: 7100 7101 .. table:: AMDGPU Trap Handler for Non-AMDHSA OS 7102 :name: amdgpu-trap-handler-for-non-amdhsa-os-table 7103 7104 =============== =============== =========================================== 7105 Usage Code Sequence Description 7106 =============== =============== =========================================== 7107 llvm.trap s_endpgm Causes wavefront to be terminated. 7108 llvm.debugtrap *none* Compiler warning given that there is no 7109 trap handler installed. 7110 =============== =============== =========================================== 7111 7112Source Languages 7113================ 7114 7115.. _amdgpu-opencl: 7116 7117OpenCL 7118------ 7119 7120When the language is OpenCL the following differences occur: 7121 71221. The OpenCL memory model is used (see :ref:`amdgpu-amdhsa-memory-model`). 71232. The AMDGPU backend appends additional arguments to the kernel's explicit 7124 arguments for the AMDHSA OS (see 7125 :ref:`opencl-kernel-implicit-arguments-appended-for-amdhsa-os-table`). 71263. Additional metadata is generated 7127 (see :ref:`amdgpu-amdhsa-code-object-metadata`). 7128 7129 .. table:: OpenCL kernel implicit arguments appended for AMDHSA OS 7130 :name: opencl-kernel-implicit-arguments-appended-for-amdhsa-os-table 7131 7132 ======== ==== ========= =========================================== 7133 Position Byte Byte Description 7134 Size Alignment 7135 ======== ==== ========= =========================================== 7136 1 8 8 OpenCL Global Offset X 7137 2 8 8 OpenCL Global Offset Y 7138 3 8 8 OpenCL Global Offset Z 7139 4 8 8 OpenCL address of printf buffer 7140 5 8 8 OpenCL address of virtual queue used by 7141 enqueue_kernel. 7142 6 8 8 OpenCL address of AqlWrap struct used by 7143 enqueue_kernel. 7144 7 8 8 Pointer argument used for Multi-gird 7145 synchronization. 7146 ======== ==== ========= =========================================== 7147 7148.. _amdgpu-hcc: 7149 7150HCC 7151--- 7152 7153When the language is HCC the following differences occur: 7154 71551. The HSA memory model is used (see :ref:`amdgpu-amdhsa-memory-model`). 7156 7157.. _amdgpu-assembler: 7158 7159Assembler 7160--------- 7161 7162AMDGPU backend has LLVM-MC based assembler which is currently in development. 7163It supports AMDGCN GFX6-GFX10. 7164 7165This section describes general syntax for instructions and operands. 7166 7167Instructions 7168~~~~~~~~~~~~ 7169 7170An instruction has the following :doc:`syntax<AMDGPUInstructionSyntax>`: 7171 7172 | ``<``\ *opcode*\ ``> <``\ *operand0*\ ``>, <``\ *operand1*\ ``>,... 7173 <``\ *modifier0*\ ``> <``\ *modifier1*\ ``>...`` 7174 7175:doc:`Operands<AMDGPUOperandSyntax>` are comma-separated while 7176:doc:`modifiers<AMDGPUModifierSyntax>` are space-separated. 7177 7178The order of operands and modifiers is fixed. 7179Most modifiers are optional and may be omitted. 7180 7181Links to detailed instruction syntax description may be found in the following 7182table. Note that features under development are not included 7183in this description. 7184 7185 =================================== ======================================= 7186 Core ISA ISA Extensions 7187 =================================== ======================================= 7188 :doc:`GFX7<AMDGPU/AMDGPUAsmGFX7>` \- 7189 :doc:`GFX8<AMDGPU/AMDGPUAsmGFX8>` \- 7190 :doc:`GFX9<AMDGPU/AMDGPUAsmGFX9>` :doc:`gfx900<AMDGPU/AMDGPUAsmGFX900>` 7191 7192 :doc:`gfx902<AMDGPU/AMDGPUAsmGFX900>` 7193 7194 :doc:`gfx904<AMDGPU/AMDGPUAsmGFX904>` 7195 7196 :doc:`gfx906<AMDGPU/AMDGPUAsmGFX906>` 7197 7198 :doc:`gfx908<AMDGPU/AMDGPUAsmGFX908>` 7199 7200 :doc:`gfx909<AMDGPU/AMDGPUAsmGFX900>` 7201 7202 :doc:`GFX10<AMDGPU/AMDGPUAsmGFX10>` :doc:`gfx1011<AMDGPU/AMDGPUAsmGFX1011>` 7203 7204 :doc:`gfx1012<AMDGPU/AMDGPUAsmGFX1011>` 7205 =================================== ======================================= 7206 7207For more information about instructions, their semantics and supported 7208combinations of operands, refer to one of instruction set architecture manuals 7209[AMD-GCN-GFX6]_, [AMD-GCN-GFX7]_, [AMD-GCN-GFX8]_, [AMD-GCN-GFX9]_ and 7210[AMD-GCN-GFX10]_. 7211 7212Operands 7213~~~~~~~~ 7214 7215Detailed description of operands may be found :doc:`here<AMDGPUOperandSyntax>`. 7216 7217Modifiers 7218~~~~~~~~~ 7219 7220Detailed description of modifiers may be found 7221:doc:`here<AMDGPUModifierSyntax>`. 7222 7223Instruction Examples 7224~~~~~~~~~~~~~~~~~~~~ 7225 7226DS 7227++ 7228 7229.. code-block:: nasm 7230 7231 ds_add_u32 v2, v4 offset:16 7232 ds_write_src2_b64 v2 offset0:4 offset1:8 7233 ds_cmpst_f32 v2, v4, v6 7234 ds_min_rtn_f64 v[8:9], v2, v[4:5] 7235 7236For full list of supported instructions, refer to "LDS/GDS instructions" in ISA 7237Manual. 7238 7239FLAT 7240++++ 7241 7242.. code-block:: nasm 7243 7244 flat_load_dword v1, v[3:4] 7245 flat_store_dwordx3 v[3:4], v[5:7] 7246 flat_atomic_swap v1, v[3:4], v5 glc 7247 flat_atomic_cmpswap v1, v[3:4], v[5:6] glc slc 7248 flat_atomic_fmax_x2 v[1:2], v[3:4], v[5:6] glc 7249 7250For full list of supported instructions, refer to "FLAT instructions" in ISA 7251Manual. 7252 7253MUBUF 7254+++++ 7255 7256.. code-block:: nasm 7257 7258 buffer_load_dword v1, off, s[4:7], s1 7259 buffer_store_dwordx4 v[1:4], v2, ttmp[4:7], s1 offen offset:4 glc tfe 7260 buffer_store_format_xy v[1:2], off, s[4:7], s1 7261 buffer_wbinvl1 7262 buffer_atomic_inc v1, v2, s[8:11], s4 idxen offset:4 slc 7263 7264For full list of supported instructions, refer to "MUBUF Instructions" in ISA 7265Manual. 7266 7267SMRD/SMEM 7268+++++++++ 7269 7270.. code-block:: nasm 7271 7272 s_load_dword s1, s[2:3], 0xfc 7273 s_load_dwordx8 s[8:15], s[2:3], s4 7274 s_load_dwordx16 s[88:103], s[2:3], s4 7275 s_dcache_inv_vol 7276 s_memtime s[4:5] 7277 7278For full list of supported instructions, refer to "Scalar Memory Operations" in 7279ISA Manual. 7280 7281SOP1 7282++++ 7283 7284.. code-block:: nasm 7285 7286 s_mov_b32 s1, s2 7287 s_mov_b64 s[0:1], 0x80000000 7288 s_cmov_b32 s1, 200 7289 s_wqm_b64 s[2:3], s[4:5] 7290 s_bcnt0_i32_b64 s1, s[2:3] 7291 s_swappc_b64 s[2:3], s[4:5] 7292 s_cbranch_join s[4:5] 7293 7294For full list of supported instructions, refer to "SOP1 Instructions" in ISA 7295Manual. 7296 7297SOP2 7298++++ 7299 7300.. code-block:: nasm 7301 7302 s_add_u32 s1, s2, s3 7303 s_and_b64 s[2:3], s[4:5], s[6:7] 7304 s_cselect_b32 s1, s2, s3 7305 s_andn2_b32 s2, s4, s6 7306 s_lshr_b64 s[2:3], s[4:5], s6 7307 s_ashr_i32 s2, s4, s6 7308 s_bfm_b64 s[2:3], s4, s6 7309 s_bfe_i64 s[2:3], s[4:5], s6 7310 s_cbranch_g_fork s[4:5], s[6:7] 7311 7312For full list of supported instructions, refer to "SOP2 Instructions" in ISA 7313Manual. 7314 7315SOPC 7316++++ 7317 7318.. code-block:: nasm 7319 7320 s_cmp_eq_i32 s1, s2 7321 s_bitcmp1_b32 s1, s2 7322 s_bitcmp0_b64 s[2:3], s4 7323 s_setvskip s3, s5 7324 7325For full list of supported instructions, refer to "SOPC Instructions" in ISA 7326Manual. 7327 7328SOPP 7329++++ 7330 7331.. code-block:: nasm 7332 7333 s_barrier 7334 s_nop 2 7335 s_endpgm 7336 s_waitcnt 0 ; Wait for all counters to be 0 7337 s_waitcnt vmcnt(0) & expcnt(0) & lgkmcnt(0) ; Equivalent to above 7338 s_waitcnt vmcnt(1) ; Wait for vmcnt counter to be 1. 7339 s_sethalt 9 7340 s_sleep 10 7341 s_sendmsg 0x1 7342 s_sendmsg sendmsg(MSG_INTERRUPT) 7343 s_trap 1 7344 7345For full list of supported instructions, refer to "SOPP Instructions" in ISA 7346Manual. 7347 7348Unless otherwise mentioned, little verification is performed on the operands 7349of SOPP Instructions, so it is up to the programmer to be familiar with the 7350range or acceptable values. 7351 7352VALU 7353++++ 7354 7355For vector ALU instruction opcodes (VOP1, VOP2, VOP3, VOPC, VOP_DPP, VOP_SDWA), 7356the assembler will automatically use optimal encoding based on its operands. To 7357force specific encoding, one can add a suffix to the opcode of the instruction: 7358 7359* _e32 for 32-bit VOP1/VOP2/VOPC 7360* _e64 for 64-bit VOP3 7361* _dpp for VOP_DPP 7362* _sdwa for VOP_SDWA 7363 7364VOP1/VOP2/VOP3/VOPC examples: 7365 7366.. code-block:: nasm 7367 7368 v_mov_b32 v1, v2 7369 v_mov_b32_e32 v1, v2 7370 v_nop 7371 v_cvt_f64_i32_e32 v[1:2], v2 7372 v_floor_f32_e32 v1, v2 7373 v_bfrev_b32_e32 v1, v2 7374 v_add_f32_e32 v1, v2, v3 7375 v_mul_i32_i24_e64 v1, v2, 3 7376 v_mul_i32_i24_e32 v1, -3, v3 7377 v_mul_i32_i24_e32 v1, -100, v3 7378 v_addc_u32 v1, s[0:1], v2, v3, s[2:3] 7379 v_max_f16_e32 v1, v2, v3 7380 7381VOP_DPP examples: 7382 7383.. code-block:: nasm 7384 7385 v_mov_b32 v0, v0 quad_perm:[0,2,1,1] 7386 v_sin_f32 v0, v0 row_shl:1 row_mask:0xa bank_mask:0x1 bound_ctrl:0 7387 v_mov_b32 v0, v0 wave_shl:1 7388 v_mov_b32 v0, v0 row_mirror 7389 v_mov_b32 v0, v0 row_bcast:31 7390 v_mov_b32 v0, v0 quad_perm:[1,3,0,1] row_mask:0xa bank_mask:0x1 bound_ctrl:0 7391 v_add_f32 v0, v0, |v0| row_shl:1 row_mask:0xa bank_mask:0x1 bound_ctrl:0 7392 v_max_f16 v1, v2, v3 row_shl:1 row_mask:0xa bank_mask:0x1 bound_ctrl:0 7393 7394VOP_SDWA examples: 7395 7396.. code-block:: nasm 7397 7398 v_mov_b32 v1, v2 dst_sel:BYTE_0 dst_unused:UNUSED_PRESERVE src0_sel:DWORD 7399 v_min_u32 v200, v200, v1 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:BYTE_1 src1_sel:DWORD 7400 v_sin_f32 v0, v0 dst_unused:UNUSED_PAD src0_sel:WORD_1 7401 v_fract_f32 v0, |v0| dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 7402 v_cmpx_le_u32 vcc, v1, v2 src0_sel:BYTE_2 src1_sel:WORD_0 7403 7404For full list of supported instructions, refer to "Vector ALU instructions". 7405 7406.. TODO:: 7407 7408 Remove once we switch to code object v3 by default. 7409 7410.. _amdgpu-amdhsa-assembler-predefined-symbols-v2: 7411 7412Code Object V2 Predefined Symbols (--amdhsa-code-object-version=2) 7413~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 7414 7415.. warning:: Code Object V2 is not the default code object version emitted by 7416 this version of LLVM. For a description of the predefined symbols available 7417 with the default configuration (Code Object V3) see 7418 :ref:`amdgpu-amdhsa-assembler-predefined-symbols-v3`. 7419 7420The AMDGPU assembler defines and updates some symbols automatically. These 7421symbols do not affect code generation. 7422 7423.option.machine_version_major 7424+++++++++++++++++++++++++++++ 7425 7426Set to the GFX major generation number of the target being assembled for. For 7427example, when assembling for a "GFX9" target this will be set to the integer 7428value "9". The possible GFX major generation numbers are presented in 7429:ref:`amdgpu-processors`. 7430 7431.option.machine_version_minor 7432+++++++++++++++++++++++++++++ 7433 7434Set to the GFX minor generation number of the target being assembled for. For 7435example, when assembling for a "GFX810" target this will be set to the integer 7436value "1". The possible GFX minor generation numbers are presented in 7437:ref:`amdgpu-processors`. 7438 7439.option.machine_version_stepping 7440++++++++++++++++++++++++++++++++ 7441 7442Set to the GFX stepping generation number of the target being assembled for. 7443For example, when assembling for a "GFX704" target this will be set to the 7444integer value "4". The possible GFX stepping generation numbers are presented 7445in :ref:`amdgpu-processors`. 7446 7447.kernel.vgpr_count 7448++++++++++++++++++ 7449 7450Set to zero each time a 7451:ref:`amdgpu-amdhsa-assembler-directive-amdgpu_hsa_kernel` directive is 7452encountered. At each instruction, if the current value of this symbol is less 7453than or equal to the maximum VGPR number explicitly referenced within that 7454instruction then the symbol value is updated to equal that VGPR number plus 7455one. 7456 7457.kernel.sgpr_count 7458++++++++++++++++++ 7459 7460Set to zero each time a 7461:ref:`amdgpu-amdhsa-assembler-directive-amdgpu_hsa_kernel` directive is 7462encountered. At each instruction, if the current value of this symbol is less 7463than or equal to the maximum VGPR number explicitly referenced within that 7464instruction then the symbol value is updated to equal that SGPR number plus 7465one. 7466 7467.. _amdgpu-amdhsa-assembler-directives-v2: 7468 7469Code Object V2 Directives (--amdhsa-code-object-version=2) 7470~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 7471 7472.. warning:: Code Object V2 is not the default code object version emitted by 7473 this version of LLVM. For a description of the directives supported with 7474 the default configuration (Code Object V3) see 7475 :ref:`amdgpu-amdhsa-assembler-directives-v3`. 7476 7477AMDGPU ABI defines auxiliary data in output code object. In assembly source, 7478one can specify them with assembler directives. 7479 7480.hsa_code_object_version major, minor 7481+++++++++++++++++++++++++++++++++++++ 7482 7483*major* and *minor* are integers that specify the version of the HSA code 7484object that will be generated by the assembler. 7485 7486.hsa_code_object_isa [major, minor, stepping, vendor, arch] 7487+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 7488 7489 7490*major*, *minor*, and *stepping* are all integers that describe the instruction 7491set architecture (ISA) version of the assembly program. 7492 7493*vendor* and *arch* are quoted strings. *vendor* should always be equal to 7494"AMD" and *arch* should always be equal to "AMDGPU". 7495 7496By default, the assembler will derive the ISA version, *vendor*, and *arch* 7497from the value of the -mcpu option that is passed to the assembler. 7498 7499.. _amdgpu-amdhsa-assembler-directive-amdgpu_hsa_kernel: 7500 7501.amdgpu_hsa_kernel (name) 7502+++++++++++++++++++++++++ 7503 7504This directives specifies that the symbol with given name is a kernel entry 7505point (label) and the object should contain corresponding symbol of type 7506STT_AMDGPU_HSA_KERNEL. 7507 7508.amd_kernel_code_t 7509++++++++++++++++++ 7510 7511This directive marks the beginning of a list of key / value pairs that are used 7512to specify the amd_kernel_code_t object that will be emitted by the assembler. 7513The list must be terminated by the *.end_amd_kernel_code_t* directive. For any 7514amd_kernel_code_t values that are unspecified a default value will be used. The 7515default value for all keys is 0, with the following exceptions: 7516 7517- *amd_code_version_major* defaults to 1. 7518- *amd_kernel_code_version_minor* defaults to 2. 7519- *amd_machine_kind* defaults to 1. 7520- *amd_machine_version_major*, *machine_version_minor*, and 7521 *amd_machine_version_stepping* are derived from the value of the -mcpu option 7522 that is passed to the assembler. 7523- *kernel_code_entry_byte_offset* defaults to 256. 7524- *wavefront_size* defaults 6 for all targets before GFX10. For GFX10 onwards 7525 defaults to 6 if target feature ``wavefrontsize64`` is enabled, otherwise 5. 7526 Note that wavefront size is specified as a power of two, so a value of **n** 7527 means a size of 2^ **n**. 7528- *call_convention* defaults to -1. 7529- *kernarg_segment_alignment*, *group_segment_alignment*, and 7530 *private_segment_alignment* default to 4. Note that alignments are specified 7531 as a power of 2, so a value of **n** means an alignment of 2^ **n**. 7532- *enable_wgp_mode* defaults to 1 if target feature ``cumode`` is disabled for 7533 GFX10 onwards. 7534- *enable_mem_ordered* defaults to 1 for GFX10 onwards. 7535 7536The *.amd_kernel_code_t* directive must be placed immediately after the 7537function label and before any instructions. 7538 7539For a full list of amd_kernel_code_t keys, refer to AMDGPU ABI document, 7540comments in lib/Target/AMDGPU/AmdKernelCodeT.h and test/CodeGen/AMDGPU/hsa.s. 7541 7542.. _amdgpu-amdhsa-assembler-example-v2: 7543 7544Code Object V2 Example Source Code (--amdhsa-code-object-version=2) 7545~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 7546 7547.. warning:: Code Object V2 is not the default code object version emitted by 7548 this version of LLVM. For a description of the directives supported with 7549 the default configuration (Code Object V3) see 7550 :ref:`amdgpu-amdhsa-assembler-example-v3`. 7551 7552Here is an example of a minimal assembly source file, defining one HSA kernel: 7553 7554.. code:: 7555 :number-lines: 7556 7557 .hsa_code_object_version 1,0 7558 .hsa_code_object_isa 7559 7560 .hsatext 7561 .globl hello_world 7562 .p2align 8 7563 .amdgpu_hsa_kernel hello_world 7564 7565 hello_world: 7566 7567 .amd_kernel_code_t 7568 enable_sgpr_kernarg_segment_ptr = 1 7569 is_ptr64 = 1 7570 compute_pgm_rsrc1_vgprs = 0 7571 compute_pgm_rsrc1_sgprs = 0 7572 compute_pgm_rsrc2_user_sgpr = 2 7573 compute_pgm_rsrc1_wgp_mode = 0 7574 compute_pgm_rsrc1_mem_ordered = 0 7575 compute_pgm_rsrc1_fwd_progress = 1 7576 .end_amd_kernel_code_t 7577 7578 s_load_dwordx2 s[0:1], s[0:1] 0x0 7579 v_mov_b32 v0, 3.14159 7580 s_waitcnt lgkmcnt(0) 7581 v_mov_b32 v1, s0 7582 v_mov_b32 v2, s1 7583 flat_store_dword v[1:2], v0 7584 s_endpgm 7585 .Lfunc_end0: 7586 .size hello_world, .Lfunc_end0-hello_world 7587 7588.. _amdgpu-amdhsa-assembler-predefined-symbols-v3: 7589 7590Code Object V3 Predefined Symbols (--amdhsa-code-object-version=3) 7591~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 7592 7593The AMDGPU assembler defines and updates some symbols automatically. These 7594symbols do not affect code generation. 7595 7596.amdgcn.gfx_generation_number 7597+++++++++++++++++++++++++++++ 7598 7599Set to the GFX major generation number of the target being assembled for. For 7600example, when assembling for a "GFX9" target this will be set to the integer 7601value "9". The possible GFX major generation numbers are presented in 7602:ref:`amdgpu-processors`. 7603 7604.amdgcn.gfx_generation_minor 7605++++++++++++++++++++++++++++ 7606 7607Set to the GFX minor generation number of the target being assembled for. For 7608example, when assembling for a "GFX810" target this will be set to the integer 7609value "1". The possible GFX minor generation numbers are presented in 7610:ref:`amdgpu-processors`. 7611 7612.amdgcn.gfx_generation_stepping 7613+++++++++++++++++++++++++++++++ 7614 7615Set to the GFX stepping generation number of the target being assembled for. 7616For example, when assembling for a "GFX704" target this will be set to the 7617integer value "4". The possible GFX stepping generation numbers are presented 7618in :ref:`amdgpu-processors`. 7619 7620.. _amdgpu-amdhsa-assembler-symbol-next_free_vgpr: 7621 7622.amdgcn.next_free_vgpr 7623++++++++++++++++++++++ 7624 7625Set to zero before assembly begins. At each instruction, if the current value 7626of this symbol is less than or equal to the maximum VGPR number explicitly 7627referenced within that instruction then the symbol value is updated to equal 7628that VGPR number plus one. 7629 7630May be used to set the `.amdhsa_next_free_vgpr` directive in 7631:ref:`amdhsa-kernel-directives-table`. 7632 7633May be set at any time, e.g. manually set to zero at the start of each kernel. 7634 7635.. _amdgpu-amdhsa-assembler-symbol-next_free_sgpr: 7636 7637.amdgcn.next_free_sgpr 7638++++++++++++++++++++++ 7639 7640Set to zero before assembly begins. At each instruction, if the current value 7641of this symbol is less than or equal the maximum SGPR number explicitly 7642referenced within that instruction then the symbol value is updated to equal 7643that SGPR number plus one. 7644 7645May be used to set the `.amdhsa_next_free_spgr` directive in 7646:ref:`amdhsa-kernel-directives-table`. 7647 7648May be set at any time, e.g. manually set to zero at the start of each kernel. 7649 7650.. _amdgpu-amdhsa-assembler-directives-v3: 7651 7652Code Object V3 Directives (--amdhsa-code-object-version=3) 7653~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 7654 7655Directives which begin with ``.amdgcn`` are valid for all ``amdgcn`` 7656architecture processors, and are not OS-specific. Directives which begin with 7657``.amdhsa`` are specific to ``amdgcn`` architecture processors when the 7658``amdhsa`` OS is specified. See :ref:`amdgpu-target-triples` and 7659:ref:`amdgpu-processors`. 7660 7661.amdgcn_target <target> 7662+++++++++++++++++++++++ 7663 7664Optional directive which declares the target supported by the containing 7665assembler source file. Valid values are described in 7666:ref:`amdgpu-amdhsa-code-object-target-identification`. Used by the assembler 7667to validate command-line options such as ``-triple``, ``-mcpu``, and those 7668which specify target features. 7669 7670.amdhsa_kernel <name> 7671+++++++++++++++++++++ 7672 7673Creates a correctly aligned AMDHSA kernel descriptor and a symbol, 7674``<name>.kd``, in the current location of the current section. Only valid when 7675the OS is ``amdhsa``. ``<name>`` must be a symbol that labels the first 7676instruction to execute, and does not need to be previously defined. 7677 7678Marks the beginning of a list of directives used to generate the bytes of a 7679kernel descriptor, as described in :ref:`amdgpu-amdhsa-kernel-descriptor`. 7680Directives which may appear in this list are described in 7681:ref:`amdhsa-kernel-directives-table`. Directives may appear in any order, must 7682be valid for the target being assembled for, and cannot be repeated. Directives 7683support the range of values specified by the field they reference in 7684:ref:`amdgpu-amdhsa-kernel-descriptor`. If a directive is not specified, it is 7685assumed to have its default value, unless it is marked as "Required", in which 7686case it is an error to omit the directive. This list of directives is 7687terminated by an ``.end_amdhsa_kernel`` directive. 7688 7689 .. table:: AMDHSA Kernel Assembler Directives 7690 :name: amdhsa-kernel-directives-table 7691 7692 ======================================================== =================== ============ =================== 7693 Directive Default Supported On Description 7694 ======================================================== =================== ============ =================== 7695 ``.amdhsa_group_segment_fixed_size`` 0 GFX6-GFX10 Controls GROUP_SEGMENT_FIXED_SIZE in 7696 :ref:`amdgpu-amdhsa-kernel-descriptor-gfx6-gfx10-table`. 7697 ``.amdhsa_private_segment_fixed_size`` 0 GFX6-GFX10 Controls PRIVATE_SEGMENT_FIXED_SIZE in 7698 :ref:`amdgpu-amdhsa-kernel-descriptor-gfx6-gfx10-table`. 7699 ``.amdhsa_user_sgpr_private_segment_buffer`` 0 GFX6-GFX10 Controls ENABLE_SGPR_PRIVATE_SEGMENT_BUFFER in 7700 :ref:`amdgpu-amdhsa-kernel-descriptor-gfx6-gfx10-table`. 7701 ``.amdhsa_user_sgpr_dispatch_ptr`` 0 GFX6-GFX10 Controls ENABLE_SGPR_DISPATCH_PTR in 7702 :ref:`amdgpu-amdhsa-kernel-descriptor-gfx6-gfx10-table`. 7703 ``.amdhsa_user_sgpr_queue_ptr`` 0 GFX6-GFX10 Controls ENABLE_SGPR_QUEUE_PTR in 7704 :ref:`amdgpu-amdhsa-kernel-descriptor-gfx6-gfx10-table`. 7705 ``.amdhsa_user_sgpr_kernarg_segment_ptr`` 0 GFX6-GFX10 Controls ENABLE_SGPR_KERNARG_SEGMENT_PTR in 7706 :ref:`amdgpu-amdhsa-kernel-descriptor-gfx6-gfx10-table`. 7707 ``.amdhsa_user_sgpr_dispatch_id`` 0 GFX6-GFX10 Controls ENABLE_SGPR_DISPATCH_ID in 7708 :ref:`amdgpu-amdhsa-kernel-descriptor-gfx6-gfx10-table`. 7709 ``.amdhsa_user_sgpr_flat_scratch_init`` 0 GFX6-GFX10 Controls ENABLE_SGPR_FLAT_SCRATCH_INIT in 7710 :ref:`amdgpu-amdhsa-kernel-descriptor-gfx6-gfx10-table`. 7711 ``.amdhsa_user_sgpr_private_segment_size`` 0 GFX6-GFX10 Controls ENABLE_SGPR_PRIVATE_SEGMENT_SIZE in 7712 :ref:`amdgpu-amdhsa-kernel-descriptor-gfx6-gfx10-table`. 7713 ``.amdhsa_wavefront_size32`` Target GFX10 Controls ENABLE_WAVEFRONT_SIZE32 in 7714 Feature :ref:`amdgpu-amdhsa-kernel-descriptor-gfx6-gfx10-table`. 7715 Specific 7716 (-wavefrontsize64) 7717 ``.amdhsa_system_sgpr_private_segment_wavefront_offset`` 0 GFX6-GFX10 Controls ENABLE_SGPR_PRIVATE_SEGMENT_WAVEFRONT_OFFSET in 7718 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 7719 ``.amdhsa_system_sgpr_workgroup_id_x`` 1 GFX6-GFX10 Controls ENABLE_SGPR_WORKGROUP_ID_X in 7720 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 7721 ``.amdhsa_system_sgpr_workgroup_id_y`` 0 GFX6-GFX10 Controls ENABLE_SGPR_WORKGROUP_ID_Y in 7722 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 7723 ``.amdhsa_system_sgpr_workgroup_id_z`` 0 GFX6-GFX10 Controls ENABLE_SGPR_WORKGROUP_ID_Z in 7724 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 7725 ``.amdhsa_system_sgpr_workgroup_info`` 0 GFX6-GFX10 Controls ENABLE_SGPR_WORKGROUP_INFO in 7726 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 7727 ``.amdhsa_system_vgpr_workitem_id`` 0 GFX6-GFX10 Controls ENABLE_VGPR_WORKITEM_ID in 7728 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 7729 Possible values are defined in 7730 :ref:`amdgpu-amdhsa-system-vgpr-work-item-id-enumeration-values-table`. 7731 ``.amdhsa_next_free_vgpr`` Required GFX6-GFX10 Maximum VGPR number explicitly referenced, plus one. 7732 Used to calculate GRANULATED_WORKITEM_VGPR_COUNT in 7733 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 7734 ``.amdhsa_next_free_sgpr`` Required GFX6-GFX10 Maximum SGPR number explicitly referenced, plus one. 7735 Used to calculate GRANULATED_WAVEFRONT_SGPR_COUNT in 7736 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 7737 ``.amdhsa_reserve_vcc`` 1 GFX6-GFX10 Whether the kernel may use the special VCC SGPR. 7738 Used to calculate GRANULATED_WAVEFRONT_SGPR_COUNT in 7739 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 7740 ``.amdhsa_reserve_flat_scratch`` 1 GFX7-GFX10 Whether the kernel may use flat instructions to access 7741 scratch memory. Used to calculate 7742 GRANULATED_WAVEFRONT_SGPR_COUNT in 7743 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 7744 ``.amdhsa_reserve_xnack_mask`` Target GFX8-GFX10 Whether the kernel may trigger XNACK replay. 7745 Feature Used to calculate GRANULATED_WAVEFRONT_SGPR_COUNT in 7746 Specific :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 7747 (+xnack) 7748 ``.amdhsa_float_round_mode_32`` 0 GFX6-GFX10 Controls FLOAT_ROUND_MODE_32 in 7749 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 7750 Possible values are defined in 7751 :ref:`amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table`. 7752 ``.amdhsa_float_round_mode_16_64`` 0 GFX6-GFX10 Controls FLOAT_ROUND_MODE_16_64 in 7753 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 7754 Possible values are defined in 7755 :ref:`amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table`. 7756 ``.amdhsa_float_denorm_mode_32`` 0 GFX6-GFX10 Controls FLOAT_DENORM_MODE_32 in 7757 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 7758 Possible values are defined in 7759 :ref:`amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table`. 7760 ``.amdhsa_float_denorm_mode_16_64`` 3 GFX6-GFX10 Controls FLOAT_DENORM_MODE_16_64 in 7761 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 7762 Possible values are defined in 7763 :ref:`amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table`. 7764 ``.amdhsa_dx10_clamp`` 1 GFX6-GFX10 Controls ENABLE_DX10_CLAMP in 7765 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 7766 ``.amdhsa_ieee_mode`` 1 GFX6-GFX10 Controls ENABLE_IEEE_MODE in 7767 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 7768 ``.amdhsa_fp16_overflow`` 0 GFX9-GFX10 Controls FP16_OVFL in 7769 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 7770 ``.amdhsa_workgroup_processor_mode`` Target GFX10 Controls ENABLE_WGP_MODE in 7771 Feature :ref:`amdgpu-amdhsa-kernel-descriptor-gfx6-gfx10-table`. 7772 Specific 7773 (-cumode) 7774 ``.amdhsa_memory_ordered`` 1 GFX10 Controls MEM_ORDERED in 7775 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 7776 ``.amdhsa_forward_progress`` 0 GFX10 Controls FWD_PROGRESS in 7777 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 7778 ``.amdhsa_exception_fp_ieee_invalid_op`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_INVALID_OPERATION in 7779 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 7780 ``.amdhsa_exception_fp_denorm_src`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_FP_DENORMAL_SOURCE in 7781 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 7782 ``.amdhsa_exception_fp_ieee_div_zero`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_DIVISION_BY_ZERO in 7783 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 7784 ``.amdhsa_exception_fp_ieee_overflow`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_OVERFLOW in 7785 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 7786 ``.amdhsa_exception_fp_ieee_underflow`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_UNDERFLOW in 7787 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 7788 ``.amdhsa_exception_fp_ieee_inexact`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_INEXACT in 7789 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 7790 ``.amdhsa_exception_int_div_zero`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_INT_DIVIDE_BY_ZERO in 7791 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 7792 ======================================================== =================== ============ =================== 7793 7794.amdgpu_metadata 7795++++++++++++++++ 7796 7797Optional directive which declares the contents of the ``NT_AMDGPU_METADATA`` 7798note record (see :ref:`amdgpu-elf-note-records-table-v3`). 7799 7800The contents must be in the [YAML]_ markup format, with the same structure and 7801semantics described in :ref:`amdgpu-amdhsa-code-object-metadata-v3`. 7802 7803This directive is terminated by an ``.end_amdgpu_metadata`` directive. 7804 7805.. _amdgpu-amdhsa-assembler-example-v3: 7806 7807Code Object V3 Example Source Code (--amdhsa-code-object-version=3) 7808~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 7809 7810Here is an example of a minimal assembly source file, defining one HSA kernel: 7811 7812.. code:: 7813 :number-lines: 7814 7815 .amdgcn_target "amdgcn-amd-amdhsa--gfx900+xnack" // optional 7816 7817 .text 7818 .globl hello_world 7819 .p2align 8 7820 .type hello_world,@function 7821 hello_world: 7822 s_load_dwordx2 s[0:1], s[0:1] 0x0 7823 v_mov_b32 v0, 3.14159 7824 s_waitcnt lgkmcnt(0) 7825 v_mov_b32 v1, s0 7826 v_mov_b32 v2, s1 7827 flat_store_dword v[1:2], v0 7828 s_endpgm 7829 .Lfunc_end0: 7830 .size hello_world, .Lfunc_end0-hello_world 7831 7832 .rodata 7833 .p2align 6 7834 .amdhsa_kernel hello_world 7835 .amdhsa_user_sgpr_kernarg_segment_ptr 1 7836 .amdhsa_next_free_vgpr .amdgcn.next_free_vgpr 7837 .amdhsa_next_free_sgpr .amdgcn.next_free_sgpr 7838 .end_amdhsa_kernel 7839 7840 .amdgpu_metadata 7841 --- 7842 amdhsa.version: 7843 - 1 7844 - 0 7845 amdhsa.kernels: 7846 - .name: hello_world 7847 .symbol: hello_world.kd 7848 .kernarg_segment_size: 48 7849 .group_segment_fixed_size: 0 7850 .private_segment_fixed_size: 0 7851 .kernarg_segment_align: 4 7852 .wavefront_size: 64 7853 .sgpr_count: 2 7854 .vgpr_count: 3 7855 .max_flat_workgroup_size: 256 7856 ... 7857 .end_amdgpu_metadata 7858 7859If an assembly source file contains multiple kernels and/or functions, the 7860:ref:`amdgpu-amdhsa-assembler-symbol-next_free_vgpr` and 7861:ref:`amdgpu-amdhsa-assembler-symbol-next_free_sgpr` symbols may be reset using 7862the ``.set <symbol>, <expression>`` directive. For example, in the case of two 7863kernels, where ``function1`` is only called from ``kernel1`` it is sufficient 7864to group the function with the kernel that calls it and reset the symbols 7865between the two connected components: 7866 7867.. code:: 7868 :number-lines: 7869 7870 .amdgcn_target "amdgcn-amd-amdhsa--gfx900+xnack" // optional 7871 7872 // gpr tracking symbols are implicitly set to zero 7873 7874 .text 7875 .globl kern0 7876 .p2align 8 7877 .type kern0,@function 7878 kern0: 7879 // ... 7880 s_endpgm 7881 .Lkern0_end: 7882 .size kern0, .Lkern0_end-kern0 7883 7884 .rodata 7885 .p2align 6 7886 .amdhsa_kernel kern0 7887 // ... 7888 .amdhsa_next_free_vgpr .amdgcn.next_free_vgpr 7889 .amdhsa_next_free_sgpr .amdgcn.next_free_sgpr 7890 .end_amdhsa_kernel 7891 7892 // reset symbols to begin tracking usage in func1 and kern1 7893 .set .amdgcn.next_free_vgpr, 0 7894 .set .amdgcn.next_free_sgpr, 0 7895 7896 .text 7897 .hidden func1 7898 .global func1 7899 .p2align 2 7900 .type func1,@function 7901 func1: 7902 // ... 7903 s_setpc_b64 s[30:31] 7904 .Lfunc1_end: 7905 .size func1, .Lfunc1_end-func1 7906 7907 .globl kern1 7908 .p2align 8 7909 .type kern1,@function 7910 kern1: 7911 // ... 7912 s_getpc_b64 s[4:5] 7913 s_add_u32 s4, s4, func1@rel32@lo+4 7914 s_addc_u32 s5, s5, func1@rel32@lo+4 7915 s_swappc_b64 s[30:31], s[4:5] 7916 // ... 7917 s_endpgm 7918 .Lkern1_end: 7919 .size kern1, .Lkern1_end-kern1 7920 7921 .rodata 7922 .p2align 6 7923 .amdhsa_kernel kern1 7924 // ... 7925 .amdhsa_next_free_vgpr .amdgcn.next_free_vgpr 7926 .amdhsa_next_free_sgpr .amdgcn.next_free_sgpr 7927 .end_amdhsa_kernel 7928 7929These symbols cannot identify connected components in order to automatically 7930track the usage for each kernel. However, in some cases careful organization of 7931the kernels and functions in the source file means there is minimal additional 7932effort required to accurately calculate GPR usage. 7933 7934Additional Documentation 7935======================== 7936 7937.. [AMD-GCN-GFX6] `AMD Southern Islands Series ISA <http://developer.amd.com/wordpress/media/2012/12/AMD_Southern_Islands_Instruction_Set_Architecture.pdf>`__ 7938.. [AMD-GCN-GFX7] `AMD Sea Islands Series ISA <http://developer.amd.com/wordpress/media/2013/07/AMD_Sea_Islands_Instruction_Set_Architecture.pdf>`_ 7939.. [AMD-GCN-GFX8] `AMD GCN3 Instruction Set Architecture <http://amd-dev.wpengine.netdna-cdn.com/wordpress/media/2013/12/AMD_GCN3_Instruction_Set_Architecture_rev1.1.pdf>`__ 7940.. [AMD-GCN-GFX9] `AMD "Vega" Instruction Set Architecture <http://developer.amd.com/wordpress/media/2013/12/Vega_Shader_ISA_28July2017.pdf>`__ 7941.. [AMD-GCN-GFX10] `AMD "RDNA 1.0" Instruction Set Architecture <https://gpuopen.com/wp-content/uploads/2019/08/RDNA_Shader_ISA_5August2019.pdf>`__ 7942.. [AMD-RADEON-HD-2000-3000] `AMD R6xx shader ISA <http://developer.amd.com/wordpress/media/2012/10/R600_Instruction_Set_Architecture.pdf>`__ 7943.. [AMD-RADEON-HD-4000] `AMD R7xx shader ISA <http://developer.amd.com/wordpress/media/2012/10/R700-Family_Instruction_Set_Architecture.pdf>`__ 7944.. [AMD-RADEON-HD-5000] `AMD Evergreen shader ISA <http://developer.amd.com/wordpress/media/2012/10/AMD_Evergreen-Family_Instruction_Set_Architecture.pdf>`__ 7945.. [AMD-RADEON-HD-6000] `AMD Cayman/Trinity shader ISA <http://developer.amd.com/wordpress/media/2012/10/AMD_HD_6900_Series_Instruction_Set_Architecture.pdf>`__ 7946.. [AMD-ROCm] `AMD ROCm Platform <https://rocm-documentation.readthedocs.io>`__ 7947.. [AMD-ROCm-github] `ROCm github <http://github.com/RadeonOpenCompute>`__ 7948.. [CLANG-ATTR] `Attributes in Clang <https://clang.llvm.org/docs/AttributeReference.html>`__ 7949.. [DWARF] `DWARF Debugging Information Format <http://dwarfstd.org/>`__ 7950.. [ELF] `Executable and Linkable Format (ELF) <http://www.sco.com/developers/gabi/>`__ 7951.. [HRF] `Heterogeneous-race-free Memory Models <http://benedictgaster.org/wp-content/uploads/2014/01/asplos269-FINAL.pdf>`__ 7952.. [HSA] `Heterogeneous System Architecture (HSA) Foundation <http://www.hsafoundation.com/>`__ 7953.. [MsgPack] `Message Pack <http://www.msgpack.org/>`__ 7954.. [OpenCL] `The OpenCL Specification Version 2.0 <http://www.khronos.org/registry/cl/specs/opencl-2.0.pdf>`__ 7955.. [SEMVER] `Semantic Versioning <https://semver.org/>`__ 7956.. [YAML] `YAML Ain't Markup Language (YAML™) Version 1.2 <http://www.yaml.org/spec/1.2/spec.html>`__ 7957