1============================= 2User Guide for AMDGPU Backend 3============================= 4 5.. contents:: 6 :local: 7 8.. toctree:: 9 :hidden: 10 11 AMDGPU/AMDGPUAsmGFX7 12 AMDGPU/AMDGPUAsmGFX8 13 AMDGPU/AMDGPUAsmGFX9 14 AMDGPU/AMDGPUAsmGFX900 15 AMDGPU/AMDGPUAsmGFX904 16 AMDGPU/AMDGPUAsmGFX906 17 AMDGPU/AMDGPUAsmGFX908 18 AMDGPU/AMDGPUAsmGFX10 19 AMDGPU/AMDGPUAsmGFX1011 20 AMDGPUModifierSyntax 21 AMDGPUOperandSyntax 22 AMDGPUInstructionSyntax 23 AMDGPUInstructionNotation 24 AMDGPUDwarfExtensionsForHeterogeneousDebugging 25 26Introduction 27============ 28 29The AMDGPU backend provides ISA code generation for AMD GPUs, starting with the 30R600 family up until the current GCN families. It lives in the 31``llvm/lib/Target/AMDGPU`` directory. 32 33LLVM 34==== 35 36.. _amdgpu-target-triples: 37 38Target Triples 39-------------- 40 41Use the ``clang -target <Architecture>-<Vendor>-<OS>-<Environment>`` option to 42specify the target triple: 43 44 .. table:: AMDGPU Architectures 45 :name: amdgpu-architecture-table 46 47 ============ ============================================================== 48 Architecture Description 49 ============ ============================================================== 50 ``r600`` AMD GPUs HD2XXX-HD6XXX for graphics and compute shaders. 51 ``amdgcn`` AMD GPUs GCN GFX6 onwards for graphics and compute shaders. 52 ============ ============================================================== 53 54 .. table:: AMDGPU Vendors 55 :name: amdgpu-vendor-table 56 57 ============ ============================================================== 58 Vendor Description 59 ============ ============================================================== 60 ``amd`` Can be used for all AMD GPU usage. 61 ``mesa3d`` Can be used if the OS is ``mesa3d``. 62 ============ ============================================================== 63 64 .. table:: AMDGPU Operating Systems 65 :name: amdgpu-os-table 66 67 ============== ============================================================ 68 OS Description 69 ============== ============================================================ 70 *<empty>* Defaults to the *unknown* OS. 71 ``amdhsa`` Compute kernels executed on HSA [HSA]_ compatible runtimes 72 such as AMD's ROCm [AMD-ROCm]_. 73 ``amdpal`` Graphic shaders and compute kernels executed on AMD PAL 74 runtime. 75 ``mesa3d`` Graphic shaders and compute kernels executed on Mesa 3D 76 runtime. 77 ============== ============================================================ 78 79 .. table:: AMDGPU Environments 80 :name: amdgpu-environment-table 81 82 ============ ============================================================== 83 Environment Description 84 ============ ============================================================== 85 *<empty>* Default. 86 ============ ============================================================== 87 88.. _amdgpu-processors: 89 90Processors 91---------- 92 93Use the ``clang -mcpu <Processor>`` option to specify the AMDGPU processor. The 94names from both the *Processor* and *Alternative Processor* can be used. 95 96 .. table:: AMDGPU Processors 97 :name: amdgpu-processor-table 98 99 =========== =============== ============ ===== ============================= ======= ====================== 100 Processor Alternative Target dGPU/ Target ROCm Example 101 Processor Triple APU Features Support Products 102 Architecture Supported 103 [Default] 104 =========== =============== ============ ===== ============================= ======= ====================== 105 **Radeon HD 2000/3000 Series (R600)** [AMD-RADEON-HD-2000-3000]_ 106 ----------------------------------------------------------------------------------------------------------- 107 ``r600`` ``r600`` dGPU 108 ``r630`` ``r600`` dGPU 109 ``rs880`` ``r600`` dGPU 110 ``rv670`` ``r600`` dGPU 111 **Radeon HD 4000 Series (R700)** [AMD-RADEON-HD-4000]_ 112 ----------------------------------------------------------------------------------------------------------- 113 ``rv710`` ``r600`` dGPU 114 ``rv730`` ``r600`` dGPU 115 ``rv770`` ``r600`` dGPU 116 **Radeon HD 5000 Series (Evergreen)** [AMD-RADEON-HD-5000]_ 117 ----------------------------------------------------------------------------------------------------------- 118 ``cedar`` ``r600`` dGPU 119 ``cypress`` ``r600`` dGPU 120 ``juniper`` ``r600`` dGPU 121 ``redwood`` ``r600`` dGPU 122 ``sumo`` ``r600`` dGPU 123 **Radeon HD 6000 Series (Northern Islands)** [AMD-RADEON-HD-6000]_ 124 ----------------------------------------------------------------------------------------------------------- 125 ``barts`` ``r600`` dGPU 126 ``caicos`` ``r600`` dGPU 127 ``cayman`` ``r600`` dGPU 128 ``turks`` ``r600`` dGPU 129 **GCN GFX6 (Southern Islands (SI))** [AMD-GCN-GFX6]_ 130 ----------------------------------------------------------------------------------------------------------- 131 ``gfx600`` - ``tahiti`` ``amdgcn`` dGPU 132 ``gfx601`` - ``pitcairn`` ``amdgcn`` dGPU 133 - ``verde`` 134 ``gfx602`` - ``hainan`` ``amdgcn`` dGPU 135 - ``oland`` 136 **GCN GFX7 (Sea Islands (CI))** [AMD-GCN-GFX7]_ 137 ----------------------------------------------------------------------------------------------------------- 138 ``gfx700`` - ``kaveri`` ``amdgcn`` APU - A6-7000 139 - A6 Pro-7050B 140 - A8-7100 141 - A8 Pro-7150B 142 - A10-7300 143 - A10 Pro-7350B 144 - FX-7500 145 - A8-7200P 146 - A10-7400P 147 - FX-7600P 148 ``gfx701`` - ``hawaii`` ``amdgcn`` dGPU ROCm - FirePro W8100 149 - FirePro W9100 150 - FirePro S9150 151 - FirePro S9170 152 ``gfx702`` ``amdgcn`` dGPU ROCm - Radeon R9 290 153 - Radeon R9 290x 154 - Radeon R390 155 - Radeon R390x 156 ``gfx703`` - ``kabini`` ``amdgcn`` APU - E1-2100 157 - ``mullins`` - E1-2200 158 - E1-2500 159 - E2-3000 160 - E2-3800 161 - A4-5000 162 - A4-5100 163 - A6-5200 164 - A4 Pro-3340B 165 ``gfx704`` - ``bonaire`` ``amdgcn`` dGPU - Radeon HD 7790 166 - Radeon HD 8770 167 - R7 260 168 - R7 260X 169 ``gfx705`` ``amdgcn`` APU *TBA* 170 171 .. TODO:: 172 173 Add product 174 names. 175 176 **GCN GFX8 (Volcanic Islands (VI))** [AMD-GCN-GFX8]_ 177 ----------------------------------------------------------------------------------------------------------- 178 ``gfx801`` - ``carrizo`` ``amdgcn`` APU - xnack - A6-8500P 179 [on] - Pro A6-8500B 180 - A8-8600P 181 - Pro A8-8600B 182 - FX-8800P 183 - Pro A12-8800B 184 \ ``amdgcn`` APU - xnack ROCm - A10-8700P 185 [on] - Pro A10-8700B 186 - A10-8780P 187 \ ``amdgcn`` APU - xnack - A10-9600P 188 [on] - A10-9630P 189 - A12-9700P 190 - A12-9730P 191 - FX-9800P 192 - FX-9830P 193 \ ``amdgcn`` APU - xnack - E2-9010 194 [on] - A6-9210 195 - A9-9410 196 ``gfx802`` - ``iceland`` ``amdgcn`` dGPU - xnack ROCm - Radeon R285 197 - ``tonga`` [off] - Radeon R9 380 198 - Radeon R9 385 199 ``gfx803`` - ``fiji`` ``amdgcn`` dGPU - xnack ROCm - Radeon R9 Nano 200 [off] - Radeon R9 Fury 201 - Radeon R9 FuryX 202 - Radeon Pro Duo 203 - FirePro S9300x2 204 - Radeon Instinct MI8 205 \ - ``polaris10`` ``amdgcn`` dGPU - xnack ROCm - Radeon RX 470 206 [off] - Radeon RX 480 207 - Radeon Instinct MI6 208 \ - ``polaris11`` ``amdgcn`` dGPU - xnack ROCm - Radeon RX 460 209 [off] 210 ``gfx805`` - ``tongapro`` ``amdgcn`` dGPU - xnack ROCm - FirePro S7150 211 [off] - FirePro S7100 212 - FirePro W7100 213 - Mobile FirePro 214 M7170 215 ``gfx810`` - ``stoney`` ``amdgcn`` APU - xnack *TBA* 216 [on] 217 .. TODO:: 218 219 Add product 220 names. 221 222 **GCN GFX9** [AMD-GCN-GFX9]_ 223 ----------------------------------------------------------------------------------------------------------- 224 ``gfx900`` ``amdgcn`` dGPU - xnack ROCm - Radeon Vega 225 [off] Frontier Edition 226 - Radeon RX Vega 56 227 - Radeon RX Vega 64 228 - Radeon RX Vega 64 229 Liquid 230 - Radeon Instinct MI25 231 ``gfx902`` ``amdgcn`` APU - xnack - Ryzen 3 2200G 232 [on] - Ryzen 5 2400G 233 ``gfx904`` ``amdgcn`` dGPU - xnack *TBA* 234 [off] 235 .. TODO:: 236 237 Add product 238 names. 239 240 ``gfx906`` ``amdgcn`` dGPU - xnack - Radeon Instinct MI50 241 [off] - Radeon Instinct MI60 242 - sram-ecc - Radeon VII 243 [off] - Radeon Pro VII 244 ``gfx908`` ``amdgcn`` dGPU - xnack *TBA* 245 [off] 246 - sram-ecc 247 [on] 248 .. TODO:: 249 250 Add product 251 names. 252 253 ``gfx909`` ``amdgcn`` APU - xnack *TBA* 254 [off] 255 .. TODO:: 256 257 Add product 258 names. 259 260 **GCN GFX10** [AMD-GCN-GFX10]_ 261 ----------------------------------------------------------------------------------------------------------- 262 ``gfx1010`` ``amdgcn`` dGPU - xnack - Radeon RX 5700 263 [off] - Radeon RX 5700 XT 264 - wavefrontsize64 - Radeon Pro 5600 XT 265 [off] - Radeon Pro 5600M 266 - cumode 267 [off] 268 ``gfx1011`` ``amdgcn`` dGPU - xnack *TBA* 269 [off] 270 - wavefrontsize64 271 [off] 272 - cumode 273 [off] 274 .. TODO:: 275 276 Add product 277 names. 278 279 ``gfx1012`` ``amdgcn`` dGPU - xnack - Radeon RX 5500 280 [off] - Radeon RX 5500 XT 281 - wavefrontsize64 282 [off] 283 - cumode 284 [off] 285 ``gfx1030`` ``amdgcn`` dGPU - wavefrontsize64 *TBA* 286 [off] 287 - cumode 288 [off] 289 .. TODO:: 290 291 Add product 292 names. 293 294 ``gfx1031`` ``amdgcn`` dGPU - wavefrontsize64 *TBA* 295 [off] 296 - cumode 297 [off] 298 .. TODO:: 299 300 Add product 301 names. 302 303 ``gfx1032`` ``amdgcn`` dGPU - wavefrontsize64 *TBA* 304 [off] 305 - cumode 306 [off] 307 .. TODO:: 308 309 Add product 310 names. 311 312 =========== =============== ============ ===== ============================= ======= ====================== 313 314.. _amdgpu-target-features: 315 316Target Features 317--------------- 318 319Target features control how code is generated to support certain 320processor specific features. Not all target features are supported by 321all processors. The runtime must ensure that the features supported by 322the device used to execute the code match the features enabled when 323generating the code. A mismatch of features may result in incorrect 324execution, or a reduction in performance. 325 326The target features supported by each processor, and the default value 327used if not specified explicitly, is listed in 328:ref:`amdgpu-processor-table`. 329 330Use the ``clang -m[no-]<TargetFeature>`` option to specify the AMDGPU 331target features. 332 333For example: 334 335``-mxnack`` 336 Enable the ``xnack`` feature. 337``-mno-xnack`` 338 Disable the ``xnack`` feature. 339 340 .. table:: AMDGPU Target Features 341 :name: amdgpu-target-feature-table 342 343 ====================== ================================================== 344 Target Feature Description 345 ====================== ================================================== 346 -m[no-]xnack Enable/disable generating code that has 347 memory clauses that are compatible with 348 having XNACK replay enabled. 349 350 This is used for demand paging and page 351 migration. If XNACK replay is enabled in 352 the device, then if a page fault occurs 353 the code may execute incorrectly if the 354 ``xnack`` feature is not enabled. Executing 355 code that has the feature enabled on a 356 device that does not have XNACK replay 357 enabled will execute correctly but may 358 be less performant than code with the 359 feature disabled. 360 361 -m[no-]sram-ecc Enable/disable generating code that assumes SRAM 362 ECC is enabled/disabled. 363 364 -m[no-]wavefrontsize64 Control the default wavefront size used when 365 generating code for kernels. When disabled 366 native wavefront size 32 is used, when enabled 367 wavefront size 64 is used. 368 369 -m[no-]cumode Control the default wavefront execution mode used 370 when generating code for kernels. When disabled 371 native WGP wavefront execution mode is used, 372 when enabled CU wavefront execution mode is used 373 (see :ref:`amdgpu-amdhsa-memory-model`). 374 ====================== ================================================== 375 376.. _amdgpu-address-spaces: 377 378Address Spaces 379-------------- 380 381The AMDGPU architecture supports a number of memory address spaces. The address 382space names use the OpenCL standard names, with some additions. 383 384The AMDGPU address spaces correspond to target architecture specific LLVM 385address space numbers used in LLVM IR. 386 387The AMDGPU address spaces are described in 388:ref:`amdgpu-address-spaces-table`. Only 64-bit process address spaces are 389supported for the ``amdgcn`` target. 390 391 .. table:: AMDGPU Address Spaces 392 :name: amdgpu-address-spaces-table 393 394 ================================= =============== =========== ================ ======= ============================ 395 .. 64-Bit Process Address Space 396 --------------------------------- --------------- ----------- ---------------- ------------------------------------ 397 Address Space Name LLVM IR Address HSA Segment Hardware Address NULL Value 398 Space Number Name Name Size 399 ================================= =============== =========== ================ ======= ============================ 400 Generic 0 flat flat 64 0x0000000000000000 401 Global 1 global global 64 0x0000000000000000 402 Region 2 N/A GDS 32 *not implemented for AMDHSA* 403 Local 3 group LDS 32 0xFFFFFFFF 404 Constant 4 constant *same as global* 64 0x0000000000000000 405 Private 5 private scratch 32 0xFFFFFFFF 406 Constant 32-bit 6 *TODO* 0x00000000 407 Buffer Fat Pointer (experimental) 7 *TODO* 408 ================================= =============== =========== ================ ======= ============================ 409 410**Generic** 411 The generic address space uses the hardware flat address support available in 412 GFX7-GFX10. This uses two fixed ranges of virtual addresses (the private and 413 local apertures), that are outside the range of addressable global memory, to 414 map from a flat address to a private or local address. 415 416 FLAT instructions can take a flat address and access global, private 417 (scratch), and group (LDS) memory depending on if the address is within one 418 of the aperture ranges. Flat access to scratch requires hardware aperture 419 setup and setup in the kernel prologue (see 420 :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`). Flat access to LDS requires 421 hardware aperture setup and M0 (GFX7-GFX8) register setup (see 422 :ref:`amdgpu-amdhsa-kernel-prolog-m0`). 423 424 To convert between a private or group address space address (termed a segment 425 address) and a flat address the base address of the corresponding aperture 426 can be used. For GFX7-GFX8 these are available in the 427 :ref:`amdgpu-amdhsa-hsa-aql-queue` the address of which can be obtained with 428 Queue Ptr SGPR (see :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). For 429 GFX9-GFX10 the aperture base addresses are directly available as inline 430 constant registers ``SRC_SHARED_BASE/LIMIT`` and ``SRC_PRIVATE_BASE/LIMIT``. 431 In 64-bit address mode the aperture sizes are 2^32 bytes and the base is 432 aligned to 2^32 which makes it easier to convert from flat to segment or 433 segment to flat. 434 435 A global address space address has the same value when used as a flat address 436 so no conversion is needed. 437 438**Global and Constant** 439 The global and constant address spaces both use global virtual addresses, 440 which are the same virtual address space used by the CPU. However, some 441 virtual addresses may only be accessible to the CPU, some only accessible 442 by the GPU, and some by both. 443 444 Using the constant address space indicates that the data will not change 445 during the execution of the kernel. This allows scalar read instructions to 446 be used. The vector and scalar L1 caches are invalidated of volatile data 447 before each kernel dispatch execution to allow constant memory to change 448 values between kernel dispatches. 449 450**Region** 451 The region address space uses the hardware Global Data Store (GDS). All 452 wavefronts executing on the same device will access the same memory for any 453 given region address. However, the same region address accessed by wavefronts 454 executing on different devices will access different memory. It is higher 455 performance than global memory. It is allocated by the runtime. The data 456 store (DS) instructions can be used to access it. 457 458**Local** 459 The local address space uses the hardware Local Data Store (LDS) which is 460 automatically allocated when the hardware creates the wavefronts of a 461 work-group, and freed when all the wavefronts of a work-group have 462 terminated. All wavefronts belonging to the same work-group will access the 463 same memory for any given local address. However, the same local address 464 accessed by wavefronts belonging to different work-groups will access 465 different memory. It is higher performance than global memory. The data store 466 (DS) instructions can be used to access it. 467 468**Private** 469 The private address space uses the hardware scratch memory support which 470 automatically allocates memory when it creates a wavefront and frees it when 471 a wavefronts terminates. The memory accessed by a lane of a wavefront for any 472 given private address will be different to the memory accessed by another lane 473 of the same or different wavefront for the same private address. 474 475 If a kernel dispatch uses scratch, then the hardware allocates memory from a 476 pool of backing memory allocated by the runtime for each wavefront. The lanes 477 of the wavefront access this using dword (4 byte) interleaving. The mapping 478 used from private address to backing memory address is: 479 480 ``wavefront-scratch-base + 481 ((private-address / 4) * wavefront-size * 4) + 482 (wavefront-lane-id * 4) + (private-address % 4)`` 483 484 If each lane of a wavefront accesses the same private address, the 485 interleaving results in adjacent dwords being accessed and hence requires 486 fewer cache lines to be fetched. 487 488 There are different ways that the wavefront scratch base address is 489 determined by a wavefront (see 490 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 491 492 Scratch memory can be accessed in an interleaved manner using buffer 493 instructions with the scratch buffer descriptor and per wavefront scratch 494 offset, by the scratch instructions, or by flat instructions. Multi-dword 495 access is not supported except by flat and scratch instructions in 496 GFX9-GFX10. 497 498**Constant 32-bit** 499 *TODO* 500 501**Buffer Fat Pointer** 502 The buffer fat pointer is an experimental address space that is currently 503 unsupported in the backend. It exposes a non-integral pointer that is in 504 the future intended to support the modelling of 128-bit buffer descriptors 505 plus a 32-bit offset into the buffer (in total encapsulating a 160-bit 506 *pointer*), allowing normal LLVM load/store/atomic operations to be used to 507 model the buffer descriptors used heavily in graphics workloads targeting 508 the backend. 509 510.. _amdgpu-memory-scopes: 511 512Memory Scopes 513------------- 514 515This section provides LLVM memory synchronization scopes supported by the AMDGPU 516backend memory model when the target triple OS is ``amdhsa`` (see 517:ref:`amdgpu-amdhsa-memory-model` and :ref:`amdgpu-target-triples`). 518 519The memory model supported is based on the HSA memory model [HSA]_ which is 520based in turn on HRF-indirect with scope inclusion [HRF]_. The happens-before 521relation is transitive over the synchronizes-with relation independent of scope 522and synchronizes-with allows the memory scope instances to be inclusive (see 523table :ref:`amdgpu-amdhsa-llvm-sync-scopes-table`). 524 525This is different to the OpenCL [OpenCL]_ memory model which does not have scope 526inclusion and requires the memory scopes to exactly match. However, this 527is conservatively correct for OpenCL. 528 529 .. table:: AMDHSA LLVM Sync Scopes 530 :name: amdgpu-amdhsa-llvm-sync-scopes-table 531 532 ======================= =================================================== 533 LLVM Sync Scope Description 534 ======================= =================================================== 535 *none* The default: ``system``. 536 537 Synchronizes with, and participates in modification 538 and seq_cst total orderings with, other operations 539 (except image operations) for all address spaces 540 (except private, or generic that accesses private) 541 provided the other operation's sync scope is: 542 543 - ``system``. 544 - ``agent`` and executed by a thread on the same 545 agent. 546 - ``workgroup`` and executed by a thread in the 547 same work-group. 548 - ``wavefront`` and executed by a thread in the 549 same wavefront. 550 551 ``agent`` Synchronizes with, and participates in modification 552 and seq_cst total orderings with, other operations 553 (except image operations) for all address spaces 554 (except private, or generic that accesses private) 555 provided the other operation's sync scope is: 556 557 - ``system`` or ``agent`` and executed by a thread 558 on the same agent. 559 - ``workgroup`` and executed by a thread in the 560 same work-group. 561 - ``wavefront`` and executed by a thread in the 562 same wavefront. 563 564 ``workgroup`` Synchronizes with, and participates in modification 565 and seq_cst total orderings with, other operations 566 (except image operations) for all address spaces 567 (except private, or generic that accesses private) 568 provided the other operation's sync scope is: 569 570 - ``system``, ``agent`` or ``workgroup`` and 571 executed by a thread in the same work-group. 572 - ``wavefront`` and executed by a thread in the 573 same wavefront. 574 575 ``wavefront`` Synchronizes with, and participates in modification 576 and seq_cst total orderings with, other operations 577 (except image operations) for all address spaces 578 (except private, or generic that accesses private) 579 provided the other operation's sync scope is: 580 581 - ``system``, ``agent``, ``workgroup`` or 582 ``wavefront`` and executed by a thread in the 583 same wavefront. 584 585 ``singlethread`` Only synchronizes with and participates in 586 modification and seq_cst total orderings with, 587 other operations (except image operations) running 588 in the same thread for all address spaces (for 589 example, in signal handlers). 590 591 ``one-as`` Same as ``system`` but only synchronizes with other 592 operations within the same address space. 593 594 ``agent-one-as`` Same as ``agent`` but only synchronizes with other 595 operations within the same address space. 596 597 ``workgroup-one-as`` Same as ``workgroup`` but only synchronizes with 598 other operations within the same address space. 599 600 ``wavefront-one-as`` Same as ``wavefront`` but only synchronizes with 601 other operations within the same address space. 602 603 ``singlethread-one-as`` Same as ``singlethread`` but only synchronizes with 604 other operations within the same address space. 605 ======================= =================================================== 606 607LLVM IR Intrinsics 608------------------ 609 610The AMDGPU backend implements the following LLVM IR intrinsics. 611 612*This section is WIP.* 613 614.. TODO:: 615 616 List AMDGPU intrinsics. 617 618LLVM IR Attributes 619------------------ 620 621The AMDGPU backend supports the following LLVM IR attributes. 622 623 .. table:: AMDGPU LLVM IR Attributes 624 :name: amdgpu-llvm-ir-attributes-table 625 626 ======================================= ========================================================== 627 LLVM Attribute Description 628 ======================================= ========================================================== 629 "amdgpu-flat-work-group-size"="min,max" Specify the minimum and maximum flat work group sizes that 630 will be specified when the kernel is dispatched. Generated 631 by the ``amdgpu_flat_work_group_size`` CLANG attribute [CLANG-ATTR]_. 632 "amdgpu-implicitarg-num-bytes"="n" Number of kernel argument bytes to add to the kernel 633 argument block size for the implicit arguments. This 634 varies by OS and language (for OpenCL see 635 :ref:`opencl-kernel-implicit-arguments-appended-for-amdhsa-os-table`). 636 "amdgpu-num-sgpr"="n" Specifies the number of SGPRs to use. Generated by 637 the ``amdgpu_num_sgpr`` CLANG attribute [CLANG-ATTR]_. 638 "amdgpu-num-vgpr"="n" Specifies the number of VGPRs to use. Generated by the 639 ``amdgpu_num_vgpr`` CLANG attribute [CLANG-ATTR]_. 640 "amdgpu-waves-per-eu"="m,n" Specify the minimum and maximum number of waves per 641 execution unit. Generated by the ``amdgpu_waves_per_eu`` 642 CLANG attribute [CLANG-ATTR]_. 643 "amdgpu-ieee" true/false. Specify whether the function expects the IEEE field of the 644 mode register to be set on entry. Overrides the default for 645 the calling convention. 646 "amdgpu-dx10-clamp" true/false. Specify whether the function expects the DX10_CLAMP field of 647 the mode register to be set on entry. Overrides the default 648 for the calling convention. 649 ======================================= ========================================================== 650 651.. _amdgpu-elf-code-object: 652 653ELF Code Object 654=============== 655 656The AMDGPU backend generates a standard ELF [ELF]_ relocatable code object that 657can be linked by ``lld`` to produce a standard ELF shared code object which can 658be loaded and executed on an AMDGPU target. 659 660.. _amdgpu-elf-header: 661 662Header 663------ 664 665The AMDGPU backend uses the following ELF header: 666 667 .. table:: AMDGPU ELF Header 668 :name: amdgpu-elf-header-table 669 670 ========================== =============================== 671 Field Value 672 ========================== =============================== 673 ``e_ident[EI_CLASS]`` ``ELFCLASS64`` 674 ``e_ident[EI_DATA]`` ``ELFDATA2LSB`` 675 ``e_ident[EI_OSABI]`` - ``ELFOSABI_NONE`` 676 - ``ELFOSABI_AMDGPU_HSA`` 677 - ``ELFOSABI_AMDGPU_PAL`` 678 - ``ELFOSABI_AMDGPU_MESA3D`` 679 ``e_ident[EI_ABIVERSION]`` - ``ELFABIVERSION_AMDGPU_HSA`` 680 - ``ELFABIVERSION_AMDGPU_PAL`` 681 - ``ELFABIVERSION_AMDGPU_MESA3D`` 682 ``e_type`` - ``ET_REL`` 683 - ``ET_DYN`` 684 ``e_machine`` ``EM_AMDGPU`` 685 ``e_entry`` 0 686 ``e_flags`` See :ref:`amdgpu-elf-header-e_flags-table` 687 ========================== =============================== 688 689.. 690 691 .. table:: AMDGPU ELF Header Enumeration Values 692 :name: amdgpu-elf-header-enumeration-values-table 693 694 =============================== ===== 695 Name Value 696 =============================== ===== 697 ``EM_AMDGPU`` 224 698 ``ELFOSABI_NONE`` 0 699 ``ELFOSABI_AMDGPU_HSA`` 64 700 ``ELFOSABI_AMDGPU_PAL`` 65 701 ``ELFOSABI_AMDGPU_MESA3D`` 66 702 ``ELFABIVERSION_AMDGPU_HSA`` 1 703 ``ELFABIVERSION_AMDGPU_PAL`` 0 704 ``ELFABIVERSION_AMDGPU_MESA3D`` 0 705 =============================== ===== 706 707``e_ident[EI_CLASS]`` 708 The ELF class is: 709 710 * ``ELFCLASS32`` for ``r600`` architecture. 711 712 * ``ELFCLASS64`` for ``amdgcn`` architecture which only supports 64-bit 713 process address space applications. 714 715``e_ident[EI_DATA]`` 716 All AMDGPU targets use ``ELFDATA2LSB`` for little-endian byte ordering. 717 718``e_ident[EI_OSABI]`` 719 One of the following AMDGPU target architecture specific OS ABIs 720 (see :ref:`amdgpu-os-table`): 721 722 * ``ELFOSABI_NONE`` for *unknown* OS. 723 724 * ``ELFOSABI_AMDGPU_HSA`` for ``amdhsa`` OS. 725 726 * ``ELFOSABI_AMDGPU_PAL`` for ``amdpal`` OS. 727 728 * ``ELFOSABI_AMDGPU_MESA3D`` for ``mesa3D`` OS. 729 730``e_ident[EI_ABIVERSION]`` 731 The ABI version of the AMDGPU target architecture specific OS ABI to which the code 732 object conforms: 733 734 * ``ELFABIVERSION_AMDGPU_HSA`` is used to specify the version of AMD HSA 735 runtime ABI. 736 737 * ``ELFABIVERSION_AMDGPU_PAL`` is used to specify the version of AMD PAL 738 runtime ABI. 739 740 * ``ELFABIVERSION_AMDGPU_MESA3D`` is used to specify the version of AMD MESA 741 3D runtime ABI. 742 743``e_type`` 744 Can be one of the following values: 745 746 747 ``ET_REL`` 748 The type produced by the AMDGPU backend compiler as it is relocatable code 749 object. 750 751 ``ET_DYN`` 752 The type produced by the linker as it is a shared code object. 753 754 The AMD HSA runtime loader requires a ``ET_DYN`` code object. 755 756``e_machine`` 757 The value ``EM_AMDGPU`` is used for the machine for all processors supported 758 by the ``r600`` and ``amdgcn`` architectures (see 759 :ref:`amdgpu-processor-table`). The specific processor is specified in the 760 ``EF_AMDGPU_MACH`` bit field of the ``e_flags`` (see 761 :ref:`amdgpu-elf-header-e_flags-table`). 762 763``e_entry`` 764 The entry point is 0 as the entry points for individual kernels must be 765 selected in order to invoke them through AQL packets. 766 767``e_flags`` 768 The AMDGPU backend uses the following ELF header flags: 769 770 .. table:: AMDGPU ELF Header ``e_flags`` 771 :name: amdgpu-elf-header-e_flags-table 772 773 ================================= ========== ============================= 774 Name Value Description 775 ================================= ========== ============================= 776 **AMDGPU Processor Flag** See :ref:`amdgpu-processor-table`. 777 -------------------------------------------- ----------------------------- 778 ``EF_AMDGPU_MACH`` 0x000000ff AMDGPU processor selection 779 mask for 780 ``EF_AMDGPU_MACH_xxx`` values 781 defined in 782 :ref:`amdgpu-ef-amdgpu-mach-table`. 783 ``EF_AMDGPU_XNACK`` 0x00000100 Indicates if the ``xnack`` 784 target feature is 785 enabled for all code 786 contained in the code object. 787 If the processor 788 does not support the 789 ``xnack`` target 790 feature then must 791 be 0. 792 See 793 :ref:`amdgpu-target-features`. 794 ``EF_AMDGPU_SRAM_ECC`` 0x00000200 Indicates if the ``sram-ecc`` 795 target feature is 796 enabled for all code 797 contained in the code object. 798 If the processor 799 does not support the 800 ``sram-ecc`` target 801 feature then must 802 be 0. 803 See 804 :ref:`amdgpu-target-features`. 805 ================================= ========== ============================= 806 807 .. table:: AMDGPU ``EF_AMDGPU_MACH`` Values 808 :name: amdgpu-ef-amdgpu-mach-table 809 810 ==================================== ========== ============================= 811 Name Value Description (see 812 :ref:`amdgpu-processor-table`) 813 ==================================== ========== ============================= 814 ``EF_AMDGPU_MACH_NONE`` 0x000 *not specified* 815 ``EF_AMDGPU_MACH_R600_R600`` 0x001 ``r600`` 816 ``EF_AMDGPU_MACH_R600_R630`` 0x002 ``r630`` 817 ``EF_AMDGPU_MACH_R600_RS880`` 0x003 ``rs880`` 818 ``EF_AMDGPU_MACH_R600_RV670`` 0x004 ``rv670`` 819 ``EF_AMDGPU_MACH_R600_RV710`` 0x005 ``rv710`` 820 ``EF_AMDGPU_MACH_R600_RV730`` 0x006 ``rv730`` 821 ``EF_AMDGPU_MACH_R600_RV770`` 0x007 ``rv770`` 822 ``EF_AMDGPU_MACH_R600_CEDAR`` 0x008 ``cedar`` 823 ``EF_AMDGPU_MACH_R600_CYPRESS`` 0x009 ``cypress`` 824 ``EF_AMDGPU_MACH_R600_JUNIPER`` 0x00a ``juniper`` 825 ``EF_AMDGPU_MACH_R600_REDWOOD`` 0x00b ``redwood`` 826 ``EF_AMDGPU_MACH_R600_SUMO`` 0x00c ``sumo`` 827 ``EF_AMDGPU_MACH_R600_BARTS`` 0x00d ``barts`` 828 ``EF_AMDGPU_MACH_R600_CAICOS`` 0x00e ``caicos`` 829 ``EF_AMDGPU_MACH_R600_CAYMAN`` 0x00f ``cayman`` 830 ``EF_AMDGPU_MACH_R600_TURKS`` 0x010 ``turks`` 831 *reserved* 0x011 - Reserved for ``r600`` 832 0x01f architecture processors. 833 ``EF_AMDGPU_MACH_AMDGCN_GFX600`` 0x020 ``gfx600`` 834 ``EF_AMDGPU_MACH_AMDGCN_GFX601`` 0x021 ``gfx601`` 835 ``EF_AMDGPU_MACH_AMDGCN_GFX700`` 0x022 ``gfx700`` 836 ``EF_AMDGPU_MACH_AMDGCN_GFX701`` 0x023 ``gfx701`` 837 ``EF_AMDGPU_MACH_AMDGCN_GFX702`` 0x024 ``gfx702`` 838 ``EF_AMDGPU_MACH_AMDGCN_GFX703`` 0x025 ``gfx703`` 839 ``EF_AMDGPU_MACH_AMDGCN_GFX704`` 0x026 ``gfx704`` 840 *reserved* 0x027 Reserved. 841 ``EF_AMDGPU_MACH_AMDGCN_GFX801`` 0x028 ``gfx801`` 842 ``EF_AMDGPU_MACH_AMDGCN_GFX802`` 0x029 ``gfx802`` 843 ``EF_AMDGPU_MACH_AMDGCN_GFX803`` 0x02a ``gfx803`` 844 ``EF_AMDGPU_MACH_AMDGCN_GFX810`` 0x02b ``gfx810`` 845 ``EF_AMDGPU_MACH_AMDGCN_GFX900`` 0x02c ``gfx900`` 846 ``EF_AMDGPU_MACH_AMDGCN_GFX902`` 0x02d ``gfx902`` 847 ``EF_AMDGPU_MACH_AMDGCN_GFX904`` 0x02e ``gfx904`` 848 ``EF_AMDGPU_MACH_AMDGCN_GFX906`` 0x02f ``gfx906`` 849 ``EF_AMDGPU_MACH_AMDGCN_GFX908`` 0x030 ``gfx908`` 850 ``EF_AMDGPU_MACH_AMDGCN_GFX909`` 0x031 ``gfx909`` 851 *reserved* 0x032 Reserved. 852 ``EF_AMDGPU_MACH_AMDGCN_GFX1010`` 0x033 ``gfx1010`` 853 ``EF_AMDGPU_MACH_AMDGCN_GFX1011`` 0x034 ``gfx1011`` 854 ``EF_AMDGPU_MACH_AMDGCN_GFX1012`` 0x035 ``gfx1012`` 855 ``EF_AMDGPU_MACH_AMDGCN_GFX1030`` 0x036 ``gfx1030`` 856 ``EF_AMDGPU_MACH_AMDGCN_GFX1031`` 0x037 ``gfx1031`` 857 ``EF_AMDGPU_MACH_AMDGCN_GFX1032`` 0x038 ``gfx1032`` 858 *reserved* 0x039 Reserved. 859 ``EF_AMDGPU_MACH_AMDGCN_GFX602`` 0x03a ``gfx602`` 860 ``EF_AMDGPU_MACH_AMDGCN_GFX705`` 0x03b ``gfx705`` 861 ``EF_AMDGPU_MACH_AMDGCN_GFX805`` 0x03c ``gfx805`` 862 ==================================== ========== ============================= 863 864Sections 865-------- 866 867An AMDGPU target ELF code object has the standard ELF sections which include: 868 869 .. table:: AMDGPU ELF Sections 870 :name: amdgpu-elf-sections-table 871 872 ================== ================ ================================= 873 Name Type Attributes 874 ================== ================ ================================= 875 ``.bss`` ``SHT_NOBITS`` ``SHF_ALLOC`` + ``SHF_WRITE`` 876 ``.data`` ``SHT_PROGBITS`` ``SHF_ALLOC`` + ``SHF_WRITE`` 877 ``.debug_``\ *\** ``SHT_PROGBITS`` *none* 878 ``.dynamic`` ``SHT_DYNAMIC`` ``SHF_ALLOC`` 879 ``.dynstr`` ``SHT_PROGBITS`` ``SHF_ALLOC`` 880 ``.dynsym`` ``SHT_PROGBITS`` ``SHF_ALLOC`` 881 ``.got`` ``SHT_PROGBITS`` ``SHF_ALLOC`` + ``SHF_WRITE`` 882 ``.hash`` ``SHT_HASH`` ``SHF_ALLOC`` 883 ``.note`` ``SHT_NOTE`` *none* 884 ``.rela``\ *name* ``SHT_RELA`` *none* 885 ``.rela.dyn`` ``SHT_RELA`` *none* 886 ``.rodata`` ``SHT_PROGBITS`` ``SHF_ALLOC`` 887 ``.shstrtab`` ``SHT_STRTAB`` *none* 888 ``.strtab`` ``SHT_STRTAB`` *none* 889 ``.symtab`` ``SHT_SYMTAB`` *none* 890 ``.text`` ``SHT_PROGBITS`` ``SHF_ALLOC`` + ``SHF_EXECINSTR`` 891 ================== ================ ================================= 892 893These sections have their standard meanings (see [ELF]_) and are only generated 894if needed. 895 896``.debug``\ *\** 897 The standard DWARF sections. See :ref:`amdgpu-dwarf-debug-information` for 898 information on the DWARF produced by the AMDGPU backend. 899 900``.dynamic``, ``.dynstr``, ``.dynsym``, ``.hash`` 901 The standard sections used by a dynamic loader. 902 903``.note`` 904 See :ref:`amdgpu-note-records` for the note records supported by the AMDGPU 905 backend. 906 907``.rela``\ *name*, ``.rela.dyn`` 908 For relocatable code objects, *name* is the name of the section that the 909 relocation records apply. For example, ``.rela.text`` is the section name for 910 relocation records associated with the ``.text`` section. 911 912 For linked shared code objects, ``.rela.dyn`` contains all the relocation 913 records from each of the relocatable code object's ``.rela``\ *name* sections. 914 915 See :ref:`amdgpu-relocation-records` for the relocation records supported by 916 the AMDGPU backend. 917 918``.text`` 919 The executable machine code for the kernels and functions they call. Generated 920 as position independent code. See :ref:`amdgpu-code-conventions` for 921 information on conventions used in the isa generation. 922 923.. _amdgpu-note-records: 924 925Note Records 926------------ 927 928The AMDGPU backend code object contains ELF note records in the ``.note`` 929section. The set of generated notes and their semantics depend on the code 930object version; see :ref:`amdgpu-note-records-v2` and 931:ref:`amdgpu-note-records-v3`. 932 933As required by ``ELFCLASS32`` and ``ELFCLASS64``, minimal zero-byte padding 934must be generated after the ``name`` field to ensure the ``desc`` field is 4 935byte aligned. In addition, minimal zero-byte padding must be generated to 936ensure the ``desc`` field size is a multiple of 4 bytes. The ``sh_addralign`` 937field of the ``.note`` section must be at least 4 to indicate at least 8 byte 938alignment. 939 940.. _amdgpu-note-records-v2: 941 942Code Object V2 Note Records (--amdhsa-code-object-version=2) 943~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 944 945.. warning:: Code Object V2 is not the default code object version emitted by 946 this version of LLVM. For a description of the notes generated with the 947 default configuration (Code Object V3) see :ref:`amdgpu-note-records-v3`. 948 949The AMDGPU backend code object uses the following ELF note record in the 950``.note`` section when compiling for Code Object V2 951(--amdhsa-code-object-version=2). 952 953Additional note records may be present, but any which are not documented here 954are deprecated and should not be used. 955 956 .. table:: AMDGPU Code Object V2 ELF Note Records 957 :name: amdgpu-elf-note-records-table-v2 958 959 ===== ============================== ====================================== 960 Name Type Description 961 ===== ============================== ====================================== 962 "AMD" ``NT_AMD_AMDGPU_HSA_METADATA`` <metadata null terminated string> 963 ===== ============================== ====================================== 964 965.. 966 967 .. table:: AMDGPU Code Object V2 ELF Note Record Enumeration Values 968 :name: amdgpu-elf-note-record-enumeration-values-table-v2 969 970 ============================== ===== 971 Name Value 972 ============================== ===== 973 *reserved* 0-9 974 ``NT_AMD_AMDGPU_HSA_METADATA`` 10 975 *reserved* 11 976 ============================== ===== 977 978``NT_AMD_AMDGPU_HSA_METADATA`` 979 Specifies extensible metadata associated with the code objects executed on HSA 980 [HSA]_ compatible runtimes such as AMD's ROCm [AMD-ROCm]_. It is required when 981 the target triple OS is ``amdhsa`` (see :ref:`amdgpu-target-triples`). See 982 :ref:`amdgpu-amdhsa-code-object-metadata-v2` for the syntax of the code 983 object metadata string. 984 985.. _amdgpu-note-records-v3: 986 987Code Object V3 Note Records (--amdhsa-code-object-version=3) 988~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 989 990The AMDGPU backend code object uses the following ELF note record in the 991``.note`` section when compiling for Code Object V3 992(--amdhsa-code-object-version=3). 993 994Additional note records may be present, but any which are not documented here 995are deprecated and should not be used. 996 997 .. table:: AMDGPU Code Object V3 ELF Note Records 998 :name: amdgpu-elf-note-records-table-v3 999 1000 ======== ============================== ====================================== 1001 Name Type Description 1002 ======== ============================== ====================================== 1003 "AMDGPU" ``NT_AMDGPU_METADATA`` Metadata in Message Pack [MsgPack]_ 1004 binary format. 1005 ======== ============================== ====================================== 1006 1007.. 1008 1009 .. table:: AMDGPU Code Object V3 ELF Note Record Enumeration Values 1010 :name: amdgpu-elf-note-record-enumeration-values-table-v3 1011 1012 ============================== ===== 1013 Name Value 1014 ============================== ===== 1015 *reserved* 0-31 1016 ``NT_AMDGPU_METADATA`` 32 1017 ============================== ===== 1018 1019``NT_AMDGPU_METADATA`` 1020 Specifies extensible metadata associated with an AMDGPU code 1021 object. It is encoded as a map in the Message Pack [MsgPack]_ binary 1022 data format. See :ref:`amdgpu-amdhsa-code-object-metadata-v3` for the 1023 map keys defined for the ``amdhsa`` OS. 1024 1025.. _amdgpu-symbols: 1026 1027Symbols 1028------- 1029 1030Symbols include the following: 1031 1032 .. table:: AMDGPU ELF Symbols 1033 :name: amdgpu-elf-symbols-table 1034 1035 ===================== ================== ================ ================== 1036 Name Type Section Description 1037 ===================== ================== ================ ================== 1038 *link-name* ``STT_OBJECT`` - ``.data`` Global variable 1039 - ``.rodata`` 1040 - ``.bss`` 1041 *link-name*\ ``.kd`` ``STT_OBJECT`` - ``.rodata`` Kernel descriptor 1042 *link-name* ``STT_FUNC`` - ``.text`` Kernel entry point 1043 *link-name* ``STT_OBJECT`` - SHN_AMDGPU_LDS Global variable in LDS 1044 ===================== ================== ================ ================== 1045 1046Global variable 1047 Global variables both used and defined by the compilation unit. 1048 1049 If the symbol is defined in the compilation unit then it is allocated in the 1050 appropriate section according to if it has initialized data or is readonly. 1051 1052 If the symbol is external then its section is ``STN_UNDEF`` and the loader 1053 will resolve relocations using the definition provided by another code object 1054 or explicitly defined by the runtime. 1055 1056 If the symbol resides in local/group memory (LDS) then its section is the 1057 special processor specific section name ``SHN_AMDGPU_LDS``, and the 1058 ``st_value`` field describes alignment requirements as it does for common 1059 symbols. 1060 1061 .. TODO:: 1062 1063 Add description of linked shared object symbols. Seems undefined symbols 1064 are marked as STT_NOTYPE. 1065 1066Kernel descriptor 1067 Every HSA kernel has an associated kernel descriptor. It is the address of the 1068 kernel descriptor that is used in the AQL dispatch packet used to invoke the 1069 kernel, not the kernel entry point. The layout of the HSA kernel descriptor is 1070 defined in :ref:`amdgpu-amdhsa-kernel-descriptor`. 1071 1072Kernel entry point 1073 Every HSA kernel also has a symbol for its machine code entry point. 1074 1075.. _amdgpu-relocation-records: 1076 1077Relocation Records 1078------------------ 1079 1080AMDGPU backend generates ``Elf64_Rela`` relocation records. Supported 1081relocatable fields are: 1082 1083``word32`` 1084 This specifies a 32-bit field occupying 4 bytes with arbitrary byte 1085 alignment. These values use the same byte order as other word values in the 1086 AMDGPU architecture. 1087 1088``word64`` 1089 This specifies a 64-bit field occupying 8 bytes with arbitrary byte 1090 alignment. These values use the same byte order as other word values in the 1091 AMDGPU architecture. 1092 1093Following notations are used for specifying relocation calculations: 1094 1095**A** 1096 Represents the addend used to compute the value of the relocatable field. 1097 1098**G** 1099 Represents the offset into the global offset table at which the relocation 1100 entry's symbol will reside during execution. 1101 1102**GOT** 1103 Represents the address of the global offset table. 1104 1105**P** 1106 Represents the place (section offset for ``et_rel`` or address for ``et_dyn``) 1107 of the storage unit being relocated (computed using ``r_offset``). 1108 1109**S** 1110 Represents the value of the symbol whose index resides in the relocation 1111 entry. Relocations not using this must specify a symbol index of 1112 ``STN_UNDEF``. 1113 1114**B** 1115 Represents the base address of a loaded executable or shared object which is 1116 the difference between the ELF address and the actual load address. 1117 Relocations using this are only valid in executable or shared objects. 1118 1119The following relocation types are supported: 1120 1121 .. table:: AMDGPU ELF Relocation Records 1122 :name: amdgpu-elf-relocation-records-table 1123 1124 ========================== ======= ===== ========== ============================== 1125 Relocation Type Kind Value Field Calculation 1126 ========================== ======= ===== ========== ============================== 1127 ``R_AMDGPU_NONE`` 0 *none* *none* 1128 ``R_AMDGPU_ABS32_LO`` Static, 1 ``word32`` (S + A) & 0xFFFFFFFF 1129 Dynamic 1130 ``R_AMDGPU_ABS32_HI`` Static, 2 ``word32`` (S + A) >> 32 1131 Dynamic 1132 ``R_AMDGPU_ABS64`` Static, 3 ``word64`` S + A 1133 Dynamic 1134 ``R_AMDGPU_REL32`` Static 4 ``word32`` S + A - P 1135 ``R_AMDGPU_REL64`` Static 5 ``word64`` S + A - P 1136 ``R_AMDGPU_ABS32`` Static, 6 ``word32`` S + A 1137 Dynamic 1138 ``R_AMDGPU_GOTPCREL`` Static 7 ``word32`` G + GOT + A - P 1139 ``R_AMDGPU_GOTPCREL32_LO`` Static 8 ``word32`` (G + GOT + A - P) & 0xFFFFFFFF 1140 ``R_AMDGPU_GOTPCREL32_HI`` Static 9 ``word32`` (G + GOT + A - P) >> 32 1141 ``R_AMDGPU_REL32_LO`` Static 10 ``word32`` (S + A - P) & 0xFFFFFFFF 1142 ``R_AMDGPU_REL32_HI`` Static 11 ``word32`` (S + A - P) >> 32 1143 *reserved* 12 1144 ``R_AMDGPU_RELATIVE64`` Dynamic 13 ``word64`` B + A 1145 ========================== ======= ===== ========== ============================== 1146 1147``R_AMDGPU_ABS32_LO`` and ``R_AMDGPU_ABS32_HI`` are only supported by 1148the ``mesa3d`` OS, which does not support ``R_AMDGPU_ABS64``. 1149 1150There is no current OS loader support for 32-bit programs and so 1151``R_AMDGPU_ABS32`` is not used. 1152 1153.. _amdgpu-loaded-code-object-path-uniform-resource-identifier: 1154 1155Loaded Code Object Path Uniform Resource Identifier (URI) 1156--------------------------------------------------------- 1157 1158The AMD GPU code object loader represents the path of the ELF shared object from 1159which the code object was loaded as a textual Unifom Resource Identifier (URI). 1160Note that the code object is the in memory loaded relocated form of the ELF 1161shared object. Multiple code objects may be loaded at different memory 1162addresses in the same process from the same ELF shared object. 1163 1164The loaded code object path URI syntax is defined by the following BNF syntax: 1165 1166.. code:: 1167 1168 code_object_uri ::== file_uri | memory_uri 1169 file_uri ::== "file://" file_path [ range_specifier ] 1170 memory_uri ::== "memory://" process_id range_specifier 1171 range_specifier ::== [ "#" | "?" ] "offset=" number "&" "size=" number 1172 file_path ::== URI_ENCODED_OS_FILE_PATH 1173 process_id ::== DECIMAL_NUMBER 1174 number ::== HEX_NUMBER | DECIMAL_NUMBER | OCTAL_NUMBER 1175 1176**number** 1177 Is a C integral literal where hexadecimal values are prefixed by "0x" or "0X", 1178 and octal values by "0". 1179 1180**file_path** 1181 Is the file's path specified as a URI encoded UTF-8 string. In URI encoding, 1182 every character that is not in the regular expression ``[a-zA-Z0-9/_.~-]`` is 1183 encoded as two uppercase hexadecimal digits proceeded by "%". Directories in 1184 the path are separated by "/". 1185 1186**offset** 1187 Is a 0-based byte offset to the start of the code object. For a file URI, it 1188 is from the start of the file specified by the ``file_path``, and if omitted 1189 defaults to 0. For a memory URI, it is the memory address and is required. 1190 1191**size** 1192 Is the number of bytes in the code object. For a file URI, if omitted it 1193 defaults to the size of the file. It is required for a memory URI. 1194 1195**process_id** 1196 Is the identity of the process owning the memory. For Linux it is the C 1197 unsigned integral decimal literal for the process ID (PID). 1198 1199For example: 1200 1201.. code:: 1202 1203 file:///dir1/dir2/file1 1204 file:///dir3/dir4/file2#offset=0x2000&size=3000 1205 memory://1234#offset=0x20000&size=3000 1206 1207.. _amdgpu-dwarf-debug-information: 1208 1209DWARF Debug Information 1210======================= 1211 1212.. warning:: 1213 1214 This section describes **provisional support** for AMDGPU DWARF [DWARF]_ that 1215 is not currently fully implemented and is subject to change. 1216 1217AMDGPU generates DWARF [DWARF]_ debugging information ELF sections (see 1218:ref:`amdgpu-elf-code-object`) which contain information that maps the code 1219object executable code and data to the source language constructs. It can be 1220used by tools such as debuggers and profilers. It uses features defined in 1221:doc:`AMDGPUDwarfExtensionsForHeterogeneousDebugging` that are made available in 1222DWARF Version 4 and DWARF Version 5 as an LLVM vendor extension. 1223 1224This section defines the AMDGPU target architecture specific DWARF mappings. 1225 1226.. _amdgpu-dwarf-register-identifier: 1227 1228Register Identifier 1229------------------- 1230 1231This section defines the AMDGPU target architecture register numbers used in 1232DWARF operation expressions (see DWARF Version 5 section 2.5 and 1233:ref:`amdgpu-dwarf-operation-expressions`) and Call Frame Information 1234instructions (see DWARF Version 5 section 6.4 and 1235:ref:`amdgpu-dwarf-call-frame-information`). 1236 1237A single code object can contain code for kernels that have different wavefront 1238sizes. The vector registers and some scalar registers are based on the wavefront 1239size. AMDGPU defines distinct DWARF registers for each wavefront size. This 1240simplifies the consumer of the DWARF so that each register has a fixed size, 1241rather than being dynamic according to the wavefront size mode. Similarly, 1242distinct DWARF registers are defined for those registers that vary in size 1243according to the process address size. This allows a consumer to treat a 1244specific AMDGPU processor as a single architecture regardless of how it is 1245configured at run time. The compiler explicitly specifies the DWARF registers 1246that match the mode in which the code it is generating will be executed. 1247 1248DWARF registers are encoded as numbers, which are mapped to architecture 1249registers. The mapping for AMDGPU is defined in 1250:ref:`amdgpu-dwarf-register-mapping-table`. All AMDGPU targets use the same 1251mapping. 1252 1253.. table:: AMDGPU DWARF Register Mapping 1254 :name: amdgpu-dwarf-register-mapping-table 1255 1256 ============== ================= ======== ================================== 1257 DWARF Register AMDGPU Register Bit Size Description 1258 ============== ================= ======== ================================== 1259 0 PC_32 32 Program Counter (PC) when 1260 executing in a 32-bit process 1261 address space. Used in the CFI to 1262 describe the PC of the calling 1263 frame. 1264 1 EXEC_MASK_32 32 Execution Mask Register when 1265 executing in wavefront 32 mode. 1266 2-15 *Reserved* *Reserved for highly accessed 1267 registers using DWARF shortcut.* 1268 16 PC_64 64 Program Counter (PC) when 1269 executing in a 64-bit process 1270 address space. Used in the CFI to 1271 describe the PC of the calling 1272 frame. 1273 17 EXEC_MASK_64 64 Execution Mask Register when 1274 executing in wavefront 64 mode. 1275 18-31 *Reserved* *Reserved for highly accessed 1276 registers using DWARF shortcut.* 1277 32-95 SGPR0-SGPR63 32 Scalar General Purpose 1278 Registers. 1279 96-127 *Reserved* *Reserved for frequently accessed 1280 registers using DWARF 1-byte ULEB.* 1281 128 STATUS 32 Status Register. 1282 129-511 *Reserved* *Reserved for future Scalar 1283 Architectural Registers.* 1284 512 VCC_32 32 Vector Condition Code Register 1285 when executing in wavefront 32 1286 mode. 1287 513-1023 *Reserved* *Reserved for future Vector 1288 Architectural Registers when 1289 executing in wavefront 32 mode.* 1290 768 VCC_64 64 Vector Condition Code Register 1291 when executing in wavefront 64 1292 mode. 1293 769-1023 *Reserved* *Reserved for future Vector 1294 Architectural Registers when 1295 executing in wavefront 64 mode.* 1296 1024-1087 *Reserved* *Reserved for padding.* 1297 1088-1129 SGPR64-SGPR105 32 Scalar General Purpose Registers. 1298 1130-1535 *Reserved* *Reserved for future Scalar 1299 General Purpose Registers.* 1300 1536-1791 VGPR0-VGPR255 32*32 Vector General Purpose Registers 1301 when executing in wavefront 32 1302 mode. 1303 1792-2047 *Reserved* *Reserved for future Vector 1304 General Purpose Registers when 1305 executing in wavefront 32 mode.* 1306 2048-2303 AGPR0-AGPR255 32*32 Vector Accumulation Registers 1307 when executing in wavefront 32 1308 mode. 1309 2304-2559 *Reserved* *Reserved for future Vector 1310 Accumulation Registers when 1311 executing in wavefront 32 mode.* 1312 2560-2815 VGPR0-VGPR255 64*32 Vector General Purpose Registers 1313 when executing in wavefront 64 1314 mode. 1315 2816-3071 *Reserved* *Reserved for future Vector 1316 General Purpose Registers when 1317 executing in wavefront 64 mode.* 1318 3072-3327 AGPR0-AGPR255 64*32 Vector Accumulation Registers 1319 when executing in wavefront 64 1320 mode. 1321 3328-3583 *Reserved* *Reserved for future Vector 1322 Accumulation Registers when 1323 executing in wavefront 64 mode.* 1324 ============== ================= ======== ================================== 1325 1326The vector registers are represented as the full size for the wavefront. They 1327are organized as consecutive dwords (32-bits), one per lane, with the dword at 1328the least significant bit position corresponding to lane 0 and so forth. DWARF 1329location expressions involving the ``DW_OP_LLVM_offset`` and 1330``DW_OP_LLVM_push_lane`` operations are used to select the part of the vector 1331register corresponding to the lane that is executing the current thread of 1332execution in languages that are implemented using a SIMD or SIMT execution 1333model. 1334 1335If the wavefront size is 32 lanes then the wavefront 32 mode register 1336definitions are used. If the wavefront size is 64 lanes then the wavefront 64 1337mode register definitions are used. Some AMDGPU targets support executing in 1338both wavefront 32 and wavefront 64 mode. The register definitions corresponding 1339to the wavefront mode of the generated code will be used. 1340 1341If code is generated to execute in a 32-bit process address space, then the 134232-bit process address space register definitions are used. If code is generated 1343to execute in a 64-bit process address space, then the 64-bit process address 1344space register definitions are used. The ``amdgcn`` target only supports the 134564-bit process address space. 1346 1347.. _amdgpu-dwarf-address-class-identifier: 1348 1349Address Class Identifier 1350------------------------ 1351 1352The DWARF address class represents the source language memory space. See DWARF 1353Version 5 section 2.12 which is updated by the *DWARF Extensions For 1354Heterogeneous Debugging* section :ref:`amdgpu-dwarf-segment_addresses`. 1355 1356The DWARF address class mapping used for AMDGPU is defined in 1357:ref:`amdgpu-dwarf-address-class-mapping-table`. 1358 1359.. table:: AMDGPU DWARF Address Class Mapping 1360 :name: amdgpu-dwarf-address-class-mapping-table 1361 1362 ========================= ====== ================= 1363 DWARF AMDGPU 1364 -------------------------------- ----------------- 1365 Address Class Name Value Address Space 1366 ========================= ====== ================= 1367 ``DW_ADDR_none`` 0x0000 Generic (Flat) 1368 ``DW_ADDR_LLVM_global`` 0x0001 Global 1369 ``DW_ADDR_LLVM_constant`` 0x0002 Global 1370 ``DW_ADDR_LLVM_group`` 0x0003 Local (group/LDS) 1371 ``DW_ADDR_LLVM_private`` 0x0004 Private (Scratch) 1372 ``DW_ADDR_AMDGPU_region`` 0x8000 Region (GDS) 1373 ========================= ====== ================= 1374 1375The DWARF address class values defined in the *DWARF Extensions For 1376Heterogeneous Debugging* section :ref:`amdgpu-dwarf-segment_addresses` are used. 1377 1378In addition, ``DW_ADDR_AMDGPU_region`` is encoded as a vendor extension. This is 1379available for use for the AMD extension for access to the hardware GDS memory 1380which is scratchpad memory allocated per device. 1381 1382For AMDGPU if no ``DW_AT_address_class`` attribute is present, then the default 1383address class of ``DW_ADDR_none`` is used. 1384 1385See :ref:`amdgpu-dwarf-address-space-identifier` for information on the AMDGPU 1386mapping of DWARF address classes to DWARF address spaces, including address size 1387and NULL value. 1388 1389.. _amdgpu-dwarf-address-space-identifier: 1390 1391Address Space Identifier 1392------------------------ 1393 1394DWARF address spaces correspond to target architecture specific linear 1395addressable memory areas. See DWARF Version 5 section 2.12 and *DWARF Extensions 1396For Heterogeneous Debugging* section :ref:`amdgpu-dwarf-segment_addresses`. 1397 1398The DWARF address space mapping used for AMDGPU is defined in 1399:ref:`amdgpu-dwarf-address-space-mapping-table`. 1400 1401.. table:: AMDGPU DWARF Address Space Mapping 1402 :name: amdgpu-dwarf-address-space-mapping-table 1403 1404 ======================================= ===== ======= ======== ================= ======================= 1405 DWARF AMDGPU Notes 1406 --------------------------------------- ----- ---------------- ----------------- ----------------------- 1407 Address Space Name Value Address Bit Size Address Space 1408 --------------------------------------- ----- ------- -------- ----------------- ----------------------- 1409 .. 64-bit 32-bit 1410 process process 1411 address address 1412 space space 1413 ======================================= ===== ======= ======== ================= ======================= 1414 ``DW_ASPACE_none`` 0x00 64 32 Global *default address space* 1415 ``DW_ASPACE_AMDGPU_generic`` 0x01 64 32 Generic (Flat) 1416 ``DW_ASPACE_AMDGPU_region`` 0x02 32 32 Region (GDS) 1417 ``DW_ASPACE_AMDGPU_local`` 0x03 32 32 Local (group/LDS) 1418 *Reserved* 0x04 1419 ``DW_ASPACE_AMDGPU_private_lane`` 0x05 32 32 Private (Scratch) *focused lane* 1420 ``DW_ASPACE_AMDGPU_private_wave`` 0x06 32 32 Private (Scratch) *unswizzled wavefront* 1421 ======================================= ===== ======= ======== ================= ======================= 1422 1423See :ref:`amdgpu-address-spaces` for information on the AMDGPU address spaces 1424including address size and NULL value. 1425 1426The ``DW_ASPACE_none`` address space is the default target architecture address 1427space used in DWARF operations that do not specify an address space. It 1428therefore has to map to the global address space so that the ``DW_OP_addr*`` and 1429related operations can refer to addresses in the program code. 1430 1431The ``DW_ASPACE_AMDGPU_generic`` address space allows location expressions to 1432specify the flat address space. If the address corresponds to an address in the 1433local address space, then it corresponds to the wavefront that is executing the 1434focused thread of execution. If the address corresponds to an address in the 1435private address space, then it corresponds to the lane that is executing the 1436focused thread of execution for languages that are implemented using a SIMD or 1437SIMT execution model. 1438 1439.. note:: 1440 1441 CUDA-like languages such as HIP that do not have address spaces in the 1442 language type system, but do allow variables to be allocated in different 1443 address spaces, need to explicitly specify the ``DW_ASPACE_AMDGPU_generic`` 1444 address space in the DWARF expression operations as the default address space 1445 is the global address space. 1446 1447The ``DW_ASPACE_AMDGPU_local`` address space allows location expressions to 1448specify the local address space corresponding to the wavefront that is executing 1449the focused thread of execution. 1450 1451The ``DW_ASPACE_AMDGPU_private_lane`` address space allows location expressions 1452to specify the private address space corresponding to the lane that is executing 1453the focused thread of execution for languages that are implemented using a SIMD 1454or SIMT execution model. 1455 1456The ``DW_ASPACE_AMDGPU_private_wave`` address space allows location expressions 1457to specify the unswizzled private address space corresponding to the wavefront 1458that is executing the focused thread of execution. The wavefront view of private 1459memory is the per wavefront unswizzled backing memory layout defined in 1460:ref:`amdgpu-address-spaces`, such that address 0 corresponds to the first 1461location for the backing memory of the wavefront (namely the address is not 1462offset by ``wavefront-scratch-base``). The following formula can be used to 1463convert from a ``DW_ASPACE_AMDGPU_private_lane`` address to a 1464``DW_ASPACE_AMDGPU_private_wave`` address: 1465 1466:: 1467 1468 private-address-wavefront = 1469 ((private-address-lane / 4) * wavefront-size * 4) + 1470 (wavefront-lane-id * 4) + (private-address-lane % 4) 1471 1472If the ``DW_ASPACE_AMDGPU_private_lane`` address is dword aligned, and the start 1473of the dwords for each lane starting with lane 0 is required, then this 1474simplifies to: 1475 1476:: 1477 1478 private-address-wavefront = 1479 private-address-lane * wavefront-size 1480 1481A compiler can use the ``DW_ASPACE_AMDGPU_private_wave`` address space to read a 1482complete spilled vector register back into a complete vector register in the 1483CFI. The frame pointer can be a private lane address which is dword aligned, 1484which can be shifted to multiply by the wavefront size, and then used to form a 1485private wavefront address that gives a location for a contiguous set of dwords, 1486one per lane, where the vector register dwords are spilled. The compiler knows 1487the wavefront size since it generates the code. Note that the type of the 1488address may have to be converted as the size of a 1489``DW_ASPACE_AMDGPU_private_lane`` address may be smaller than the size of a 1490``DW_ASPACE_AMDGPU_private_wave`` address. 1491 1492.. _amdgpu-dwarf-lane-identifier: 1493 1494Lane identifier 1495--------------- 1496 1497DWARF lane identifies specify a target architecture lane position for hardware 1498that executes in a SIMD or SIMT manner, and on which a source language maps its 1499threads of execution onto those lanes. The DWARF lane identifier is pushed by 1500the ``DW_OP_LLVM_push_lane`` DWARF expression operation. See DWARF Version 5 1501section 2.5 which is updated by *DWARF Extensions For Heterogeneous Debugging* 1502section :ref:`amdgpu-dwarf-operation-expressions`. 1503 1504For AMDGPU, the lane identifier corresponds to the hardware lane ID of a 1505wavefront. It is numbered from 0 to the wavefront size minus 1. 1506 1507Operation Expressions 1508--------------------- 1509 1510DWARF expressions are used to compute program values and the locations of 1511program objects. See DWARF Version 5 section 2.5 and 1512:ref:`amdgpu-dwarf-operation-expressions`. 1513 1514DWARF location descriptions describe how to access storage which includes memory 1515and registers. When accessing storage on AMDGPU, bytes are ordered with least 1516significant bytes first, and bits are ordered within bytes with least 1517significant bits first. 1518 1519For AMDGPU CFI expressions, ``DW_OP_LLVM_select_bit_piece`` is used to describe 1520unwinding vector registers that are spilled under the execution mask to memory: 1521the zero-single location description is the vector register, and the one-single 1522location description is the spilled memory location description. The 1523``DW_OP_LLVM_form_aspace_address`` is used to specify the address space of the 1524memory location description. 1525 1526In AMDGPU expressions, ``DW_OP_LLVM_select_bit_piece`` is used by the 1527``DW_AT_LLVM_lane_pc`` attribute expression where divergent control flow is 1528controlled by the execution mask. An undefined location description together 1529with ``DW_OP_LLVM_extend`` is used to indicate the lane was not active on entry 1530to the subprogram. See :ref:`amdgpu-dwarf-dw-at-llvm-lane-pc` for an example. 1531 1532Debugger Information Entry Attributes 1533------------------------------------- 1534 1535This section describes how certain debugger information entry attributes are 1536used by AMDGPU. See the sections in DWARF Version 5 section 2 which are updated 1537by *DWARF Extensions For Heterogeneous Debugging* section 1538:ref:`amdgpu-dwarf-debugging-information-entry-attributes`. 1539 1540.. _amdgpu-dwarf-dw-at-llvm-lane-pc: 1541 1542``DW_AT_LLVM_lane_pc`` 1543~~~~~~~~~~~~~~~~~~~~~~ 1544 1545For AMDGPU, the ``DW_AT_LLVM_lane_pc`` attribute is used to specify the program 1546location of the separate lanes of a SIMT thread. 1547 1548If the lane is an active lane then this will be the same as the current program 1549location. 1550 1551If the lane is inactive, but was active on entry to the subprogram, then this is 1552the program location in the subprogram at which execution of the lane is 1553conceptual positioned. 1554 1555If the lane was not active on entry to the subprogram, then this will be the 1556undefined location. A client debugger can check if the lane is part of a valid 1557work-group by checking that the lane is in the range of the associated 1558work-group within the grid, accounting for partial work-groups. If it is not, 1559then the debugger can omit any information for the lane. Otherwise, the debugger 1560may repeatedly unwind the stack and inspect the ``DW_AT_LLVM_lane_pc`` of the 1561calling subprogram until it finds a non-undefined location. Conceptually the 1562lane only has the call frames that it has a non-undefined 1563``DW_AT_LLVM_lane_pc``. 1564 1565The following example illustrates how the AMDGPU backend can generate a DWARF 1566location list expression for the nested ``IF/THEN/ELSE`` structures of the 1567following subprogram pseudo code for a target with 64 lanes per wavefront. 1568 1569.. code:: 1570 :number-lines: 1571 1572 SUBPROGRAM X 1573 BEGIN 1574 a; 1575 IF (c1) THEN 1576 b; 1577 IF (c2) THEN 1578 c; 1579 ELSE 1580 d; 1581 ENDIF 1582 e; 1583 ELSE 1584 f; 1585 ENDIF 1586 g; 1587 END 1588 1589The AMDGPU backend may generate the following pseudo LLVM MIR to manipulate the 1590execution mask (``EXEC``) to linearize the control flow. The condition is 1591evaluated to make a mask of the lanes for which the condition evaluates to true. 1592First the ``THEN`` region is executed by setting the ``EXEC`` mask to the 1593logical ``AND`` of the current ``EXEC`` mask with the condition mask. Then the 1594``ELSE`` region is executed by negating the ``EXEC`` mask and logical ``AND`` of 1595the saved ``EXEC`` mask at the start of the region. After the ``IF/THEN/ELSE`` 1596region the ``EXEC`` mask is restored to the value it had at the beginning of the 1597region. This is shown below. Other approaches are possible, but the basic 1598concept is the same. 1599 1600.. code:: 1601 :number-lines: 1602 1603 $lex_start: 1604 a; 1605 %1 = EXEC 1606 %2 = c1 1607 $lex_1_start: 1608 EXEC = %1 & %2 1609 $if_1_then: 1610 b; 1611 %3 = EXEC 1612 %4 = c2 1613 $lex_1_1_start: 1614 EXEC = %3 & %4 1615 $lex_1_1_then: 1616 c; 1617 EXEC = ~EXEC & %3 1618 $lex_1_1_else: 1619 d; 1620 EXEC = %3 1621 $lex_1_1_end: 1622 e; 1623 EXEC = ~EXEC & %1 1624 $lex_1_else: 1625 f; 1626 EXEC = %1 1627 $lex_1_end: 1628 g; 1629 $lex_end: 1630 1631To create the DWARF location list expression that defines the location 1632description of a vector of lane program locations, the LLVM MIR ``DBG_VALUE`` 1633pseudo instruction can be used to annotate the linearized control flow. This can 1634be done by defining an artificial variable for the lane PC. The DWARF location 1635list expression created for it is used as the value of the 1636``DW_AT_LLVM_lane_pc`` attribute on the subprogram's debugger information entry. 1637 1638A DWARF procedure is defined for each well nested structured control flow region 1639which provides the conceptual lane program location for a lane if it is not 1640active (namely it is divergent). The DWARF operation expression for each region 1641conceptually inherits the value of the immediately enclosing region and modifies 1642it according to the semantics of the region. 1643 1644For an ``IF/THEN/ELSE`` region the divergent program location is at the start of 1645the region for the ``THEN`` region since it is executed first. For the ``ELSE`` 1646region the divergent program location is at the end of the ``IF/THEN/ELSE`` 1647region since the ``THEN`` region has completed. 1648 1649The lane PC artificial variable is assigned at each region transition. It uses 1650the immediately enclosing region's DWARF procedure to compute the program 1651location for each lane assuming they are divergent, and then modifies the result 1652by inserting the current program location for each lane that the ``EXEC`` mask 1653indicates is active. 1654 1655By having separate DWARF procedures for each region, they can be reused to 1656define the value for any nested region. This reduces the total size of the DWARF 1657operation expressions. 1658 1659The following provides an example using pseudo LLVM MIR. 1660 1661.. code:: 1662 :number-lines: 1663 1664 $lex_start: 1665 DEFINE_DWARF %__uint_64 = DW_TAG_base_type[ 1666 DW_AT_name = "__uint64"; 1667 DW_AT_byte_size = 8; 1668 DW_AT_encoding = DW_ATE_unsigned; 1669 ]; 1670 DEFINE_DWARF %__active_lane_pc = DW_TAG_dwarf_procedure[ 1671 DW_AT_name = "__active_lane_pc"; 1672 DW_AT_location = [ 1673 DW_OP_regx PC; 1674 DW_OP_LLVM_extend 64, 64; 1675 DW_OP_regval_type EXEC, %uint_64; 1676 DW_OP_LLVM_select_bit_piece 64, 64; 1677 ]; 1678 ]; 1679 DEFINE_DWARF %__divergent_lane_pc = DW_TAG_dwarf_procedure[ 1680 DW_AT_name = "__divergent_lane_pc"; 1681 DW_AT_location = [ 1682 DW_OP_LLVM_undefined; 1683 DW_OP_LLVM_extend 64, 64; 1684 ]; 1685 ]; 1686 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 1687 DW_OP_call_ref %__divergent_lane_pc; 1688 DW_OP_call_ref %__active_lane_pc; 1689 ]; 1690 a; 1691 %1 = EXEC; 1692 DBG_VALUE %1, $noreg, %__lex_1_save_exec; 1693 %2 = c1; 1694 $lex_1_start: 1695 EXEC = %1 & %2; 1696 $lex_1_then: 1697 DEFINE_DWARF %__divergent_lane_pc_1_then = DW_TAG_dwarf_procedure[ 1698 DW_AT_name = "__divergent_lane_pc_1_then"; 1699 DW_AT_location = DIExpression[ 1700 DW_OP_call_ref %__divergent_lane_pc; 1701 DW_OP_addrx &lex_1_start; 1702 DW_OP_stack_value; 1703 DW_OP_LLVM_extend 64, 64; 1704 DW_OP_call_ref %__lex_1_save_exec; 1705 DW_OP_deref_type 64, %__uint_64; 1706 DW_OP_LLVM_select_bit_piece 64, 64; 1707 ]; 1708 ]; 1709 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 1710 DW_OP_call_ref %__divergent_lane_pc_1_then; 1711 DW_OP_call_ref %__active_lane_pc; 1712 ]; 1713 b; 1714 %3 = EXEC; 1715 DBG_VALUE %3, %__lex_1_1_save_exec; 1716 %4 = c2; 1717 $lex_1_1_start: 1718 EXEC = %3 & %4; 1719 $lex_1_1_then: 1720 DEFINE_DWARF %__divergent_lane_pc_1_1_then = DW_TAG_dwarf_procedure[ 1721 DW_AT_name = "__divergent_lane_pc_1_1_then"; 1722 DW_AT_location = DIExpression[ 1723 DW_OP_call_ref %__divergent_lane_pc_1_then; 1724 DW_OP_addrx &lex_1_1_start; 1725 DW_OP_stack_value; 1726 DW_OP_LLVM_extend 64, 64; 1727 DW_OP_call_ref %__lex_1_1_save_exec; 1728 DW_OP_deref_type 64, %__uint_64; 1729 DW_OP_LLVM_select_bit_piece 64, 64; 1730 ]; 1731 ]; 1732 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 1733 DW_OP_call_ref %__divergent_lane_pc_1_1_then; 1734 DW_OP_call_ref %__active_lane_pc; 1735 ]; 1736 c; 1737 EXEC = ~EXEC & %3; 1738 $lex_1_1_else: 1739 DEFINE_DWARF %__divergent_lane_pc_1_1_else = DW_TAG_dwarf_procedure[ 1740 DW_AT_name = "__divergent_lane_pc_1_1_else"; 1741 DW_AT_location = DIExpression[ 1742 DW_OP_call_ref %__divergent_lane_pc_1_then; 1743 DW_OP_addrx &lex_1_1_end; 1744 DW_OP_stack_value; 1745 DW_OP_LLVM_extend 64, 64; 1746 DW_OP_call_ref %__lex_1_1_save_exec; 1747 DW_OP_deref_type 64, %__uint_64; 1748 DW_OP_LLVM_select_bit_piece 64, 64; 1749 ]; 1750 ]; 1751 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 1752 DW_OP_call_ref %__divergent_lane_pc_1_1_else; 1753 DW_OP_call_ref %__active_lane_pc; 1754 ]; 1755 d; 1756 EXEC = %3; 1757 $lex_1_1_end: 1758 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 1759 DW_OP_call_ref %__divergent_lane_pc; 1760 DW_OP_call_ref %__active_lane_pc; 1761 ]; 1762 e; 1763 EXEC = ~EXEC & %1; 1764 $lex_1_else: 1765 DEFINE_DWARF %__divergent_lane_pc_1_else = DW_TAG_dwarf_procedure[ 1766 DW_AT_name = "__divergent_lane_pc_1_else"; 1767 DW_AT_location = DIExpression[ 1768 DW_OP_call_ref %__divergent_lane_pc; 1769 DW_OP_addrx &lex_1_end; 1770 DW_OP_stack_value; 1771 DW_OP_LLVM_extend 64, 64; 1772 DW_OP_call_ref %__lex_1_save_exec; 1773 DW_OP_deref_type 64, %__uint_64; 1774 DW_OP_LLVM_select_bit_piece 64, 64; 1775 ]; 1776 ]; 1777 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 1778 DW_OP_call_ref %__divergent_lane_pc_1_else; 1779 DW_OP_call_ref %__active_lane_pc; 1780 ]; 1781 f; 1782 EXEC = %1; 1783 $lex_1_end: 1784 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc DIExpression[ 1785 DW_OP_call_ref %__divergent_lane_pc; 1786 DW_OP_call_ref %__active_lane_pc; 1787 ]; 1788 g; 1789 $lex_end: 1790 1791The DWARF procedure ``%__active_lane_pc`` is used to update the lane pc elements 1792that are active, with the current program location. 1793 1794Artificial variables %__lex_1_save_exec and %__lex_1_1_save_exec are created for 1795the execution masks saved on entry to a region. Using the ``DBG_VALUE`` pseudo 1796instruction, location list entries will be created that describe where the 1797artificial variables are allocated at any given program location. The compiler 1798may allocate them to registers or spill them to memory. 1799 1800The DWARF procedures for each region use the values of the saved execution mask 1801artificial variables to only update the lanes that are active on entry to the 1802region. All other lanes retain the value of the enclosing region where they were 1803last active. If they were not active on entry to the subprogram, then will have 1804the undefined location description. 1805 1806Other structured control flow regions can be handled similarly. For example, 1807loops would set the divergent program location for the region at the end of the 1808loop. Any lanes active will be in the loop, and any lanes not active must have 1809exited the loop. 1810 1811An ``IF/THEN/ELSEIF/ELSEIF/...`` region can be treated as a nest of 1812``IF/THEN/ELSE`` regions. 1813 1814The DWARF procedures can use the active lane artificial variable described in 1815:ref:`amdgpu-dwarf-amdgpu-dw-at-llvm-active-lane` rather than the actual 1816``EXEC`` mask in order to support whole or quad wavefront mode. 1817 1818.. _amdgpu-dwarf-amdgpu-dw-at-llvm-active-lane: 1819 1820``DW_AT_LLVM_active_lane`` 1821~~~~~~~~~~~~~~~~~~~~~~~~~~ 1822 1823The ``DW_AT_LLVM_active_lane`` attribute on a subprogram debugger information 1824entry is used to specify the lanes that are conceptually active for a SIMT 1825thread. 1826 1827The execution mask may be modified to implement whole or quad wavefront mode 1828operations. For example, all lanes may need to temporarily be made active to 1829execute a whole wavefront operation. Such regions would save the ``EXEC`` mask, 1830update it to enable the necessary lanes, perform the operations, and then 1831restore the ``EXEC`` mask from the saved value. While executing the whole 1832wavefront region, the conceptual execution mask is the saved value, not the 1833``EXEC`` value. 1834 1835This is handled by defining an artificial variable for the active lane mask. The 1836active lane mask artificial variable would be the actual ``EXEC`` mask for 1837normal regions, and the saved execution mask for regions where the mask is 1838temporarily updated. The location list expression created for this artificial 1839variable is used to define the value of the ``DW_AT_LLVM_active_lane`` 1840attribute. 1841 1842``DW_AT_LLVM_augmentation`` 1843~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1844 1845For AMDGPU, the ``DW_AT_LLVM_augmentation`` attribute of a compilation unit 1846debugger information entry has the following value for the augmentation string: 1847 1848:: 1849 1850 [amdgpu:v0.0] 1851 1852The "vX.Y" specifies the major X and minor Y version number of the AMDGPU 1853extensions used in the DWARF of the compilation unit. The version number 1854conforms to [SEMVER]_. 1855 1856Call Frame Information 1857---------------------- 1858 1859DWARF Call Frame Information (CFI) describes how a consumer can virtually 1860*unwind* call frames in a running process or core dump. See DWARF Version 5 1861section 6.4 and :ref:`amdgpu-dwarf-call-frame-information`. 1862 1863For AMDGPU, the Common Information Entry (CIE) fields have the following values: 1864 18651. ``augmentation`` string contains the following null-terminated UTF-8 string: 1866 1867 :: 1868 1869 [amd:v0.0] 1870 1871 The ``vX.Y`` specifies the major X and minor Y version number of the AMDGPU 1872 extensions used in this CIE or to the FDEs that use it. The version number 1873 conforms to [SEMVER]_. 1874 18752. ``address_size`` for the ``Global`` address space is defined in 1876 :ref:`amdgpu-dwarf-address-space-identifier`. 1877 18783. ``segment_selector_size`` is 0 as AMDGPU does not use a segment selector. 1879 18804. ``code_alignment_factor`` is 4 bytes. 1881 1882 .. TODO:: 1883 1884 Add to :ref:`amdgpu-processor-table` table. 1885 18865. ``data_alignment_factor`` is 4 bytes. 1887 1888 .. TODO:: 1889 1890 Add to :ref:`amdgpu-processor-table` table. 1891 18926. ``return_address_register`` is ``PC_32`` for 32-bit processes and ``PC_64`` 1893 for 64-bit processes defined in :ref:`amdgpu-dwarf-register-identifier`. 1894 18957. ``initial_instructions`` Since a subprogram X with fewer registers can be 1896 called from subprogram Y that has more allocated, X will not change any of 1897 the extra registers as it cannot access them. Therefore, the default rule 1898 for all columns is ``same value``. 1899 1900For AMDGPU the register number follows the numbering defined in 1901:ref:`amdgpu-dwarf-register-identifier`. 1902 1903For AMDGPU the instructions are variable size. A consumer can subtract 1 from 1904the return address to get the address of a byte within the call site 1905instructions. See DWARF Version 5 section 6.4.4. 1906 1907Accelerated Access 1908------------------ 1909 1910See DWARF Version 5 section 6.1. 1911 1912Lookup By Name Section Header 1913~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1914 1915See DWARF Version 5 section 6.1.1.4.1 and :ref:`amdgpu-dwarf-lookup-by-name`. 1916 1917For AMDGPU the lookup by name section header table: 1918 1919``augmentation_string_size`` (uword) 1920 1921 Set to the length of the ``augmentation_string`` value which is always a 1922 multiple of 4. 1923 1924``augmentation_string`` (sequence of UTF-8 characters) 1925 1926 Contains the following UTF-8 string null padded to a multiple of 4 bytes: 1927 1928 :: 1929 1930 [amdgpu:v0.0] 1931 1932 The "vX.Y" specifies the major X and minor Y version number of the AMDGPU 1933 extensions used in the DWARF of this index. The version number conforms to 1934 [SEMVER]_. 1935 1936 .. note:: 1937 1938 This is different to the DWARF Version 5 definition that requires the first 1939 4 characters to be the vendor ID. But this is consistent with the other 1940 augmentation strings and does allow multiple vendor contributions. However, 1941 backwards compatibility may be more desirable. 1942 1943Lookup By Address Section Header 1944~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1945 1946See DWARF Version 5 section 6.1.2. 1947 1948For AMDGPU the lookup by address section header table: 1949 1950``address_size`` (ubyte) 1951 1952 Match the address size for the ``Global`` address space defined in 1953 :ref:`amdgpu-dwarf-address-space-identifier`. 1954 1955``segment_selector_size`` (ubyte) 1956 1957 AMDGPU does not use a segment selector so this is 0. The entries in the 1958 ``.debug_aranges`` do not have a segment selector. 1959 1960Line Number Information 1961----------------------- 1962 1963See DWARF Version 5 section 6.2 and :ref:`amdgpu-dwarf-line-number-information`. 1964 1965AMDGPU does not use the ``isa`` state machine registers and always sets it to 0. 1966The instruction set must be obtained from the ELF file header ``e_flags`` field 1967in the ``EF_AMDGPU_MACH`` bit position (see :ref:`ELF Header 1968<amdgpu-elf-header>`). See DWARF Version 5 section 6.2.2. 1969 1970.. TODO:: 1971 1972 Should the ``isa`` state machine register be used to indicate if the code is 1973 in wavefront32 or wavefront64 mode? Or used to specify the architecture ISA? 1974 1975For AMDGPU the line number program header fields have the following values (see 1976DWARF Version 5 section 6.2.4): 1977 1978``address_size`` (ubyte) 1979 Matches the address size for the ``Global`` address space defined in 1980 :ref:`amdgpu-dwarf-address-space-identifier`. 1981 1982``segment_selector_size`` (ubyte) 1983 AMDGPU does not use a segment selector so this is 0. 1984 1985``minimum_instruction_length`` (ubyte) 1986 For GFX9-GFX10 this is 4. 1987 1988``maximum_operations_per_instruction`` (ubyte) 1989 For GFX9-GFX10 this is 1. 1990 1991Source text for online-compiled programs (for example, those compiled by the 1992OpenCL language runtime) may be embedded into the DWARF Version 5 line table. 1993See DWARF Version 5 section 6.2.4.1 which is updated by *DWARF Extensions For 1994Heterogeneous Debugging* section :ref:`DW_LNCT_LLVM_source 1995<amdgpu-dwarf-line-number-information-dw-lnct-llvm-source>`. 1996 1997The Clang option used to control source embedding in AMDGPU is defined in 1998:ref:`amdgpu-clang-debug-options-table`. 1999 2000 .. table:: AMDGPU Clang Debug Options 2001 :name: amdgpu-clang-debug-options-table 2002 2003 ==================== ================================================== 2004 Debug Flag Description 2005 ==================== ================================================== 2006 -g[no-]embed-source Enable/disable embedding source text in DWARF 2007 debug sections. Useful for environments where 2008 source cannot be written to disk, such as 2009 when performing online compilation. 2010 ==================== ================================================== 2011 2012For example: 2013 2014``-gembed-source`` 2015 Enable the embedded source. 2016 2017``-gno-embed-source`` 2018 Disable the embedded source. 2019 202032-Bit and 64-Bit DWARF Formats 2021------------------------------- 2022 2023See DWARF Version 5 section 7.4 and 2024:ref:`amdgpu-dwarf-32-bit-and-64-bit-dwarf-formats`. 2025 2026For AMDGPU: 2027 2028* For the ``amdgcn`` target architecture only the 64-bit process address space 2029 is supported. 2030 2031* The producer can generate either 32-bit or 64-bit DWARF format. LLVM generates 2032 the 32-bit DWARF format. 2033 2034Unit Headers 2035------------ 2036 2037For AMDGPU the following values apply for each of the unit headers described in 2038DWARF Version 5 sections 7.5.1.1, 7.5.1.2, and 7.5.1.3: 2039 2040``address_size`` (ubyte) 2041 Matches the address size for the ``Global`` address space defined in 2042 :ref:`amdgpu-dwarf-address-space-identifier`. 2043 2044.. _amdgpu-code-conventions: 2045 2046Code Conventions 2047================ 2048 2049This section provides code conventions used for each supported target triple OS 2050(see :ref:`amdgpu-target-triples`). 2051 2052AMDHSA 2053------ 2054 2055This section provides code conventions used when the target triple OS is 2056``amdhsa`` (see :ref:`amdgpu-target-triples`). 2057 2058.. _amdgpu-amdhsa-code-object-target-identification: 2059 2060Code Object Target Identification 2061~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2062 2063The AMDHSA OS uses the following syntax to specify the code object 2064target as a single string: 2065 2066 ``<Architecture>-<Vendor>-<OS>-<Environment>-<Processor><Target Features>`` 2067 2068Where: 2069 2070 - ``<Architecture>``, ``<Vendor>``, ``<OS>`` and ``<Environment>`` 2071 are the same as the *Target Triple* (see 2072 :ref:`amdgpu-target-triples`). 2073 2074 - ``<Processor>`` is the same as the *Processor* (see 2075 :ref:`amdgpu-processors`). 2076 2077 - ``<Target Features>`` is a list of the enabled *Target Features* 2078 (see :ref:`amdgpu-target-features`), each prefixed by a plus, that 2079 apply to *Processor*. The list must be in the same order as listed 2080 in the table :ref:`amdgpu-target-feature-table`. Note that *Target 2081 Features* must be included in the list if they are enabled even if 2082 that is the default for *Processor*. 2083 2084For example: 2085 2086 ``"amdgcn-amd-amdhsa--gfx902+xnack"`` 2087 2088.. _amdgpu-amdhsa-code-object-metadata: 2089 2090Code Object Metadata 2091~~~~~~~~~~~~~~~~~~~~ 2092 2093The code object metadata specifies extensible metadata associated with the code 2094objects executed on HSA [HSA]_ compatible runtimes such as AMD's ROCm 2095[AMD-ROCm]_. The encoding and semantics of this metadata depends on the code 2096object version; see :ref:`amdgpu-amdhsa-code-object-metadata-v2` and 2097:ref:`amdgpu-amdhsa-code-object-metadata-v3`. 2098 2099Code object metadata is specified in a note record (see 2100:ref:`amdgpu-note-records`) and is required when the target triple OS is 2101``amdhsa`` (see :ref:`amdgpu-target-triples`). It must contain the minimum 2102information necessary to support the ROCM kernel queries. For example, the 2103segment sizes needed in a dispatch packet. In addition, a high-level language 2104runtime may require other information to be included. For example, the AMD 2105OpenCL runtime records kernel argument information. 2106 2107.. _amdgpu-amdhsa-code-object-metadata-v2: 2108 2109Code Object V2 Metadata (--amdhsa-code-object-version=2) 2110++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 2111 2112.. warning:: Code Object V2 is not the default code object version emitted by 2113 this version of LLVM. For a description of the metadata generated with the 2114 default configuration (Code Object V3) see 2115 :ref:`amdgpu-amdhsa-code-object-metadata-v3`. 2116 2117Code object V2 metadata is specified by the ``NT_AMD_AMDGPU_METADATA`` note 2118record (see :ref:`amdgpu-note-records-v2`). 2119 2120The metadata is specified as a YAML formatted string (see [YAML]_ and 2121:doc:`YamlIO`). 2122 2123.. TODO:: 2124 2125 Is the string null terminated? It probably should not if YAML allows it to 2126 contain null characters, otherwise it should be. 2127 2128The metadata is represented as a single YAML document comprised of the mapping 2129defined in table :ref:`amdgpu-amdhsa-code-object-metadata-map-table-v2` and 2130referenced tables. 2131 2132For boolean values, the string values of ``false`` and ``true`` are used for 2133false and true respectively. 2134 2135Additional information can be added to the mappings. To avoid conflicts, any 2136non-AMD key names should be prefixed by "*vendor-name*.". 2137 2138 .. table:: AMDHSA Code Object V2 Metadata Map 2139 :name: amdgpu-amdhsa-code-object-metadata-map-table-v2 2140 2141 ========== ============== ========= ======================================= 2142 String Key Value Type Required? Description 2143 ========== ============== ========= ======================================= 2144 "Version" sequence of Required - The first integer is the major 2145 2 integers version. Currently 1. 2146 - The second integer is the minor 2147 version. Currently 0. 2148 "Printf" sequence of Each string is encoded information 2149 strings about a printf function call. The 2150 encoded information is organized as 2151 fields separated by colon (':'): 2152 2153 ``ID:N:S[0]:S[1]:...:S[N-1]:FormatString`` 2154 2155 where: 2156 2157 ``ID`` 2158 A 32-bit integer as a unique id for 2159 each printf function call 2160 2161 ``N`` 2162 A 32-bit integer equal to the number 2163 of arguments of printf function call 2164 minus 1 2165 2166 ``S[i]`` (where i = 0, 1, ... , N-1) 2167 32-bit integers for the size in bytes 2168 of the i-th FormatString argument of 2169 the printf function call 2170 2171 FormatString 2172 The format string passed to the 2173 printf function call. 2174 "Kernels" sequence of Required Sequence of the mappings for each 2175 mapping kernel in the code object. See 2176 :ref:`amdgpu-amdhsa-code-object-kernel-metadata-map-table-v2` 2177 for the definition of the mapping. 2178 ========== ============== ========= ======================================= 2179 2180.. 2181 2182 .. table:: AMDHSA Code Object V2 Kernel Metadata Map 2183 :name: amdgpu-amdhsa-code-object-kernel-metadata-map-table-v2 2184 2185 ================= ============== ========= ================================ 2186 String Key Value Type Required? Description 2187 ================= ============== ========= ================================ 2188 "Name" string Required Source name of the kernel. 2189 "SymbolName" string Required Name of the kernel 2190 descriptor ELF symbol. 2191 "Language" string Source language of the kernel. 2192 Values include: 2193 2194 - "OpenCL C" 2195 - "OpenCL C++" 2196 - "HCC" 2197 - "OpenMP" 2198 2199 "LanguageVersion" sequence of - The first integer is the major 2200 2 integers version. 2201 - The second integer is the 2202 minor version. 2203 "Attrs" mapping Mapping of kernel attributes. 2204 See 2205 :ref:`amdgpu-amdhsa-code-object-kernel-attribute-metadata-map-table-v2` 2206 for the mapping definition. 2207 "Args" sequence of Sequence of mappings of the 2208 mapping kernel arguments. See 2209 :ref:`amdgpu-amdhsa-code-object-kernel-argument-metadata-map-table-v2` 2210 for the definition of the mapping. 2211 "CodeProps" mapping Mapping of properties related to 2212 the kernel code. See 2213 :ref:`amdgpu-amdhsa-code-object-kernel-code-properties-metadata-map-table-v2` 2214 for the mapping definition. 2215 ================= ============== ========= ================================ 2216 2217.. 2218 2219 .. table:: AMDHSA Code Object V2 Kernel Attribute Metadata Map 2220 :name: amdgpu-amdhsa-code-object-kernel-attribute-metadata-map-table-v2 2221 2222 =================== ============== ========= ============================== 2223 String Key Value Type Required? Description 2224 =================== ============== ========= ============================== 2225 "ReqdWorkGroupSize" sequence of If not 0, 0, 0 then all values 2226 3 integers must be >=1 and the dispatch 2227 work-group size X, Y, Z must 2228 correspond to the specified 2229 values. Defaults to 0, 0, 0. 2230 2231 Corresponds to the OpenCL 2232 ``reqd_work_group_size`` 2233 attribute. 2234 "WorkGroupSizeHint" sequence of The dispatch work-group size 2235 3 integers X, Y, Z is likely to be the 2236 specified values. 2237 2238 Corresponds to the OpenCL 2239 ``work_group_size_hint`` 2240 attribute. 2241 "VecTypeHint" string The name of a scalar or vector 2242 type. 2243 2244 Corresponds to the OpenCL 2245 ``vec_type_hint`` attribute. 2246 2247 "RuntimeHandle" string The external symbol name 2248 associated with a kernel. 2249 OpenCL runtime allocates a 2250 global buffer for the symbol 2251 and saves the kernel's address 2252 to it, which is used for 2253 device side enqueueing. Only 2254 available for device side 2255 enqueued kernels. 2256 =================== ============== ========= ============================== 2257 2258.. 2259 2260 .. table:: AMDHSA Code Object V2 Kernel Argument Metadata Map 2261 :name: amdgpu-amdhsa-code-object-kernel-argument-metadata-map-table-v2 2262 2263 ================= ============== ========= ================================ 2264 String Key Value Type Required? Description 2265 ================= ============== ========= ================================ 2266 "Name" string Kernel argument name. 2267 "TypeName" string Kernel argument type name. 2268 "Size" integer Required Kernel argument size in bytes. 2269 "Align" integer Required Kernel argument alignment in 2270 bytes. Must be a power of two. 2271 "ValueKind" string Required Kernel argument kind that 2272 specifies how to set up the 2273 corresponding argument. 2274 Values include: 2275 2276 "ByValue" 2277 The argument is copied 2278 directly into the kernarg. 2279 2280 "GlobalBuffer" 2281 A global address space pointer 2282 to the buffer data is passed 2283 in the kernarg. 2284 2285 "DynamicSharedPointer" 2286 A group address space pointer 2287 to dynamically allocated LDS 2288 is passed in the kernarg. 2289 2290 "Sampler" 2291 A global address space 2292 pointer to a S# is passed in 2293 the kernarg. 2294 2295 "Image" 2296 A global address space 2297 pointer to a T# is passed in 2298 the kernarg. 2299 2300 "Pipe" 2301 A global address space pointer 2302 to an OpenCL pipe is passed in 2303 the kernarg. 2304 2305 "Queue" 2306 A global address space pointer 2307 to an OpenCL device enqueue 2308 queue is passed in the 2309 kernarg. 2310 2311 "HiddenGlobalOffsetX" 2312 The OpenCL grid dispatch 2313 global offset for the X 2314 dimension is passed in the 2315 kernarg. 2316 2317 "HiddenGlobalOffsetY" 2318 The OpenCL grid dispatch 2319 global offset for the Y 2320 dimension is passed in the 2321 kernarg. 2322 2323 "HiddenGlobalOffsetZ" 2324 The OpenCL grid dispatch 2325 global offset for the Z 2326 dimension is passed in the 2327 kernarg. 2328 2329 "HiddenNone" 2330 An argument that is not used 2331 by the kernel. Space needs to 2332 be left for it, but it does 2333 not need to be set up. 2334 2335 "HiddenPrintfBuffer" 2336 A global address space pointer 2337 to the runtime printf buffer 2338 is passed in kernarg. 2339 2340 "HiddenHostcallBuffer" 2341 A global address space pointer 2342 to the runtime hostcall buffer 2343 is passed in kernarg. 2344 2345 "HiddenDefaultQueue" 2346 A global address space pointer 2347 to the OpenCL device enqueue 2348 queue that should be used by 2349 the kernel by default is 2350 passed in the kernarg. 2351 2352 "HiddenCompletionAction" 2353 A global address space pointer 2354 to help link enqueued kernels into 2355 the ancestor tree for determining 2356 when the parent kernel has finished. 2357 2358 "HiddenMultiGridSyncArg" 2359 A global address space pointer for 2360 multi-grid synchronization is 2361 passed in the kernarg. 2362 2363 "ValueType" string Unused and deprecated. This should no longer 2364 be emitted, but is accepted for compatibility. 2365 2366 2367 "PointeeAlign" integer Alignment in bytes of pointee 2368 type for pointer type kernel 2369 argument. Must be a power 2370 of 2. Only present if 2371 "ValueKind" is 2372 "DynamicSharedPointer". 2373 "AddrSpaceQual" string Kernel argument address space 2374 qualifier. Only present if 2375 "ValueKind" is "GlobalBuffer" or 2376 "DynamicSharedPointer". Values 2377 are: 2378 2379 - "Private" 2380 - "Global" 2381 - "Constant" 2382 - "Local" 2383 - "Generic" 2384 - "Region" 2385 2386 .. TODO:: 2387 2388 Is GlobalBuffer only Global 2389 or Constant? Is 2390 DynamicSharedPointer always 2391 Local? Can HCC allow Generic? 2392 How can Private or Region 2393 ever happen? 2394 2395 "AccQual" string Kernel argument access 2396 qualifier. Only present if 2397 "ValueKind" is "Image" or 2398 "Pipe". Values 2399 are: 2400 2401 - "ReadOnly" 2402 - "WriteOnly" 2403 - "ReadWrite" 2404 2405 .. TODO:: 2406 2407 Does this apply to 2408 GlobalBuffer? 2409 2410 "ActualAccQual" string The actual memory accesses 2411 performed by the kernel on the 2412 kernel argument. Only present if 2413 "ValueKind" is "GlobalBuffer", 2414 "Image", or "Pipe". This may be 2415 more restrictive than indicated 2416 by "AccQual" to reflect what the 2417 kernel actual does. If not 2418 present then the runtime must 2419 assume what is implied by 2420 "AccQual" and "IsConst". Values 2421 are: 2422 2423 - "ReadOnly" 2424 - "WriteOnly" 2425 - "ReadWrite" 2426 2427 "IsConst" boolean Indicates if the kernel argument 2428 is const qualified. Only present 2429 if "ValueKind" is 2430 "GlobalBuffer". 2431 2432 "IsRestrict" boolean Indicates if the kernel argument 2433 is restrict qualified. Only 2434 present if "ValueKind" is 2435 "GlobalBuffer". 2436 2437 "IsVolatile" boolean Indicates if the kernel argument 2438 is volatile qualified. Only 2439 present if "ValueKind" is 2440 "GlobalBuffer". 2441 2442 "IsPipe" boolean Indicates if the kernel argument 2443 is pipe qualified. Only present 2444 if "ValueKind" is "Pipe". 2445 2446 .. TODO:: 2447 2448 Can GlobalBuffer be pipe 2449 qualified? 2450 2451 ================= ============== ========= ================================ 2452 2453.. 2454 2455 .. table:: AMDHSA Code Object V2 Kernel Code Properties Metadata Map 2456 :name: amdgpu-amdhsa-code-object-kernel-code-properties-metadata-map-table-v2 2457 2458 ============================ ============== ========= ===================== 2459 String Key Value Type Required? Description 2460 ============================ ============== ========= ===================== 2461 "KernargSegmentSize" integer Required The size in bytes of 2462 the kernarg segment 2463 that holds the values 2464 of the arguments to 2465 the kernel. 2466 "GroupSegmentFixedSize" integer Required The amount of group 2467 segment memory 2468 required by a 2469 work-group in 2470 bytes. This does not 2471 include any 2472 dynamically allocated 2473 group segment memory 2474 that may be added 2475 when the kernel is 2476 dispatched. 2477 "PrivateSegmentFixedSize" integer Required The amount of fixed 2478 private address space 2479 memory required for a 2480 work-item in 2481 bytes. If the kernel 2482 uses a dynamic call 2483 stack then additional 2484 space must be added 2485 to this value for the 2486 call stack. 2487 "KernargSegmentAlign" integer Required The maximum byte 2488 alignment of 2489 arguments in the 2490 kernarg segment. Must 2491 be a power of 2. 2492 "WavefrontSize" integer Required Wavefront size. Must 2493 be a power of 2. 2494 "NumSGPRs" integer Required Number of scalar 2495 registers used by a 2496 wavefront for 2497 GFX6-GFX10. This 2498 includes the special 2499 SGPRs for VCC, Flat 2500 Scratch (GFX7-GFX10) 2501 and XNACK (for 2502 GFX8-GFX10). It does 2503 not include the 16 2504 SGPR added if a trap 2505 handler is 2506 enabled. It is not 2507 rounded up to the 2508 allocation 2509 granularity. 2510 "NumVGPRs" integer Required Number of vector 2511 registers used by 2512 each work-item for 2513 GFX6-GFX10 2514 "MaxFlatWorkGroupSize" integer Required Maximum flat 2515 work-group size 2516 supported by the 2517 kernel in work-items. 2518 Must be >=1 and 2519 consistent with 2520 ReqdWorkGroupSize if 2521 not 0, 0, 0. 2522 "NumSpilledSGPRs" integer Number of stores from 2523 a scalar register to 2524 a register allocator 2525 created spill 2526 location. 2527 "NumSpilledVGPRs" integer Number of stores from 2528 a vector register to 2529 a register allocator 2530 created spill 2531 location. 2532 ============================ ============== ========= ===================== 2533 2534.. _amdgpu-amdhsa-code-object-metadata-v3: 2535 2536Code Object V3 Metadata (--amdhsa-code-object-version=3) 2537++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 2538 2539Code object V3 metadata is specified by the ``NT_AMDGPU_METADATA`` note record 2540(see :ref:`amdgpu-note-records-v3`). 2541 2542The metadata is represented as Message Pack formatted binary data (see 2543[MsgPack]_). The top level is a Message Pack map that includes the 2544keys defined in table 2545:ref:`amdgpu-amdhsa-code-object-metadata-map-table-v3` and referenced 2546tables. 2547 2548Additional information can be added to the maps. To avoid conflicts, 2549any key names should be prefixed by "*vendor-name*." where 2550``vendor-name`` can be the name of the vendor and specific vendor 2551tool that generates the information. The prefix is abbreviated to 2552simply "." when it appears within a map that has been added by the 2553same *vendor-name*. 2554 2555 .. table:: AMDHSA Code Object V3 Metadata Map 2556 :name: amdgpu-amdhsa-code-object-metadata-map-table-v3 2557 2558 ================= ============== ========= ======================================= 2559 String Key Value Type Required? Description 2560 ================= ============== ========= ======================================= 2561 "amdhsa.version" sequence of Required - The first integer is the major 2562 2 integers version. Currently 1. 2563 - The second integer is the minor 2564 version. Currently 0. 2565 "amdhsa.printf" sequence of Each string is encoded information 2566 strings about a printf function call. The 2567 encoded information is organized as 2568 fields separated by colon (':'): 2569 2570 ``ID:N:S[0]:S[1]:...:S[N-1]:FormatString`` 2571 2572 where: 2573 2574 ``ID`` 2575 A 32-bit integer as a unique id for 2576 each printf function call 2577 2578 ``N`` 2579 A 32-bit integer equal to the number 2580 of arguments of printf function call 2581 minus 1 2582 2583 ``S[i]`` (where i = 0, 1, ... , N-1) 2584 32-bit integers for the size in bytes 2585 of the i-th FormatString argument of 2586 the printf function call 2587 2588 FormatString 2589 The format string passed to the 2590 printf function call. 2591 "amdhsa.kernels" sequence of Required Sequence of the maps for each 2592 map kernel in the code object. See 2593 :ref:`amdgpu-amdhsa-code-object-kernel-metadata-map-table-v3` 2594 for the definition of the keys included 2595 in that map. 2596 ================= ============== ========= ======================================= 2597 2598.. 2599 2600 .. table:: AMDHSA Code Object V3 Kernel Metadata Map 2601 :name: amdgpu-amdhsa-code-object-kernel-metadata-map-table-v3 2602 2603 =================================== ============== ========= ================================ 2604 String Key Value Type Required? Description 2605 =================================== ============== ========= ================================ 2606 ".name" string Required Source name of the kernel. 2607 ".symbol" string Required Name of the kernel 2608 descriptor ELF symbol. 2609 ".language" string Source language of the kernel. 2610 Values include: 2611 2612 - "OpenCL C" 2613 - "OpenCL C++" 2614 - "HCC" 2615 - "HIP" 2616 - "OpenMP" 2617 - "Assembler" 2618 2619 ".language_version" sequence of - The first integer is the major 2620 2 integers version. 2621 - The second integer is the 2622 minor version. 2623 ".args" sequence of Sequence of maps of the 2624 map kernel arguments. See 2625 :ref:`amdgpu-amdhsa-code-object-kernel-argument-metadata-map-table-v3` 2626 for the definition of the keys 2627 included in that map. 2628 ".reqd_workgroup_size" sequence of If not 0, 0, 0 then all values 2629 3 integers must be >=1 and the dispatch 2630 work-group size X, Y, Z must 2631 correspond to the specified 2632 values. Defaults to 0, 0, 0. 2633 2634 Corresponds to the OpenCL 2635 ``reqd_work_group_size`` 2636 attribute. 2637 ".workgroup_size_hint" sequence of The dispatch work-group size 2638 3 integers X, Y, Z is likely to be the 2639 specified values. 2640 2641 Corresponds to the OpenCL 2642 ``work_group_size_hint`` 2643 attribute. 2644 ".vec_type_hint" string The name of a scalar or vector 2645 type. 2646 2647 Corresponds to the OpenCL 2648 ``vec_type_hint`` attribute. 2649 2650 ".device_enqueue_symbol" string The external symbol name 2651 associated with a kernel. 2652 OpenCL runtime allocates a 2653 global buffer for the symbol 2654 and saves the kernel's address 2655 to it, which is used for 2656 device side enqueueing. Only 2657 available for device side 2658 enqueued kernels. 2659 ".kernarg_segment_size" integer Required The size in bytes of 2660 the kernarg segment 2661 that holds the values 2662 of the arguments to 2663 the kernel. 2664 ".group_segment_fixed_size" integer Required The amount of group 2665 segment memory 2666 required by a 2667 work-group in 2668 bytes. This does not 2669 include any 2670 dynamically allocated 2671 group segment memory 2672 that may be added 2673 when the kernel is 2674 dispatched. 2675 ".private_segment_fixed_size" integer Required The amount of fixed 2676 private address space 2677 memory required for a 2678 work-item in 2679 bytes. If the kernel 2680 uses a dynamic call 2681 stack then additional 2682 space must be added 2683 to this value for the 2684 call stack. 2685 ".kernarg_segment_align" integer Required The maximum byte 2686 alignment of 2687 arguments in the 2688 kernarg segment. Must 2689 be a power of 2. 2690 ".wavefront_size" integer Required Wavefront size. Must 2691 be a power of 2. 2692 ".sgpr_count" integer Required Number of scalar 2693 registers required by a 2694 wavefront for 2695 GFX6-GFX9. A register 2696 is required if it is 2697 used explicitly, or 2698 if a higher numbered 2699 register is used 2700 explicitly. This 2701 includes the special 2702 SGPRs for VCC, Flat 2703 Scratch (GFX7-GFX9) 2704 and XNACK (for 2705 GFX8-GFX9). It does 2706 not include the 16 2707 SGPR added if a trap 2708 handler is 2709 enabled. It is not 2710 rounded up to the 2711 allocation 2712 granularity. 2713 ".vgpr_count" integer Required Number of vector 2714 registers required by 2715 each work-item for 2716 GFX6-GFX9. A register 2717 is required if it is 2718 used explicitly, or 2719 if a higher numbered 2720 register is used 2721 explicitly. 2722 ".max_flat_workgroup_size" integer Required Maximum flat 2723 work-group size 2724 supported by the 2725 kernel in work-items. 2726 Must be >=1 and 2727 consistent with 2728 ReqdWorkGroupSize if 2729 not 0, 0, 0. 2730 ".sgpr_spill_count" integer Number of stores from 2731 a scalar register to 2732 a register allocator 2733 created spill 2734 location. 2735 ".vgpr_spill_count" integer Number of stores from 2736 a vector register to 2737 a register allocator 2738 created spill 2739 location. 2740 =================================== ============== ========= ================================ 2741 2742.. 2743 2744 .. table:: AMDHSA Code Object V3 Kernel Argument Metadata Map 2745 :name: amdgpu-amdhsa-code-object-kernel-argument-metadata-map-table-v3 2746 2747 ====================== ============== ========= ================================ 2748 String Key Value Type Required? Description 2749 ====================== ============== ========= ================================ 2750 ".name" string Kernel argument name. 2751 ".type_name" string Kernel argument type name. 2752 ".size" integer Required Kernel argument size in bytes. 2753 ".offset" integer Required Kernel argument offset in 2754 bytes. The offset must be a 2755 multiple of the alignment 2756 required by the argument. 2757 ".value_kind" string Required Kernel argument kind that 2758 specifies how to set up the 2759 corresponding argument. 2760 Values include: 2761 2762 "by_value" 2763 The argument is copied 2764 directly into the kernarg. 2765 2766 "global_buffer" 2767 A global address space pointer 2768 to the buffer data is passed 2769 in the kernarg. 2770 2771 "dynamic_shared_pointer" 2772 A group address space pointer 2773 to dynamically allocated LDS 2774 is passed in the kernarg. 2775 2776 "sampler" 2777 A global address space 2778 pointer to a S# is passed in 2779 the kernarg. 2780 2781 "image" 2782 A global address space 2783 pointer to a T# is passed in 2784 the kernarg. 2785 2786 "pipe" 2787 A global address space pointer 2788 to an OpenCL pipe is passed in 2789 the kernarg. 2790 2791 "queue" 2792 A global address space pointer 2793 to an OpenCL device enqueue 2794 queue is passed in the 2795 kernarg. 2796 2797 "hidden_global_offset_x" 2798 The OpenCL grid dispatch 2799 global offset for the X 2800 dimension is passed in the 2801 kernarg. 2802 2803 "hidden_global_offset_y" 2804 The OpenCL grid dispatch 2805 global offset for the Y 2806 dimension is passed in the 2807 kernarg. 2808 2809 "hidden_global_offset_z" 2810 The OpenCL grid dispatch 2811 global offset for the Z 2812 dimension is passed in the 2813 kernarg. 2814 2815 "hidden_none" 2816 An argument that is not used 2817 by the kernel. Space needs to 2818 be left for it, but it does 2819 not need to be set up. 2820 2821 "hidden_printf_buffer" 2822 A global address space pointer 2823 to the runtime printf buffer 2824 is passed in kernarg. 2825 2826 "hidden_hostcall_buffer" 2827 A global address space pointer 2828 to the runtime hostcall buffer 2829 is passed in kernarg. 2830 2831 "hidden_default_queue" 2832 A global address space pointer 2833 to the OpenCL device enqueue 2834 queue that should be used by 2835 the kernel by default is 2836 passed in the kernarg. 2837 2838 "hidden_completion_action" 2839 A global address space pointer 2840 to help link enqueued kernels into 2841 the ancestor tree for determining 2842 when the parent kernel has finished. 2843 2844 "hidden_multigrid_sync_arg" 2845 A global address space pointer for 2846 multi-grid synchronization is 2847 passed in the kernarg. 2848 2849 ".value_type" string Unused and deprecated. This should no longer 2850 be emitted, but is accepted for compatibility. 2851 2852 ".pointee_align" integer Alignment in bytes of pointee 2853 type for pointer type kernel 2854 argument. Must be a power 2855 of 2. Only present if 2856 ".value_kind" is 2857 "dynamic_shared_pointer". 2858 ".address_space" string Kernel argument address space 2859 qualifier. Only present if 2860 ".value_kind" is "global_buffer" or 2861 "dynamic_shared_pointer". Values 2862 are: 2863 2864 - "private" 2865 - "global" 2866 - "constant" 2867 - "local" 2868 - "generic" 2869 - "region" 2870 2871 .. TODO:: 2872 2873 Is "global_buffer" only "global" 2874 or "constant"? Is 2875 "dynamic_shared_pointer" always 2876 "local"? Can HCC allow "generic"? 2877 How can "private" or "region" 2878 ever happen? 2879 2880 ".access" string Kernel argument access 2881 qualifier. Only present if 2882 ".value_kind" is "image" or 2883 "pipe". Values 2884 are: 2885 2886 - "read_only" 2887 - "write_only" 2888 - "read_write" 2889 2890 .. TODO:: 2891 2892 Does this apply to 2893 "global_buffer"? 2894 2895 ".actual_access" string The actual memory accesses 2896 performed by the kernel on the 2897 kernel argument. Only present if 2898 ".value_kind" is "global_buffer", 2899 "image", or "pipe". This may be 2900 more restrictive than indicated 2901 by ".access" to reflect what the 2902 kernel actual does. If not 2903 present then the runtime must 2904 assume what is implied by 2905 ".access" and ".is_const" . Values 2906 are: 2907 2908 - "read_only" 2909 - "write_only" 2910 - "read_write" 2911 2912 ".is_const" boolean Indicates if the kernel argument 2913 is const qualified. Only present 2914 if ".value_kind" is 2915 "global_buffer". 2916 2917 ".is_restrict" boolean Indicates if the kernel argument 2918 is restrict qualified. Only 2919 present if ".value_kind" is 2920 "global_buffer". 2921 2922 ".is_volatile" boolean Indicates if the kernel argument 2923 is volatile qualified. Only 2924 present if ".value_kind" is 2925 "global_buffer". 2926 2927 ".is_pipe" boolean Indicates if the kernel argument 2928 is pipe qualified. Only present 2929 if ".value_kind" is "pipe". 2930 2931 .. TODO:: 2932 2933 Can "global_buffer" be pipe 2934 qualified? 2935 2936 ====================== ============== ========= ================================ 2937 2938.. 2939 2940Kernel Dispatch 2941~~~~~~~~~~~~~~~ 2942 2943The HSA architected queuing language (AQL) defines a user space memory interface 2944that can be used to control the dispatch of kernels, in an agent independent 2945way. An agent can have zero or more AQL queues created for it using the ROCm 2946runtime, in which AQL packets (all of which are 64 bytes) can be placed. See the 2947*HSA Platform System Architecture Specification* [HSA]_ for the AQL queue 2948mechanics and packet layouts. 2949 2950The packet processor of a kernel agent is responsible for detecting and 2951dispatching HSA kernels from the AQL queues associated with it. For AMD GPUs the 2952packet processor is implemented by the hardware command processor (CP), 2953asynchronous dispatch controller (ADC) and shader processor input controller 2954(SPI). 2955 2956The ROCm runtime can be used to allocate an AQL queue object. It uses the kernel 2957mode driver to initialize and register the AQL queue with CP. 2958 2959To dispatch a kernel the following actions are performed. This can occur in the 2960CPU host program, or from an HSA kernel executing on a GPU. 2961 29621. A pointer to an AQL queue for the kernel agent on which the kernel is to be 2963 executed is obtained. 29642. A pointer to the kernel descriptor (see 2965 :ref:`amdgpu-amdhsa-kernel-descriptor`) of the kernel to execute is obtained. 2966 It must be for a kernel that is contained in a code object that that was 2967 loaded by the ROCm runtime on the kernel agent with which the AQL queue is 2968 associated. 29693. Space is allocated for the kernel arguments using the ROCm runtime allocator 2970 for a memory region with the kernarg property for the kernel agent that will 2971 execute the kernel. It must be at least 16-byte aligned. 29724. Kernel argument values are assigned to the kernel argument memory 2973 allocation. The layout is defined in the *HSA Programmer's Language 2974 Reference* [HSA]_. For AMDGPU the kernel execution directly accesses the 2975 kernel argument memory in the same way constant memory is accessed. (Note 2976 that the HSA specification allows an implementation to copy the kernel 2977 argument contents to another location that is accessed by the kernel.) 29785. An AQL kernel dispatch packet is created on the AQL queue. The ROCm runtime 2979 api uses 64-bit atomic operations to reserve space in the AQL queue for the 2980 packet. The packet must be set up, and the final write must use an atomic 2981 store release to set the packet kind to ensure the packet contents are 2982 visible to the kernel agent. AQL defines a doorbell signal mechanism to 2983 notify the kernel agent that the AQL queue has been updated. These rules, and 2984 the layout of the AQL queue and kernel dispatch packet is defined in the *HSA 2985 System Architecture Specification* [HSA]_. 29866. A kernel dispatch packet includes information about the actual dispatch, 2987 such as grid and work-group size, together with information from the code 2988 object about the kernel, such as segment sizes. The ROCm runtime queries on 2989 the kernel symbol can be used to obtain the code object values which are 2990 recorded in the :ref:`amdgpu-amdhsa-code-object-metadata`. 29917. CP executes micro-code and is responsible for detecting and setting up the 2992 GPU to execute the wavefronts of a kernel dispatch. 29938. CP ensures that when the a wavefront starts executing the kernel machine 2994 code, the scalar general purpose registers (SGPR) and vector general purpose 2995 registers (VGPR) are set up as required by the machine code. The required 2996 setup is defined in the :ref:`amdgpu-amdhsa-kernel-descriptor`. The initial 2997 register state is defined in 2998 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`. 29999. The prolog of the kernel machine code (see 3000 :ref:`amdgpu-amdhsa-kernel-prolog`) sets up the machine state as necessary 3001 before continuing executing the machine code that corresponds to the kernel. 300210. When the kernel dispatch has completed execution, CP signals the completion 3003 signal specified in the kernel dispatch packet if not 0. 3004 3005.. _amdgpu-amdhsa-memory-spaces: 3006 3007Memory Spaces 3008~~~~~~~~~~~~~ 3009 3010The memory space properties are: 3011 3012 .. table:: AMDHSA Memory Spaces 3013 :name: amdgpu-amdhsa-memory-spaces-table 3014 3015 ================= =========== ======== ======= ================== 3016 Memory Space Name HSA Segment Hardware Address NULL Value 3017 Name Name Size 3018 ================= =========== ======== ======= ================== 3019 Private private scratch 32 0x00000000 3020 Local group LDS 32 0xFFFFFFFF 3021 Global global global 64 0x0000000000000000 3022 Constant constant *same as 64 0x0000000000000000 3023 global* 3024 Generic flat flat 64 0x0000000000000000 3025 Region N/A GDS 32 *not implemented 3026 for AMDHSA* 3027 ================= =========== ======== ======= ================== 3028 3029The global and constant memory spaces both use global virtual addresses, which 3030are the same virtual address space used by the CPU. However, some virtual 3031addresses may only be accessible to the CPU, some only accessible by the GPU, 3032and some by both. 3033 3034Using the constant memory space indicates that the data will not change during 3035the execution of the kernel. This allows scalar read instructions to be 3036used. The vector and scalar L1 caches are invalidated of volatile data before 3037each kernel dispatch execution to allow constant memory to change values between 3038kernel dispatches. 3039 3040The local memory space uses the hardware Local Data Store (LDS) which is 3041automatically allocated when the hardware creates work-groups of wavefronts, and 3042freed when all the wavefronts of a work-group have terminated. The data store 3043(DS) instructions can be used to access it. 3044 3045The private memory space uses the hardware scratch memory support. If the kernel 3046uses scratch, then the hardware allocates memory that is accessed using 3047wavefront lane dword (4 byte) interleaving. The mapping used from private 3048address to physical address is: 3049 3050 ``wavefront-scratch-base + 3051 (private-address * wavefront-size * 4) + 3052 (wavefront-lane-id * 4)`` 3053 3054There are different ways that the wavefront scratch base address is determined 3055by a wavefront (see :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). This 3056memory can be accessed in an interleaved manner using buffer instruction with 3057the scratch buffer descriptor and per wavefront scratch offset, by the scratch 3058instructions, or by flat instructions. If each lane of a wavefront accesses the 3059same private address, the interleaving results in adjacent dwords being accessed 3060and hence requires fewer cache lines to be fetched. Multi-dword access is not 3061supported except by flat and scratch instructions in GFX9-GFX10. 3062 3063The generic address space uses the hardware flat address support available in 3064GFX7-GFX10. This uses two fixed ranges of virtual addresses (the private and 3065local apertures), that are outside the range of addressible global memory, to 3066map from a flat address to a private or local address. 3067 3068FLAT instructions can take a flat address and access global, private (scratch) 3069and group (LDS) memory depending in if the address is within one of the 3070aperture ranges. Flat access to scratch requires hardware aperture setup and 3071setup in the kernel prologue (see 3072:ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`). Flat access to LDS requires 3073hardware aperture setup and M0 (GFX7-GFX8) register setup (see 3074:ref:`amdgpu-amdhsa-kernel-prolog-m0`). 3075 3076To convert between a segment address and a flat address the base address of the 3077apertures address can be used. For GFX7-GFX8 these are available in the 3078:ref:`amdgpu-amdhsa-hsa-aql-queue` the address of which can be obtained with 3079Queue Ptr SGPR (see :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). For 3080GFX9-GFX10 the aperture base addresses are directly available as inline constant 3081registers ``SRC_SHARED_BASE/LIMIT`` and ``SRC_PRIVATE_BASE/LIMIT``. In 64 bit 3082address mode the aperture sizes are 2^32 bytes and the base is aligned to 2^32 3083which makes it easier to convert from flat to segment or segment to flat. 3084 3085Image and Samplers 3086~~~~~~~~~~~~~~~~~~ 3087 3088Image and sample handles created by the ROCm runtime are 64-bit addresses of a 3089hardware 32-byte V# and 48 byte S# object respectively. In order to support the 3090HSA ``query_sampler`` operations two extra dwords are used to store the HSA BRIG 3091enumeration values for the queries that are not trivially deducible from the S# 3092representation. 3093 3094HSA Signals 3095~~~~~~~~~~~ 3096 3097HSA signal handles created by the ROCm runtime are 64-bit addresses of a 3098structure allocated in memory accessible from both the CPU and GPU. The 3099structure is defined by the ROCm runtime and subject to change between releases 3100(see [AMD-ROCm-github]_). 3101 3102.. _amdgpu-amdhsa-hsa-aql-queue: 3103 3104HSA AQL Queue 3105~~~~~~~~~~~~~ 3106 3107The HSA AQL queue structure is defined by the ROCm runtime and subject to change 3108between releases (see [AMD-ROCm-github]_). For some processors it contains 3109fields needed to implement certain language features such as the flat address 3110aperture bases. It also contains fields used by CP such as managing the 3111allocation of scratch memory. 3112 3113.. _amdgpu-amdhsa-kernel-descriptor: 3114 3115Kernel Descriptor 3116~~~~~~~~~~~~~~~~~ 3117 3118A kernel descriptor consists of the information needed by CP to initiate the 3119execution of a kernel, including the entry point address of the machine code 3120that implements the kernel. 3121 3122Kernel Descriptor for GFX6-GFX10 3123++++++++++++++++++++++++++++++++ 3124 3125CP microcode requires the Kernel descriptor to be allocated on 64-byte 3126alignment. 3127 3128 .. table:: Kernel Descriptor for GFX6-GFX10 3129 :name: amdgpu-amdhsa-kernel-descriptor-gfx6-gfx10-table 3130 3131 ======= ======= =============================== ============================ 3132 Bits Size Field Name Description 3133 ======= ======= =============================== ============================ 3134 31:0 4 bytes GROUP_SEGMENT_FIXED_SIZE The amount of fixed local 3135 address space memory 3136 required for a work-group 3137 in bytes. This does not 3138 include any dynamically 3139 allocated local address 3140 space memory that may be 3141 added when the kernel is 3142 dispatched. 3143 63:32 4 bytes PRIVATE_SEGMENT_FIXED_SIZE The amount of fixed 3144 private address space 3145 memory required for a 3146 work-item in bytes. If 3147 is_dynamic_callstack is 1 3148 then additional space must 3149 be added to this value for 3150 the call stack. 3151 127:64 8 bytes Reserved, must be 0. 3152 191:128 8 bytes KERNEL_CODE_ENTRY_BYTE_OFFSET Byte offset (possibly 3153 negative) from base 3154 address of kernel 3155 descriptor to kernel's 3156 entry point instruction 3157 which must be 256 byte 3158 aligned. 3159 351:272 20 Reserved, must be 0. 3160 bytes 3161 383:352 4 bytes COMPUTE_PGM_RSRC3 GFX6-9 3162 Reserved, must be 0. 3163 GFX10 3164 Compute Shader (CS) 3165 program settings used by 3166 CP to set up 3167 ``COMPUTE_PGM_RSRC3`` 3168 configuration 3169 register. See 3170 :ref:`amdgpu-amdhsa-compute_pgm_rsrc3-gfx10-table`. 3171 415:384 4 bytes COMPUTE_PGM_RSRC1 Compute Shader (CS) 3172 program settings used by 3173 CP to set up 3174 ``COMPUTE_PGM_RSRC1`` 3175 configuration 3176 register. See 3177 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 3178 447:416 4 bytes COMPUTE_PGM_RSRC2 Compute Shader (CS) 3179 program settings used by 3180 CP to set up 3181 ``COMPUTE_PGM_RSRC2`` 3182 configuration 3183 register. See 3184 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 3185 448 1 bit ENABLE_SGPR_PRIVATE_SEGMENT Enable the setup of the 3186 _BUFFER SGPR user data registers 3187 (see 3188 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 3189 3190 The total number of SGPR 3191 user data registers 3192 requested must not exceed 3193 16 and match value in 3194 ``compute_pgm_rsrc2.user_sgpr.user_sgpr_count``. 3195 Any requests beyond 16 3196 will be ignored. 3197 449 1 bit ENABLE_SGPR_DISPATCH_PTR *see above* 3198 450 1 bit ENABLE_SGPR_QUEUE_PTR *see above* 3199 451 1 bit ENABLE_SGPR_KERNARG_SEGMENT_PTR *see above* 3200 452 1 bit ENABLE_SGPR_DISPATCH_ID *see above* 3201 453 1 bit ENABLE_SGPR_FLAT_SCRATCH_INIT *see above* 3202 454 1 bit ENABLE_SGPR_PRIVATE_SEGMENT *see above* 3203 _SIZE 3204 457:455 3 bits Reserved, must be 0. 3205 458 1 bit ENABLE_WAVEFRONT_SIZE32 GFX6-9 3206 Reserved, must be 0. 3207 GFX10 3208 - If 0 execute in 3209 wavefront size 64 mode. 3210 - If 1 execute in 3211 native wavefront size 3212 32 mode. 3213 463:459 5 bits Reserved, must be 0. 3214 511:464 6 bytes Reserved, must be 0. 3215 512 **Total size 64 bytes.** 3216 ======= ==================================================================== 3217 3218.. 3219 3220 .. table:: compute_pgm_rsrc1 for GFX6-GFX10 3221 :name: amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table 3222 3223 ======= ======= =============================== =========================================================================== 3224 Bits Size Field Name Description 3225 ======= ======= =============================== =========================================================================== 3226 5:0 6 bits GRANULATED_WORKITEM_VGPR_COUNT Number of vector register 3227 blocks used by each work-item; 3228 granularity is device 3229 specific: 3230 3231 GFX6-GFX9 3232 - vgprs_used 0..256 3233 - max(0, ceil(vgprs_used / 4) - 1) 3234 GFX10 (wavefront size 64) 3235 - max_vgpr 1..256 3236 - max(0, ceil(vgprs_used / 4) - 1) 3237 GFX10 (wavefront size 32) 3238 - max_vgpr 1..256 3239 - max(0, ceil(vgprs_used / 8) - 1) 3240 3241 Where vgprs_used is defined 3242 as the highest VGPR number 3243 explicitly referenced plus 3244 one. 3245 3246 Used by CP to set up 3247 ``COMPUTE_PGM_RSRC1.VGPRS``. 3248 3249 The 3250 :ref:`amdgpu-assembler` 3251 calculates this 3252 automatically for the 3253 selected processor from 3254 values provided to the 3255 `.amdhsa_kernel` directive 3256 by the 3257 `.amdhsa_next_free_vgpr` 3258 nested directive (see 3259 :ref:`amdhsa-kernel-directives-table`). 3260 9:6 4 bits GRANULATED_WAVEFRONT_SGPR_COUNT Number of scalar register 3261 blocks used by a wavefront; 3262 granularity is device 3263 specific: 3264 3265 GFX6-GFX8 3266 - sgprs_used 0..112 3267 - max(0, ceil(sgprs_used / 8) - 1) 3268 GFX9 3269 - sgprs_used 0..112 3270 - 2 * max(0, ceil(sgprs_used / 16) - 1) 3271 GFX10 3272 Reserved, must be 0. 3273 (128 SGPRs always 3274 allocated.) 3275 3276 Where sgprs_used is 3277 defined as the highest 3278 SGPR number explicitly 3279 referenced plus one, plus 3280 a target specific number 3281 of additional special 3282 SGPRs for VCC, 3283 FLAT_SCRATCH (GFX7+) and 3284 XNACK_MASK (GFX8+), and 3285 any additional 3286 target specific 3287 limitations. It does not 3288 include the 16 SGPRs added 3289 if a trap handler is 3290 enabled. 3291 3292 The target specific 3293 limitations and special 3294 SGPR layout are defined in 3295 the hardware 3296 documentation, which can 3297 be found in the 3298 :ref:`amdgpu-processors` 3299 table. 3300 3301 Used by CP to set up 3302 ``COMPUTE_PGM_RSRC1.SGPRS``. 3303 3304 The 3305 :ref:`amdgpu-assembler` 3306 calculates this 3307 automatically for the 3308 selected processor from 3309 values provided to the 3310 `.amdhsa_kernel` directive 3311 by the 3312 `.amdhsa_next_free_sgpr` 3313 and `.amdhsa_reserve_*` 3314 nested directives (see 3315 :ref:`amdhsa-kernel-directives-table`). 3316 11:10 2 bits PRIORITY Must be 0. 3317 3318 Start executing wavefront 3319 at the specified priority. 3320 3321 CP is responsible for 3322 filling in 3323 ``COMPUTE_PGM_RSRC1.PRIORITY``. 3324 13:12 2 bits FLOAT_ROUND_MODE_32 Wavefront starts execution 3325 with specified rounding 3326 mode for single (32 3327 bit) floating point 3328 precision floating point 3329 operations. 3330 3331 Floating point rounding 3332 mode values are defined in 3333 :ref:`amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table`. 3334 3335 Used by CP to set up 3336 ``COMPUTE_PGM_RSRC1.FLOAT_MODE``. 3337 15:14 2 bits FLOAT_ROUND_MODE_16_64 Wavefront starts execution 3338 with specified rounding 3339 denorm mode for half/double (16 3340 and 64-bit) floating point 3341 precision floating point 3342 operations. 3343 3344 Floating point rounding 3345 mode values are defined in 3346 :ref:`amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table`. 3347 3348 Used by CP to set up 3349 ``COMPUTE_PGM_RSRC1.FLOAT_MODE``. 3350 17:16 2 bits FLOAT_DENORM_MODE_32 Wavefront starts execution 3351 with specified denorm mode 3352 for single (32 3353 bit) floating point 3354 precision floating point 3355 operations. 3356 3357 Floating point denorm mode 3358 values are defined in 3359 :ref:`amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table`. 3360 3361 Used by CP to set up 3362 ``COMPUTE_PGM_RSRC1.FLOAT_MODE``. 3363 19:18 2 bits FLOAT_DENORM_MODE_16_64 Wavefront starts execution 3364 with specified denorm mode 3365 for half/double (16 3366 and 64-bit) floating point 3367 precision floating point 3368 operations. 3369 3370 Floating point denorm mode 3371 values are defined in 3372 :ref:`amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table`. 3373 3374 Used by CP to set up 3375 ``COMPUTE_PGM_RSRC1.FLOAT_MODE``. 3376 20 1 bit PRIV Must be 0. 3377 3378 Start executing wavefront 3379 in privilege trap handler 3380 mode. 3381 3382 CP is responsible for 3383 filling in 3384 ``COMPUTE_PGM_RSRC1.PRIV``. 3385 21 1 bit ENABLE_DX10_CLAMP Wavefront starts execution 3386 with DX10 clamp mode 3387 enabled. Used by the vector 3388 ALU to force DX10 style 3389 treatment of NaN's (when 3390 set, clamp NaN to zero, 3391 otherwise pass NaN 3392 through). 3393 3394 Used by CP to set up 3395 ``COMPUTE_PGM_RSRC1.DX10_CLAMP``. 3396 22 1 bit DEBUG_MODE Must be 0. 3397 3398 Start executing wavefront 3399 in single step mode. 3400 3401 CP is responsible for 3402 filling in 3403 ``COMPUTE_PGM_RSRC1.DEBUG_MODE``. 3404 23 1 bit ENABLE_IEEE_MODE Wavefront starts execution 3405 with IEEE mode 3406 enabled. Floating point 3407 opcodes that support 3408 exception flag gathering 3409 will quiet and propagate 3410 signaling-NaN inputs per 3411 IEEE 754-2008. Min_dx10 and 3412 max_dx10 become IEEE 3413 754-2008 compliant due to 3414 signaling-NaN propagation 3415 and quieting. 3416 3417 Used by CP to set up 3418 ``COMPUTE_PGM_RSRC1.IEEE_MODE``. 3419 24 1 bit BULKY Must be 0. 3420 3421 Only one work-group allowed 3422 to execute on a compute 3423 unit. 3424 3425 CP is responsible for 3426 filling in 3427 ``COMPUTE_PGM_RSRC1.BULKY``. 3428 25 1 bit CDBG_USER Must be 0. 3429 3430 Flag that can be used to 3431 control debugging code. 3432 3433 CP is responsible for 3434 filling in 3435 ``COMPUTE_PGM_RSRC1.CDBG_USER``. 3436 26 1 bit FP16_OVFL GFX6-GFX8 3437 Reserved, must be 0. 3438 GFX9-GFX10 3439 Wavefront starts execution 3440 with specified fp16 overflow 3441 mode. 3442 3443 - If 0, fp16 overflow generates 3444 +/-INF values. 3445 - If 1, fp16 overflow that is the 3446 result of an +/-INF input value 3447 or divide by 0 produces a +/-INF, 3448 otherwise clamps computed 3449 overflow to +/-MAX_FP16 as 3450 appropriate. 3451 3452 Used by CP to set up 3453 ``COMPUTE_PGM_RSRC1.FP16_OVFL``. 3454 28:27 2 bits Reserved, must be 0. 3455 29 1 bit WGP_MODE GFX6-GFX9 3456 Reserved, must be 0. 3457 GFX10 3458 - If 0 execute work-groups in 3459 CU wavefront execution mode. 3460 - If 1 execute work-groups on 3461 in WGP wavefront execution mode. 3462 3463 See :ref:`amdgpu-amdhsa-memory-model`. 3464 3465 Used by CP to set up 3466 ``COMPUTE_PGM_RSRC1.WGP_MODE``. 3467 30 1 bit MEM_ORDERED GFX6-9 3468 Reserved, must be 0. 3469 GFX10 3470 Controls the behavior of the 3471 waitcnt's vmcnt and vscnt 3472 counters. 3473 3474 - If 0 vmcnt reports completion 3475 of load and atomic with return 3476 out of order with sample 3477 instructions, and the vscnt 3478 reports the completion of 3479 store and atomic without 3480 return in order. 3481 - If 1 vmcnt reports completion 3482 of load, atomic with return 3483 and sample instructions in 3484 order, and the vscnt reports 3485 the completion of store and 3486 atomic without return in order. 3487 3488 Used by CP to set up 3489 ``COMPUTE_PGM_RSRC1.MEM_ORDERED``. 3490 31 1 bit FWD_PROGRESS GFX6-9 3491 Reserved, must be 0. 3492 GFX10 3493 - If 0 execute SIMD wavefronts 3494 using oldest first policy. 3495 - If 1 execute SIMD wavefronts to 3496 ensure wavefronts will make some 3497 forward progress. 3498 3499 Used by CP to set up 3500 ``COMPUTE_PGM_RSRC1.FWD_PROGRESS``. 3501 32 **Total size 4 bytes** 3502 ======= =================================================================================================================== 3503 3504.. 3505 3506 .. table:: compute_pgm_rsrc2 for GFX6-GFX10 3507 :name: amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table 3508 3509 ======= ======= =============================== =========================================================================== 3510 Bits Size Field Name Description 3511 ======= ======= =============================== =========================================================================== 3512 0 1 bit ENABLE_SGPR_PRIVATE_SEGMENT Enable the setup of the 3513 _WAVEFRONT_OFFSET SGPR wavefront scratch offset 3514 system register (see 3515 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 3516 3517 Used by CP to set up 3518 ``COMPUTE_PGM_RSRC2.SCRATCH_EN``. 3519 5:1 5 bits USER_SGPR_COUNT The total number of SGPR 3520 user data registers 3521 requested. This number must 3522 match the number of user 3523 data registers enabled. 3524 3525 Used by CP to set up 3526 ``COMPUTE_PGM_RSRC2.USER_SGPR``. 3527 6 1 bit ENABLE_TRAP_HANDLER Must be 0. 3528 3529 This bit represents 3530 ``COMPUTE_PGM_RSRC2.TRAP_PRESENT``, 3531 which is set by the CP if 3532 the runtime has installed a 3533 trap handler. 3534 7 1 bit ENABLE_SGPR_WORKGROUP_ID_X Enable the setup of the 3535 system SGPR register for 3536 the work-group id in the X 3537 dimension (see 3538 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 3539 3540 Used by CP to set up 3541 ``COMPUTE_PGM_RSRC2.TGID_X_EN``. 3542 8 1 bit ENABLE_SGPR_WORKGROUP_ID_Y Enable the setup of the 3543 system SGPR register for 3544 the work-group id in the Y 3545 dimension (see 3546 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 3547 3548 Used by CP to set up 3549 ``COMPUTE_PGM_RSRC2.TGID_Y_EN``. 3550 9 1 bit ENABLE_SGPR_WORKGROUP_ID_Z Enable the setup of the 3551 system SGPR register for 3552 the work-group id in the Z 3553 dimension (see 3554 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 3555 3556 Used by CP to set up 3557 ``COMPUTE_PGM_RSRC2.TGID_Z_EN``. 3558 10 1 bit ENABLE_SGPR_WORKGROUP_INFO Enable the setup of the 3559 system SGPR register for 3560 work-group information (see 3561 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 3562 3563 Used by CP to set up 3564 ``COMPUTE_PGM_RSRC2.TGID_SIZE_EN``. 3565 12:11 2 bits ENABLE_VGPR_WORKITEM_ID Enable the setup of the 3566 VGPR system registers used 3567 for the work-item ID. 3568 :ref:`amdgpu-amdhsa-system-vgpr-work-item-id-enumeration-values-table` 3569 defines the values. 3570 3571 Used by CP to set up 3572 ``COMPUTE_PGM_RSRC2.TIDIG_CMP_CNT``. 3573 13 1 bit ENABLE_EXCEPTION_ADDRESS_WATCH Must be 0. 3574 3575 Wavefront starts execution 3576 with address watch 3577 exceptions enabled which 3578 are generated when L1 has 3579 witnessed a thread access 3580 an *address of 3581 interest*. 3582 3583 CP is responsible for 3584 filling in the address 3585 watch bit in 3586 ``COMPUTE_PGM_RSRC2.EXCP_EN_MSB`` 3587 according to what the 3588 runtime requests. 3589 14 1 bit ENABLE_EXCEPTION_MEMORY Must be 0. 3590 3591 Wavefront starts execution 3592 with memory violation 3593 exceptions exceptions 3594 enabled which are generated 3595 when a memory violation has 3596 occurred for this wavefront from 3597 L1 or LDS 3598 (write-to-read-only-memory, 3599 mis-aligned atomic, LDS 3600 address out of range, 3601 illegal address, etc.). 3602 3603 CP sets the memory 3604 violation bit in 3605 ``COMPUTE_PGM_RSRC2.EXCP_EN_MSB`` 3606 according to what the 3607 runtime requests. 3608 23:15 9 bits GRANULATED_LDS_SIZE Must be 0. 3609 3610 CP uses the rounded value 3611 from the dispatch packet, 3612 not this value, as the 3613 dispatch may contain 3614 dynamically allocated group 3615 segment memory. CP writes 3616 directly to 3617 ``COMPUTE_PGM_RSRC2.LDS_SIZE``. 3618 3619 Amount of group segment 3620 (LDS) to allocate for each 3621 work-group. Granularity is 3622 device specific: 3623 3624 GFX6: 3625 roundup(lds-size / (64 * 4)) 3626 GFX7-GFX10: 3627 roundup(lds-size / (128 * 4)) 3628 3629 24 1 bit ENABLE_EXCEPTION_IEEE_754_FP Wavefront starts execution 3630 _INVALID_OPERATION with specified exceptions 3631 enabled. 3632 3633 Used by CP to set up 3634 ``COMPUTE_PGM_RSRC2.EXCP_EN`` 3635 (set from bits 0..6). 3636 3637 IEEE 754 FP Invalid 3638 Operation 3639 25 1 bit ENABLE_EXCEPTION_FP_DENORMAL FP Denormal one or more 3640 _SOURCE input operands is a 3641 denormal number 3642 26 1 bit ENABLE_EXCEPTION_IEEE_754_FP IEEE 754 FP Division by 3643 _DIVISION_BY_ZERO Zero 3644 27 1 bit ENABLE_EXCEPTION_IEEE_754_FP IEEE 754 FP FP Overflow 3645 _OVERFLOW 3646 28 1 bit ENABLE_EXCEPTION_IEEE_754_FP IEEE 754 FP Underflow 3647 _UNDERFLOW 3648 29 1 bit ENABLE_EXCEPTION_IEEE_754_FP IEEE 754 FP Inexact 3649 _INEXACT 3650 30 1 bit ENABLE_EXCEPTION_INT_DIVIDE_BY Integer Division by Zero 3651 _ZERO (rcp_iflag_f32 instruction 3652 only) 3653 31 1 bit Reserved, must be 0. 3654 32 **Total size 4 bytes.** 3655 ======= =================================================================================================================== 3656 3657.. 3658 3659 .. table:: compute_pgm_rsrc3 for GFX10 3660 :name: amdgpu-amdhsa-compute_pgm_rsrc3-gfx10-table 3661 3662 ======= ======= =============================== =========================================================================== 3663 Bits Size Field Name Description 3664 ======= ======= =============================== =========================================================================== 3665 3:0 4 bits SHARED_VGPR_COUNT Number of shared VGPRs for wavefront size 64. Granularity 8. Value 0-120. 3666 compute_pgm_rsrc1.vgprs + shared_vgpr_cnt cannot exceed 64. 3667 31:4 28 Reserved, must be 0. 3668 bits 3669 32 **Total size 4 bytes.** 3670 ======= =================================================================================================================== 3671 3672.. 3673 3674 .. table:: Floating Point Rounding Mode Enumeration Values 3675 :name: amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table 3676 3677 ====================================== ===== ============================== 3678 Enumeration Name Value Description 3679 ====================================== ===== ============================== 3680 FLOAT_ROUND_MODE_NEAR_EVEN 0 Round Ties To Even 3681 FLOAT_ROUND_MODE_PLUS_INFINITY 1 Round Toward +infinity 3682 FLOAT_ROUND_MODE_MINUS_INFINITY 2 Round Toward -infinity 3683 FLOAT_ROUND_MODE_ZERO 3 Round Toward 0 3684 ====================================== ===== ============================== 3685 3686.. 3687 3688 .. table:: Floating Point Denorm Mode Enumeration Values 3689 :name: amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table 3690 3691 ====================================== ===== ============================== 3692 Enumeration Name Value Description 3693 ====================================== ===== ============================== 3694 FLOAT_DENORM_MODE_FLUSH_SRC_DST 0 Flush Source and Destination 3695 Denorms 3696 FLOAT_DENORM_MODE_FLUSH_DST 1 Flush Output Denorms 3697 FLOAT_DENORM_MODE_FLUSH_SRC 2 Flush Source Denorms 3698 FLOAT_DENORM_MODE_FLUSH_NONE 3 No Flush 3699 ====================================== ===== ============================== 3700 3701.. 3702 3703 .. table:: System VGPR Work-Item ID Enumeration Values 3704 :name: amdgpu-amdhsa-system-vgpr-work-item-id-enumeration-values-table 3705 3706 ======================================== ===== ============================ 3707 Enumeration Name Value Description 3708 ======================================== ===== ============================ 3709 SYSTEM_VGPR_WORKITEM_ID_X 0 Set work-item X dimension 3710 ID. 3711 SYSTEM_VGPR_WORKITEM_ID_X_Y 1 Set work-item X and Y 3712 dimensions ID. 3713 SYSTEM_VGPR_WORKITEM_ID_X_Y_Z 2 Set work-item X, Y and Z 3714 dimensions ID. 3715 SYSTEM_VGPR_WORKITEM_ID_UNDEFINED 3 Undefined. 3716 ======================================== ===== ============================ 3717 3718.. _amdgpu-amdhsa-initial-kernel-execution-state: 3719 3720Initial Kernel Execution State 3721~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 3722 3723This section defines the register state that will be set up by the packet 3724processor prior to the start of execution of every wavefront. This is limited by 3725the constraints of the hardware controllers of CP/ADC/SPI. 3726 3727The order of the SGPR registers is defined, but the compiler can specify which 3728ones are actually setup in the kernel descriptor using the ``enable_sgpr_*`` bit 3729fields (see :ref:`amdgpu-amdhsa-kernel-descriptor`). The register numbers used 3730for enabled registers are dense starting at SGPR0: the first enabled register is 3731SGPR0, the next enabled register is SGPR1 etc.; disabled registers do not have 3732an SGPR number. 3733 3734The initial SGPRs comprise up to 16 User SRGPs that are set by CP and apply to 3735all wavefronts of the grid. It is possible to specify more than 16 User SGPRs 3736using the ``enable_sgpr_*`` bit fields, in which case only the first 16 are 3737actually initialized. These are then immediately followed by the System SGPRs 3738that are set up by ADC/SPI and can have different values for each wavefront of 3739the grid dispatch. 3740 3741SGPR register initial state is defined in 3742:ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 3743 3744 .. table:: SGPR Register Set Up Order 3745 :name: amdgpu-amdhsa-sgpr-register-set-up-order-table 3746 3747 ========== ========================== ====== ============================== 3748 SGPR Order Name Number Description 3749 (kernel descriptor enable of 3750 field) SGPRs 3751 ========== ========================== ====== ============================== 3752 First Private Segment Buffer 4 V# that can be used, together 3753 (enable_sgpr_private with Scratch Wavefront Offset 3754 _segment_buffer) as an offset, to access the 3755 private memory space using a 3756 segment address. 3757 3758 CP uses the value provided by 3759 the runtime. 3760 then Dispatch Ptr 2 64-bit address of AQL dispatch 3761 (enable_sgpr_dispatch_ptr) packet for kernel dispatch 3762 actually executing. 3763 then Queue Ptr 2 64-bit address of amd_queue_t 3764 (enable_sgpr_queue_ptr) object for AQL queue on which 3765 the dispatch packet was 3766 queued. 3767 then Kernarg Segment Ptr 2 64-bit address of Kernarg 3768 (enable_sgpr_kernarg segment. This is directly 3769 _segment_ptr) copied from the 3770 kernarg_address in the kernel 3771 dispatch packet. 3772 3773 Having CP load it once avoids 3774 loading it at the beginning of 3775 every wavefront. 3776 then Dispatch Id 2 64-bit Dispatch ID of the 3777 (enable_sgpr_dispatch_id) dispatch packet being 3778 executed. 3779 then Flat Scratch Init 2 This is 2 SGPRs: 3780 (enable_sgpr_flat_scratch 3781 _init) GFX6 3782 Not supported. 3783 GFX7-GFX8 3784 The first SGPR is a 32-bit 3785 byte offset from 3786 ``SH_HIDDEN_PRIVATE_BASE_VIMID`` 3787 to per SPI base of memory 3788 for scratch for the queue 3789 executing the kernel 3790 dispatch. CP obtains this 3791 from the runtime. (The 3792 Scratch Segment Buffer base 3793 address is 3794 ``SH_HIDDEN_PRIVATE_BASE_VIMID`` 3795 plus this offset.) The value 3796 of Scratch Wavefront Offset must 3797 be added to this offset by 3798 the kernel machine code, 3799 right shifted by 8, and 3800 moved to the FLAT_SCRATCH_HI 3801 SGPR register. 3802 FLAT_SCRATCH_HI corresponds 3803 to SGPRn-4 on GFX7, and 3804 SGPRn-6 on GFX8 (where SGPRn 3805 is the highest numbered SGPR 3806 allocated to the wavefront). 3807 FLAT_SCRATCH_HI is 3808 multiplied by 256 (as it is 3809 in units of 256 bytes) and 3810 added to 3811 ``SH_HIDDEN_PRIVATE_BASE_VIMID`` 3812 to calculate the per wavefront 3813 FLAT SCRATCH BASE in flat 3814 memory instructions that 3815 access the scratch 3816 aperture. 3817 3818 The second SGPR is 32-bit 3819 byte size of a single 3820 work-item's scratch memory 3821 usage. CP obtains this from 3822 the runtime, and it is 3823 always a multiple of DWORD. 3824 CP checks that the value in 3825 the kernel dispatch packet 3826 Private Segment Byte Size is 3827 not larger and requests the 3828 runtime to increase the 3829 queue's scratch size if 3830 necessary. The kernel code 3831 must move it to 3832 FLAT_SCRATCH_LO which is 3833 SGPRn-3 on GFX7 and SGPRn-5 3834 on GFX8. FLAT_SCRATCH_LO is 3835 used as the FLAT SCRATCH 3836 SIZE in flat memory 3837 instructions. Having CP load 3838 it once avoids loading it at 3839 the beginning of every 3840 wavefront. 3841 GFX9-GFX10 3842 This is the 3843 64-bit base address of the 3844 per SPI scratch backing 3845 memory managed by SPI for 3846 the queue executing the 3847 kernel dispatch. CP obtains 3848 this from the runtime (and 3849 divides it if there are 3850 multiple Shader Arrays each 3851 with its own SPI). The value 3852 of Scratch Wavefront Offset must 3853 be added by the kernel 3854 machine code and the result 3855 moved to the FLAT_SCRATCH 3856 SGPR which is SGPRn-6 and 3857 SGPRn-5. It is used as the 3858 FLAT SCRATCH BASE in flat 3859 memory instructions. 3860 then Private Segment Size 1 The 32-bit byte size of a 3861 (enable_sgpr_private single 3862 work-item's 3863 scratch_segment_size) memory 3864 allocation. This is the 3865 value from the kernel 3866 dispatch packet Private 3867 Segment Byte Size rounded up 3868 by CP to a multiple of 3869 DWORD. 3870 3871 Having CP load it once avoids 3872 loading it at the beginning of 3873 every wavefront. 3874 3875 This is not used for 3876 GFX7-GFX8 since it is the same 3877 value as the second SGPR of 3878 Flat Scratch Init. However, it 3879 may be needed for GFX9-GFX10 which 3880 changes the meaning of the 3881 Flat Scratch Init value. 3882 then Grid Work-Group Count X 1 32-bit count of the number of 3883 (enable_sgpr_grid work-groups in the X dimension 3884 _workgroup_count_X) for the grid being 3885 executed. Computed from the 3886 fields in the kernel dispatch 3887 packet as ((grid_size.x + 3888 workgroup_size.x - 1) / 3889 workgroup_size.x). 3890 then Grid Work-Group Count Y 1 32-bit count of the number of 3891 (enable_sgpr_grid work-groups in the Y dimension 3892 _workgroup_count_Y && for the grid being 3893 less than 16 previous executed. Computed from the 3894 SGPRs) fields in the kernel dispatch 3895 packet as ((grid_size.y + 3896 workgroup_size.y - 1) / 3897 workgroupSize.y). 3898 3899 Only initialized if <16 3900 previous SGPRs initialized. 3901 then Grid Work-Group Count Z 1 32-bit count of the number of 3902 (enable_sgpr_grid work-groups in the Z dimension 3903 _workgroup_count_Z && for the grid being 3904 less than 16 previous executed. Computed from the 3905 SGPRs) fields in the kernel dispatch 3906 packet as ((grid_size.z + 3907 workgroup_size.z - 1) / 3908 workgroupSize.z). 3909 3910 Only initialized if <16 3911 previous SGPRs initialized. 3912 then Work-Group Id X 1 32-bit work-group id in X 3913 (enable_sgpr_workgroup_id dimension of grid for 3914 _X) wavefront. 3915 then Work-Group Id Y 1 32-bit work-group id in Y 3916 (enable_sgpr_workgroup_id dimension of grid for 3917 _Y) wavefront. 3918 then Work-Group Id Z 1 32-bit work-group id in Z 3919 (enable_sgpr_workgroup_id dimension of grid for 3920 _Z) wavefront. 3921 then Work-Group Info 1 {first_wavefront, 14'b0000, 3922 (enable_sgpr_workgroup ordered_append_term[10:0], 3923 _info) threadgroup_size_in_wavefronts[5:0]} 3924 then Scratch Wavefront Offset 1 32-bit byte offset from base 3925 (enable_sgpr_private of scratch base of queue 3926 _segment_wavefront_offset) executing the kernel 3927 dispatch. Must be used as an 3928 offset with Private 3929 segment address when using 3930 Scratch Segment Buffer. It 3931 must be used to set up FLAT 3932 SCRATCH for flat addressing 3933 (see 3934 :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`). 3935 ========== ========================== ====== ============================== 3936 3937The order of the VGPR registers is defined, but the compiler can specify which 3938ones are actually setup in the kernel descriptor using the ``enable_vgpr*`` bit 3939fields (see :ref:`amdgpu-amdhsa-kernel-descriptor`). The register numbers used 3940for enabled registers are dense starting at VGPR0: the first enabled register is 3941VGPR0, the next enabled register is VGPR1 etc.; disabled registers do not have a 3942VGPR number. 3943 3944VGPR register initial state is defined in 3945:ref:`amdgpu-amdhsa-vgpr-register-set-up-order-table`. 3946 3947 .. table:: VGPR Register Set Up Order 3948 :name: amdgpu-amdhsa-vgpr-register-set-up-order-table 3949 3950 ========== ========================== ====== ============================== 3951 VGPR Order Name Number Description 3952 (kernel descriptor enable of 3953 field) VGPRs 3954 ========== ========================== ====== ============================== 3955 First Work-Item Id X 1 32-bit work-item id in X 3956 (Always initialized) dimension of work-group for 3957 wavefront lane. 3958 then Work-Item Id Y 1 32-bit work-item id in Y 3959 (enable_vgpr_workitem_id dimension of work-group for 3960 > 0) wavefront lane. 3961 then Work-Item Id Z 1 32-bit work-item id in Z 3962 (enable_vgpr_workitem_id dimension of work-group for 3963 > 1) wavefront lane. 3964 ========== ========================== ====== ============================== 3965 3966The setting of registers is done by GPU CP/ADC/SPI hardware as follows: 3967 39681. SGPRs before the Work-Group Ids are set by CP using the 16 User Data 3969 registers. 39702. Work-group Id registers X, Y, Z are set by ADC which supports any 3971 combination including none. 39723. Scratch Wavefront Offset is set by SPI in a per wavefront basis which is why 3973 its value cannot be included with the flat scratch init value which is per 3974 queue. 39754. The VGPRs are set by SPI which only supports specifying either (X), (X, Y) 3976 or (X, Y, Z). 3977 3978Flat Scratch register pair are adjacent SGPRs so they can be moved as a 64-bit 3979value to the hardware required SGPRn-3 and SGPRn-4 respectively. 3980 3981The global segment can be accessed either using buffer instructions (GFX6 which 3982has V# 64-bit address support), flat instructions (GFX7-GFX10), or global 3983instructions (GFX9-GFX10). 3984 3985If buffer operations are used, then the compiler can generate a V# with the 3986following properties: 3987 3988* base address of 0 3989* no swizzle 3990* ATC: 1 if IOMMU present (such as APU) 3991* ptr64: 1 3992* MTYPE set to support memory coherence that matches the runtime (such as CC for 3993 APU and NC for dGPU). 3994 3995.. _amdgpu-amdhsa-kernel-prolog: 3996 3997Kernel Prolog 3998~~~~~~~~~~~~~ 3999 4000The compiler performs initialization in the kernel prologue depending on the 4001target and information about things like stack usage in the kernel and called 4002functions. Some of this initialization requires the compiler to request certain 4003User and System SGPRs be present in the 4004:ref:`amdgpu-amdhsa-initial-kernel-execution-state` via the 4005:ref:`amdgpu-amdhsa-kernel-descriptor`. 4006 4007.. _amdgpu-amdhsa-kernel-prolog-cfi: 4008 4009CFI 4010+++ 4011 40121. The CFI return address is undefined. 4013 40142. The CFI CFA is defined using an expression which evaluates to a location 4015 description that comprises one memory location description for the 4016 ``DW_ASPACE_AMDGPU_private_lane`` address space address ``0``. 4017 4018.. _amdgpu-amdhsa-kernel-prolog-m0: 4019 4020M0 4021++ 4022 4023GFX6-GFX8 4024 The M0 register must be initialized with a value at least the total LDS size 4025 if the kernel may access LDS via DS or flat operations. Total LDS size is 4026 available in dispatch packet. For M0, it is also possible to use maximum 4027 possible value of LDS for given target (0x7FFF for GFX6 and 0xFFFF for 4028 GFX7-GFX8). 4029GFX9-GFX10 4030 The M0 register is not used for range checking LDS accesses and so does not 4031 need to be initialized in the prolog. 4032 4033.. _amdgpu-amdhsa-kernel-prolog-stack-pointer: 4034 4035Stack Pointer 4036+++++++++++++ 4037 4038If the kernel has function calls it must set up the ABI stack pointer described 4039in :ref:`amdgpu-amdhsa-function-call-convention-non-kernel-functions` by setting 4040SGPR32 to the unswizzled scratch offset of the address past the last local 4041allocation. 4042 4043.. _amdgpu-amdhsa-kernel-prolog-frame-pointer: 4044 4045Frame Pointer 4046+++++++++++++ 4047 4048If the kernel needs a frame pointer for the reasons defined in 4049``SIFrameLowering`` then SGPR33 is used and is always set to ``0`` in the 4050kernel prolog. If a frame pointer is not required then all uses of the frame 4051pointer are replaced with immediate ``0`` offsets. 4052 4053.. _amdgpu-amdhsa-kernel-prolog-flat-scratch: 4054 4055Flat Scratch 4056++++++++++++ 4057 4058If the kernel or any function it calls may use flat operations to access 4059scratch memory, the prolog code must set up the FLAT_SCRATCH register pair 4060(FLAT_SCRATCH_LO/FLAT_SCRATCH_HI which are in SGPRn-4/SGPRn-3). Initialization 4061uses Flat Scratch Init and Scratch Wavefront Offset SGPR registers (see 4062:ref:`amdgpu-amdhsa-initial-kernel-execution-state`): 4063 4064GFX6 4065 Flat scratch is not supported. 4066 4067GFX7-GFX8 4068 4069 1. The low word of Flat Scratch Init is 32-bit byte offset from 4070 ``SH_HIDDEN_PRIVATE_BASE_VIMID`` to the base of scratch backing memory 4071 being managed by SPI for the queue executing the kernel dispatch. This is 4072 the same value used in the Scratch Segment Buffer V# base address. The 4073 prolog must add the value of Scratch Wavefront Offset to get the 4074 wavefront's byte scratch backing memory offset from 4075 ``SH_HIDDEN_PRIVATE_BASE_VIMID``. Since FLAT_SCRATCH_LO is in units of 256 4076 bytes, the offset must be right shifted by 8 before moving into 4077 FLAT_SCRATCH_LO. 4078 2. The second word of Flat Scratch Init is 32-bit byte size of a single 4079 work-items scratch memory usage. This is directly loaded from the kernel 4080 dispatch packet Private Segment Byte Size and rounded up to a multiple of 4081 DWORD. Having CP load it once avoids loading it at the beginning of every 4082 wavefront. The prolog must move it to FLAT_SCRATCH_LO for use as FLAT 4083 SCRATCH SIZE. 4084 4085GFX9-GFX10 4086 The Flat Scratch Init is the 64-bit address of the base of scratch backing 4087 memory being managed by SPI for the queue executing the kernel dispatch. The 4088 prolog must add the value of Scratch Wavefront Offset and moved to the 4089 FLAT_SCRATCH pair for use as the flat scratch base in flat memory 4090 instructions. 4091 4092.. _amdgpu-amdhsa-kernel-prolog-private-segment-buffer: 4093 4094Private Segment Buffer 4095++++++++++++++++++++++ 4096 4097A set of four SGPRs beginning at a four-aligned SGPR index are always selected 4098to serve as the scratch V# for the kernel as follows: 4099 4100 - If it is known during instruction selection that there is stack usage, 4101 SGPR0-3 is reserved for use as the scratch V#. Stack usage is assumed if 4102 optimizations are disabled (``-O0``), if stack objects already exist (for 4103 locals, etc.), or if there are any function calls. 4104 4105 - Otherwise, four high numbered SGPRs beginning at a four-aligned SGPR index 4106 are reserved for the tentative scratch V#. These will be used if it is 4107 determined that spilling is needed. 4108 4109 - If no use is made of the tentative scratch V#, then it is unreserved, 4110 and the register count is determined ignoring it. 4111 - If use is made of the tentative scratch V#, then its register numbers 4112 are shifted to the first four-aligned SGPR index after the highest one 4113 allocated by the register allocator, and all uses are updated. The 4114 register count includes them in the shifted location. 4115 - In either case, if the processor has the SGPR allocation bug, the 4116 tentative allocation is not shifted or unreserved in order to ensure 4117 the register count is higher to workaround the bug. 4118 4119 .. note:: 4120 4121 This approach of using a tentative scratch V# and shifting the register 4122 numbers if used avoids having to perform register allocation a second 4123 time if the tentative V# is eliminated. This is more efficient and 4124 avoids the problem that the second register allocation may perform 4125 spilling which will fail as there is no longer a scratch V#. 4126 4127When the kernel prolog code is being emitted it is known whether the scratch V# 4128described above is actually used. If it is, the prolog code must set it up by 4129copying the Private Segment Buffer to the scratch V# registers and then adding 4130the Private Segment Wavefront Offset to the queue base address in the V#. The 4131result is a V# with a base address pointing to the beginning of the wavefront 4132scratch backing memory. 4133 4134The Private Segment Buffer is always requested, but the Private Segment 4135Wavefront Offset is only requested if it is used (see 4136:ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4137 4138.. _amdgpu-amdhsa-memory-model: 4139 4140Memory Model 4141~~~~~~~~~~~~ 4142 4143This section describes the mapping of LLVM memory model onto AMDGPU machine code 4144(see :ref:`memmodel`). 4145 4146The AMDGPU backend supports the memory synchronization scopes specified in 4147:ref:`amdgpu-memory-scopes`. 4148 4149The code sequences used to implement the memory model are defined in table 4150:ref:`amdgpu-amdhsa-memory-model-code-sequences-gfx6-gfx10-table`. 4151 4152The sequences specify the order of instructions that a single thread must 4153execute. The ``s_waitcnt`` and ``buffer_wbinvl1_vol`` are defined with respect 4154to other memory instructions executed by the same thread. This allows them to be 4155moved earlier or later which can allow them to be combined with other instances 4156of the same instruction, or hoisted/sunk out of loops to improve 4157performance. Only the instructions related to the memory model are given; 4158additional ``s_waitcnt`` instructions are required to ensure registers are 4159defined before being used. These may be able to be combined with the memory 4160model ``s_waitcnt`` instructions as described above. 4161 4162The AMDGPU backend supports the following memory models: 4163 4164 HSA Memory Model [HSA]_ 4165 The HSA memory model uses a single happens-before relation for all address 4166 spaces (see :ref:`amdgpu-address-spaces`). 4167 OpenCL Memory Model [OpenCL]_ 4168 The OpenCL memory model which has separate happens-before relations for the 4169 global and local address spaces. Only a fence specifying both global and 4170 local address space, and seq_cst instructions join the relationships. Since 4171 the LLVM ``memfence`` instruction does not allow an address space to be 4172 specified the OpenCL fence has to conservatively assume both local and 4173 global address space was specified. However, optimizations can often be 4174 done to eliminate the additional ``s_waitcnt`` instructions when there are 4175 no intervening memory instructions which access the corresponding address 4176 space. The code sequences in the table indicate what can be omitted for the 4177 OpenCL memory. The target triple environment is used to determine if the 4178 source language is OpenCL (see :ref:`amdgpu-opencl`). 4179 4180``ds/flat_load/store/atomic`` instructions to local memory are termed LDS 4181operations. 4182 4183``buffer/global/flat_load/store/atomic`` instructions to global memory are 4184termed vector memory operations. 4185 4186For GFX6-GFX9: 4187 4188* Each agent has multiple shader arrays (SA). 4189* Each SA has multiple compute units (CU). 4190* Each CU has multiple SIMDs that execute wavefronts. 4191* The wavefronts for a single work-group are executed in the same CU but may be 4192 executed by different SIMDs. 4193* Each CU has a single LDS memory shared by the wavefronts of the work-groups 4194 executing on it. 4195* All LDS operations of a CU are performed as wavefront wide operations in a 4196 global order and involve no caching. Completion is reported to a wavefront in 4197 execution order. 4198* The LDS memory has multiple request queues shared by the SIMDs of a 4199 CU. Therefore, the LDS operations performed by different wavefronts of a 4200 work-group can be reordered relative to each other, which can result in 4201 reordering the visibility of vector memory operations with respect to LDS 4202 operations of other wavefronts in the same work-group. A ``s_waitcnt 4203 lgkmcnt(0)`` is required to ensure synchronization between LDS operations and 4204 vector memory operations between wavefronts of a work-group, but not between 4205 operations performed by the same wavefront. 4206* The vector memory operations are performed as wavefront wide operations and 4207 completion is reported to a wavefront in execution order. The exception is 4208 that for GFX7-GFX9 ``flat_load/store/atomic`` instructions can report out of 4209 vector memory order if they access LDS memory, and out of LDS operation order 4210 if they access global memory. 4211* The vector memory operations access a single vector L1 cache shared by all 4212 SIMDs a CU. Therefore, no special action is required for coherence between the 4213 lanes of a single wavefront, or for coherence between wavefronts in the same 4214 work-group. A ``buffer_wbinvl1_vol`` is required for coherence between 4215 wavefronts executing in different work-groups as they may be executing on 4216 different CUs. 4217* The scalar memory operations access a scalar L1 cache shared by all wavefronts 4218 on a group of CUs. The scalar and vector L1 caches are not coherent. However, 4219 scalar operations are used in a restricted way so do not impact the memory 4220 model. See :ref:`amdgpu-amdhsa-memory-spaces`. 4221* The vector and scalar memory operations use an L2 cache shared by all CUs on 4222 the same agent. 4223* The L2 cache has independent channels to service disjoint ranges of virtual 4224 addresses. 4225* Each CU has a separate request queue per channel. Therefore, the vector and 4226 scalar memory operations performed by wavefronts executing in different 4227 work-groups (which may be executing on different CUs) of an agent can be 4228 reordered relative to each other. A ``s_waitcnt vmcnt(0)`` is required to 4229 ensure synchronization between vector memory operations of different CUs. It 4230 ensures a previous vector memory operation has completed before executing a 4231 subsequent vector memory or LDS operation and so can be used to meet the 4232 requirements of acquire and release. 4233* The L2 cache can be kept coherent with other agents on some targets, or ranges 4234 of virtual addresses can be set up to bypass it to ensure system coherence. 4235 4236For GFX10: 4237 4238* Each agent has multiple shader arrays (SA). 4239* Each SA has multiple work-group processors (WGP). 4240* Each WGP has multiple compute units (CU). 4241* Each CU has multiple SIMDs that execute wavefronts. 4242* The wavefronts for a single work-group are executed in the same 4243 WGP. In CU wavefront execution mode the wavefronts may be executed by 4244 different SIMDs in the same CU. In WGP wavefront execution mode the 4245 wavefronts may be executed by different SIMDs in different CUs in the same 4246 WGP. 4247* Each WGP has a single LDS memory shared by the wavefronts of the work-groups 4248 executing on it. 4249* All LDS operations of a WGP are performed as wavefront wide operations in a 4250 global order and involve no caching. Completion is reported to a wavefront in 4251 execution order. 4252* The LDS memory has multiple request queues shared by the SIMDs of a 4253 WGP. Therefore, the LDS operations performed by different wavefronts of a 4254 work-group can be reordered relative to each other, which can result in 4255 reordering the visibility of vector memory operations with respect to LDS 4256 operations of other wavefronts in the same work-group. A ``s_waitcnt 4257 lgkmcnt(0)`` is required to ensure synchronization between LDS operations and 4258 vector memory operations between wavefronts of a work-group, but not between 4259 operations performed by the same wavefront. 4260* The vector memory operations are performed as wavefront wide operations. 4261 Completion of load/store/sample operations are reported to a wavefront in 4262 execution order of other load/store/sample operations performed by that 4263 wavefront. 4264* The vector memory operations access a vector L0 cache. There is a single L0 4265 cache per CU. Each SIMD of a CU accesses the same L0 cache. Therefore, no 4266 special action is required for coherence between the lanes of a single 4267 wavefront. However, a ``buffer_gl0_inv`` is required for coherence between 4268 wavefronts executing in the same work-group as they may be executing on SIMDs 4269 of different CUs that access different L0s. A ``buffer_gl0_inv`` is also 4270 required for coherence between wavefronts executing in different work-groups 4271 as they may be executing on different WGPs. 4272* The scalar memory operations access a scalar L0 cache shared by all wavefronts 4273 on a WGP. The scalar and vector L0 caches are not coherent. However, scalar 4274 operations are used in a restricted way so do not impact the memory model. See 4275 :ref:`amdgpu-amdhsa-memory-spaces`. 4276* The vector and scalar memory L0 caches use an L1 cache shared by all WGPs on 4277 the same SA. Therefore, no special action is required for coherence between 4278 the wavefronts of a single work-group. However, a ``buffer_gl1_inv`` is 4279 required for coherence between wavefronts executing in different work-groups 4280 as they may be executing on different SAs that access different L1s. 4281* The L1 caches have independent quadrants to service disjoint ranges of virtual 4282 addresses. 4283* Each L0 cache has a separate request queue per L1 quadrant. Therefore, the 4284 vector and scalar memory operations performed by different wavefronts, whether 4285 executing in the same or different work-groups (which may be executing on 4286 different CUs accessing different L0s), can be reordered relative to each 4287 other. A ``s_waitcnt vmcnt(0) & vscnt(0)`` is required to ensure 4288 synchronization between vector memory operations of different wavefronts. It 4289 ensures a previous vector memory operation has completed before executing a 4290 subsequent vector memory or LDS operation and so can be used to meet the 4291 requirements of acquire, release and sequential consistency. 4292* The L1 caches use an L2 cache shared by all SAs on the same agent. 4293* The L2 cache has independent channels to service disjoint ranges of virtual 4294 addresses. 4295* Each L1 quadrant of a single SA accesses a different L2 channel. Each L1 4296 quadrant has a separate request queue per L2 channel. Therefore, the vector 4297 and scalar memory operations performed by wavefronts executing in different 4298 work-groups (which may be executing on different SAs) of an agent can be 4299 reordered relative to each other. A ``s_waitcnt vmcnt(0) & vscnt(0)`` is 4300 required to ensure synchronization between vector memory operations of 4301 different SAs. It ensures a previous vector memory operation has completed 4302 before executing a subsequent vector memory and so can be used to meet the 4303 requirements of acquire, release and sequential consistency. 4304* The L2 cache can be kept coherent with other agents on some targets, or ranges 4305 of virtual addresses can be set up to bypass it to ensure system coherence. 4306 4307Private address space uses ``buffer_load/store`` using the scratch V# 4308(GFX6-GFX8), or ``scratch_load/store`` (GFX9-GFX10). Since only a single thread 4309is accessing the memory, atomic memory orderings are not meaningful, and all 4310accesses are treated as non-atomic. 4311 4312Constant address space uses ``buffer/global_load`` instructions (or equivalent 4313scalar memory instructions). Since the constant address space contents do not 4314change during the execution of a kernel dispatch it is not legal to perform 4315stores, and atomic memory orderings are not meaningful, and all access are 4316treated as non-atomic. 4317 4318A memory synchronization scope wider than work-group is not meaningful for the 4319group (LDS) address space and is treated as work-group. 4320 4321The memory model does not support the region address space which is treated as 4322non-atomic. 4323 4324Acquire memory ordering is not meaningful on store atomic instructions and is 4325treated as non-atomic. 4326 4327Release memory ordering is not meaningful on load atomic instructions and is 4328treated a non-atomic. 4329 4330Acquire-release memory ordering is not meaningful on load or store atomic 4331instructions and is treated as acquire and release respectively. 4332 4333AMDGPU backend only uses scalar memory operations to access memory that is 4334proven to not change during the execution of the kernel dispatch. This includes 4335constant address space and global address space for program scope const 4336variables. Therefore, the kernel machine code does not have to maintain the 4337scalar L1 cache to ensure it is coherent with the vector L1 cache. The scalar 4338and vector L1 caches are invalidated between kernel dispatches by CP since 4339constant address space data may change between kernel dispatch executions. See 4340:ref:`amdgpu-amdhsa-memory-spaces`. 4341 4342The one exception is if scalar writes are used to spill SGPR registers. In this 4343case the AMDGPU backend ensures the memory location used to spill is never 4344accessed by vector memory operations at the same time. If scalar writes are used 4345then a ``s_dcache_wb`` is inserted before the ``s_endpgm`` and before a function 4346return since the locations may be used for vector memory instructions by a 4347future wavefront that uses the same scratch area, or a function call that 4348creates a frame at the same address, respectively. There is no need for a 4349``s_dcache_inv`` as all scalar writes are write-before-read in the same thread. 4350 4351For GFX6-GFX9, scratch backing memory (which is used for the private address 4352space) is accessed with MTYPE NC_NV (non-coherent non-volatile). Since the 4353private address space is only accessed by a single thread, and is always 4354write-before-read, there is never a need to invalidate these entries from the L1 4355cache. Hence all cache invalidates are done as ``*_vol`` to only invalidate the 4356volatile cache lines. 4357 4358For GFX10, scratch backing memory (which is used for the private address space) 4359is accessed with MTYPE NC (non-coherent). Since the private address space is 4360only accessed by a single thread, and is always write-before-read, there is 4361never a need to invalidate these entries from the L0 or L1 caches. 4362 4363For GFX10, wavefronts are executed in native mode with in-order reporting of 4364loads and sample instructions. In this mode vmcnt reports completion of load, 4365atomic with return and sample instructions in order, and the vscnt reports the 4366completion of store and atomic without return in order. See ``MEM_ORDERED`` 4367field in :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 4368 4369In GFX10, wavefronts can be executed in WGP or CU wavefront execution mode: 4370 4371* In WGP wavefront execution mode the wavefronts of a work-group are executed 4372 on the SIMDs of both CUs of the WGP. Therefore, explicit management of the per 4373 CU L0 caches is required for work-group synchronization. Also accesses to L1 4374 at work-group scope need to be explicitly ordered as the accesses from 4375 different CUs are not ordered. 4376* In CU wavefront execution mode the wavefronts of a work-group are executed on 4377 the SIMDs of a single CU of the WGP. Therefore, all global memory access by 4378 the work-group access the same L0 which in turn ensures L1 accesses are 4379 ordered and so do not require explicit management of the caches for 4380 work-group synchronization. 4381 4382See ``WGP_MODE`` field in 4383:ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table` and 4384:ref:`amdgpu-target-features`. 4385 4386On dGPU the kernarg backing memory is accessed as UC (uncached) to avoid needing 4387to invalidate the L2 cache. For GFX6-GFX9, this also causes it to be treated as 4388non-volatile and so is not invalidated by ``*_vol``. On APU it is accessed as CC 4389(cache coherent) and so the L2 cache will be coherent with the CPU and other 4390agents. 4391 4392 .. table:: AMDHSA Memory Model Code Sequences GFX6-GFX10 4393 :name: amdgpu-amdhsa-memory-model-code-sequences-gfx6-gfx10-table 4394 4395 ============ ============ ============== ========== ================================ ================================ 4396 LLVM Instr LLVM Memory LLVM Memory AMDGPU AMDGPU Machine Code AMDGPU Machine Code 4397 Ordering Sync Scope Address GFX6-9 GFX10 4398 Space 4399 ============ ============ ============== ========== ================================ ================================ 4400 **Non-Atomic** 4401 --------------------------------------------------------------------------------------------------------------------- 4402 load *none* *none* - global - !volatile & !nontemporal - !volatile & !nontemporal 4403 - generic 4404 - private 1. buffer/global/flat_load 1. buffer/global/flat_load 4405 - constant 4406 - volatile & !nontemporal - volatile & !nontemporal 4407 4408 1. buffer/global/flat_load 1. buffer/global/flat_load 4409 glc=1 glc=1 dlc=1 4410 4411 - nontemporal - nontemporal 4412 4413 1. buffer/global/flat_load 1. buffer/global/flat_load 4414 glc=1 slc=1 slc=1 4415 4416 load *none* *none* - local 1. ds_load 1. ds_load 4417 store *none* *none* - global - !nontemporal - !nontemporal 4418 - generic 4419 - private 1. buffer/global/flat_store 1. buffer/global/flat_store 4420 - constant 4421 - nontemporal - nontemporal 4422 4423 1. buffer/global/flat_store 1. buffer/global/flat_store 4424 glc=1 slc=1 slc=1 4425 4426 store *none* *none* - local 1. ds_store 1. ds_store 4427 **Unordered Atomic** 4428 --------------------------------------------------------------------------------------------------------------------- 4429 load atomic unordered *any* *any* *Same as non-atomic*. *Same as non-atomic*. 4430 store atomic unordered *any* *any* *Same as non-atomic*. *Same as non-atomic*. 4431 atomicrmw unordered *any* *any* *Same as monotonic *Same as monotonic 4432 atomic*. atomic*. 4433 **Monotonic Atomic** 4434 --------------------------------------------------------------------------------------------------------------------- 4435 load atomic monotonic - singlethread - global 1. buffer/global/flat_load 1. buffer/global/flat_load 4436 - wavefront - generic 4437 load atomic monotonic - workgroup - global 1. buffer/global/flat_load 1. buffer/global/flat_load 4438 - generic glc=1 4439 4440 - If CU wavefront execution 4441 mode, omit glc=1. 4442 4443 load atomic monotonic - singlethread - local 1. ds_load 1. ds_load 4444 - wavefront 4445 - workgroup 4446 load atomic monotonic - agent - global 1. buffer/global/flat_load 1. buffer/global/flat_load 4447 - system - generic glc=1 glc=1 dlc=1 4448 store atomic monotonic - singlethread - global 1. buffer/global/flat_store 1. buffer/global/flat_store 4449 - wavefront - generic 4450 - workgroup 4451 - agent 4452 - system 4453 store atomic monotonic - singlethread - local 1. ds_store 1. ds_store 4454 - wavefront 4455 - workgroup 4456 atomicrmw monotonic - singlethread - global 1. buffer/global/flat_atomic 1. buffer/global/flat_atomic 4457 - wavefront - generic 4458 - workgroup 4459 - agent 4460 - system 4461 atomicrmw monotonic - singlethread - local 1. ds_atomic 1. ds_atomic 4462 - wavefront 4463 - workgroup 4464 **Acquire Atomic** 4465 --------------------------------------------------------------------------------------------------------------------- 4466 load atomic acquire - singlethread - global 1. buffer/global/ds/flat_load 1. buffer/global/ds/flat_load 4467 - wavefront - local 4468 - generic 4469 load atomic acquire - workgroup - global 1. buffer/global_load 1. buffer/global_load glc=1 4470 4471 - If CU wavefront execution 4472 mode, omit glc=1. 4473 4474 2. s_waitcnt vmcnt(0) 4475 4476 - If CU wavefront execution 4477 mode, omit. 4478 - Must happen before 4479 the following buffer_gl0_inv 4480 and before any following 4481 global/generic 4482 load/load 4483 atomic/store/store 4484 atomic/atomicrmw. 4485 4486 3. buffer_gl0_inv 4487 4488 - If CU wavefront execution 4489 mode, omit. 4490 - Ensures that 4491 following 4492 loads will not see 4493 stale data. 4494 4495 load atomic acquire - workgroup - local 1. ds_load 1. ds_load 4496 2. s_waitcnt lgkmcnt(0) 2. s_waitcnt lgkmcnt(0) 4497 4498 - If OpenCL, omit. - If OpenCL, omit. 4499 - Must happen before - Must happen before 4500 any following the following buffer_gl0_inv 4501 global/generic and before any following 4502 load/load global/generic load/load 4503 atomic/store/store atomic/store/store 4504 atomic/atomicrmw. atomic/atomicrmw. 4505 - Ensures any - Ensures any 4506 following global following global 4507 data read is no data read is no 4508 older than the load older than the load 4509 atomic value being atomic value being 4510 acquired. acquired. 4511 4512 3. buffer_gl0_inv 4513 4514 - If CU wavefront execution 4515 mode, omit. 4516 - If OpenCL, omit. 4517 - Ensures that 4518 following 4519 loads will not see 4520 stale data. 4521 4522 load atomic acquire - workgroup - generic 1. flat_load 1. flat_load glc=1 4523 4524 - If CU wavefront execution 4525 mode, omit glc=1. 4526 4527 2. s_waitcnt lgkmcnt(0) 2. s_waitcnt lgkmcnt(0) & 4528 vmcnt(0) 4529 4530 - If CU wavefront execution 4531 mode, omit vmcnt(0). 4532 - If OpenCL, omit. - If OpenCL, omit 4533 lgkmcnt(0). 4534 - Must happen before - Must happen before 4535 any following the following 4536 global/generic buffer_gl0_inv and any 4537 load/load following global/generic 4538 atomic/store/store load/load 4539 atomic/atomicrmw. atomic/store/store 4540 atomic/atomicrmw. 4541 - Ensures any - Ensures any 4542 following global following global 4543 data read is no data read is no 4544 older than the load older than the load 4545 atomic value being atomic value being 4546 acquired. acquired. 4547 4548 3. buffer_gl0_inv 4549 4550 - If CU wavefront execution 4551 mode, omit. 4552 - Ensures that 4553 following 4554 loads will not see 4555 stale data. 4556 4557 load atomic acquire - agent - global 1. buffer/global_load 1. buffer/global_load 4558 - system glc=1 glc=1 dlc=1 4559 2. s_waitcnt vmcnt(0) 2. s_waitcnt vmcnt(0) 4560 4561 - Must happen before - Must happen before 4562 following following 4563 buffer_wbinvl1_vol. buffer_gl*_inv. 4564 - Ensures the load - Ensures the load 4565 has completed has completed 4566 before invalidating before invalidating 4567 the cache. the caches. 4568 4569 3. buffer_wbinvl1_vol 3. buffer_gl0_inv; 4570 buffer_gl1_inv 4571 4572 - Must happen before - Must happen before 4573 any following any following 4574 global/generic global/generic 4575 load/load load/load 4576 atomic/atomicrmw. atomic/atomicrmw. 4577 - Ensures that - Ensures that 4578 following following 4579 loads will not see loads will not see 4580 stale global data. stale global data. 4581 4582 load atomic acquire - agent - generic 1. flat_load glc=1 1. flat_load glc=1 dlc=1 4583 - system 2. s_waitcnt vmcnt(0) & 2. s_waitcnt vmcnt(0) & 4584 lgkmcnt(0) lgkmcnt(0) 4585 4586 - If OpenCL omit - If OpenCL omit 4587 lgkmcnt(0). lgkmcnt(0). 4588 - Must happen before - Must happen before 4589 following following 4590 buffer_wbinvl1_vol. buffer_gl*_invl. 4591 - Ensures the flat_load - Ensures the flat_load 4592 has completed has completed 4593 before invalidating before invalidating 4594 the cache. the caches. 4595 4596 3. buffer_wbinvl1_vol 3. buffer_gl0_inv; 4597 buffer_gl1_inv 4598 4599 - Must happen before - Must happen before 4600 any following any following 4601 global/generic global/generic 4602 load/load load/load 4603 atomic/atomicrmw. atomic/atomicrmw. 4604 - Ensures that - Ensures that 4605 following loads following loads 4606 will not see stale will not see stale 4607 global data. global data. 4608 4609 atomicrmw acquire - singlethread - global 1. buffer/global/ds/flat_atomic 1. buffer/global/ds/flat_atomic 4610 - wavefront - local 4611 - generic 4612 atomicrmw acquire - workgroup - global 1. buffer/global_atomic 1. buffer/global_atomic 4613 2. s_waitcnt vm/vscnt(0) 4614 4615 - If CU wavefront execution 4616 mode, omit. 4617 - Use vmcnt(0) if atomic with 4618 return and vscnt(0) if 4619 atomic with no-return. 4620 - Must happen before 4621 the following buffer_gl0_inv 4622 and before any following 4623 global/generic 4624 load/load 4625 atomic/store/store 4626 atomic/atomicrmw. 4627 4628 3. buffer_gl0_inv 4629 4630 - If CU wavefront execution 4631 mode, omit. 4632 - Ensures that 4633 following 4634 loads will not see 4635 stale data. 4636 4637 atomicrmw acquire - workgroup - local 1. ds_atomic 1. ds_atomic 4638 2. waitcnt lgkmcnt(0) 2. waitcnt lgkmcnt(0) 4639 4640 - If OpenCL, omit. - If OpenCL, omit. 4641 - Must happen before - Must happen before 4642 any following the following 4643 global/generic buffer_gl0_inv. 4644 load/load 4645 atomic/store/store 4646 atomic/atomicrmw. 4647 - Ensures any - Ensures any 4648 following global following global 4649 data read is no data read is no 4650 older than the older than the 4651 atomicrmw value atomicrmw value 4652 being acquired. being acquired. 4653 4654 3. buffer_gl0_inv 4655 4656 - If OpenCL omit. 4657 - Ensures that 4658 following 4659 loads will not see 4660 stale data. 4661 4662 atomicrmw acquire - workgroup - generic 1. flat_atomic 1. flat_atomic 4663 2. waitcnt lgkmcnt(0) 2. waitcnt lgkmcnt(0) & 4664 vm/vscnt(0) 4665 4666 - If CU wavefront execution 4667 mode, omit vm/vscnt(0). 4668 - If OpenCL, omit. - If OpenCL, omit 4669 waitcnt lgkmcnt(0). 4670 - Use vmcnt(0) if atomic with 4671 return and vscnt(0) if 4672 atomic with no-return. 4673 - Must happen before - Must happen before 4674 any following the following 4675 global/generic buffer_gl0_inv. 4676 load/load 4677 atomic/store/store 4678 atomic/atomicrmw. 4679 - Ensures any - Ensures any 4680 following global following global 4681 data read is no data read is no 4682 older than the older than the 4683 atomicrmw value atomicrmw value 4684 being acquired. being acquired. 4685 4686 3. buffer_gl0_inv 4687 4688 - If CU wavefront execution 4689 mode, omit. 4690 - Ensures that 4691 following 4692 loads will not see 4693 stale data. 4694 4695 atomicrmw acquire - agent - global 1. buffer/global_atomic 1. buffer/global_atomic 4696 - system 2. s_waitcnt vmcnt(0) 2. s_waitcnt vm/vscnt(0) 4697 4698 - Use vmcnt(0) if atomic with 4699 return and vscnt(0) if 4700 atomic with no-return. 4701 waitcnt lgkmcnt(0). 4702 - Must happen before - Must happen before 4703 following following 4704 buffer_wbinvl1_vol. buffer_gl*_inv. 4705 - Ensures the - Ensures the 4706 atomicrmw has atomicrmw has 4707 completed before completed before 4708 invalidating the invalidating the 4709 cache. caches. 4710 4711 3. buffer_wbinvl1_vol 3. buffer_gl0_inv; 4712 buffer_gl1_inv 4713 4714 - Must happen before - Must happen before 4715 any following any following 4716 global/generic global/generic 4717 load/load load/load 4718 atomic/atomicrmw. atomic/atomicrmw. 4719 - Ensures that - Ensures that 4720 following loads following loads 4721 will not see stale will not see stale 4722 global data. global data. 4723 4724 atomicrmw acquire - agent - generic 1. flat_atomic 1. flat_atomic 4725 - system 2. s_waitcnt vmcnt(0) & 2. s_waitcnt vm/vscnt(0) & 4726 lgkmcnt(0) lgkmcnt(0) 4727 4728 - If OpenCL, omit - If OpenCL, omit 4729 lgkmcnt(0). lgkmcnt(0). 4730 - Use vmcnt(0) if atomic with 4731 return and vscnt(0) if 4732 atomic with no-return. 4733 - Must happen before - Must happen before 4734 following following 4735 buffer_wbinvl1_vol. buffer_gl*_inv. 4736 - Ensures the - Ensures the 4737 atomicrmw has atomicrmw has 4738 completed before completed before 4739 invalidating the invalidating the 4740 cache. caches. 4741 4742 3. buffer_wbinvl1_vol 3. buffer_gl0_inv; 4743 buffer_gl1_inv 4744 4745 - Must happen before - Must happen before 4746 any following any following 4747 global/generic global/generic 4748 load/load load/load 4749 atomic/atomicrmw. atomic/atomicrmw. 4750 - Ensures that - Ensures that 4751 following loads following loads 4752 will not see stale will not see stale 4753 global data. global data. 4754 4755 fence acquire - singlethread *none* *none* *none* 4756 - wavefront 4757 fence acquire - workgroup *none* 1. s_waitcnt lgkmcnt(0) 1. s_waitcnt lgkmcnt(0) & 4758 vmcnt(0) & vscnt(0) 4759 4760 - If CU wavefront execution 4761 mode, omit vmcnt(0) and 4762 vscnt(0). 4763 - If OpenCL and - If OpenCL and 4764 address space is address space is 4765 not generic, omit. not generic, omit 4766 lgkmcnt(0). 4767 - If OpenCL and 4768 address space is 4769 local, omit 4770 vmcnt(0) and vscnt(0). 4771 - However, since LLVM - However, since LLVM 4772 currently has no currently has no 4773 address space on address space on 4774 the fence need to the fence need to 4775 conservatively conservatively 4776 always generate. If always generate. If 4777 fence had an fence had an 4778 address space then address space then 4779 set to address set to address 4780 space of OpenCL space of OpenCL 4781 fence flag, or to fence flag, or to 4782 generic if both generic if both 4783 local and global local and global 4784 flags are flags are 4785 specified. specified. 4786 - Must happen after 4787 any preceding 4788 local/generic load 4789 atomic/atomicrmw 4790 with an equal or 4791 wider sync scope 4792 and memory ordering 4793 stronger than 4794 unordered (this is 4795 termed the 4796 fence-paired-atomic). 4797 - Must happen before 4798 any following 4799 global/generic 4800 load/load 4801 atomic/store/store 4802 atomic/atomicrmw. 4803 - Ensures any 4804 following global 4805 data read is no 4806 older than the 4807 value read by the 4808 fence-paired-atomic. 4809 - Could be split into 4810 separate s_waitcnt 4811 vmcnt(0), s_waitcnt 4812 vscnt(0) and s_waitcnt 4813 lgkmcnt(0) to allow 4814 them to be 4815 independently moved 4816 according to the 4817 following rules. 4818 - s_waitcnt vmcnt(0) 4819 must happen after 4820 any preceding 4821 global/generic load 4822 atomic/ 4823 atomicrmw-with-return-value 4824 with an equal or 4825 wider sync scope 4826 and memory ordering 4827 stronger than 4828 unordered (this is 4829 termed the 4830 fence-paired-atomic). 4831 - s_waitcnt vscnt(0) 4832 must happen after 4833 any preceding 4834 global/generic 4835 atomicrmw-no-return-value 4836 with an equal or 4837 wider sync scope 4838 and memory ordering 4839 stronger than 4840 unordered (this is 4841 termed the 4842 fence-paired-atomic). 4843 - s_waitcnt lgkmcnt(0) 4844 must happen after 4845 any preceding 4846 local/generic load 4847 atomic/atomicrmw 4848 with an equal or 4849 wider sync scope 4850 and memory ordering 4851 stronger than 4852 unordered (this is 4853 termed the 4854 fence-paired-atomic). 4855 - Must happen before 4856 the following 4857 buffer_gl0_inv. 4858 - Ensures that the 4859 fence-paired atomic 4860 has completed 4861 before invalidating 4862 the 4863 cache. Therefore 4864 any following 4865 locations read must 4866 be no older than 4867 the value read by 4868 the 4869 fence-paired-atomic. 4870 4871 3. buffer_gl0_inv 4872 4873 - If CU wavefront execution 4874 mode, omit. 4875 - Ensures that 4876 following 4877 loads will not see 4878 stale data. 4879 4880 fence acquire - agent *none* 1. s_waitcnt lgkmcnt(0) & 1. s_waitcnt lgkmcnt(0) & 4881 - system vmcnt(0) vmcnt(0) & vscnt(0) 4882 4883 - If OpenCL and - If OpenCL and 4884 address space is address space is 4885 not generic, omit not generic, omit 4886 lgkmcnt(0). lgkmcnt(0). 4887 - If OpenCL and 4888 address space is 4889 local, omit 4890 vmcnt(0) and vscnt(0). 4891 - However, since LLVM - However, since LLVM 4892 currently has no currently has no 4893 address space on address space on 4894 the fence need to the fence need to 4895 conservatively conservatively 4896 always generate always generate 4897 (see comment for (see comment for 4898 previous fence). previous fence). 4899 - Could be split into 4900 separate s_waitcnt 4901 vmcnt(0) and 4902 s_waitcnt 4903 lgkmcnt(0) to allow 4904 them to be 4905 independently moved 4906 according to the 4907 following rules. 4908 - s_waitcnt vmcnt(0) 4909 must happen after 4910 any preceding 4911 global/generic load 4912 atomic/atomicrmw 4913 with an equal or 4914 wider sync scope 4915 and memory ordering 4916 stronger than 4917 unordered (this is 4918 termed the 4919 fence-paired-atomic). 4920 - s_waitcnt lgkmcnt(0) 4921 must happen after 4922 any preceding 4923 local/generic load 4924 atomic/atomicrmw 4925 with an equal or 4926 wider sync scope 4927 and memory ordering 4928 stronger than 4929 unordered (this is 4930 termed the 4931 fence-paired-atomic). 4932 - Must happen before 4933 the following 4934 buffer_wbinvl1_vol. 4935 - Ensures that the 4936 fence-paired atomic 4937 has completed 4938 before invalidating 4939 the 4940 cache. Therefore 4941 any following 4942 locations read must 4943 be no older than 4944 the value read by 4945 the 4946 fence-paired-atomic. 4947 - Could be split into 4948 separate s_waitcnt 4949 vmcnt(0), s_waitcnt 4950 vscnt(0) and s_waitcnt 4951 lgkmcnt(0) to allow 4952 them to be 4953 independently moved 4954 according to the 4955 following rules. 4956 - s_waitcnt vmcnt(0) 4957 must happen after 4958 any preceding 4959 global/generic load 4960 atomic/ 4961 atomicrmw-with-return-value 4962 with an equal or 4963 wider sync scope 4964 and memory ordering 4965 stronger than 4966 unordered (this is 4967 termed the 4968 fence-paired-atomic). 4969 - s_waitcnt vscnt(0) 4970 must happen after 4971 any preceding 4972 global/generic 4973 atomicrmw-no-return-value 4974 with an equal or 4975 wider sync scope 4976 and memory ordering 4977 stronger than 4978 unordered (this is 4979 termed the 4980 fence-paired-atomic). 4981 - s_waitcnt lgkmcnt(0) 4982 must happen after 4983 any preceding 4984 local/generic load 4985 atomic/atomicrmw 4986 with an equal or 4987 wider sync scope 4988 and memory ordering 4989 stronger than 4990 unordered (this is 4991 termed the 4992 fence-paired-atomic). 4993 - Must happen before 4994 the following 4995 buffer_gl*_inv. 4996 - Ensures that the 4997 fence-paired atomic 4998 has completed 4999 before invalidating 5000 the 5001 caches. Therefore 5002 any following 5003 locations read must 5004 be no older than 5005 the value read by 5006 the 5007 fence-paired-atomic. 5008 5009 2. buffer_wbinvl1_vol 2. buffer_gl0_inv; 5010 buffer_gl1_inv 5011 5012 - Must happen before any - Must happen before any 5013 following global/generic following global/generic 5014 load/load load/load 5015 atomic/store/store atomic/store/store 5016 atomic/atomicrmw. atomic/atomicrmw. 5017 - Ensures that - Ensures that 5018 following loads following loads 5019 will not see stale will not see stale 5020 global data. global data. 5021 5022 **Release Atomic** 5023 --------------------------------------------------------------------------------------------------------------------- 5024 store atomic release - singlethread - global 1. buffer/global/ds/flat_store 1. buffer/global/ds/flat_store 5025 - wavefront - local 5026 - generic 5027 store atomic release - workgroup - global 1. s_waitcnt lgkmcnt(0) 1. s_waitcnt lgkmcnt(0) & 5028 vmcnt(0) & vscnt(0) 5029 5030 - If CU wavefront execution 5031 mode, omit vmcnt(0) and 5032 vscnt(0). 5033 - If OpenCL, omit. - If OpenCL, omit 5034 lgkmcnt(0). 5035 - Must happen after 5036 any preceding 5037 local/generic 5038 load/store/load 5039 atomic/store 5040 atomic/atomicrmw. 5041 - Could be split into 5042 separate s_waitcnt 5043 vmcnt(0), s_waitcnt 5044 vscnt(0) and s_waitcnt 5045 lgkmcnt(0) to allow 5046 them to be 5047 independently moved 5048 according to the 5049 following rules. 5050 - s_waitcnt vmcnt(0) 5051 must happen after 5052 any preceding 5053 global/generic load/load 5054 atomic/ 5055 atomicrmw-with-return-value. 5056 - s_waitcnt vscnt(0) 5057 must happen after 5058 any preceding 5059 global/generic 5060 store/store 5061 atomic/ 5062 atomicrmw-no-return-value. 5063 - s_waitcnt lgkmcnt(0) 5064 must happen after 5065 any preceding 5066 local/generic 5067 load/store/load 5068 atomic/store 5069 atomic/atomicrmw. 5070 - Must happen before - Must happen before 5071 the following the following 5072 store. store. 5073 - Ensures that all - Ensures that all 5074 memory operations memory operations 5075 to local have have 5076 completed before completed before 5077 performing the performing the 5078 store that is being store that is being 5079 released. released. 5080 5081 2. buffer/global_store 2. buffer/global_store 5082 store atomic release - workgroup - local 1. waitcnt vmcnt(0) & vscnt(0) 5083 5084 - If CU wavefront execution 5085 mode, omit. 5086 - If OpenCL, omit. 5087 - Could be split into 5088 separate s_waitcnt 5089 vmcnt(0) and s_waitcnt 5090 vscnt(0) to allow 5091 them to be 5092 independently moved 5093 according to the 5094 following rules. 5095 - s_waitcnt vmcnt(0) 5096 must happen after 5097 any preceding 5098 global/generic load/load 5099 atomic/ 5100 atomicrmw-with-return-value. 5101 - s_waitcnt vscnt(0) 5102 must happen after 5103 any preceding 5104 global/generic 5105 store/store atomic/ 5106 atomicrmw-no-return-value. 5107 - Must happen before 5108 the following 5109 store. 5110 - Ensures that all 5111 global memory 5112 operations have 5113 completed before 5114 performing the 5115 store that is being 5116 released. 5117 5118 1. ds_store 2. ds_store 5119 store atomic release - workgroup - generic 1. s_waitcnt lgkmcnt(0) 1. s_waitcnt lgkmcnt(0) & 5120 vmcnt(0) & vscnt(0) 5121 5122 - If CU wavefront execution 5123 mode, omit vmcnt(0) and 5124 vscnt(0). 5125 - If OpenCL, omit. - If OpenCL, omit 5126 lgkmcnt(0). 5127 - Must happen after 5128 any preceding 5129 local/generic 5130 load/store/load 5131 atomic/store 5132 atomic/atomicrmw. 5133 - Could be split into 5134 separate s_waitcnt 5135 vmcnt(0), s_waitcnt 5136 vscnt(0) and s_waitcnt 5137 lgkmcnt(0) to allow 5138 them to be 5139 independently moved 5140 according to the 5141 following rules. 5142 - s_waitcnt vmcnt(0) 5143 must happen after 5144 any preceding 5145 global/generic load/load 5146 atomic/ 5147 atomicrmw-with-return-value. 5148 - s_waitcnt vscnt(0) 5149 must happen after 5150 any preceding 5151 global/generic 5152 store/store 5153 atomic/ 5154 atomicrmw-no-return-value. 5155 - s_waitcnt lgkmcnt(0) 5156 must happen after 5157 any preceding 5158 local/generic 5159 load/store/load 5160 atomic/store 5161 atomic/atomicrmw. 5162 - Must happen before - Must happen before 5163 the following the following 5164 store. store. 5165 - Ensures that all - Ensures that all 5166 memory operations memory operations 5167 to local have have 5168 completed before completed before 5169 performing the performing the 5170 store that is being store that is being 5171 released. released. 5172 5173 2. flat_store 2. flat_store 5174 store atomic release - agent - global 1. s_waitcnt lgkmcnt(0) & 1. s_waitcnt lgkmcnt(0) & 5175 - system - generic vmcnt(0) vmcnt(0) & vscnt(0) 5176 5177 - If OpenCL, omit - If OpenCL, omit 5178 lgkmcnt(0). lgkmcnt(0). 5179 - Could be split into - Could be split into 5180 separate s_waitcnt separate s_waitcnt 5181 vmcnt(0) and vmcnt(0), s_waitcnt vscnt(0) 5182 s_waitcnt and s_waitcnt 5183 lgkmcnt(0) to allow lgkmcnt(0) to allow 5184 them to be them to be 5185 independently moved independently moved 5186 according to the according to the 5187 following rules. following rules. 5188 - s_waitcnt vmcnt(0) - s_waitcnt vmcnt(0) 5189 must happen after must happen after 5190 any preceding any preceding 5191 global/generic global/generic 5192 load/store/load load/load 5193 atomic/store atomic/ 5194 atomic/atomicrmw. atomicrmw-with-return-value. 5195 - s_waitcnt vscnt(0) 5196 must happen after 5197 any preceding 5198 global/generic 5199 store/store atomic/ 5200 atomicrmw-no-return-value. 5201 - s_waitcnt lgkmcnt(0) - s_waitcnt lgkmcnt(0) 5202 must happen after must happen after 5203 any preceding any preceding 5204 local/generic local/generic 5205 load/store/load load/store/load 5206 atomic/store atomic/store 5207 atomic/atomicrmw. atomic/atomicrmw. 5208 - Must happen before - Must happen before 5209 the following the following 5210 store. store. 5211 - Ensures that all - Ensures that all 5212 memory operations memory operations 5213 to memory have to memory have 5214 completed before completed before 5215 performing the performing the 5216 store that is being store that is being 5217 released. released. 5218 5219 2. buffer/global/flat_store 2. buffer/global/flat_store 5220 atomicrmw release - singlethread - global 1. buffer/global/ds/flat_atomic 1. buffer/global/ds/flat_atomic 5221 - wavefront - local 5222 - generic 5223 atomicrmw release - workgroup - global 1. s_waitcnt lgkmcnt(0) 1. s_waitcnt lgkmcnt(0) & 5224 vmcnt(0) & vscnt(0) 5225 5226 - If CU wavefront execution 5227 mode, omit vmcnt(0) and 5228 vscnt(0). 5229 - If OpenCL, omit. 5230 5231 - Must happen after 5232 any preceding 5233 local/generic 5234 load/store/load 5235 atomic/store 5236 atomic/atomicrmw. 5237 - Could be split into 5238 separate s_waitcnt 5239 vmcnt(0), s_waitcnt 5240 vscnt(0) and s_waitcnt 5241 lgkmcnt(0) to allow 5242 them to be 5243 independently moved 5244 according to the 5245 following rules. 5246 - s_waitcnt vmcnt(0) 5247 must happen after 5248 any preceding 5249 global/generic load/load 5250 atomic/ 5251 atomicrmw-with-return-value. 5252 - s_waitcnt vscnt(0) 5253 must happen after 5254 any preceding 5255 global/generic 5256 store/store 5257 atomic/ 5258 atomicrmw-no-return-value. 5259 - s_waitcnt lgkmcnt(0) 5260 must happen after 5261 any preceding 5262 local/generic 5263 load/store/load 5264 atomic/store 5265 atomic/atomicrmw. 5266 - Must happen before - Must happen before 5267 the following the following 5268 atomicrmw. atomicrmw. 5269 - Ensures that all - Ensures that all 5270 memory operations memory operations 5271 to local have have 5272 completed before completed before 5273 performing the performing the 5274 atomicrmw that is atomicrmw that is 5275 being released. being released. 5276 5277 2. buffer/global_atomic 2. buffer/global_atomic 5278 atomicrmw release - workgroup - local 1. waitcnt vmcnt(0) & vscnt(0) 5279 5280 - If CU wavefront execution 5281 mode, omit. 5282 - If OpenCL, omit. 5283 - Could be split into 5284 separate s_waitcnt 5285 vmcnt(0) and s_waitcnt 5286 vscnt(0) to allow 5287 them to be 5288 independently moved 5289 according to the 5290 following rules. 5291 - s_waitcnt vmcnt(0) 5292 must happen after 5293 any preceding 5294 global/generic load/load 5295 atomic/ 5296 atomicrmw-with-return-value. 5297 - s_waitcnt vscnt(0) 5298 must happen after 5299 any preceding 5300 global/generic 5301 store/store atomic/ 5302 atomicrmw-no-return-value. 5303 - Must happen before 5304 the following 5305 store. 5306 - Ensures that all 5307 global memory 5308 operations have 5309 completed before 5310 performing the 5311 store that is being 5312 released. 5313 5314 1. ds_atomic 2. ds_atomic 5315 atomicrmw release - workgroup - generic 1. s_waitcnt lgkmcnt(0) 1. s_waitcnt lgkmcnt(0) & 5316 vmcnt(0) & vscnt(0) 5317 5318 - If CU wavefront execution 5319 mode, omit vmcnt(0) and 5320 vscnt(0). 5321 - If OpenCL, omit. - If OpenCL, omit 5322 waitcnt lgkmcnt(0). 5323 - Must happen after 5324 any preceding 5325 local/generic 5326 load/store/load 5327 atomic/store 5328 atomic/atomicrmw. 5329 - Could be split into 5330 separate s_waitcnt 5331 vmcnt(0), s_waitcnt 5332 vscnt(0) and s_waitcnt 5333 lgkmcnt(0) to allow 5334 them to be 5335 independently moved 5336 according to the 5337 following rules. 5338 - s_waitcnt vmcnt(0) 5339 must happen after 5340 any preceding 5341 global/generic load/load 5342 atomic/ 5343 atomicrmw-with-return-value. 5344 - s_waitcnt vscnt(0) 5345 must happen after 5346 any preceding 5347 global/generic 5348 store/store 5349 atomic/ 5350 atomicrmw-no-return-value. 5351 - s_waitcnt lgkmcnt(0) 5352 must happen after 5353 any preceding 5354 local/generic 5355 load/store/load 5356 atomic/store 5357 atomic/atomicrmw. 5358 - Must happen before - Must happen before 5359 the following the following 5360 atomicrmw. atomicrmw. 5361 - Ensures that all - Ensures that all 5362 memory operations memory operations 5363 to local have have 5364 completed before completed before 5365 performing the performing the 5366 atomicrmw that is atomicrmw that is 5367 being released. being released. 5368 5369 2. flat_atomic 2. flat_atomic 5370 atomicrmw release - agent - global 1. s_waitcnt lgkmcnt(0) & 1. s_waitcnt lkkmcnt(0) & 5371 - system - generic vmcnt(0) vmcnt(0) & vscnt(0) 5372 5373 - If OpenCL, omit - If OpenCL, omit 5374 lgkmcnt(0). lgkmcnt(0). 5375 - Could be split into - Could be split into 5376 separate s_waitcnt separate s_waitcnt 5377 vmcnt(0) and vmcnt(0), s_waitcnt 5378 s_waitcnt vscnt(0) and s_waitcnt 5379 lgkmcnt(0) to allow lgkmcnt(0) to allow 5380 them to be them to be 5381 independently moved independently moved 5382 according to the according to the 5383 following rules. following rules. 5384 - s_waitcnt vmcnt(0) - s_waitcnt vmcnt(0) 5385 must happen after must happen after 5386 any preceding any preceding 5387 global/generic global/generic 5388 load/store/load load/load atomic/ 5389 atomic/store atomicrmw-with-return-value. 5390 atomic/atomicrmw. 5391 - s_waitcnt vscnt(0) 5392 must happen after 5393 any preceding 5394 global/generic 5395 store/store atomic/ 5396 atomicrmw-no-return-value. 5397 - s_waitcnt lgkmcnt(0) - s_waitcnt lgkmcnt(0) 5398 must happen after must happen after 5399 any preceding any preceding 5400 local/generic local/generic 5401 load/store/load load/store/load 5402 atomic/store atomic/store 5403 atomic/atomicrmw. atomic/atomicrmw. 5404 - Must happen before - Must happen before 5405 the following the following 5406 atomicrmw. atomicrmw. 5407 - Ensures that all - Ensures that all 5408 memory operations memory operations 5409 to global and local to global and local 5410 have completed have completed 5411 before performing before performing 5412 the atomicrmw that the atomicrmw that 5413 is being released. is being released. 5414 5415 2. buffer/global/flat_atomic 2. buffer/global/flat_atomic 5416 fence release - singlethread *none* *none* *none* 5417 - wavefront 5418 fence release - workgroup *none* 1. s_waitcnt lgkmcnt(0) 1. s_waitcnt lgkmcnt(0) & 5419 vmcnt(0) & vscnt(0) 5420 5421 - If CU wavefront execution 5422 mode, omit vmcnt(0) and 5423 vscnt(0). 5424 - If OpenCL and - If OpenCL and 5425 address space is address space is 5426 not generic, omit. not generic, omit 5427 lgkmcnt(0). 5428 - If OpenCL and 5429 address space is 5430 local, omit 5431 vmcnt(0) and vscnt(0). 5432 - However, since LLVM - However, since LLVM 5433 currently has no currently has no 5434 address space on address space on 5435 the fence need to the fence need to 5436 conservatively conservatively 5437 always generate. If always generate. If 5438 fence had an fence had an 5439 address space then address space then 5440 set to address set to address 5441 space of OpenCL space of OpenCL 5442 fence flag, or to fence flag, or to 5443 generic if both generic if both 5444 local and global local and global 5445 flags are flags are 5446 specified. specified. 5447 - Must happen after 5448 any preceding 5449 local/generic 5450 load/load 5451 atomic/store/store 5452 atomic/atomicrmw. 5453 - Could be split into 5454 separate s_waitcnt 5455 vmcnt(0), s_waitcnt 5456 vscnt(0) and s_waitcnt 5457 lgkmcnt(0) to allow 5458 them to be 5459 independently moved 5460 according to the 5461 following rules. 5462 - s_waitcnt vmcnt(0) 5463 must happen after 5464 any preceding 5465 global/generic 5466 load/load 5467 atomic/ 5468 atomicrmw-with-return-value. 5469 - s_waitcnt vscnt(0) 5470 must happen after 5471 any preceding 5472 global/generic 5473 store/store atomic/ 5474 atomicrmw-no-return-value. 5475 - s_waitcnt lgkmcnt(0) 5476 must happen after 5477 any preceding 5478 local/generic 5479 load/store/load 5480 atomic/store atomic/ 5481 atomicrmw. 5482 - Must happen before - Must happen before 5483 any following store any following store 5484 atomic/atomicrmw atomic/atomicrmw 5485 with an equal or with an equal or 5486 wider sync scope wider sync scope 5487 and memory ordering and memory ordering 5488 stronger than stronger than 5489 unordered (this is unordered (this is 5490 termed the termed the 5491 fence-paired-atomic). fence-paired-atomic). 5492 - Ensures that all - Ensures that all 5493 memory operations memory operations 5494 to local have have 5495 completed before completed before 5496 performing the performing the 5497 following following 5498 fence-paired-atomic. fence-paired-atomic. 5499 5500 fence release - agent *none* 1. s_waitcnt lgkmcnt(0) & 1. s_waitcnt lgkmcnt(0) & 5501 - system vmcnt(0) vmcnt(0) & vscnt(0) 5502 5503 - If OpenCL and - If OpenCL and 5504 address space is address space is 5505 not generic, omit not generic, omit 5506 lgkmcnt(0). lgkmcnt(0). 5507 - If OpenCL and - If OpenCL and 5508 address space is address space is 5509 local, omit local, omit 5510 vmcnt(0). vmcnt(0) and vscnt(0). 5511 - However, since LLVM - However, since LLVM 5512 currently has no currently has no 5513 address space on address space on 5514 the fence need to the fence need to 5515 conservatively conservatively 5516 always generate. If always generate. If 5517 fence had an fence had an 5518 address space then address space then 5519 set to address set to address 5520 space of OpenCL space of OpenCL 5521 fence flag, or to fence flag, or to 5522 generic if both generic if both 5523 local and global local and global 5524 flags are flags are 5525 specified. specified. 5526 - Could be split into - Could be split into 5527 separate s_waitcnt separate s_waitcnt 5528 vmcnt(0) and vmcnt(0), s_waitcnt 5529 s_waitcnt vscnt(0) and s_waitcnt 5530 lgkmcnt(0) to allow lgkmcnt(0) to allow 5531 them to be them to be 5532 independently moved independently moved 5533 according to the according to the 5534 following rules. following rules. 5535 - s_waitcnt vmcnt(0) - s_waitcnt vmcnt(0) 5536 must happen after must happen after 5537 any preceding any preceding 5538 global/generic global/generic 5539 load/store/load load/load atomic/ 5540 atomic/store atomicrmw-with-return-value. 5541 atomic/atomicrmw. 5542 - s_waitcnt vscnt(0) 5543 must happen after 5544 any preceding 5545 global/generic 5546 store/store atomic/ 5547 atomicrmw-no-return-value. 5548 - s_waitcnt lgkmcnt(0) - s_waitcnt lgkmcnt(0) 5549 must happen after must happen after 5550 any preceding any preceding 5551 local/generic local/generic 5552 load/store/load load/store/load 5553 atomic/store atomic/store 5554 atomic/atomicrmw. atomic/atomicrmw. 5555 - Must happen before - Must happen before 5556 any following store any following store 5557 atomic/atomicrmw atomic/atomicrmw 5558 with an equal or with an equal or 5559 wider sync scope wider sync scope 5560 and memory ordering and memory ordering 5561 stronger than stronger than 5562 unordered (this is unordered (this is 5563 termed the termed the 5564 fence-paired-atomic). fence-paired-atomic). 5565 - Ensures that all - Ensures that all 5566 memory operations memory operations 5567 have have 5568 completed before completed before 5569 performing the performing the 5570 following following 5571 fence-paired-atomic. fence-paired-atomic. 5572 5573 **Acquire-Release Atomic** 5574 --------------------------------------------------------------------------------------------------------------------- 5575 atomicrmw acq_rel - singlethread - global 1. buffer/global/ds/flat_atomic 1. buffer/global/ds/flat_atomic 5576 - wavefront - local 5577 - generic 5578 atomicrmw acq_rel - workgroup - global 1. s_waitcnt lgkmcnt(0) 1. s_waitcnt lgkmcnt(0) & 5579 vmcnt(0) & vscnt(0) 5580 5581 - If CU wavefront execution 5582 mode, omit vmcnt(0) and 5583 vscnt(0). 5584 - If OpenCL, omit. - If OpenCL, omit 5585 s_waitcnt lgkmcnt(0). 5586 - Must happen after - Must happen after 5587 any preceding any preceding 5588 local/generic local/generic 5589 load/store/load load/store/load 5590 atomic/store atomic/store 5591 atomic/atomicrmw. atomic/atomicrmw. 5592 - Could be split into 5593 separate s_waitcnt 5594 vmcnt(0), s_waitcnt 5595 vscnt(0) and s_waitcnt 5596 lgkmcnt(0) to allow 5597 them to be 5598 independently moved 5599 according to the 5600 following rules. 5601 - s_waitcnt vmcnt(0) 5602 must happen after 5603 any preceding 5604 global/generic load/load 5605 atomic/ 5606 atomicrmw-with-return-value. 5607 - s_waitcnt vscnt(0) 5608 must happen after 5609 any preceding 5610 global/generic 5611 store/store 5612 atomic/ 5613 atomicrmw-no-return-value. 5614 - s_waitcnt lgkmcnt(0) 5615 must happen after 5616 any preceding 5617 local/generic 5618 load/store/load 5619 atomic/store 5620 atomic/atomicrmw. 5621 - Must happen before - Must happen before 5622 the following the following 5623 atomicrmw. atomicrmw. 5624 - Ensures that all - Ensures that all 5625 memory operations memory operations 5626 to local have have 5627 completed before completed before 5628 performing the performing the 5629 atomicrmw that is atomicrmw that is 5630 being released. being released. 5631 5632 2. buffer/global_atomic 2. buffer/global_atomic 5633 3. s_waitcnt vm/vscnt(0) 5634 5635 - If CU wavefront execution 5636 mode, omit vm/vscnt(0). 5637 - Use vmcnt(0) if atomic with 5638 return and vscnt(0) if 5639 atomic with no-return. 5640 waitcnt lgkmcnt(0). 5641 - Must happen before 5642 the following 5643 buffer_gl0_inv. 5644 - Ensures any 5645 following global 5646 data read is no 5647 older than the 5648 atomicrmw value 5649 being acquired. 5650 5651 4. buffer_gl0_inv 5652 5653 - If CU wavefront execution 5654 mode, omit. 5655 - Ensures that 5656 following 5657 loads will not see 5658 stale data. 5659 5660 atomicrmw acq_rel - workgroup - local 1. waitcnt vmcnt(0) & vscnt(0) 5661 5662 - If CU wavefront execution 5663 mode, omit. 5664 - If OpenCL, omit. 5665 - Could be split into 5666 separate s_waitcnt 5667 vmcnt(0) and s_waitcnt 5668 vscnt(0) to allow 5669 them to be 5670 independently moved 5671 according to the 5672 following rules. 5673 - s_waitcnt vmcnt(0) 5674 must happen after 5675 any preceding 5676 global/generic load/load 5677 atomic/ 5678 atomicrmw-with-return-value. 5679 - s_waitcnt vscnt(0) 5680 must happen after 5681 any preceding 5682 global/generic 5683 store/store atomic/ 5684 atomicrmw-no-return-value. 5685 - Must happen before 5686 the following 5687 store. 5688 - Ensures that all 5689 global memory 5690 operations have 5691 completed before 5692 performing the 5693 store that is being 5694 released. 5695 5696 1. ds_atomic 2. ds_atomic 5697 2. s_waitcnt lgkmcnt(0) 3. s_waitcnt lgkmcnt(0) 5698 5699 - If OpenCL, omit. - If OpenCL, omit. 5700 - Must happen before - Must happen before 5701 any following the following 5702 global/generic buffer_gl0_inv. 5703 load/load 5704 atomic/store/store 5705 atomic/atomicrmw. 5706 - Ensures any - Ensures any 5707 following global following global 5708 data read is no data read is no 5709 older than the load older than the load 5710 atomic value being atomic value being 5711 acquired. acquired. 5712 5713 4. buffer_gl0_inv 5714 5715 - If CU wavefront execution 5716 mode, omit. 5717 - If OpenCL omit. 5718 - Ensures that 5719 following 5720 loads will not see 5721 stale data. 5722 5723 atomicrmw acq_rel - workgroup - generic 1. s_waitcnt lgkmcnt(0) 1. s_waitcnt lgkmcnt(0) & 5724 vmcnt(0) & vscnt(0) 5725 5726 - If CU wavefront execution 5727 mode, omit vmcnt(0) and 5728 vscnt(0). 5729 - If OpenCL, omit. - If OpenCL, omit 5730 waitcnt lgkmcnt(0). 5731 - Must happen after 5732 any preceding 5733 local/generic 5734 load/store/load 5735 atomic/store 5736 atomic/atomicrmw. 5737 - Could be split into 5738 separate s_waitcnt 5739 vmcnt(0), s_waitcnt 5740 vscnt(0) and s_waitcnt 5741 lgkmcnt(0) to allow 5742 them to be 5743 independently moved 5744 according to the 5745 following rules. 5746 - s_waitcnt vmcnt(0) 5747 must happen after 5748 any preceding 5749 global/generic load/load 5750 atomic/ 5751 atomicrmw-with-return-value. 5752 - s_waitcnt vscnt(0) 5753 must happen after 5754 any preceding 5755 global/generic 5756 store/store 5757 atomic/ 5758 atomicrmw-no-return-value. 5759 - s_waitcnt lgkmcnt(0) 5760 must happen after 5761 any preceding 5762 local/generic 5763 load/store/load 5764 atomic/store 5765 atomic/atomicrmw. 5766 - Must happen before - Must happen before 5767 the following the following 5768 atomicrmw. atomicrmw. 5769 - Ensures that all - Ensures that all 5770 memory operations memory operations 5771 to local have have 5772 completed before completed before 5773 performing the performing the 5774 atomicrmw that is atomicrmw that is 5775 being released. being released. 5776 5777 2. flat_atomic 2. flat_atomic 5778 3. s_waitcnt lgkmcnt(0) 3. s_waitcnt lgkmcnt(0) & 5779 vm/vscnt(0) 5780 5781 - If CU wavefront execution 5782 mode, omit vm/vscnt(0). 5783 - If OpenCL, omit. - If OpenCL, omit 5784 waitcnt lgkmcnt(0). 5785 - Must happen before - Must happen before 5786 any following the following 5787 global/generic buffer_gl0_inv. 5788 load/load 5789 atomic/store/store 5790 atomic/atomicrmw. 5791 - Ensures any - Ensures any 5792 following global following global 5793 data read is no data read is no 5794 older than the load older than the load 5795 atomic value being atomic value being 5796 acquired. acquired. 5797 5798 3. buffer_gl0_inv 5799 5800 - If CU wavefront execution 5801 mode, omit. 5802 - Ensures that 5803 following 5804 loads will not see 5805 stale data. 5806 5807 atomicrmw acq_rel - agent - global 1. s_waitcnt lgkmcnt(0) & 1. s_waitcnt lgkmcnt(0) & 5808 - system vmcnt(0) vmcnt(0) & vscnt(0) 5809 5810 - If OpenCL, omit - If OpenCL, omit 5811 lgkmcnt(0). lgkmcnt(0). 5812 - Could be split into - Could be split into 5813 separate s_waitcnt separate s_waitcnt 5814 vmcnt(0) and vmcnt(0), s_waitcnt 5815 s_waitcnt vscnt(0) and s_waitcnt 5816 lgkmcnt(0) to allow lgkmcnt(0) to allow 5817 them to be them to be 5818 independently moved independently moved 5819 according to the according to the 5820 following rules. following rules. 5821 - s_waitcnt vmcnt(0) - s_waitcnt vmcnt(0) 5822 must happen after must happen after 5823 any preceding any preceding 5824 global/generic global/generic 5825 load/store/load load/load atomic/ 5826 atomic/store atomicrmw-with-return-value. 5827 atomic/atomicrmw. 5828 - s_waitcnt vscnt(0) 5829 must happen after 5830 any preceding 5831 global/generic 5832 store/store atomic/ 5833 atomicrmw-no-return-value. 5834 - s_waitcnt lgkmcnt(0) - s_waitcnt lgkmcnt(0) 5835 must happen after must happen after 5836 any preceding any preceding 5837 local/generic local/generic 5838 load/store/load load/store/load 5839 atomic/store atomic/store 5840 atomic/atomicrmw. atomic/atomicrmw. 5841 - Must happen before - Must happen before 5842 the following the following 5843 atomicrmw. atomicrmw. 5844 - Ensures that all - Ensures that all 5845 memory operations memory operations 5846 to global have to global have 5847 completed before completed before 5848 performing the performing the 5849 atomicrmw that is atomicrmw that is 5850 being released. being released. 5851 5852 2. buffer/global_atomic 2. buffer/global_atomic 5853 3. s_waitcnt vmcnt(0) 3. s_waitcnt vm/vscnt(0) 5854 5855 - Use vmcnt(0) if atomic with 5856 return and vscnt(0) if 5857 atomic with no-return. 5858 waitcnt lgkmcnt(0). 5859 - Must happen before - Must happen before 5860 following following 5861 buffer_wbinvl1_vol. buffer_gl*_inv. 5862 - Ensures the - Ensures the 5863 atomicrmw has atomicrmw has 5864 completed before completed before 5865 invalidating the invalidating the 5866 cache. caches. 5867 5868 4. buffer_wbinvl1_vol 4. buffer_gl0_inv; 5869 buffer_gl1_inv 5870 5871 - Must happen before - Must happen before 5872 any following any following 5873 global/generic global/generic 5874 load/load load/load 5875 atomic/atomicrmw. atomic/atomicrmw. 5876 - Ensures that - Ensures that 5877 following loads following loads 5878 will not see stale will not see stale 5879 global data. global data. 5880 5881 atomicrmw acq_rel - agent - generic 1. s_waitcnt lgkmcnt(0) & 1. s_waitcnt lgkmcnt(0) & 5882 - system vmcnt(0) vmcnt(0) & vscnt(0) 5883 5884 - If OpenCL, omit - If OpenCL, omit 5885 lgkmcnt(0). lgkmcnt(0). 5886 - Could be split into - Could be split into 5887 separate s_waitcnt separate s_waitcnt 5888 vmcnt(0) and vmcnt(0), s_waitcnt 5889 s_waitcnt vscnt(0) and s_waitcnt 5890 lgkmcnt(0) to allow lgkmcnt(0) to allow 5891 them to be them to be 5892 independently moved independently moved 5893 according to the according to the 5894 following rules. following rules. 5895 - s_waitcnt vmcnt(0) - s_waitcnt vmcnt(0) 5896 must happen after must happen after 5897 any preceding any preceding 5898 global/generic global/generic 5899 load/store/load load/load atomic 5900 atomic/store atomicrmw-with-return-value. 5901 atomic/atomicrmw. 5902 - s_waitcnt vscnt(0) 5903 must happen after 5904 any preceding 5905 global/generic 5906 store/store atomic/ 5907 atomicrmw-no-return-value. 5908 - s_waitcnt lgkmcnt(0) - s_waitcnt lgkmcnt(0) 5909 must happen after must happen after 5910 any preceding any preceding 5911 local/generic local/generic 5912 load/store/load load/store/load 5913 atomic/store atomic/store 5914 atomic/atomicrmw. atomic/atomicrmw. 5915 - Must happen before - Must happen before 5916 the following the following 5917 atomicrmw. atomicrmw. 5918 - Ensures that all - Ensures that all 5919 memory operations memory operations 5920 to global have have 5921 completed before completed before 5922 performing the performing the 5923 atomicrmw that is atomicrmw that is 5924 being released. being released. 5925 5926 2. flat_atomic 2. flat_atomic 5927 3. s_waitcnt vmcnt(0) & 3. s_waitcnt vm/vscnt(0) & 5928 lgkmcnt(0) lgkmcnt(0) 5929 5930 - If OpenCL, omit - If OpenCL, omit 5931 lgkmcnt(0). lgkmcnt(0). 5932 - Use vmcnt(0) if atomic with 5933 return and vscnt(0) if 5934 atomic with no-return. 5935 - Must happen before - Must happen before 5936 following following 5937 buffer_wbinvl1_vol. buffer_gl*_inv. 5938 - Ensures the - Ensures the 5939 atomicrmw has atomicrmw has 5940 completed before completed before 5941 invalidating the invalidating the 5942 cache. caches. 5943 5944 4. buffer_wbinvl1_vol 4. buffer_gl0_inv; 5945 buffer_gl1_inv 5946 5947 - Must happen before - Must happen before 5948 any following any following 5949 global/generic global/generic 5950 load/load load/load 5951 atomic/atomicrmw. atomic/atomicrmw. 5952 - Ensures that - Ensures that 5953 following loads following loads 5954 will not see stale will not see stale 5955 global data. global data. 5956 5957 fence acq_rel - singlethread *none* *none* *none* 5958 - wavefront 5959 fence acq_rel - workgroup *none* 1. s_waitcnt lgkmcnt(0) 1. s_waitcnt lgkmcnt(0) & 5960 vmcnt(0) & vscnt(0) 5961 5962 - If CU wavefront execution 5963 mode, omit vmcnt(0) and 5964 vscnt(0). 5965 - If OpenCL and - If OpenCL and 5966 address space is address space is 5967 not generic, omit. not generic, omit 5968 lgkmcnt(0). 5969 - If OpenCL and 5970 address space is 5971 local, omit 5972 vmcnt(0) and vscnt(0). 5973 - However, - However, 5974 since LLVM since LLVM 5975 currently has no currently has no 5976 address space on address space on 5977 the fence need to the fence need to 5978 conservatively conservatively 5979 always generate always generate 5980 (see comment for (see comment for 5981 previous fence). previous fence). 5982 - Must happen after 5983 any preceding 5984 local/generic 5985 load/load 5986 atomic/store/store 5987 atomic/atomicrmw. 5988 - Could be split into 5989 separate s_waitcnt 5990 vmcnt(0), s_waitcnt 5991 vscnt(0) and s_waitcnt 5992 lgkmcnt(0) to allow 5993 them to be 5994 independently moved 5995 according to the 5996 following rules. 5997 - s_waitcnt vmcnt(0) 5998 must happen after 5999 any preceding 6000 global/generic 6001 load/load 6002 atomic/ 6003 atomicrmw-with-return-value. 6004 - s_waitcnt vscnt(0) 6005 must happen after 6006 any preceding 6007 global/generic 6008 store/store atomic/ 6009 atomicrmw-no-return-value. 6010 - s_waitcnt lgkmcnt(0) 6011 must happen after 6012 any preceding 6013 local/generic 6014 load/store/load 6015 atomic/store atomic/ 6016 atomicrmw. 6017 - Must happen before - Must happen before 6018 any following any following 6019 global/generic global/generic 6020 load/load load/load 6021 atomic/store/store atomic/store/store 6022 atomic/atomicrmw. atomic/atomicrmw. 6023 - Ensures that all - Ensures that all 6024 memory operations memory operations 6025 to local have have 6026 completed before completed before 6027 performing any performing any 6028 following global following global 6029 memory operations. memory operations. 6030 - Ensures that the - Ensures that the 6031 preceding preceding 6032 local/generic load local/generic load 6033 atomic/atomicrmw atomic/atomicrmw 6034 with an equal or with an equal or 6035 wider sync scope wider sync scope 6036 and memory ordering and memory ordering 6037 stronger than stronger than 6038 unordered (this is unordered (this is 6039 termed the termed the 6040 acquire-fence-paired-atomic acquire-fence-paired-atomic 6041 ) has completed ) has completed 6042 before following before following 6043 global memory global memory 6044 operations. This operations. This 6045 satisfies the satisfies the 6046 requirements of requirements of 6047 acquire. acquire. 6048 - Ensures that all - Ensures that all 6049 previous memory previous memory 6050 operations have operations have 6051 completed before a completed before a 6052 following following 6053 local/generic store local/generic store 6054 atomic/atomicrmw atomic/atomicrmw 6055 with an equal or with an equal or 6056 wider sync scope wider sync scope 6057 and memory ordering and memory ordering 6058 stronger than stronger than 6059 unordered (this is unordered (this is 6060 termed the termed the 6061 release-fence-paired-atomic release-fence-paired-atomic 6062 ). This satisfies the ). This satisfies the 6063 requirements of requirements of 6064 release. release. 6065 - Must happen before 6066 the following 6067 buffer_gl0_inv. 6068 - Ensures that the 6069 acquire-fence-paired 6070 atomic has completed 6071 before invalidating 6072 the 6073 cache. Therefore 6074 any following 6075 locations read must 6076 be no older than 6077 the value read by 6078 the 6079 acquire-fence-paired-atomic. 6080 6081 3. buffer_gl0_inv 6082 6083 - If CU wavefront execution 6084 mode, omit. 6085 - Ensures that 6086 following 6087 loads will not see 6088 stale data. 6089 6090 fence acq_rel - agent *none* 1. s_waitcnt lgkmcnt(0) & 1. s_waitcnt lgkmcnt(0) & 6091 - system vmcnt(0) vmcnt(0) & vscnt(0) 6092 6093 - If OpenCL and - If OpenCL and 6094 address space is address space is 6095 not generic, omit not generic, omit 6096 lgkmcnt(0). lgkmcnt(0). 6097 - If OpenCL and 6098 address space is 6099 local, omit 6100 vmcnt(0) and vscnt(0). 6101 - However, since LLVM - However, since LLVM 6102 currently has no currently has no 6103 address space on address space on 6104 the fence need to the fence need to 6105 conservatively conservatively 6106 always generate always generate 6107 (see comment for (see comment for 6108 previous fence). previous fence). 6109 - Could be split into - Could be split into 6110 separate s_waitcnt separate s_waitcnt 6111 vmcnt(0) and vmcnt(0), s_waitcnt 6112 s_waitcnt vscnt(0) and s_waitcnt 6113 lgkmcnt(0) to allow lgkmcnt(0) to allow 6114 them to be them to be 6115 independently moved independently moved 6116 according to the according to the 6117 following rules. following rules. 6118 - s_waitcnt vmcnt(0) - s_waitcnt vmcnt(0) 6119 must happen after must happen after 6120 any preceding any preceding 6121 global/generic global/generic 6122 load/store/load load/load 6123 atomic/store atomic/ 6124 atomic/atomicrmw. atomicrmw-with-return-value. 6125 - s_waitcnt vscnt(0) 6126 must happen after 6127 any preceding 6128 global/generic 6129 store/store atomic/ 6130 atomicrmw-no-return-value. 6131 - s_waitcnt lgkmcnt(0) - s_waitcnt lgkmcnt(0) 6132 must happen after must happen after 6133 any preceding any preceding 6134 local/generic local/generic 6135 load/store/load load/store/load 6136 atomic/store atomic/store 6137 atomic/atomicrmw. atomic/atomicrmw. 6138 - Must happen before - Must happen before 6139 the following the following 6140 buffer_wbinvl1_vol. buffer_gl*_inv. 6141 - Ensures that the - Ensures that the 6142 preceding preceding 6143 global/local/generic global/local/generic 6144 load load 6145 atomic/atomicrmw atomic/atomicrmw 6146 with an equal or with an equal or 6147 wider sync scope wider sync scope 6148 and memory ordering and memory ordering 6149 stronger than stronger than 6150 unordered (this is unordered (this is 6151 termed the termed the 6152 acquire-fence-paired-atomic acquire-fence-paired-atomic 6153 ) has completed ) has completed 6154 before invalidating before invalidating 6155 the cache. This the caches. This 6156 satisfies the satisfies the 6157 requirements of requirements of 6158 acquire. acquire. 6159 - Ensures that all - Ensures that all 6160 previous memory previous memory 6161 operations have operations have 6162 completed before a completed before a 6163 following following 6164 global/local/generic global/local/generic 6165 store store 6166 atomic/atomicrmw atomic/atomicrmw 6167 with an equal or with an equal or 6168 wider sync scope wider sync scope 6169 and memory ordering and memory ordering 6170 stronger than stronger than 6171 unordered (this is unordered (this is 6172 termed the termed the 6173 release-fence-paired-atomic release-fence-paired-atomic 6174 ). This satisfies the ). This satisfies the 6175 requirements of requirements of 6176 release. release. 6177 6178 2. buffer_wbinvl1_vol 2. buffer_gl0_inv; 6179 buffer_gl1_inv 6180 6181 - Must happen before - Must happen before 6182 any following any following 6183 global/generic global/generic 6184 load/load load/load 6185 atomic/store/store atomic/store/store 6186 atomic/atomicrmw. atomic/atomicrmw. 6187 - Ensures that - Ensures that 6188 following loads following loads 6189 will not see stale will not see stale 6190 global data. This global data. This 6191 satisfies the satisfies the 6192 requirements of requirements of 6193 acquire. acquire. 6194 6195 **Sequential Consistent Atomic** 6196 --------------------------------------------------------------------------------------------------------------------- 6197 load atomic seq_cst - singlethread - global *Same as corresponding *Same as corresponding 6198 - wavefront - local load atomic acquire, load atomic acquire, 6199 - generic except must generated except must generated 6200 all instructions even all instructions even 6201 for OpenCL.* for OpenCL.* 6202 load atomic seq_cst - workgroup - global 1. s_waitcnt lgkmcnt(0) 1. s_waitcnt lgkmcnt(0) & 6203 - generic vmcnt(0) & vscnt(0) 6204 6205 - If CU wavefront execution 6206 mode, omit vmcnt(0) and 6207 vscnt(0). 6208 - Could be split into 6209 separate s_waitcnt 6210 vmcnt(0), s_waitcnt 6211 vscnt(0) and s_waitcnt 6212 lgkmcnt(0) to allow 6213 them to be 6214 independently moved 6215 according to the 6216 following rules. 6217 - Must - waitcnt lgkmcnt(0) must 6218 happen after happen after 6219 preceding preceding 6220 global/generic load local load 6221 atomic/store atomic/store 6222 atomic/atomicrmw atomic/atomicrmw 6223 with memory with memory 6224 ordering of seq_cst ordering of seq_cst 6225 and with equal or and with equal or 6226 wider sync scope. wider sync scope. 6227 (Note that seq_cst (Note that seq_cst 6228 fences have their fences have their 6229 own s_waitcnt own s_waitcnt 6230 lgkmcnt(0) and so do lgkmcnt(0) and so do 6231 not need to be not need to be 6232 considered.) considered.) 6233 - waitcnt vmcnt(0) 6234 Must happen after 6235 preceding 6236 global/generic load 6237 atomic/ 6238 atomicrmw-with-return-value 6239 with memory 6240 ordering of seq_cst 6241 and with equal or 6242 wider sync scope. 6243 (Note that seq_cst 6244 fences have their 6245 own s_waitcnt 6246 vmcnt(0) and so do 6247 not need to be 6248 considered.) 6249 - waitcnt vscnt(0) 6250 Must happen after 6251 preceding 6252 global/generic store 6253 atomic/ 6254 atomicrmw-no-return-value 6255 with memory 6256 ordering of seq_cst 6257 and with equal or 6258 wider sync scope. 6259 (Note that seq_cst 6260 fences have their 6261 own s_waitcnt 6262 vscnt(0) and so do 6263 not need to be 6264 considered.) 6265 - Ensures any - Ensures any 6266 preceding preceding 6267 sequential sequential 6268 consistent local consistent global/local 6269 memory instructions memory instructions 6270 have completed have completed 6271 before executing before executing 6272 this sequentially this sequentially 6273 consistent consistent 6274 instruction. This instruction. This 6275 prevents reordering prevents reordering 6276 a seq_cst store a seq_cst store 6277 followed by a followed by a 6278 seq_cst load. (Note seq_cst load. (Note 6279 that seq_cst is that seq_cst is 6280 stronger than stronger than 6281 acquire/release as acquire/release as 6282 the reordering of the reordering of 6283 load acquire load acquire 6284 followed by a store followed by a store 6285 release is release is 6286 prevented by the prevented by the 6287 waitcnt of waitcnt of 6288 the release, but the release, but 6289 there is nothing there is nothing 6290 preventing a store preventing a store 6291 release followed by release followed by 6292 load acquire from load acquire from 6293 completing out of completing out of 6294 order. The waitcnt order. The waitcnt 6295 could be placed after could be placed after 6296 seq_store or before seq_store or before 6297 the seq_load. We the seq_load. We 6298 choose the load to choose the load to 6299 make the waitcnt be make the waitcnt be 6300 as late as possible as late as possible 6301 so that the store so that the store 6302 may have already may have already 6303 completed.) completed.) 6304 6305 2. *Following 2. *Following 6306 instructions same as instructions same as 6307 corresponding load corresponding load 6308 atomic acquire, atomic acquire, 6309 except must generated except must generated 6310 all instructions even all instructions even 6311 for OpenCL.* for OpenCL.* 6312 load atomic seq_cst - workgroup - local *Same as corresponding 6313 load atomic acquire, 6314 except must generated 6315 all instructions even 6316 for OpenCL.* 6317 6318 1. s_waitcnt vmcnt(0) & vscnt(0) 6319 6320 - If CU wavefront execution 6321 mode, omit. 6322 - Could be split into 6323 separate s_waitcnt 6324 vmcnt(0) and s_waitcnt 6325 vscnt(0) to allow 6326 them to be 6327 independently moved 6328 according to the 6329 following rules. 6330 - waitcnt vmcnt(0) 6331 Must happen after 6332 preceding 6333 global/generic load 6334 atomic/ 6335 atomicrmw-with-return-value 6336 with memory 6337 ordering of seq_cst 6338 and with equal or 6339 wider sync scope. 6340 (Note that seq_cst 6341 fences have their 6342 own s_waitcnt 6343 vmcnt(0) and so do 6344 not need to be 6345 considered.) 6346 - waitcnt vscnt(0) 6347 Must happen after 6348 preceding 6349 global/generic store 6350 atomic/ 6351 atomicrmw-no-return-value 6352 with memory 6353 ordering of seq_cst 6354 and with equal or 6355 wider sync scope. 6356 (Note that seq_cst 6357 fences have their 6358 own s_waitcnt 6359 vscnt(0) and so do 6360 not need to be 6361 considered.) 6362 - Ensures any 6363 preceding 6364 sequential 6365 consistent global 6366 memory instructions 6367 have completed 6368 before executing 6369 this sequentially 6370 consistent 6371 instruction. This 6372 prevents reordering 6373 a seq_cst store 6374 followed by a 6375 seq_cst load. (Note 6376 that seq_cst is 6377 stronger than 6378 acquire/release as 6379 the reordering of 6380 load acquire 6381 followed by a store 6382 release is 6383 prevented by the 6384 waitcnt of 6385 the release, but 6386 there is nothing 6387 preventing a store 6388 release followed by 6389 load acquire from 6390 completing out of 6391 order. The waitcnt 6392 could be placed after 6393 seq_store or before 6394 the seq_load. We 6395 choose the load to 6396 make the waitcnt be 6397 as late as possible 6398 so that the store 6399 may have already 6400 completed.) 6401 6402 2. *Following 6403 instructions same as 6404 corresponding load 6405 atomic acquire, 6406 except must generated 6407 all instructions even 6408 for OpenCL.* 6409 6410 load atomic seq_cst - agent - global 1. s_waitcnt lgkmcnt(0) & 1. s_waitcnt lgkmcnt(0) & 6411 - system - generic vmcnt(0) vmcnt(0) & vscnt(0) 6412 6413 - Could be split into - Could be split into 6414 separate s_waitcnt separate s_waitcnt 6415 vmcnt(0) vmcnt(0), s_waitcnt 6416 and s_waitcnt vscnt(0) and s_waitcnt 6417 lgkmcnt(0) to allow lgkmcnt(0) to allow 6418 them to be them to be 6419 independently moved independently moved 6420 according to the according to the 6421 following rules. following rules. 6422 - waitcnt lgkmcnt(0) - waitcnt lgkmcnt(0) 6423 must happen after must happen after 6424 preceding preceding 6425 global/generic load local load 6426 atomic/store atomic/store 6427 atomic/atomicrmw atomic/atomicrmw 6428 with memory with memory 6429 ordering of seq_cst ordering of seq_cst 6430 and with equal or and with equal or 6431 wider sync scope. wider sync scope. 6432 (Note that seq_cst (Note that seq_cst 6433 fences have their fences have their 6434 own s_waitcnt own s_waitcnt 6435 lgkmcnt(0) and so do lgkmcnt(0) and so do 6436 not need to be not need to be 6437 considered.) considered.) 6438 - waitcnt vmcnt(0) - waitcnt vmcnt(0) 6439 must happen after must happen after 6440 preceding preceding 6441 global/generic load global/generic load 6442 atomic/store atomic/ 6443 atomic/atomicrmw atomicrmw-with-return-value 6444 with memory with memory 6445 ordering of seq_cst ordering of seq_cst 6446 and with equal or and with equal or 6447 wider sync scope. wider sync scope. 6448 (Note that seq_cst (Note that seq_cst 6449 fences have their fences have their 6450 own s_waitcnt own s_waitcnt 6451 vmcnt(0) and so do vmcnt(0) and so do 6452 not need to be not need to be 6453 considered.) considered.) 6454 - waitcnt vscnt(0) 6455 Must happen after 6456 preceding 6457 global/generic store 6458 atomic/ 6459 atomicrmw-no-return-value 6460 with memory 6461 ordering of seq_cst 6462 and with equal or 6463 wider sync scope. 6464 (Note that seq_cst 6465 fences have their 6466 own s_waitcnt 6467 vscnt(0) and so do 6468 not need to be 6469 considered.) 6470 - Ensures any - Ensures any 6471 preceding preceding 6472 sequential sequential 6473 consistent global consistent global 6474 memory instructions memory instructions 6475 have completed have completed 6476 before executing before executing 6477 this sequentially this sequentially 6478 consistent consistent 6479 instruction. This instruction. This 6480 prevents reordering prevents reordering 6481 a seq_cst store a seq_cst store 6482 followed by a followed by a 6483 seq_cst load. (Note seq_cst load. (Note 6484 that seq_cst is that seq_cst is 6485 stronger than stronger than 6486 acquire/release as acquire/release as 6487 the reordering of the reordering of 6488 load acquire load acquire 6489 followed by a store followed by a store 6490 release is release is 6491 prevented by the prevented by the 6492 waitcnt of waitcnt of 6493 the release, but the release, but 6494 there is nothing there is nothing 6495 preventing a store preventing a store 6496 release followed by release followed by 6497 load acquire from load acquire from 6498 completing out of completing out of 6499 order. The waitcnt order. The waitcnt 6500 could be placed after could be placed after 6501 seq_store or before seq_store or before 6502 the seq_load. We the seq_load. We 6503 choose the load to choose the load to 6504 make the waitcnt be make the waitcnt be 6505 as late as possible as late as possible 6506 so that the store so that the store 6507 may have already may have already 6508 completed.) completed.) 6509 6510 2. *Following 2. *Following 6511 instructions same as instructions same as 6512 corresponding load corresponding load 6513 atomic acquire, atomic acquire, 6514 except must generated except must generated 6515 all instructions even all instructions even 6516 for OpenCL.* for OpenCL.* 6517 store atomic seq_cst - singlethread - global *Same as corresponding *Same as corresponding 6518 - wavefront - local store atomic release, store atomic release, 6519 - workgroup - generic except must generated except must generated 6520 all instructions even all instructions even 6521 for OpenCL.* for OpenCL.* 6522 store atomic seq_cst - agent - global *Same as corresponding *Same as corresponding 6523 - system - generic store atomic release, store atomic release, 6524 except must generated except must generated 6525 all instructions even all instructions even 6526 for OpenCL.* for OpenCL.* 6527 atomicrmw seq_cst - singlethread - global *Same as corresponding *Same as corresponding 6528 - wavefront - local atomicrmw acq_rel, atomicrmw acq_rel, 6529 - workgroup - generic except must generated except must generated 6530 all instructions even all instructions even 6531 for OpenCL.* for OpenCL.* 6532 atomicrmw seq_cst - agent - global *Same as corresponding *Same as corresponding 6533 - system - generic atomicrmw acq_rel, atomicrmw acq_rel, 6534 except must generated except must generated 6535 all instructions even all instructions even 6536 for OpenCL.* for OpenCL.* 6537 fence seq_cst - singlethread *none* *Same as corresponding *Same as corresponding 6538 - wavefront fence acq_rel, fence acq_rel, 6539 - workgroup except must generated except must generated 6540 - agent all instructions even all instructions even 6541 - system for OpenCL.* for OpenCL.* 6542 ============ ============ ============== ========== ================================ ================================ 6543 6544The memory order also adds the single thread optimization constrains defined in 6545table 6546:ref:`amdgpu-amdhsa-memory-model-single-thread-optimization-constraints-gfx6-gfx10-table`. 6547 6548 .. table:: AMDHSA Memory Model Single Thread Optimization Constraints GFX6-GFX10 6549 :name: amdgpu-amdhsa-memory-model-single-thread-optimization-constraints-gfx6-gfx10-table 6550 6551 ============ ============================================================== 6552 LLVM Memory Optimization Constraints 6553 Ordering 6554 ============ ============================================================== 6555 unordered *none* 6556 monotonic *none* 6557 acquire - If a load atomic/atomicrmw then no following load/load 6558 atomic/store/ store atomic/atomicrmw/fence instruction can 6559 be moved before the acquire. 6560 - If a fence then same as load atomic, plus no preceding 6561 associated fence-paired-atomic can be moved after the fence. 6562 release - If a store atomic/atomicrmw then no preceding load/load 6563 atomic/store/ store atomic/atomicrmw/fence instruction can 6564 be moved after the release. 6565 - If a fence then same as store atomic, plus no following 6566 associated fence-paired-atomic can be moved before the 6567 fence. 6568 acq_rel Same constraints as both acquire and release. 6569 seq_cst - If a load atomic then same constraints as acquire, plus no 6570 preceding sequentially consistent load atomic/store 6571 atomic/atomicrmw/fence instruction can be moved after the 6572 seq_cst. 6573 - If a store atomic then the same constraints as release, plus 6574 no following sequentially consistent load atomic/store 6575 atomic/atomicrmw/fence instruction can be moved before the 6576 seq_cst. 6577 - If an atomicrmw/fence then same constraints as acq_rel. 6578 ============ ============================================================== 6579 6580Trap Handler ABI 6581~~~~~~~~~~~~~~~~ 6582 6583For code objects generated by AMDGPU backend for HSA [HSA]_ compatible runtimes 6584(such as ROCm [AMD-ROCm]_), the runtime installs a trap handler that supports 6585the ``s_trap`` instruction with the following usage: 6586 6587 .. table:: AMDGPU Trap Handler for AMDHSA OS 6588 :name: amdgpu-trap-handler-for-amdhsa-os-table 6589 6590 =================== =============== =============== ======================= 6591 Usage Code Sequence Trap Handler Description 6592 Inputs 6593 =================== =============== =============== ======================= 6594 reserved ``s_trap 0x00`` Reserved by hardware. 6595 ``debugtrap(arg)`` ``s_trap 0x01`` ``SGPR0-1``: Reserved for HSA 6596 ``queue_ptr`` ``debugtrap`` 6597 ``VGPR0``: intrinsic (not 6598 ``arg`` implemented). 6599 ``llvm.trap`` ``s_trap 0x02`` ``SGPR0-1``: Causes dispatch to be 6600 ``queue_ptr`` terminated and its 6601 associated queue put 6602 into the error state. 6603 ``llvm.debugtrap`` ``s_trap 0x03`` - If debugger not 6604 installed then 6605 behaves as a 6606 no-operation. The 6607 trap handler is 6608 entered and 6609 immediately returns 6610 to continue 6611 execution of the 6612 wavefront. 6613 - If the debugger is 6614 installed, causes 6615 the debug trap to be 6616 reported by the 6617 debugger and the 6618 wavefront is put in 6619 the halt state until 6620 resumed by the 6621 debugger. 6622 reserved ``s_trap 0x04`` Reserved. 6623 reserved ``s_trap 0x05`` Reserved. 6624 reserved ``s_trap 0x06`` Reserved. 6625 debugger breakpoint ``s_trap 0x07`` Reserved for debugger 6626 breakpoints. 6627 reserved ``s_trap 0x08`` Reserved. 6628 reserved ``s_trap 0xfe`` Reserved. 6629 reserved ``s_trap 0xff`` Reserved. 6630 =================== =============== =============== ======================= 6631 6632.. _amdgpu-amdhsa-function-call-convention: 6633 6634Call Convention 6635~~~~~~~~~~~~~~~ 6636 6637.. note:: 6638 6639 This section is currently incomplete and has inaccuracies. It is WIP that will 6640 be updated as information is determined. 6641 6642See :ref:`amdgpu-dwarf-address-space-identifier` for information on swizzled 6643addresses. Unswizzled addresses are normal linear addresses. 6644 6645.. _amdgpu-amdhsa-function-call-convention-kernel-functions: 6646 6647Kernel Functions 6648++++++++++++++++ 6649 6650This section describes the call convention ABI for the outer kernel function. 6651 6652See :ref:`amdgpu-amdhsa-initial-kernel-execution-state` for the kernel call 6653convention. 6654 6655The following is not part of the AMDGPU kernel calling convention but describes 6656how the AMDGPU implements function calls: 6657 66581. Clang decides the kernarg layout to match the *HSA Programmer's Language 6659 Reference* [HSA]_. 6660 6661 - All structs are passed directly. 6662 - Lambda values are passed *TBA*. 6663 6664 .. TODO:: 6665 6666 - Does this really follow HSA rules? Or are structs >16 bytes passed 6667 by-value struct? 6668 - What is ABI for lambda values? 6669 66704. The kernel performs certain setup in its prolog, as described in 6671 :ref:`amdgpu-amdhsa-kernel-prolog`. 6672 6673.. _amdgpu-amdhsa-function-call-convention-non-kernel-functions: 6674 6675Non-Kernel Functions 6676++++++++++++++++++++ 6677 6678This section describes the call convention ABI for functions other than the 6679outer kernel function. 6680 6681If a kernel has function calls then scratch is always allocated and used for 6682the call stack which grows from low address to high address using the swizzled 6683scratch address space. 6684 6685On entry to a function: 6686 66871. SGPR0-3 contain a V# with the following properties (see 6688 :ref:`amdgpu-amdhsa-kernel-prolog-private-segment-buffer`): 6689 6690 * Base address pointing to the beginning of the wavefront scratch backing 6691 memory. 6692 * Swizzled with dword element size and stride of wavefront size elements. 6693 66942. The FLAT_SCRATCH register pair is setup. See 6695 :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`. 66963. GFX6-8: M0 register set to the size of LDS in bytes. See 6697 :ref:`amdgpu-amdhsa-kernel-prolog-m0`. 66984. The EXEC register is set to the lanes active on entry to the function. 66995. MODE register: *TBD* 67006. VGPR0-31 and SGPR4-29 are used to pass function input arguments as described 6701 below. 67027. SGPR30-31 return address (RA). The code address that the function must 6703 return to when it completes. The value is undefined if the function is *no 6704 return*. 67058. SGPR32 is used for the stack pointer (SP). It is an unswizzled scratch 6706 offset relative to the beginning of the wavefront scratch backing memory. 6707 6708 The unswizzled SP can be used with buffer instructions as an unswizzled SGPR 6709 offset with the scratch V# in SGPR0-3 to access the stack in a swizzled 6710 manner. 6711 6712 The unswizzled SP value can be converted into the swizzled SP value by: 6713 6714 | swizzled SP = unswizzled SP / wavefront size 6715 6716 This may be used to obtain the private address space address of stack 6717 objects and to convert this address to a flat address by adding the flat 6718 scratch aperture base address. 6719 6720 The swizzled SP value is always 4 bytes aligned for the ``r600`` 6721 architecture and 16 byte aligned for the ``amdgcn`` architecture. 6722 6723 .. note:: 6724 6725 The ``amdgcn`` value is selected to avoid dynamic stack alignment for the 6726 OpenCL language which has the largest base type defined as 16 bytes. 6727 6728 On entry, the swizzled SP value is the address of the first function 6729 argument passed on the stack. Other stack passed arguments are positive 6730 offsets from the entry swizzled SP value. 6731 6732 The function may use positive offsets beyond the last stack passed argument 6733 for stack allocated local variables and register spill slots. If necessary, 6734 the function may align these to greater alignment than 16 bytes. After these 6735 the function may dynamically allocate space for such things as runtime sized 6736 ``alloca`` local allocations. 6737 6738 If the function calls another function, it will place any stack allocated 6739 arguments after the last local allocation and adjust SGPR32 to the address 6740 after the last local allocation. 6741 67429. All other registers are unspecified. 674310. Any necessary ``waitcnt`` has been performed to ensure memory is available 6744 to the function. 6745 6746On exit from a function: 6747 67481. VGPR0-31 and SGPR4-29 are used to pass function result arguments as 6749 described below. Any registers used are considered clobbered registers. 67502. The following registers are preserved and have the same value as on entry: 6751 6752 * FLAT_SCRATCH 6753 * EXEC 6754 * GFX6-8: M0 6755 * All SGPR registers except the clobbered registers of SGPR4-31. 6756 * VGPR40-47 6757 VGPR56-63 6758 VGPR72-79 6759 VGPR88-95 6760 VGPR104-111 6761 VGPR120-127 6762 VGPR136-143 6763 VGPR152-159 6764 VGPR168-175 6765 VGPR184-191 6766 VGPR200-207 6767 VGPR216-223 6768 VGPR232-239 6769 VGPR248-255 6770 6771 *Except the argument registers, the VGPR clobbered and the preserved 6772 registers are intermixed at regular intervals in order to 6773 get a better occupancy.* 6774 6775 For the AMDGPU backend, an inter-procedural register allocation (IPRA) 6776 optimization may mark some of clobbered SGPR and VGPR registers as 6777 preserved if it can be determined that the called function does not change 6778 their value. 6779 67802. The PC is set to the RA provided on entry. 67813. MODE register: *TBD*. 67824. All other registers are clobbered. 67835. Any necessary ``waitcnt`` has been performed to ensure memory accessed by 6784 function is available to the caller. 6785 6786.. TODO:: 6787 6788 - On gfx908 are all ACC registers clobbered? 6789 6790 - How are function results returned? The address of structured types is passed 6791 by reference, but what about other types? 6792 6793The function input arguments are made up of the formal arguments explicitly 6794declared by the source language function plus the implicit input arguments used 6795by the implementation. 6796 6797The source language input arguments are: 6798 67991. Any source language implicit ``this`` or ``self`` argument comes first as a 6800 pointer type. 68012. Followed by the function formal arguments in left to right source order. 6802 6803The source language result arguments are: 6804 68051. The function result argument. 6806 6807The source language input or result struct type arguments that are less than or 6808equal to 16 bytes, are decomposed recursively into their base type fields, and 6809each field is passed as if a separate argument. For input arguments, if the 6810called function requires the struct to be in memory, for example because its 6811address is taken, then the function body is responsible for allocating a stack 6812location and copying the field arguments into it. Clang terms this *direct 6813struct*. 6814 6815The source language input struct type arguments that are greater than 16 bytes, 6816are passed by reference. The caller is responsible for allocating a stack 6817location to make a copy of the struct value and pass the address as the input 6818argument. The called function is responsible to perform the dereference when 6819accessing the input argument. Clang terms this *by-value struct*. 6820 6821A source language result struct type argument that is greater than 16 bytes, is 6822returned by reference. The caller is responsible for allocating a stack location 6823to hold the result value and passes the address as the last input argument 6824(before the implicit input arguments). In this case there are no result 6825arguments. The called function is responsible to perform the dereference when 6826storing the result value. Clang terms this *structured return (sret)*. 6827 6828*TODO: correct the ``sret`` definition.* 6829 6830.. TODO:: 6831 6832 Is this definition correct? Or is ``sret`` only used if passing in registers, and 6833 pass as non-decomposed struct as stack argument? Or something else? Is the 6834 memory location in the caller stack frame, or a stack memory argument and so 6835 no address is passed as the caller can directly write to the argument stack 6836 location? But then the stack location is still live after return. If an 6837 argument stack location is it the first stack argument or the last one? 6838 6839Lambda argument types are treated as struct types with an implementation defined 6840set of fields. 6841 6842.. TODO:: 6843 6844 Need to specify the ABI for lambda types for AMDGPU. 6845 6846For AMDGPU backend all source language arguments (including the decomposed 6847struct type arguments) are passed in VGPRs unless marked ``inreg`` in which case 6848they are passed in SGPRs. 6849 6850The AMDGPU backend walks the function call graph from the leaves to determine 6851which implicit input arguments are used, propagating to each caller of the 6852function. The used implicit arguments are appended to the function arguments 6853after the source language arguments in the following order: 6854 6855.. TODO:: 6856 6857 Is recursion or external functions supported? 6858 68591. Work-Item ID (1 VGPR) 6860 6861 The X, Y and Z work-item ID are packed into a single VGRP with the following 6862 layout. Only fields actually used by the function are set. The other bits 6863 are undefined. 6864 6865 The values come from the initial kernel execution state. See 6866 :ref:`amdgpu-amdhsa-vgpr-register-set-up-order-table`. 6867 6868 .. table:: Work-item implicit argument layout 6869 :name: amdgpu-amdhsa-workitem-implicit-argument-layout-table 6870 6871 ======= ======= ============== 6872 Bits Size Field Name 6873 ======= ======= ============== 6874 9:0 10 bits X Work-Item ID 6875 19:10 10 bits Y Work-Item ID 6876 29:20 10 bits Z Work-Item ID 6877 31:30 2 bits Unused 6878 ======= ======= ============== 6879 68802. Dispatch Ptr (2 SGPRs) 6881 6882 The value comes from the initial kernel execution state. See 6883 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 6884 68853. Queue Ptr (2 SGPRs) 6886 6887 The value comes from the initial kernel execution state. See 6888 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 6889 68904. Kernarg Segment Ptr (2 SGPRs) 6891 6892 The value comes from the initial kernel execution state. See 6893 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 6894 68955. Dispatch id (2 SGPRs) 6896 6897 The value comes from the initial kernel execution state. See 6898 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 6899 69006. Work-Group ID X (1 SGPR) 6901 6902 The value comes from the initial kernel execution state. See 6903 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 6904 69057. Work-Group ID Y (1 SGPR) 6906 6907 The value comes from the initial kernel execution state. See 6908 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 6909 69108. Work-Group ID Z (1 SGPR) 6911 6912 The value comes from the initial kernel execution state. See 6913 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 6914 69159. Implicit Argument Ptr (2 SGPRs) 6916 6917 The value is computed by adding an offset to Kernarg Segment Ptr to get the 6918 global address space pointer to the first kernarg implicit argument. 6919 6920The input and result arguments are assigned in order in the following manner: 6921 6922.. note:: 6923 6924 There are likely some errors and omissions in the following description that 6925 need correction. 6926 6927 .. TODO:: 6928 6929 Check the clang source code to decipher how function arguments and return 6930 results are handled. Also see the AMDGPU specific values used. 6931 6932* VGPR arguments are assigned to consecutive VGPRs starting at VGPR0 up to 6933 VGPR31. 6934 6935 If there are more arguments than will fit in these registers, the remaining 6936 arguments are allocated on the stack in order on naturally aligned 6937 addresses. 6938 6939 .. TODO:: 6940 6941 How are overly aligned structures allocated on the stack? 6942 6943* SGPR arguments are assigned to consecutive SGPRs starting at SGPR0 up to 6944 SGPR29. 6945 6946 If there are more arguments than will fit in these registers, the remaining 6947 arguments are allocated on the stack in order on naturally aligned 6948 addresses. 6949 6950Note that decomposed struct type arguments may have some fields passed in 6951registers and some in memory. 6952 6953.. TODO:: 6954 6955 So, a struct which can pass some fields as decomposed register arguments, will 6956 pass the rest as decomposed stack elements? But an argument that will not start 6957 in registers will not be decomposed and will be passed as a non-decomposed 6958 stack value? 6959 6960The following is not part of the AMDGPU function calling convention but 6961describes how the AMDGPU implements function calls: 6962 69631. SGPR33 is used as a frame pointer (FP) if necessary. Like the SP it is an 6964 unswizzled scratch address. It is only needed if runtime sized ``alloca`` 6965 are used, or for the reasons defined in ``SIFrameLowering``. 69662. Runtime stack alignment is supported. SGPR34 is used as a base pointer (BP) 6967 to access the incoming stack arguments in the function. The BP is needed 6968 only when the function requires the runtime stack alignment. 6969 69703. Allocating SGPR arguments on the stack are not supported. 6971 69724. No CFI is currently generated. See 6973 :ref:`amdgpu-dwarf-call-frame-information`. 6974 6975 .. note:: 6976 6977 CFI will be generated that defines the CFA as the unswizzled address 6978 relative to the wave scratch base in the unswizzled private address space 6979 of the lowest address stack allocated local variable. 6980 6981 ``DW_AT_frame_base`` will be defined as the swizzled address in the 6982 swizzled private address space by dividing the CFA by the wavefront size 6983 (since CFA is always at least dword aligned which matches the scratch 6984 swizzle element size). 6985 6986 If no dynamic stack alignment was performed, the stack allocated arguments 6987 are accessed as negative offsets relative to ``DW_AT_frame_base``, and the 6988 local variables and register spill slots are accessed as positive offsets 6989 relative to ``DW_AT_frame_base``. 6990 69915. Function argument passing is implemented by copying the input physical 6992 registers to virtual registers on entry. The register allocator can spill if 6993 necessary. These are copied back to physical registers at call sites. The 6994 net effect is that each function call can have these values in entirely 6995 distinct locations. The IPRA can help avoid shuffling argument registers. 69966. Call sites are implemented by setting up the arguments at positive offsets 6997 from SP. Then SP is incremented to account for the known frame size before 6998 the call and decremented after the call. 6999 7000 .. note:: 7001 7002 The CFI will reflect the changed calculation needed to compute the CFA 7003 from SP. 7004 70057. 4 byte spill slots are used in the stack frame. One slot is allocated for an 7006 emergency spill slot. Buffer instructions are used for stack accesses and 7007 not the ``flat_scratch`` instruction. 7008 7009 .. TODO:: 7010 7011 Explain when the emergency spill slot is used. 7012 7013.. TODO:: 7014 7015 Possible broken issues: 7016 7017 - Stack arguments must be aligned to required alignment. 7018 - Stack is aligned to max(16, max formal argument alignment) 7019 - Direct argument < 64 bits should check register budget. 7020 - Register budget calculation should respect ``inreg`` for SGPR. 7021 - SGPR overflow is not handled. 7022 - struct with 1 member unpeeling is not checking size of member. 7023 - ``sret`` is after ``this`` pointer. 7024 - Caller is not implementing stack realignment: need an extra pointer. 7025 - Should say AMDGPU passes FP rather than SP. 7026 - Should CFI define CFA as address of locals or arguments. Difference is 7027 apparent when have implemented dynamic alignment. 7028 - If ``SCRATCH`` instruction could allow negative offsets, then can make FP be 7029 highest address of stack frame and use negative offset for locals. Would 7030 allow SP to be the same as FP and could support signal-handler-like as now 7031 have a real SP for the top of the stack. 7032 - How is ``sret`` passed on the stack? In argument stack area? Can it overlay 7033 arguments? 7034 7035AMDPAL 7036------ 7037 7038This section provides code conventions used when the target triple OS is 7039``amdpal`` (see :ref:`amdgpu-target-triples`) for passing runtime parameters 7040from the application/runtime to each invocation of a hardware shader. These 7041parameters include both generic, application-controlled parameters called 7042*user data* as well as system-generated parameters that are a product of the 7043draw or dispatch execution. 7044 7045User Data 7046~~~~~~~~~ 7047 7048Each hardware stage has a set of 32-bit *user data registers* which can be 7049written from a command buffer and then loaded into SGPRs when waves are launched 7050via a subsequent dispatch or draw operation. This is the way most arguments are 7051passed from the application/runtime to a hardware shader. 7052 7053Compute User Data 7054~~~~~~~~~~~~~~~~~ 7055 7056Compute shader user data mappings are simpler than graphics shaders and have a 7057fixed mapping. 7058 7059Note that there are always 10 available *user data entries* in registers - 7060entries beyond that limit must be fetched from memory (via the spill table 7061pointer) by the shader. 7062 7063 .. table:: PAL Compute Shader User Data Registers 7064 :name: pal-compute-user-data-registers 7065 7066 ============= ================================ 7067 User Register Description 7068 ============= ================================ 7069 0 Global Internal Table (32-bit pointer) 7070 1 Per-Shader Internal Table (32-bit pointer) 7071 2 - 11 Application-Controlled User Data (10 32-bit values) 7072 12 Spill Table (32-bit pointer) 7073 13 - 14 Thread Group Count (64-bit pointer) 7074 15 GDS Range 7075 ============= ================================ 7076 7077Graphics User Data 7078~~~~~~~~~~~~~~~~~~ 7079 7080Graphics pipelines support a much more flexible user data mapping: 7081 7082 .. table:: PAL Graphics Shader User Data Registers 7083 :name: pal-graphics-user-data-registers 7084 7085 ============= ================================ 7086 User Register Description 7087 ============= ================================ 7088 0 Global Internal Table (32-bit pointer) 7089 + Per-Shader Internal Table (32-bit pointer) 7090 + 1-15 Application Controlled User Data 7091 (1-15 Contiguous 32-bit Values in Registers) 7092 + Spill Table (32-bit pointer) 7093 + Draw Index (First Stage Only) 7094 + Vertex Offset (First Stage Only) 7095 + Instance Offset (First Stage Only) 7096 ============= ================================ 7097 7098 The placement of the global internal table remains fixed in the first *user 7099 data SGPR register*. Otherwise all parameters are optional, and can be mapped 7100 to any desired *user data SGPR register*, with the following restrictions: 7101 7102 * Draw Index, Vertex Offset, and Instance Offset can only be used by the first 7103 active hardware stage in a graphics pipeline (i.e. where the API vertex 7104 shader runs). 7105 7106 * Application-controlled user data must be mapped into a contiguous range of 7107 user data registers. 7108 7109 * The application-controlled user data range supports compaction remapping, so 7110 only *entries* that are actually consumed by the shader must be assigned to 7111 corresponding *registers*. Note that in order to support an efficient runtime 7112 implementation, the remapping must pack *registers* in the same order as 7113 *entries*, with unused *entries* removed. 7114 7115.. _pal_global_internal_table: 7116 7117Global Internal Table 7118~~~~~~~~~~~~~~~~~~~~~ 7119 7120The global internal table is a table of *shader resource descriptors* (SRDs) 7121that define how certain engine-wide, runtime-managed resources should be 7122accessed from a shader. The majority of these resources have HW-defined formats, 7123and it is up to the compiler to write/read data as required by the target 7124hardware. 7125 7126The following table illustrates the required format: 7127 7128 .. table:: PAL Global Internal Table 7129 :name: pal-git-table 7130 7131 ============= ================================ 7132 Offset Description 7133 ============= ================================ 7134 0-3 Graphics Scratch SRD 7135 4-7 Compute Scratch SRD 7136 8-11 ES/GS Ring Output SRD 7137 12-15 ES/GS Ring Input SRD 7138 16-19 GS/VS Ring Output #0 7139 20-23 GS/VS Ring Output #1 7140 24-27 GS/VS Ring Output #2 7141 28-31 GS/VS Ring Output #3 7142 32-35 GS/VS Ring Input SRD 7143 36-39 Tessellation Factor Buffer SRD 7144 40-43 Off-Chip LDS Buffer SRD 7145 44-47 Off-Chip Param Cache Buffer SRD 7146 48-51 Sample Position Buffer SRD 7147 52 vaRange::ShadowDescriptorTable High Bits 7148 ============= ================================ 7149 7150 The pointer to the global internal table passed to the shader as user data 7151 is a 32-bit pointer. The top 32 bits should be assumed to be the same as 7152 the top 32 bits of the pipeline, so the shader may use the program 7153 counter's top 32 bits. 7154 7155Unspecified OS 7156-------------- 7157 7158This section provides code conventions used when the target triple OS is 7159empty (see :ref:`amdgpu-target-triples`). 7160 7161Trap Handler ABI 7162~~~~~~~~~~~~~~~~ 7163 7164For code objects generated by AMDGPU backend for non-amdhsa OS, the runtime does 7165not install a trap handler. The ``llvm.trap`` and ``llvm.debugtrap`` 7166instructions are handled as follows: 7167 7168 .. table:: AMDGPU Trap Handler for Non-AMDHSA OS 7169 :name: amdgpu-trap-handler-for-non-amdhsa-os-table 7170 7171 =============== =============== =========================================== 7172 Usage Code Sequence Description 7173 =============== =============== =========================================== 7174 llvm.trap s_endpgm Causes wavefront to be terminated. 7175 llvm.debugtrap *none* Compiler warning given that there is no 7176 trap handler installed. 7177 =============== =============== =========================================== 7178 7179Source Languages 7180================ 7181 7182.. _amdgpu-opencl: 7183 7184OpenCL 7185------ 7186 7187When the language is OpenCL the following differences occur: 7188 71891. The OpenCL memory model is used (see :ref:`amdgpu-amdhsa-memory-model`). 71902. The AMDGPU backend appends additional arguments to the kernel's explicit 7191 arguments for the AMDHSA OS (see 7192 :ref:`opencl-kernel-implicit-arguments-appended-for-amdhsa-os-table`). 71933. Additional metadata is generated 7194 (see :ref:`amdgpu-amdhsa-code-object-metadata`). 7195 7196 .. table:: OpenCL kernel implicit arguments appended for AMDHSA OS 7197 :name: opencl-kernel-implicit-arguments-appended-for-amdhsa-os-table 7198 7199 ======== ==== ========= =========================================== 7200 Position Byte Byte Description 7201 Size Alignment 7202 ======== ==== ========= =========================================== 7203 1 8 8 OpenCL Global Offset X 7204 2 8 8 OpenCL Global Offset Y 7205 3 8 8 OpenCL Global Offset Z 7206 4 8 8 OpenCL address of printf buffer 7207 5 8 8 OpenCL address of virtual queue used by 7208 enqueue_kernel. 7209 6 8 8 OpenCL address of AqlWrap struct used by 7210 enqueue_kernel. 7211 7 8 8 Pointer argument used for Multi-gird 7212 synchronization. 7213 ======== ==== ========= =========================================== 7214 7215.. _amdgpu-hcc: 7216 7217HCC 7218--- 7219 7220When the language is HCC the following differences occur: 7221 72221. The HSA memory model is used (see :ref:`amdgpu-amdhsa-memory-model`). 7223 7224.. _amdgpu-assembler: 7225 7226Assembler 7227--------- 7228 7229AMDGPU backend has LLVM-MC based assembler which is currently in development. 7230It supports AMDGCN GFX6-GFX10. 7231 7232This section describes general syntax for instructions and operands. 7233 7234Instructions 7235~~~~~~~~~~~~ 7236 7237An instruction has the following :doc:`syntax<AMDGPUInstructionSyntax>`: 7238 7239 | ``<``\ *opcode*\ ``> <``\ *operand0*\ ``>, <``\ *operand1*\ ``>,... 7240 <``\ *modifier0*\ ``> <``\ *modifier1*\ ``>...`` 7241 7242:doc:`Operands<AMDGPUOperandSyntax>` are comma-separated while 7243:doc:`modifiers<AMDGPUModifierSyntax>` are space-separated. 7244 7245The order of operands and modifiers is fixed. 7246Most modifiers are optional and may be omitted. 7247 7248Links to detailed instruction syntax description may be found in the following 7249table. Note that features under development are not included 7250in this description. 7251 7252 =================================== ======================================= 7253 Core ISA ISA Extensions 7254 =================================== ======================================= 7255 :doc:`GFX7<AMDGPU/AMDGPUAsmGFX7>` \- 7256 :doc:`GFX8<AMDGPU/AMDGPUAsmGFX8>` \- 7257 :doc:`GFX9<AMDGPU/AMDGPUAsmGFX9>` :doc:`gfx900<AMDGPU/AMDGPUAsmGFX900>` 7258 7259 :doc:`gfx902<AMDGPU/AMDGPUAsmGFX900>` 7260 7261 :doc:`gfx904<AMDGPU/AMDGPUAsmGFX904>` 7262 7263 :doc:`gfx906<AMDGPU/AMDGPUAsmGFX906>` 7264 7265 :doc:`gfx908<AMDGPU/AMDGPUAsmGFX908>` 7266 7267 :doc:`gfx909<AMDGPU/AMDGPUAsmGFX900>` 7268 7269 :doc:`GFX10<AMDGPU/AMDGPUAsmGFX10>` :doc:`gfx1011<AMDGPU/AMDGPUAsmGFX1011>` 7270 7271 :doc:`gfx1012<AMDGPU/AMDGPUAsmGFX1011>` 7272 =================================== ======================================= 7273 7274For more information about instructions, their semantics and supported 7275combinations of operands, refer to one of instruction set architecture manuals 7276[AMD-GCN-GFX6]_, [AMD-GCN-GFX7]_, [AMD-GCN-GFX8]_, [AMD-GCN-GFX9]_ and 7277[AMD-GCN-GFX10]_. 7278 7279Operands 7280~~~~~~~~ 7281 7282Detailed description of operands may be found :doc:`here<AMDGPUOperandSyntax>`. 7283 7284Modifiers 7285~~~~~~~~~ 7286 7287Detailed description of modifiers may be found 7288:doc:`here<AMDGPUModifierSyntax>`. 7289 7290Instruction Examples 7291~~~~~~~~~~~~~~~~~~~~ 7292 7293DS 7294++ 7295 7296.. code-block:: nasm 7297 7298 ds_add_u32 v2, v4 offset:16 7299 ds_write_src2_b64 v2 offset0:4 offset1:8 7300 ds_cmpst_f32 v2, v4, v6 7301 ds_min_rtn_f64 v[8:9], v2, v[4:5] 7302 7303For full list of supported instructions, refer to "LDS/GDS instructions" in ISA 7304Manual. 7305 7306FLAT 7307++++ 7308 7309.. code-block:: nasm 7310 7311 flat_load_dword v1, v[3:4] 7312 flat_store_dwordx3 v[3:4], v[5:7] 7313 flat_atomic_swap v1, v[3:4], v5 glc 7314 flat_atomic_cmpswap v1, v[3:4], v[5:6] glc slc 7315 flat_atomic_fmax_x2 v[1:2], v[3:4], v[5:6] glc 7316 7317For full list of supported instructions, refer to "FLAT instructions" in ISA 7318Manual. 7319 7320MUBUF 7321+++++ 7322 7323.. code-block:: nasm 7324 7325 buffer_load_dword v1, off, s[4:7], s1 7326 buffer_store_dwordx4 v[1:4], v2, ttmp[4:7], s1 offen offset:4 glc tfe 7327 buffer_store_format_xy v[1:2], off, s[4:7], s1 7328 buffer_wbinvl1 7329 buffer_atomic_inc v1, v2, s[8:11], s4 idxen offset:4 slc 7330 7331For full list of supported instructions, refer to "MUBUF Instructions" in ISA 7332Manual. 7333 7334SMRD/SMEM 7335+++++++++ 7336 7337.. code-block:: nasm 7338 7339 s_load_dword s1, s[2:3], 0xfc 7340 s_load_dwordx8 s[8:15], s[2:3], s4 7341 s_load_dwordx16 s[88:103], s[2:3], s4 7342 s_dcache_inv_vol 7343 s_memtime s[4:5] 7344 7345For full list of supported instructions, refer to "Scalar Memory Operations" in 7346ISA Manual. 7347 7348SOP1 7349++++ 7350 7351.. code-block:: nasm 7352 7353 s_mov_b32 s1, s2 7354 s_mov_b64 s[0:1], 0x80000000 7355 s_cmov_b32 s1, 200 7356 s_wqm_b64 s[2:3], s[4:5] 7357 s_bcnt0_i32_b64 s1, s[2:3] 7358 s_swappc_b64 s[2:3], s[4:5] 7359 s_cbranch_join s[4:5] 7360 7361For full list of supported instructions, refer to "SOP1 Instructions" in ISA 7362Manual. 7363 7364SOP2 7365++++ 7366 7367.. code-block:: nasm 7368 7369 s_add_u32 s1, s2, s3 7370 s_and_b64 s[2:3], s[4:5], s[6:7] 7371 s_cselect_b32 s1, s2, s3 7372 s_andn2_b32 s2, s4, s6 7373 s_lshr_b64 s[2:3], s[4:5], s6 7374 s_ashr_i32 s2, s4, s6 7375 s_bfm_b64 s[2:3], s4, s6 7376 s_bfe_i64 s[2:3], s[4:5], s6 7377 s_cbranch_g_fork s[4:5], s[6:7] 7378 7379For full list of supported instructions, refer to "SOP2 Instructions" in ISA 7380Manual. 7381 7382SOPC 7383++++ 7384 7385.. code-block:: nasm 7386 7387 s_cmp_eq_i32 s1, s2 7388 s_bitcmp1_b32 s1, s2 7389 s_bitcmp0_b64 s[2:3], s4 7390 s_setvskip s3, s5 7391 7392For full list of supported instructions, refer to "SOPC Instructions" in ISA 7393Manual. 7394 7395SOPP 7396++++ 7397 7398.. code-block:: nasm 7399 7400 s_barrier 7401 s_nop 2 7402 s_endpgm 7403 s_waitcnt 0 ; Wait for all counters to be 0 7404 s_waitcnt vmcnt(0) & expcnt(0) & lgkmcnt(0) ; Equivalent to above 7405 s_waitcnt vmcnt(1) ; Wait for vmcnt counter to be 1. 7406 s_sethalt 9 7407 s_sleep 10 7408 s_sendmsg 0x1 7409 s_sendmsg sendmsg(MSG_INTERRUPT) 7410 s_trap 1 7411 7412For full list of supported instructions, refer to "SOPP Instructions" in ISA 7413Manual. 7414 7415Unless otherwise mentioned, little verification is performed on the operands 7416of SOPP Instructions, so it is up to the programmer to be familiar with the 7417range or acceptable values. 7418 7419VALU 7420++++ 7421 7422For vector ALU instruction opcodes (VOP1, VOP2, VOP3, VOPC, VOP_DPP, VOP_SDWA), 7423the assembler will automatically use optimal encoding based on its operands. To 7424force specific encoding, one can add a suffix to the opcode of the instruction: 7425 7426* _e32 for 32-bit VOP1/VOP2/VOPC 7427* _e64 for 64-bit VOP3 7428* _dpp for VOP_DPP 7429* _sdwa for VOP_SDWA 7430 7431VOP1/VOP2/VOP3/VOPC examples: 7432 7433.. code-block:: nasm 7434 7435 v_mov_b32 v1, v2 7436 v_mov_b32_e32 v1, v2 7437 v_nop 7438 v_cvt_f64_i32_e32 v[1:2], v2 7439 v_floor_f32_e32 v1, v2 7440 v_bfrev_b32_e32 v1, v2 7441 v_add_f32_e32 v1, v2, v3 7442 v_mul_i32_i24_e64 v1, v2, 3 7443 v_mul_i32_i24_e32 v1, -3, v3 7444 v_mul_i32_i24_e32 v1, -100, v3 7445 v_addc_u32 v1, s[0:1], v2, v3, s[2:3] 7446 v_max_f16_e32 v1, v2, v3 7447 7448VOP_DPP examples: 7449 7450.. code-block:: nasm 7451 7452 v_mov_b32 v0, v0 quad_perm:[0,2,1,1] 7453 v_sin_f32 v0, v0 row_shl:1 row_mask:0xa bank_mask:0x1 bound_ctrl:0 7454 v_mov_b32 v0, v0 wave_shl:1 7455 v_mov_b32 v0, v0 row_mirror 7456 v_mov_b32 v0, v0 row_bcast:31 7457 v_mov_b32 v0, v0 quad_perm:[1,3,0,1] row_mask:0xa bank_mask:0x1 bound_ctrl:0 7458 v_add_f32 v0, v0, |v0| row_shl:1 row_mask:0xa bank_mask:0x1 bound_ctrl:0 7459 v_max_f16 v1, v2, v3 row_shl:1 row_mask:0xa bank_mask:0x1 bound_ctrl:0 7460 7461VOP_SDWA examples: 7462 7463.. code-block:: nasm 7464 7465 v_mov_b32 v1, v2 dst_sel:BYTE_0 dst_unused:UNUSED_PRESERVE src0_sel:DWORD 7466 v_min_u32 v200, v200, v1 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:BYTE_1 src1_sel:DWORD 7467 v_sin_f32 v0, v0 dst_unused:UNUSED_PAD src0_sel:WORD_1 7468 v_fract_f32 v0, |v0| dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 7469 v_cmpx_le_u32 vcc, v1, v2 src0_sel:BYTE_2 src1_sel:WORD_0 7470 7471For full list of supported instructions, refer to "Vector ALU instructions". 7472 7473.. TODO:: 7474 7475 Remove once we switch to code object v3 by default. 7476 7477.. _amdgpu-amdhsa-assembler-predefined-symbols-v2: 7478 7479Code Object V2 Predefined Symbols (--amdhsa-code-object-version=2) 7480~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 7481 7482.. warning:: Code Object V2 is not the default code object version emitted by 7483 this version of LLVM. For a description of the predefined symbols available 7484 with the default configuration (Code Object V3) see 7485 :ref:`amdgpu-amdhsa-assembler-predefined-symbols-v3`. 7486 7487The AMDGPU assembler defines and updates some symbols automatically. These 7488symbols do not affect code generation. 7489 7490.option.machine_version_major 7491+++++++++++++++++++++++++++++ 7492 7493Set to the GFX major generation number of the target being assembled for. For 7494example, when assembling for a "GFX9" target this will be set to the integer 7495value "9". The possible GFX major generation numbers are presented in 7496:ref:`amdgpu-processors`. 7497 7498.option.machine_version_minor 7499+++++++++++++++++++++++++++++ 7500 7501Set to the GFX minor generation number of the target being assembled for. For 7502example, when assembling for a "GFX810" target this will be set to the integer 7503value "1". The possible GFX minor generation numbers are presented in 7504:ref:`amdgpu-processors`. 7505 7506.option.machine_version_stepping 7507++++++++++++++++++++++++++++++++ 7508 7509Set to the GFX stepping generation number of the target being assembled for. 7510For example, when assembling for a "GFX704" target this will be set to the 7511integer value "4". The possible GFX stepping generation numbers are presented 7512in :ref:`amdgpu-processors`. 7513 7514.kernel.vgpr_count 7515++++++++++++++++++ 7516 7517Set to zero each time a 7518:ref:`amdgpu-amdhsa-assembler-directive-amdgpu_hsa_kernel` directive is 7519encountered. At each instruction, if the current value of this symbol is less 7520than or equal to the maximum VGPR number explicitly referenced within that 7521instruction then the symbol value is updated to equal that VGPR number plus 7522one. 7523 7524.kernel.sgpr_count 7525++++++++++++++++++ 7526 7527Set to zero each time a 7528:ref:`amdgpu-amdhsa-assembler-directive-amdgpu_hsa_kernel` directive is 7529encountered. At each instruction, if the current value of this symbol is less 7530than or equal to the maximum VGPR number explicitly referenced within that 7531instruction then the symbol value is updated to equal that SGPR number plus 7532one. 7533 7534.. _amdgpu-amdhsa-assembler-directives-v2: 7535 7536Code Object V2 Directives (--amdhsa-code-object-version=2) 7537~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 7538 7539.. warning:: Code Object V2 is not the default code object version emitted by 7540 this version of LLVM. For a description of the directives supported with 7541 the default configuration (Code Object V3) see 7542 :ref:`amdgpu-amdhsa-assembler-directives-v3`. 7543 7544AMDGPU ABI defines auxiliary data in output code object. In assembly source, 7545one can specify them with assembler directives. 7546 7547.hsa_code_object_version major, minor 7548+++++++++++++++++++++++++++++++++++++ 7549 7550*major* and *minor* are integers that specify the version of the HSA code 7551object that will be generated by the assembler. 7552 7553.hsa_code_object_isa [major, minor, stepping, vendor, arch] 7554+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 7555 7556 7557*major*, *minor*, and *stepping* are all integers that describe the instruction 7558set architecture (ISA) version of the assembly program. 7559 7560*vendor* and *arch* are quoted strings. *vendor* should always be equal to 7561"AMD" and *arch* should always be equal to "AMDGPU". 7562 7563By default, the assembler will derive the ISA version, *vendor*, and *arch* 7564from the value of the -mcpu option that is passed to the assembler. 7565 7566.. _amdgpu-amdhsa-assembler-directive-amdgpu_hsa_kernel: 7567 7568.amdgpu_hsa_kernel (name) 7569+++++++++++++++++++++++++ 7570 7571This directives specifies that the symbol with given name is a kernel entry 7572point (label) and the object should contain corresponding symbol of type 7573STT_AMDGPU_HSA_KERNEL. 7574 7575.amd_kernel_code_t 7576++++++++++++++++++ 7577 7578This directive marks the beginning of a list of key / value pairs that are used 7579to specify the amd_kernel_code_t object that will be emitted by the assembler. 7580The list must be terminated by the *.end_amd_kernel_code_t* directive. For any 7581amd_kernel_code_t values that are unspecified a default value will be used. The 7582default value for all keys is 0, with the following exceptions: 7583 7584- *amd_code_version_major* defaults to 1. 7585- *amd_kernel_code_version_minor* defaults to 2. 7586- *amd_machine_kind* defaults to 1. 7587- *amd_machine_version_major*, *machine_version_minor*, and 7588 *amd_machine_version_stepping* are derived from the value of the -mcpu option 7589 that is passed to the assembler. 7590- *kernel_code_entry_byte_offset* defaults to 256. 7591- *wavefront_size* defaults 6 for all targets before GFX10. For GFX10 onwards 7592 defaults to 6 if target feature ``wavefrontsize64`` is enabled, otherwise 5. 7593 Note that wavefront size is specified as a power of two, so a value of **n** 7594 means a size of 2^ **n**. 7595- *call_convention* defaults to -1. 7596- *kernarg_segment_alignment*, *group_segment_alignment*, and 7597 *private_segment_alignment* default to 4. Note that alignments are specified 7598 as a power of 2, so a value of **n** means an alignment of 2^ **n**. 7599- *enable_wgp_mode* defaults to 1 if target feature ``cumode`` is disabled for 7600 GFX10 onwards. 7601- *enable_mem_ordered* defaults to 1 for GFX10 onwards. 7602 7603The *.amd_kernel_code_t* directive must be placed immediately after the 7604function label and before any instructions. 7605 7606For a full list of amd_kernel_code_t keys, refer to AMDGPU ABI document, 7607comments in lib/Target/AMDGPU/AmdKernelCodeT.h and test/CodeGen/AMDGPU/hsa.s. 7608 7609.. _amdgpu-amdhsa-assembler-example-v2: 7610 7611Code Object V2 Example Source Code (--amdhsa-code-object-version=2) 7612~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 7613 7614.. warning:: Code Object V2 is not the default code object version emitted by 7615 this version of LLVM. For a description of the directives supported with 7616 the default configuration (Code Object V3) see 7617 :ref:`amdgpu-amdhsa-assembler-example-v3`. 7618 7619Here is an example of a minimal assembly source file, defining one HSA kernel: 7620 7621.. code:: 7622 :number-lines: 7623 7624 .hsa_code_object_version 1,0 7625 .hsa_code_object_isa 7626 7627 .hsatext 7628 .globl hello_world 7629 .p2align 8 7630 .amdgpu_hsa_kernel hello_world 7631 7632 hello_world: 7633 7634 .amd_kernel_code_t 7635 enable_sgpr_kernarg_segment_ptr = 1 7636 is_ptr64 = 1 7637 compute_pgm_rsrc1_vgprs = 0 7638 compute_pgm_rsrc1_sgprs = 0 7639 compute_pgm_rsrc2_user_sgpr = 2 7640 compute_pgm_rsrc1_wgp_mode = 0 7641 compute_pgm_rsrc1_mem_ordered = 0 7642 compute_pgm_rsrc1_fwd_progress = 1 7643 .end_amd_kernel_code_t 7644 7645 s_load_dwordx2 s[0:1], s[0:1] 0x0 7646 v_mov_b32 v0, 3.14159 7647 s_waitcnt lgkmcnt(0) 7648 v_mov_b32 v1, s0 7649 v_mov_b32 v2, s1 7650 flat_store_dword v[1:2], v0 7651 s_endpgm 7652 .Lfunc_end0: 7653 .size hello_world, .Lfunc_end0-hello_world 7654 7655.. _amdgpu-amdhsa-assembler-predefined-symbols-v3: 7656 7657Code Object V3 Predefined Symbols (--amdhsa-code-object-version=3) 7658~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 7659 7660The AMDGPU assembler defines and updates some symbols automatically. These 7661symbols do not affect code generation. 7662 7663.amdgcn.gfx_generation_number 7664+++++++++++++++++++++++++++++ 7665 7666Set to the GFX major generation number of the target being assembled for. For 7667example, when assembling for a "GFX9" target this will be set to the integer 7668value "9". The possible GFX major generation numbers are presented in 7669:ref:`amdgpu-processors`. 7670 7671.amdgcn.gfx_generation_minor 7672++++++++++++++++++++++++++++ 7673 7674Set to the GFX minor generation number of the target being assembled for. For 7675example, when assembling for a "GFX810" target this will be set to the integer 7676value "1". The possible GFX minor generation numbers are presented in 7677:ref:`amdgpu-processors`. 7678 7679.amdgcn.gfx_generation_stepping 7680+++++++++++++++++++++++++++++++ 7681 7682Set to the GFX stepping generation number of the target being assembled for. 7683For example, when assembling for a "GFX704" target this will be set to the 7684integer value "4". The possible GFX stepping generation numbers are presented 7685in :ref:`amdgpu-processors`. 7686 7687.. _amdgpu-amdhsa-assembler-symbol-next_free_vgpr: 7688 7689.amdgcn.next_free_vgpr 7690++++++++++++++++++++++ 7691 7692Set to zero before assembly begins. At each instruction, if the current value 7693of this symbol is less than or equal to the maximum VGPR number explicitly 7694referenced within that instruction then the symbol value is updated to equal 7695that VGPR number plus one. 7696 7697May be used to set the `.amdhsa_next_free_vgpr` directive in 7698:ref:`amdhsa-kernel-directives-table`. 7699 7700May be set at any time, e.g. manually set to zero at the start of each kernel. 7701 7702.. _amdgpu-amdhsa-assembler-symbol-next_free_sgpr: 7703 7704.amdgcn.next_free_sgpr 7705++++++++++++++++++++++ 7706 7707Set to zero before assembly begins. At each instruction, if the current value 7708of this symbol is less than or equal the maximum SGPR number explicitly 7709referenced within that instruction then the symbol value is updated to equal 7710that SGPR number plus one. 7711 7712May be used to set the `.amdhsa_next_free_spgr` directive in 7713:ref:`amdhsa-kernel-directives-table`. 7714 7715May be set at any time, e.g. manually set to zero at the start of each kernel. 7716 7717.. _amdgpu-amdhsa-assembler-directives-v3: 7718 7719Code Object V3 Directives (--amdhsa-code-object-version=3) 7720~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 7721 7722Directives which begin with ``.amdgcn`` are valid for all ``amdgcn`` 7723architecture processors, and are not OS-specific. Directives which begin with 7724``.amdhsa`` are specific to ``amdgcn`` architecture processors when the 7725``amdhsa`` OS is specified. See :ref:`amdgpu-target-triples` and 7726:ref:`amdgpu-processors`. 7727 7728.amdgcn_target <target> 7729+++++++++++++++++++++++ 7730 7731Optional directive which declares the target supported by the containing 7732assembler source file. Valid values are described in 7733:ref:`amdgpu-amdhsa-code-object-target-identification`. Used by the assembler 7734to validate command-line options such as ``-triple``, ``-mcpu``, and those 7735which specify target features. 7736 7737.amdhsa_kernel <name> 7738+++++++++++++++++++++ 7739 7740Creates a correctly aligned AMDHSA kernel descriptor and a symbol, 7741``<name>.kd``, in the current location of the current section. Only valid when 7742the OS is ``amdhsa``. ``<name>`` must be a symbol that labels the first 7743instruction to execute, and does not need to be previously defined. 7744 7745Marks the beginning of a list of directives used to generate the bytes of a 7746kernel descriptor, as described in :ref:`amdgpu-amdhsa-kernel-descriptor`. 7747Directives which may appear in this list are described in 7748:ref:`amdhsa-kernel-directives-table`. Directives may appear in any order, must 7749be valid for the target being assembled for, and cannot be repeated. Directives 7750support the range of values specified by the field they reference in 7751:ref:`amdgpu-amdhsa-kernel-descriptor`. If a directive is not specified, it is 7752assumed to have its default value, unless it is marked as "Required", in which 7753case it is an error to omit the directive. This list of directives is 7754terminated by an ``.end_amdhsa_kernel`` directive. 7755 7756 .. table:: AMDHSA Kernel Assembler Directives 7757 :name: amdhsa-kernel-directives-table 7758 7759 ======================================================== =================== ============ =================== 7760 Directive Default Supported On Description 7761 ======================================================== =================== ============ =================== 7762 ``.amdhsa_group_segment_fixed_size`` 0 GFX6-GFX10 Controls GROUP_SEGMENT_FIXED_SIZE in 7763 :ref:`amdgpu-amdhsa-kernel-descriptor-gfx6-gfx10-table`. 7764 ``.amdhsa_private_segment_fixed_size`` 0 GFX6-GFX10 Controls PRIVATE_SEGMENT_FIXED_SIZE in 7765 :ref:`amdgpu-amdhsa-kernel-descriptor-gfx6-gfx10-table`. 7766 ``.amdhsa_user_sgpr_private_segment_buffer`` 0 GFX6-GFX10 Controls ENABLE_SGPR_PRIVATE_SEGMENT_BUFFER in 7767 :ref:`amdgpu-amdhsa-kernel-descriptor-gfx6-gfx10-table`. 7768 ``.amdhsa_user_sgpr_dispatch_ptr`` 0 GFX6-GFX10 Controls ENABLE_SGPR_DISPATCH_PTR in 7769 :ref:`amdgpu-amdhsa-kernel-descriptor-gfx6-gfx10-table`. 7770 ``.amdhsa_user_sgpr_queue_ptr`` 0 GFX6-GFX10 Controls ENABLE_SGPR_QUEUE_PTR in 7771 :ref:`amdgpu-amdhsa-kernel-descriptor-gfx6-gfx10-table`. 7772 ``.amdhsa_user_sgpr_kernarg_segment_ptr`` 0 GFX6-GFX10 Controls ENABLE_SGPR_KERNARG_SEGMENT_PTR in 7773 :ref:`amdgpu-amdhsa-kernel-descriptor-gfx6-gfx10-table`. 7774 ``.amdhsa_user_sgpr_dispatch_id`` 0 GFX6-GFX10 Controls ENABLE_SGPR_DISPATCH_ID in 7775 :ref:`amdgpu-amdhsa-kernel-descriptor-gfx6-gfx10-table`. 7776 ``.amdhsa_user_sgpr_flat_scratch_init`` 0 GFX6-GFX10 Controls ENABLE_SGPR_FLAT_SCRATCH_INIT in 7777 :ref:`amdgpu-amdhsa-kernel-descriptor-gfx6-gfx10-table`. 7778 ``.amdhsa_user_sgpr_private_segment_size`` 0 GFX6-GFX10 Controls ENABLE_SGPR_PRIVATE_SEGMENT_SIZE in 7779 :ref:`amdgpu-amdhsa-kernel-descriptor-gfx6-gfx10-table`. 7780 ``.amdhsa_wavefront_size32`` Target GFX10 Controls ENABLE_WAVEFRONT_SIZE32 in 7781 Feature :ref:`amdgpu-amdhsa-kernel-descriptor-gfx6-gfx10-table`. 7782 Specific 7783 (-wavefrontsize64) 7784 ``.amdhsa_system_sgpr_private_segment_wavefront_offset`` 0 GFX6-GFX10 Controls ENABLE_SGPR_PRIVATE_SEGMENT_WAVEFRONT_OFFSET in 7785 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 7786 ``.amdhsa_system_sgpr_workgroup_id_x`` 1 GFX6-GFX10 Controls ENABLE_SGPR_WORKGROUP_ID_X in 7787 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 7788 ``.amdhsa_system_sgpr_workgroup_id_y`` 0 GFX6-GFX10 Controls ENABLE_SGPR_WORKGROUP_ID_Y in 7789 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 7790 ``.amdhsa_system_sgpr_workgroup_id_z`` 0 GFX6-GFX10 Controls ENABLE_SGPR_WORKGROUP_ID_Z in 7791 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 7792 ``.amdhsa_system_sgpr_workgroup_info`` 0 GFX6-GFX10 Controls ENABLE_SGPR_WORKGROUP_INFO in 7793 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 7794 ``.amdhsa_system_vgpr_workitem_id`` 0 GFX6-GFX10 Controls ENABLE_VGPR_WORKITEM_ID in 7795 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 7796 Possible values are defined in 7797 :ref:`amdgpu-amdhsa-system-vgpr-work-item-id-enumeration-values-table`. 7798 ``.amdhsa_next_free_vgpr`` Required GFX6-GFX10 Maximum VGPR number explicitly referenced, plus one. 7799 Used to calculate GRANULATED_WORKITEM_VGPR_COUNT in 7800 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 7801 ``.amdhsa_next_free_sgpr`` Required GFX6-GFX10 Maximum SGPR number explicitly referenced, plus one. 7802 Used to calculate GRANULATED_WAVEFRONT_SGPR_COUNT in 7803 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 7804 ``.amdhsa_reserve_vcc`` 1 GFX6-GFX10 Whether the kernel may use the special VCC SGPR. 7805 Used to calculate GRANULATED_WAVEFRONT_SGPR_COUNT in 7806 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 7807 ``.amdhsa_reserve_flat_scratch`` 1 GFX7-GFX10 Whether the kernel may use flat instructions to access 7808 scratch memory. Used to calculate 7809 GRANULATED_WAVEFRONT_SGPR_COUNT in 7810 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 7811 ``.amdhsa_reserve_xnack_mask`` Target GFX8-GFX10 Whether the kernel may trigger XNACK replay. 7812 Feature Used to calculate GRANULATED_WAVEFRONT_SGPR_COUNT in 7813 Specific :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 7814 (+xnack) 7815 ``.amdhsa_float_round_mode_32`` 0 GFX6-GFX10 Controls FLOAT_ROUND_MODE_32 in 7816 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 7817 Possible values are defined in 7818 :ref:`amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table`. 7819 ``.amdhsa_float_round_mode_16_64`` 0 GFX6-GFX10 Controls FLOAT_ROUND_MODE_16_64 in 7820 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 7821 Possible values are defined in 7822 :ref:`amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table`. 7823 ``.amdhsa_float_denorm_mode_32`` 0 GFX6-GFX10 Controls FLOAT_DENORM_MODE_32 in 7824 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 7825 Possible values are defined in 7826 :ref:`amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table`. 7827 ``.amdhsa_float_denorm_mode_16_64`` 3 GFX6-GFX10 Controls FLOAT_DENORM_MODE_16_64 in 7828 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 7829 Possible values are defined in 7830 :ref:`amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table`. 7831 ``.amdhsa_dx10_clamp`` 1 GFX6-GFX10 Controls ENABLE_DX10_CLAMP in 7832 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 7833 ``.amdhsa_ieee_mode`` 1 GFX6-GFX10 Controls ENABLE_IEEE_MODE in 7834 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 7835 ``.amdhsa_fp16_overflow`` 0 GFX9-GFX10 Controls FP16_OVFL in 7836 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 7837 ``.amdhsa_workgroup_processor_mode`` Target GFX10 Controls ENABLE_WGP_MODE in 7838 Feature :ref:`amdgpu-amdhsa-kernel-descriptor-gfx6-gfx10-table`. 7839 Specific 7840 (-cumode) 7841 ``.amdhsa_memory_ordered`` 1 GFX10 Controls MEM_ORDERED in 7842 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 7843 ``.amdhsa_forward_progress`` 0 GFX10 Controls FWD_PROGRESS in 7844 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 7845 ``.amdhsa_exception_fp_ieee_invalid_op`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_INVALID_OPERATION in 7846 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 7847 ``.amdhsa_exception_fp_denorm_src`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_FP_DENORMAL_SOURCE in 7848 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 7849 ``.amdhsa_exception_fp_ieee_div_zero`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_DIVISION_BY_ZERO in 7850 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 7851 ``.amdhsa_exception_fp_ieee_overflow`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_OVERFLOW in 7852 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 7853 ``.amdhsa_exception_fp_ieee_underflow`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_UNDERFLOW in 7854 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 7855 ``.amdhsa_exception_fp_ieee_inexact`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_INEXACT in 7856 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 7857 ``.amdhsa_exception_int_div_zero`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_INT_DIVIDE_BY_ZERO in 7858 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 7859 ======================================================== =================== ============ =================== 7860 7861.amdgpu_metadata 7862++++++++++++++++ 7863 7864Optional directive which declares the contents of the ``NT_AMDGPU_METADATA`` 7865note record (see :ref:`amdgpu-elf-note-records-table-v3`). 7866 7867The contents must be in the [YAML]_ markup format, with the same structure and 7868semantics described in :ref:`amdgpu-amdhsa-code-object-metadata-v3`. 7869 7870This directive is terminated by an ``.end_amdgpu_metadata`` directive. 7871 7872.. _amdgpu-amdhsa-assembler-example-v3: 7873 7874Code Object V3 Example Source Code (--amdhsa-code-object-version=3) 7875~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 7876 7877Here is an example of a minimal assembly source file, defining one HSA kernel: 7878 7879.. code:: 7880 :number-lines: 7881 7882 .amdgcn_target "amdgcn-amd-amdhsa--gfx900+xnack" // optional 7883 7884 .text 7885 .globl hello_world 7886 .p2align 8 7887 .type hello_world,@function 7888 hello_world: 7889 s_load_dwordx2 s[0:1], s[0:1] 0x0 7890 v_mov_b32 v0, 3.14159 7891 s_waitcnt lgkmcnt(0) 7892 v_mov_b32 v1, s0 7893 v_mov_b32 v2, s1 7894 flat_store_dword v[1:2], v0 7895 s_endpgm 7896 .Lfunc_end0: 7897 .size hello_world, .Lfunc_end0-hello_world 7898 7899 .rodata 7900 .p2align 6 7901 .amdhsa_kernel hello_world 7902 .amdhsa_user_sgpr_kernarg_segment_ptr 1 7903 .amdhsa_next_free_vgpr .amdgcn.next_free_vgpr 7904 .amdhsa_next_free_sgpr .amdgcn.next_free_sgpr 7905 .end_amdhsa_kernel 7906 7907 .amdgpu_metadata 7908 --- 7909 amdhsa.version: 7910 - 1 7911 - 0 7912 amdhsa.kernels: 7913 - .name: hello_world 7914 .symbol: hello_world.kd 7915 .kernarg_segment_size: 48 7916 .group_segment_fixed_size: 0 7917 .private_segment_fixed_size: 0 7918 .kernarg_segment_align: 4 7919 .wavefront_size: 64 7920 .sgpr_count: 2 7921 .vgpr_count: 3 7922 .max_flat_workgroup_size: 256 7923 ... 7924 .end_amdgpu_metadata 7925 7926If an assembly source file contains multiple kernels and/or functions, the 7927:ref:`amdgpu-amdhsa-assembler-symbol-next_free_vgpr` and 7928:ref:`amdgpu-amdhsa-assembler-symbol-next_free_sgpr` symbols may be reset using 7929the ``.set <symbol>, <expression>`` directive. For example, in the case of two 7930kernels, where ``function1`` is only called from ``kernel1`` it is sufficient 7931to group the function with the kernel that calls it and reset the symbols 7932between the two connected components: 7933 7934.. code:: 7935 :number-lines: 7936 7937 .amdgcn_target "amdgcn-amd-amdhsa--gfx900+xnack" // optional 7938 7939 // gpr tracking symbols are implicitly set to zero 7940 7941 .text 7942 .globl kern0 7943 .p2align 8 7944 .type kern0,@function 7945 kern0: 7946 // ... 7947 s_endpgm 7948 .Lkern0_end: 7949 .size kern0, .Lkern0_end-kern0 7950 7951 .rodata 7952 .p2align 6 7953 .amdhsa_kernel kern0 7954 // ... 7955 .amdhsa_next_free_vgpr .amdgcn.next_free_vgpr 7956 .amdhsa_next_free_sgpr .amdgcn.next_free_sgpr 7957 .end_amdhsa_kernel 7958 7959 // reset symbols to begin tracking usage in func1 and kern1 7960 .set .amdgcn.next_free_vgpr, 0 7961 .set .amdgcn.next_free_sgpr, 0 7962 7963 .text 7964 .hidden func1 7965 .global func1 7966 .p2align 2 7967 .type func1,@function 7968 func1: 7969 // ... 7970 s_setpc_b64 s[30:31] 7971 .Lfunc1_end: 7972 .size func1, .Lfunc1_end-func1 7973 7974 .globl kern1 7975 .p2align 8 7976 .type kern1,@function 7977 kern1: 7978 // ... 7979 s_getpc_b64 s[4:5] 7980 s_add_u32 s4, s4, func1@rel32@lo+4 7981 s_addc_u32 s5, s5, func1@rel32@lo+4 7982 s_swappc_b64 s[30:31], s[4:5] 7983 // ... 7984 s_endpgm 7985 .Lkern1_end: 7986 .size kern1, .Lkern1_end-kern1 7987 7988 .rodata 7989 .p2align 6 7990 .amdhsa_kernel kern1 7991 // ... 7992 .amdhsa_next_free_vgpr .amdgcn.next_free_vgpr 7993 .amdhsa_next_free_sgpr .amdgcn.next_free_sgpr 7994 .end_amdhsa_kernel 7995 7996These symbols cannot identify connected components in order to automatically 7997track the usage for each kernel. However, in some cases careful organization of 7998the kernels and functions in the source file means there is minimal additional 7999effort required to accurately calculate GPR usage. 8000 8001Additional Documentation 8002======================== 8003 8004.. [AMD-GCN-GFX6] `AMD Southern Islands Series ISA <http://developer.amd.com/wordpress/media/2012/12/AMD_Southern_Islands_Instruction_Set_Architecture.pdf>`__ 8005.. [AMD-GCN-GFX7] `AMD Sea Islands Series ISA <http://developer.amd.com/wordpress/media/2013/07/AMD_Sea_Islands_Instruction_Set_Architecture.pdf>`_ 8006.. [AMD-GCN-GFX8] `AMD GCN3 Instruction Set Architecture <http://amd-dev.wpengine.netdna-cdn.com/wordpress/media/2013/12/AMD_GCN3_Instruction_Set_Architecture_rev1.1.pdf>`__ 8007.. [AMD-GCN-GFX9] `AMD "Vega" Instruction Set Architecture <http://developer.amd.com/wordpress/media/2013/12/Vega_Shader_ISA_28July2017.pdf>`__ 8008.. [AMD-GCN-GFX10] `AMD "RDNA 1.0" Instruction Set Architecture <https://gpuopen.com/wp-content/uploads/2019/08/RDNA_Shader_ISA_5August2019.pdf>`__ 8009.. [AMD-RADEON-HD-2000-3000] `AMD R6xx shader ISA <http://developer.amd.com/wordpress/media/2012/10/R600_Instruction_Set_Architecture.pdf>`__ 8010.. [AMD-RADEON-HD-4000] `AMD R7xx shader ISA <http://developer.amd.com/wordpress/media/2012/10/R700-Family_Instruction_Set_Architecture.pdf>`__ 8011.. [AMD-RADEON-HD-5000] `AMD Evergreen shader ISA <http://developer.amd.com/wordpress/media/2012/10/AMD_Evergreen-Family_Instruction_Set_Architecture.pdf>`__ 8012.. [AMD-RADEON-HD-6000] `AMD Cayman/Trinity shader ISA <http://developer.amd.com/wordpress/media/2012/10/AMD_HD_6900_Series_Instruction_Set_Architecture.pdf>`__ 8013.. [AMD-ROCm] `AMD ROCm Platform <https://rocm-documentation.readthedocs.io>`__ 8014.. [AMD-ROCm-github] `ROCm github <http://github.com/RadeonOpenCompute>`__ 8015.. [CLANG-ATTR] `Attributes in Clang <https://clang.llvm.org/docs/AttributeReference.html>`__ 8016.. [DWARF] `DWARF Debugging Information Format <http://dwarfstd.org/>`__ 8017.. [ELF] `Executable and Linkable Format (ELF) <http://www.sco.com/developers/gabi/>`__ 8018.. [HRF] `Heterogeneous-race-free Memory Models <http://benedictgaster.org/wp-content/uploads/2014/01/asplos269-FINAL.pdf>`__ 8019.. [HSA] `Heterogeneous System Architecture (HSA) Foundation <http://www.hsafoundation.com/>`__ 8020.. [MsgPack] `Message Pack <http://www.msgpack.org/>`__ 8021.. [OpenCL] `The OpenCL Specification Version 2.0 <http://www.khronos.org/registry/cl/specs/opencl-2.0.pdf>`__ 8022.. [SEMVER] `Semantic Versioning <https://semver.org/>`__ 8023.. [YAML] `YAML Ain't Markup Language (YAML™) Version 1.2 <http://www.yaml.org/spec/1.2/spec.html>`__ 8024